Time action positioning method and system based on time sequence context maximum pooling

Abstract

The invention discloses a time action positioning method and system based on time sequence context maximum pooling. The method comprises the following steps: acquiring a to-be-identified action video, and extracting and acquiring a feature coding sequence of the to-be-identified action video; presetting a time action positioning model, inputting the feature coding sequence into the time action positioning model, and obtaining an action classification result; by introducing the time sequence context maximum pooling operation and the multi-scale time feature pyramid structure, the calculation complexity is effectively reduced, and the reasoning speed and efficiency are improved. Meanwhile, by designing a time action positioning model comprising an encoder, a long-term time context module and a decoder, the model can simultaneously capture short-time and long-time dependency, so that the accuracy of time action positioning is improved. Due to the advantages, the method has important application value in the field of time action positioning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video understanding and temporal action localization, and particularly relates to a temporal action localization method and system based on temporal context max-pooling. BACKGROUND

[0002] Temporal action localization is a challenging task in video understanding, aiming to identify and localize actions in a video sequence. Current temporal action localization algorithms are generally divided into two-stage and one-stage methods to detect actions in videos. The two-stage method first generates possible action proposals, and then classifies them to determine the action category to which they belong. The one-stage method performs action proposal generation and classification in a single processing without using a separate proposal generation step.

[0003] Recent temporal action localization research emphasizes the importance of applying long-term temporal context modeling on extracted video clip features. For example, by using self-attention mechanisms, graphs or more complex modules to help the module capture long-term dependencies, which can be useful for identifying complex actions that may unfold over a longer period of time or have more overlapping actions. Although they improve performance on benchmark tests, there is a time-consuming long-term encoding process, which incurs high computational costs, and their inference speed and effectiveness are rarely considered. SUMMARY

[0004] This section is intended to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the specification to avoid obscuring the purpose of this section, the abstract and the title, and such simplifications or omissions are not used to limit the scope of the present application.

[0005] In view of the above-mentioned existing problems, the present application is proposed.

[0006] Therefore, the present application provides a temporal action localization method and system based on temporal context max-pooling, which can solve the problems mentioned in the background art.

[0007] To solve the above technical problems, the present application provides the following technical solutions. A temporal action localization method based on temporal context max-pooling, comprising:

[0008] Obtaining a video to be identified for action, and extracting a feature encoding sequence of the obtained video to be identified for action;

[0009] A pre-set temporal action localization model, inputting the feature encoding sequence into the temporal action localization model, and obtaining an action classification result;

[0010] The temporal action localization model includes an encoder, a long-term temporal context module, and a decoder. The input of the temporal action localization model is the feature encoding sequence of the action video to be identified, and the output is the action classification result. The long-term temporal context module is used to extract feature information in a multi-dimensional vector space to form a multi-scale temporal feature pyramid.

[0011] As a preferred embodiment of the temporal action localization method based on temporal context max pooling described in this invention, the step of acquiring the action video to be identified and extracting the feature encoding sequence of the acquired action video includes:

[0012] Let X be a video of an action to be recognized. Using a pre-defined feature encoding network, X is transformed into a set of feature encoding sequences, represented as X = {x1, x2, ..., x...}. T}, where the number of discrete time steps t = {1, 2, ..., T} depends on the video length, x T This represents the feature encoding sequence corresponding to the discrete time step T;

[0013] The preset feature encoding network is an open-source model or a custom feature encoding model.

[0014] As a preferred embodiment of the temporal action localization method based on temporal context max pooling described in this invention, wherein: the preset temporal action localization model, by inputting the feature encoding sequence into the temporal action localization model, obtains action classification results including:

[0015] The time-action localization model includes an encoder, a long-term time context module, and a decoder. The encoder is a feature projection layer used to convert the feature encoding sequence into a multi-dimensional space vector.

[0016] The projection process of the encoder is represented as follows:

[0017] X P =E2(E1(Concat(X)))

[0018] Where X represents the video feature encoding sequence of the action to be identified, concatenated in the first dimension, and passed to two feature projection modules E1 and E2. Each projection module includes a one-dimensional convolutional neural network layer, a normalization layer, and a ReLU activation function. P This represents the projection result of the video feature encoding sequence of the action to be identified, i.e., a multidimensional space vector.

[0019] As a preferred embodiment of the temporal action localization method based on temporal context max pooling described in this invention, the preset temporal action localization model, by inputting the feature encoding into the temporal action localization model to obtain the action classification result, further includes:

[0020] The long-term temporal context module is used to extract feature information from the multi-dimensional vector space, forming a multi-scale temporal feature pyramid, and then... 1 =X P It is assigned as the first feature in the multi-scale time pyramid Z;

[0021] The long-term temporal context module is used to extract feature information in a multi-dimensional vector space, forming a multi-scale temporal feature pyramid, specifically represented as follows:

[0022] Z l =TemporalMaxer(Z l-1 )

[0023] TemporalMaxer represents the max pooling operation with a pooling window step size of 2;

[0024] The TemporalMaxer layer in the long-term temporal context module encodes the input features into a multi-scale temporal feature pyramid Z = {Z}. 1 Z 2 ,…,Z l};

[0025] The decoder learns to predict the sequence label Ψ = {ψ1, ψ2, ..., ψN} for each time step using a multi-scale temporal feature pyramid, where N is the number of action instances in X. It includes a classification head module and a regression head module, and all weights of the decoder are shared among different features of the multi-scale feature pyramid.

[0026] As a preferred embodiment of the temporal action localization method based on temporal context max pooling described in this invention, the classification head module includes:

[0027] The classification head module includes two 1D convolutional layers, each followed by a normalization layer and a ReLU activation function. After the two 1D convolutional layers, a single 1D convolutional layer is added to classify the actions according to a predefined category and to calculate the classification probability of each category.

[0028] The calculation process for the classification header module is as follows:

[0029] C l =F c (E4(E3(Z l )))

[0030] Where E3 and E4 represent 1D convolutional layers followed by a normalization layer and a ReLU activation function, F c 1D convolutional layer, C l Representing multi-scale temporal features Z l The results of the classification operation.

[0031] As a preferred embodiment of the temporal action localization method based on temporal context max pooling described in this invention, the regression head module includes:

[0032] The regression head module includes two 1D convolutional layers, each followed by a normalization layer and a ReLU activation function, and then a single 1D convolutional layer and a ReLU activation function are added after the two 1D convolutional layers.

[0033] The calculation process of the regression head module is as follows:

[0034] O l =ReLU(F o (E6(E5(Z l ))))

[0035] Where E5 and E6 represent 1D convolutional layers followed by a normalization layer and a ReLU activation function, F o Represents a 1D convolutional layer, O l Representing multi-scale temporal features Z l The regression results.

[0036] As a preferred embodiment of the temporal action localization method based on temporal context max pooling described in this invention, the preset temporal action localization model, by inputting the feature encoding into the temporal action localization model to obtain the action classification result, further includes:

[0037] Establish a loss function for the time-motion localization model, the loss function being expressed as:

[0038]

[0039] Among them, l cls Represents classification loss, l reg Let represent the regression loss, t represent the time step, and T1 represent the number of positive samples, i.e., the number of time steps containing the action.

[0040] A time-based action localization system based on temporal context max pooling includes:

[0041] The feature encoding sequence acquisition module is used to acquire the action video to be identified and extract the feature encoding sequence of the acquired action video;

[0042] The action localization module is used to preset a time-based action localization model, input the feature encoding sequence into the time-based action localization model, and obtain action classification results;

[0043] The temporal action localization model includes an encoder, a long-term temporal context module, and a decoder. The input of the temporal action localization model is the feature encoding sequence of the action video to be identified, and the output is the action classification result. The long-term temporal context module is used to extract feature information in a multi-dimensional vector space to form a multi-scale temporal feature pyramid.

[0044] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0045] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0046] The beneficial effects of this invention are as follows: This invention proposes a temporal action localization method and system based on temporal context max pooling. The method involves acquiring a video of an action to be identified and extracting its feature encoding sequence. A pre-defined temporal action localization model is then established, and the feature encoding sequence is input into the model to obtain action classification results. The temporal action localization model includes an encoder, a long-term temporal context module, and a decoder. The input to the model is the feature encoding sequence of the video of the action to be identified, and the output is the action classification result. The long-term temporal context module is used to extract feature information from a multi-dimensional vector space, forming a multi-scale temporal feature pyramid. By introducing temporal context max pooling and a multi-scale temporal feature pyramid structure, computational complexity is effectively reduced, and inference speed and efficiency are improved. Furthermore, by designing a temporal action localization model that includes an encoder, a long-term temporal context module, and a decoder, the model can simultaneously capture short-term and long-term dependencies, thereby improving the accuracy of temporal action localization. These advantages make this invention of significant application value in the field of temporal action localization. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0048] Figure 1 A flowchart of a time-based action localization method and system based on temporal context max pooling is provided in one embodiment of the present invention;

[0049] Figure 2This is an internal structural diagram of a computer device for a time-based action localization method and system based on temporal context max pooling, provided as an embodiment of the present invention. Detailed Implementation

[0050] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0051] Example 1

[0052] Reference Figures 1-2 This is the first embodiment of the present invention. This embodiment provides a time action localization method and system based on temporal context max pooling, including a time action localization method based on temporal context max pooling and a time action localization system based on temporal context max pooling. The time action localization method based on temporal context max pooling includes:

[0053] S101, acquire the action video to be identified, and extract the feature encoding sequence of the action video to be identified;

[0054] The process of acquiring the action video to be identified and extracting its feature encoding sequence includes: Let X be the action video to be identified; use a pre-defined feature encoding network to transform X into a set of feature encoding sequences, represented as X = {x1, x2, ..., x...}. T}, where the number of discrete time steps t = {1, 2, ..., T} depends on the video length, x T This represents the feature encoding sequence corresponding to the discrete time step T;

[0055] Specifically, the preset feature encoding network can be an open-source model or a custom feature encoding model. The preset feature encoding network can be a model trained by 3D convolutional neural network, I3D, C3D, etc.

[0056] In this embodiment, a custom feature encoding model is designed to extract the feature encoding sequence of the action video to be recognized. The specific steps are as follows:

[0057] A ResNet-3D-based architecture is established, constructing residual blocks consisting of multi-layer 3D convolutions, 3D batch normalization, and ReLU activation functions. For each residual block, the following improvements are added: a spatial attention module (such as a Squeeze-and-Excitation module) or a spatiotemporal attention module (such as a Non-local module) is added to the output of each residual block to enhance the model's attention to important spatiotemporal regions. The mathematical expression for the spatial attention module is:

[0058] SE(x)=σ(W2*ReLU(W1*GlobalAvgPool(x)))

[0059] Where x is the input feature, GlobalAvgPool is the global average pooling operation, W1 and W2 are learnable weight matrices, and σ is the sigmoid activation function.

[0060] Furthermore, a channel attention module (such as the CBAM module) can be added after the 3D convolutional layer to improve the model's ability to distinguish features from different channels.

[0061] Furthermore, dilated convolutions are used in some 3D convolutional layers to increase the receptive field and capture longer-range spatiotemporal dependencies.

[0062] Furthermore, the video of the action to be recognized is preprocessed, such as sampling a frame sequence with a fixed frame rate and adjusting the resolution, to obtain a video frame sequence V = {v1, v2, ..., vT}, where T is the frame number and vi is the i-th frame.

[0063] Furthermore, using a video frame sequence V as input, feature extraction is performed through an improved 3D-CNN network. Let the network have L layers, and the output of the l-th layer be the feature map Fl, then:

[0064] Fl=Improved3DResNet(V),l=1,2,...,L

[0065] Furthermore, global average pooling or global max pooling is applied to each feature map Fl to compress the spatial dimension into a single value, resulting in the feature vector fl:

[0066] fl=GlobalPooling(Fl),l=1,2,...,L

[0067] Furthermore, the feature vectors fl of each layer are arranged in order of network depth to form a feature encoding sequence E = {e1, e2, ..., eL}, where ei = fl.

[0068] It should be noted that using existing open-source models allows for direct utilization of existing training results, avoiding the need to train a model from scratch and saving time and computational resources. Pre-trained models can reduce training difficulty, making the extracted video features more consistent with domain characteristics. This allows for effective extraction of information from the video. Furthermore, open-source models often possess good versatility and scalability, allowing for customization and optimization based on specific task requirements.

[0069] It should be noted that custom models can be tailored and optimized based on specific task requirements and data characteristics, potentially resulting in better performance. Furthermore, custom models can be better integrated into existing systems or frameworks, enabling more efficient computation and deployment.

[0070] It should be noted that only some information in the video is related to the context of the action in time, while the rest of the information is similar between adjacent video segments. In order to preserve the most distinctive features of the video segments carrying basic information, this invention constructs the simplest long-term context module, which mainly consists of max pooling layers. This module helps the model capture long-term temporal dependencies between frames, which can be very useful for recognizing complex or overlapping actions over a sufficiently long time span. It can maximize the extraction of information from open-source models or custom feature encoding models, reduce training parameters, and reduce computational resources.

[0071] S102, Preset time-action localization model, input the feature encoding sequence into the time-action localization model to obtain action classification results;

[0072] The temporal action localization model includes an encoder, a long-term temporal context module, and a decoder. The input of the temporal action localization model is the feature encoding sequence of the action video to be identified, and the output is the action classification result. The long-term temporal context module is used to extract feature information in the multi-dimensional vector space to form a multi-scale temporal feature pyramid.

[0073] Furthermore, a preset temporal action localization model is used to input the feature encoding sequence into the temporal action localization model and obtain action classification results. The temporal action localization model includes an encoder, a long-term temporal context module, and a decoder. The encoder is a feature projection layer used to transform the feature encoding sequence into a multi-dimensional space vector.

[0074] Furthermore, the encoder's projection process can be represented as:

[0075] X P =E2(E1(Concat(X)))

[0076] Where X represents the video feature encoding sequence of the action to be identified, concatenated in the first dimension, and passed to two feature projection modules E1 and E2. Each projection module includes a one-dimensional convolutional neural network layer, a normalization layer, and a ReLU activation function. P This represents the projection result of the video feature encoding sequence of the action to be identified, i.e., a multidimensional space vector.

[0077] Furthermore, a pre-defined temporal action localization model is used to input feature encodings into the model to obtain action classification results. This also includes a long-term temporal context module for extracting feature information from a multi-dimensional vector space, forming a multi-scale temporal feature pyramid, and then... 1 =X P It is assigned as the first feature in the multi-scale time pyramid Z;

[0078] The long-term temporal context module is used to extract feature information from the multi-dimensional vector space, forming a multi-scale temporal feature pyramid, specifically represented as follows:

[0079] Z l =TemporalMaxer(Z l-1 )

[0080] TemporalMaxer represents the max pooling operation with a pooling window step size of 2;

[0081] Furthermore, the TemporalMaxer layer in the long-term temporal context module encodes the input features into a multi-scale temporal feature pyramid Z = {Z...} 1 Z 2 ,…,Z l};

[0082] Furthermore, the decoder learns to predict the sequence label Ψ = {ψ1, ψ2, ..., ψN} for each time step by using a multi-scale temporal feature pyramid, where N is the number of action instances in X, including a classification head module and a regression head module. All weights of the decoder are shared among the different features of the multi-scale feature pyramid.

[0083] Furthermore, the classification head module includes: two 1D convolutional layers, each followed by a normalization layer and a ReLU activation function, and then a single 1D convolutional layer after the two 1D convolutional layers, which classifies the actions according to the predefined categories and calculates the classification probability of each category.

[0084] The calculation process for the classification header module is as follows:

[0085] C l =F c (E4(E3(Z l )))

[0086] Where E3 and E4 represent 1D convolutional layers followed by a normalization layer and a ReLU activation function, F c 1D convolutional layer, C l Representing multi-scale temporal features Z l The results of the classification operation.

[0087] It should be noted that the designed classification head module effectively enhances the model's ability to extract deep features from temporal action sequences through the stacking of multiple 1D convolutional layers. Each convolutional layer is accompanied by a normalization layer and a ReLU activation function, which not only accelerates the model's training process but also enhances its non-linear mapping ability, enabling the model to better learn complex temporal action patterns. This classification head module classifies actions according to predefined categories and calculates the classification probability for each category. This design allows the model to provide more refined and comprehensive classification results when handling multi-class action recognition tasks, meeting the needs of practical applications for both accuracy and diversity in action recognition.

[0088] It should be noted that the classification head module works in conjunction with other parts of the decoder (such as the regression head module) to form a complete temporal action localization model. This modular design allows the model to better adapt to different task requirements and data characteristics during training and optimization, thereby improving the model's versatility and scalability.

[0089] Furthermore, the regression head module includes: two 1D convolutional layers, each followed by a normalization layer and a ReLU activation function, and then a single 1D convolutional layer and a ReLU activation function after the two 1D convolutional layers.

[0090] The calculation process of the regression head module is as follows:

[0091] O l =ReLU(F o (E6(E5(Z l ))))

[0092] Where E5 and E6 represent 1D convolutional layers followed by a normalization layer and a ReLU activation function, F o Represents a 1D convolutional layer, O l Representing multi-scale temporal features Z l The regression results.

[0093] It should be noted that the designed regression head module can extract rich temporal information from the multi-scale temporal feature pyramid, thereby more accurately predicting the start and end times of actions. Furthermore, the design of the regression head module also considers model complexity and computational efficiency, ensuring that the model maintains high performance without becoming overly complex, making it easy to deploy and use in practical applications.

[0094] Furthermore, the preset temporal action localization model, inputting feature encodings into the temporal action localization model, and obtaining action classification results also includes: establishing the loss function of the temporal action localization model, the loss function being expressed as:

[0095]

[0096] Among them, l cls Represents classification loss, l reg Let represent the regression loss, t represent the time step, and T1 represent the number of positive samples, i.e., the number of time steps containing the action.

[0097] It should be noted that this loss function combines classification and regression losses, comprehensively considering the model's performance in both action classification and action localization. The classification loss enables the model to accurately identify the action category in the video, while the regression loss enables it to accurately predict the start and end times of the action. This comprehensive approach allows the model to achieve better overall performance when handling complex temporal action localization tasks. The loss function employs a time-step weighting method, giving extra attention to positive samples (i.e., time steps containing action). This approach allows the model to focus more on time steps crucial to action classification and localization during training, thereby improving the model's robustness and accuracy. The design of this loss function also considers computational efficiency and model complexity. Through reasonable weight allocation and computational optimization, the model maintains high performance without becoming overly complex, making it easy to deploy and use in practical applications.

[0098] In an optional embodiment, the method further includes evaluating the temporal motion localization model. This evaluation can be performed using average precision, temporal intersection-over-union ratio (TIU), average time offset, and frame-level accuracy. This application improves the frame-level accuracy method to make it more suitable for evaluating the temporal motion localization model. Specifically, the improved frame-level accuracy evaluation method is as follows:

[0099] Step 1: Define basic metrics and clarify the basic frame-level accuracy calculation method.

[0100] Given a video V containing n action instances A i =(s i ,e i ,li ), where s i and e i , representing the start frame and end frame of the i-th action instance, respectively. i This represents its category label. The model predicts a set of candidate action instances P. j =(p s j, p e j, p l j). The traditional method for calculating frame-level accuracy is as follows:

[0101]

[0102] in, `m` is an indicator function that returns 1 if the condition is true and 0 otherwise; `m` is the number of predicted action instances. This formula only calculates perfectly matching action instances, ignoring the precision of start and end times.

[0103] Step Two: To focus more on the model's accurate prediction of the start and end times of actions, we can introduce the concept of time offset tolerance, defined as follows:

[0104]

[0105] Among them, |A i ∩P j | represents the number of frames that overlap between two intervals, |A i ∪P j | indicates the total number of frames after merging the two intervals. A threshold α is set (e.g., 0.5).

[0106] When TIOU(A) i ,P j When α ≥ α, it is considered that the prediction P j With real action A i There is sufficient time overlap.

[0107] Step 3: Incorporate time offset tolerance into accuracy calculation to obtain the improved frame-level accuracy:

[0108]

[0109] It should be noted that a prediction is only considered correct if the time range of the predicted action instance has sufficient overlap with the time range of the actual action instance.

[0110] Step 4: To make the evaluation focus more on the model's sensitivity to action duration, a weighting factor w is introduced, whose value is proportional to the action duration. For example:

[0111]

[0112] Where T is the total number of frames in the entire video. After weighting, the improved frame-level accuracy becomes:

[0113]

[0114] Step 5: Processing Multiple Action Instances. For videos containing multiple action instances, there may be situations where a predicted bounding box overlaps with multiple ground truth bounding boxes. To avoid double scoring, a non-maximum suppression strategy can be used. First, sort the predicted bounding boxes by TIoU, then retain the predicted bounding boxes whose overlap with the current highest-scoring ground truth bounding box exceeds a threshold, while removing other predicted bounding boxes with excessive overlap.

[0115] In an optional embodiment, two thresholds may also be set: a first threshold and a second threshold, wherein the first threshold is less than the second threshold.

[0116] When the improved frame-level accuracy falls below a first threshold, it indicates that the model's performance on the temporal action localization task has not met the expected basic standard. Possible reasons include an unsuitable model architecture, insufficient training data, and inappropriate hyperparameter selection. The following measures can be taken: Model optimization and hyperparameter tuning: Re-examine whether the model structure is suitable for the current task, and consider replacing or adjusting the model. Perform detailed hyperparameter tuning on the existing model, including learning rate, batch size, and regularization strength, to improve model performance. Data augmentation and expansion: Check the quality and quantity of the training dataset, and consider adding more samples and performing data augmentation (such as flipping, pruning, and time shifting) to improve the model's generalization ability. Preprocessing and feature engineering: Investigate whether the model's ability to recognize the start and end times of actions can be improved by improving input feature extraction and temporal feature encoding. Expert feedback and iteration: Communicate with domain experts to understand the reasons for the model's failure in specific scenarios, and adjust training strategies or collect specific types of samples to improve model performance.

[0117] When the improved frame-level accuracy is above the first threshold but below the second threshold, the model performance has met the basic requirements, but there is still room for improvement. The following can be done: Deeply analyze error cases: Carefully analyze samples misclassified or poorly located in the test set to identify common problems, such as difficulty in recognizing specific action categories, actions of specific lengths, or specific background interference, and optimize the model or adjust the training strategy accordingly. Ensemble learning and model fusion: Try using ensemble methods of multiple models (such as voting, averaging, stacking, etc.) to improve overall performance. Different models may be complementary in some aspects, and fusing their prediction results may improve the overall accuracy. Refine evaluation metrics: In addition to the overall frame-level accuracy, focus on other sub-metrics, such as the accuracy of each action category and the ability to detect long-tail actions, and perform specific optimizations for poorly performing subtasks. Continuous monitoring and online learning: Continuously collect user feedback and new data in practical applications, and use online learning or incremental learning techniques to update the model and maintain its adaptability to new scenarios.

[0118] When the improved frame-level accuracy exceeds the second threshold, and if the improved frame-level accuracy significantly surpasses the set high standard, it indicates that the model performs exceptionally well in the temporal action localization task. At this point, consider the following: Deployment and Application: The model has reached the expected performance standard and can be prepared for deployment in real-world application scenarios, such as video surveillance and intelligent video analytics systems. Resource Optimization: While ensuring performance, perform operations such as compression and quantization on the model to reduce computational resource consumption and improve inference speed, enabling it to run on resource-constrained hardware platforms. Continuous Maintenance and Version Iteration: Even with excellent model performance, it is necessary to periodically evaluate its performance on new data, monitor changes in model performance over time, and make timely fine-tuning or version upgrades.

[0119] In summary, this invention proposes a temporal action localization method based on temporal context max pooling. The method acquires a video of an action to be identified and extracts its feature encoding sequence. A pre-defined temporal action localization model is then established, and the feature encoding sequence is input into the model to obtain action classification results. The temporal action localization model includes an encoder, a long-term temporal context module, and a decoder. The input to the model is the feature encoding sequence of the video of the action to be identified, and the output is the action classification result. The long-term temporal context module is used to extract feature information in a multi-dimensional vector space, forming a multi-scale temporal feature pyramid. By introducing temporal context max pooling and a multi-scale temporal feature pyramid structure, computational complexity is effectively reduced, and inference speed and efficiency are improved. Furthermore, by designing a temporal action localization model that includes an encoder, a long-term temporal context module, and a decoder, the model can simultaneously capture short-term and long-term dependencies, thereby improving the accuracy of temporal action localization. These advantages make this invention valuable for applications in the field of temporal action localization.

[0120] In a preferred embodiment, a time-based action localization system based on temporal context max pooling includes:

[0121] The feature encoding sequence acquisition module is used to acquire the action video to be identified and extract the feature encoding sequence of the action video to be identified.

[0122] The action localization module is used to preset the time-based action localization model, input the feature encoding sequence into the time-based action localization model, and obtain the action classification result.

[0123] The temporal action localization model includes an encoder, a long-term temporal context module, and a decoder. The input of the temporal action localization model is the feature encoding sequence of the action video to be identified, and the output is the action classification result. The long-term temporal context module is used to extract feature information in the multi-dimensional vector space to form a multi-scale temporal feature pyramid.

[0124] The above-mentioned unit modules can be embedded in the processor of the computer device in hardware form or independent of it, or they can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above modules.

[0125] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 2 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a time-based action localization method based on temporal context max pooling. The display screen can be an LCD screen or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0126] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0127] Acquire the video of the action to be identified, and extract the feature encoding sequence of the video of the action to be identified;

[0128] A preset time-action localization model is used. The feature encoding sequence is input into the time-action localization model to obtain the action classification result.

[0129] The temporal action localization model includes an encoder, a long-term temporal context module, and a decoder. The input of the temporal action localization model is the feature encoding sequence of the action video to be identified, and the output is the action classification result. The long-term temporal context module is used to extract feature information in the multi-dimensional vector space to form a multi-scale temporal feature pyramid.

[0130] Example 2

[0131] For video A showing the maintenance actions of personnel inspecting power distribution towers, acquired by a drone, the following steps will be followed to process the video and ultimately achieve motion localization:

[0132] Video feature extraction (A→B):

[0133] First, a pre-trained custom feature encoding model is used to perform in-depth analysis of video A, extracting visual features closely related to maintenance actions from the video frame sequence. These features can characterize the form, trajectory, and interaction with other environmental elements of specific actions such as climbing towers, inspecting equipment, and operating tools by maintenance personnel. After 3D-CNN processing, the feature encoding vector sequence B is obtained, which is a highly abstract and condensed representation of the maintenance action information in the original video A.

[0134] Feature projection and enhancement (B→C):

[0135] The feature encoding sequence B is input into the feature projection layer (encoder). The encoder further processes B through two feature projection modules, each containing a convolutional neural network layer, a normalization layer, and a ReLU activation function. The purpose of this stage is to perform non-linear transformations and normalization on the initially extracted feature vectors from B using deep learning, thereby improving their ability to represent maintenance actions. After processing by the encoder, a feature vector sequence C is obtained, where each vector more accurately reflects the maintenance action state at a specific moment in video A.

[0136] Temporal context modeling (C→D):

[0137] The feature vector sequence C is fed into the long-term temporal context module (TemporalMaxer layer). This module captures the most critical maintenance action information from adjacent and local clip embeddings through max pooling operations. It also considers that actions may span consecutive video frames, forming a multi-scale temporal feature pyramid D. This processing helps the model understand and identify the maintenance personnel's maintenance behaviors at different time scales (such as instantaneous actions and continuous action phases), such as rapid climbing and continuous meticulous inspection.

[0138] Multi-scale feature fusion and decoding (D→E):

[0139] The multi-scale feature pyramid D contains maintenance action features at different temporal granularities. The decoder learns to use these features to predict the maintenance action sequence label at each time step. The decoder includes a classification head and a regression head. The classification head classifies predefined maintenance action categories (such as climbing, inspection, and repair) and calculates the classification probability for each category. The regression head estimates the start and end times of each maintenance action in the video. The decoder's processing result E contains both the most likely action category corresponding to each video frame at each time step and the specific time interval in which these actions occurred.

[0140] Motion localization result generation (E → final result):

[0141] Finally, based on the decoder output E, the classification results of maintenance actions at each moment in video A and the corresponding action occurrence time are obtained. These results together constitute the precise location of the maintenance actions of the distribution network tower maintenance personnel in video A, including the action type, the time point of the action, and the duration. By interpreting this location information, we can gain a detailed understanding of the maintenance personnel's workflow, work efficiency, and whether they followed safety regulations in video A, providing valuable data support for distribution network maintenance management.

[0142] In summary, for video A of maintenance personnel's actions on power distribution towers acquired by UAVs, a series of processing steps, including video feature extraction, feature projection and enhancement, temporal context modeling, and multi-scale feature fusion and decoding, were used to achieve precise localization of the maintenance personnel's actions, providing strong support for power distribution network operation and maintenance analysis.

[0143] Example 3

[0144] Reference Figures 1-2 As an embodiment of the present invention, a time action localization method and system based on temporal context max pooling is provided. To verify the beneficial effects of the present invention, comparative experiments are conducted for scientific demonstration.

[0145] The proposed method is evaluated on the THUMOS14 dataset and compared with state-of-the-art methods. The widely used Temporal Action Localization (TAL) evaluation metric, mean average precision (mAP), is used, calculated over various temporal intersection regions. tIoU is the ratio of the intersection to the union of two temporal windows, i.e., the one-dimensional Jaccard exponent. TemporalMaxer demonstrates significant performance, achieving a mean mAP of 67.7%, significantly outperforming all previous methods, both single-stage and two-stage, and increasing mAP by 1.1% at tIoU = 0.4. The proposed method takes an average of only 50 milliseconds to fully process the entire video on the THUMOS dataset. This is 1.6 times faster than the ActionFormer baseline and 3.9 times faster than TadTR.

[0146] The method of this invention demonstrates superior performance in temporal action localization tasks by efficiently achieving high inference speed and accuracy across multiple datasets. TemporalMaxer provides a simple and efficient method for temporal action localization tasks, offering excellent performance and fast inference speed, making it suitable for practical applications on large-scale video datasets.

[0147] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

[0148] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0149] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0152] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0153] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A temporal action localization method based on temporal context max pooling, characterized in that, include: Acquire the action video to be identified, and extract the feature encoding sequence of the acquired action video; A preset time-action localization model is used. The feature encoding sequence is input into the time-action localization model to obtain the action classification result. The temporal action localization model includes an encoder, a long-term temporal context module, and a decoder. The input of the temporal action localization model is the feature encoding sequence of the action video to be identified, and the output is the action classification result. The long-term temporal context module is used to extract feature information in a multi-dimensional vector space to form a multi-scale temporal feature pyramid.

2. The time-based action localization method based on temporal context max pooling as described in claim 1, characterized in that, The process of acquiring the action video to be identified and extracting the feature encoding sequence of the acquired action video includes: Let X be a video of an action to be recognized. Using a pre-defined feature encoding network, X is transformed into a set of feature encoding sequences, represented as X = {x1, x2, ..., x...}. T }, where the number of discrete time steps t = {1, 2, ..., T} depends on the video length, x T This represents the feature encoding sequence corresponding to the discrete time step T; The preset feature encoding network is an open-source model or a custom feature encoding model.

3. The time-based action localization method based on temporal context max pooling as described in claim 2, characterized in that, The preset time-action localization model inputs the feature encoding sequence into the time-action localization model to obtain action classification results, including: The time-action localization model includes an encoder, a long-term time context module, and a decoder. The encoder is a feature projection layer used to convert the feature encoding sequence into a multi-dimensional space vector. The projection process of the encoder is represented as follows: X P =E2(E1(Concat(X))) Where X represents the video feature encoding sequence of the action to be identified, concatenated in the first dimension, and passed to two feature projection modules E1 and E2. Each projection module includes a one-dimensional convolutional neural network layer, a normalization layer, and a ReLU activation function. P This represents the projection result of the video feature encoding sequence of the action to be identified, i.e., a multidimensional space vector.

4. The time-based action localization method based on temporal context max pooling as described in claim 3, characterized in that, The preset time-action localization model, by inputting the feature encoding into the time-action localization model to obtain the action classification result, further includes: The long-term temporal context module is used to extract feature information from the multi-dimensional vector space, forming a multi-scale temporal feature pyramid, and then... 1 =X P It is assigned as the first feature in the multi-scale time pyramid Z; The long-term temporal context module is used to extract feature information in a multi-dimensional vector space, forming a multi-scale temporal feature pyramid, specifically represented as follows: WITH l =TemporalMaxer(Z l-1 ) TemporalMaxer represents the max pooling operation with a pooling window step size of 2; The TemporalMaxer layer in the long-term temporal context module encodes the input features into a multi-scale temporal feature pyramid Z = {Z}. 1 Z 2 ,…,Z l }; The decoder learns to predict the sequence label Ψ = {ψ1, ψ2, ..., ψN} for each time step using a multi-scale temporal feature pyramid, where N is the number of action instances in X. It includes a classification head module and a regression head module, and all weights of the decoder are shared among different features of the multi-scale feature pyramid.

5. The time-based action localization method based on temporal context max pooling as described in claim 4, characterized in that, The classification header module includes: The classification head module includes two 1D convolutional layers, each followed by a normalization layer and a ReLU activation function. After the two 1D convolutional layers, a single 1D convolutional layer is added to classify the actions according to a predefined category and to calculate the classification probability of each category. The calculation process for the classification header module is as follows: C l =F c (E4(E3(Z l ))) Where E3 and E4 represent 1D convolutional layers followed by a normalization layer and a ReLU activation function, F c 1D convolutional layer, C l Representing multi-scale temporal features Z l The results of the classification operation.

6. The time-based action localization method based on temporal context max pooling as described in claim 4, characterized in that, The regression head module includes: The regression head module includes two 1D convolutional layers, each followed by a normalization layer and a ReLU activation function, and then a single 1D convolutional layer and a ReLU activation function are added after the two 1D convolutional layers. The calculation process of the regression head module is as follows: O l =ReLU(F o (E6(E5(Z l )))) Where E5 and E6 represent 1D convolutional layers followed by a normalization layer and a ReLU activation function, F o Represents a 1D convolutional layer, O l Representing multi-scale temporal features Z l The regression results.

7. The time-based action localization method based on temporal context max pooling as described in claim 6, characterized in that, The preset time-action localization model, by inputting the feature encoding into the time-action localization model to obtain the action classification result, further includes: Establish a loss function for the time-motion localization model, the loss function being expressed as: Among them, l cls Represents classification loss, l reg Let represent the regression loss, t represent the time step, and T1 represent the number of positive samples, i.e., the number of time steps containing the action.

8. A time-based action localization system based on temporal context max pooling, characterized in that, include: The feature encoding sequence acquisition module is used to acquire the action video to be identified and extract the feature encoding sequence of the acquired action video; The action localization module is used to preset a time-based action localization model, input the feature encoding sequence into the time-based action localization model, and obtain action classification results; The temporal action localization model includes an encoder, a long-term temporal context module, and a decoder. The input of the temporal action localization model is the feature encoding sequence of the action video to be identified, and the output is the action classification result. The long-term temporal context module is used to extract feature information in a multi-dimensional vector space to form a multi-scale temporal feature pyramid.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images