A video time sequence action positioning method, system, device and medium based on agent attention and multi-scale Transformer

Abstract

A video timing action positioning method, system, device and medium based on agent attention and multi-scale Transformer, the method: using a pre-trained feature extractor to extract features from an input video; construct a timing action positioning model based on agent self-attention and multi-scale Transformer, stack seven Transformer modules, for encoding the input video features, extract key information; different scale features output by the multi-scale Transformer module and the agent Transformer module are sent into the regression head and the classification head to obtain the positioning result, that is, the starting time, the ending time and the category label of the action instance in the uncropped video; the system, device and medium are used to realize the method; the application reduces the similarity between video features, reduces the parameter quantity and the calculation complexity of the model, improves the representation ability of the model, and further improves the video action positioning precision by combining the mixed module of convolution and Transformer architecture and the fine-grained agent self-attention module.

Description

Technical Field

[0001] This invention belongs to the field of video understanding and computer vision technology, specifically relating to a video temporal action localization method, system, device and medium based on agent attention and multi-scale Transformer. Background Technology

[0002] With the rapid development of social informatization, the scale and complexity of video data are constantly increasing, making automatic video analysis increasingly important. In the field of video content analysis, how to quickly extract effective information from videos has become a problem that researchers are paying more and more attention to. As one of the key tasks in video content analysis, video temporal motion localization technology has wide applications in many fields, including video surveillance and security, intelligent transportation systems, sports analysis, advertising and media analysis, and medical image analysis. Therefore, improvements in video temporal motion localization methods can not only promote technological progress in the field of computer vision, but also have profound significance in practical applications. Its goal is to locate all human motion segments in uncropped videos, providing support for a series of downstream tasks.

[0003] Currently, mainstream video temporal action localization methods can be divided into convolutional network-based methods and Transformer network-based methods. Traditional video temporal action localization methods typically combine hand-designed features with traditional machine learning methods. However, with the rise of deep learning, researchers have begun to utilize convolutional networks to solve video problems. Convolutional network-based video temporal action localization methods can generally be divided into two-stage and single-stage methods. Two-stage methods involve two phases: candidate action segment generation and action classification. In the first phase, the model generates candidate segments that may contain actions using a sliding window or other methods. In the second phase, the model further analyzes these candidate segments, including classification and boundary refinement, to determine the final action instance. This method involves two relatively independent phases, requiring separate optimization of the models for each phase. This increases the difficulty of algorithm design and debugging. Furthermore, the first phase typically generates a large number of candidate segments, including many redundant segments. Therefore, this method may not meet the requirements for real-time applications or when rapid processing of large-scale video data is needed. Single-stage action localization methods, on the other hand, simultaneously generate and refine action segments within a unified framework, avoiding the complexity and redundant computation of two-stage methods. However, single-stage methods that directly predict action location and category can lead to model performance instability, especially when dealing with complex scenes and actions. Furthermore, single-stage methods typically operate across the entire video sequence, requiring consideration of the contextual information of the entire sequence; therefore, the model may perform poorly when handling long videos.

[0004] In recent years, due to the excellent performance of the Transformer model in natural language processing, researchers have begun to apply it to computer vision, including video temporal action localization. Compared with convolutional network-based video temporal action localization methods, Transformer-based methods, due to their unique self-attention mechanism, have shown significant advantages in capturing subtle differences between video frames and modeling long-distance temporal dependencies between action subjects. Furthermore, the parallel processing capability of the self-attention mechanism gives Transformer a potential efficiency advantage when processing large-scale video data. Although recent research shows that Transformer has demonstrated superior performance in video sequence processing, existing Transformer-based video temporal action localization methods still have some shortcomings. For example, during model training, the attention computation of Transformer further increases the similarity between video frames, and excessively high similarity may cause rank loss. In addition, Transformer only uses simple linear transformations to represent video features, limiting the overall performance of the model.

[0005] An uncropped video typically contains multiple action instances, such as running and speaking, and the lengths of these action instances often vary considerably. Therefore, accurately locating action instances from an uncropped video is a challenging task. Convolutional methods are limited by a fixed receptive field size, resulting in poor performance when processing long sequences of data. Transformer networks, however, are not limited by a fixed receptive field and use a self-attention mechanism to capture long-distance dependencies in sequences. However, the key parameters Q (query), K (key), and V (value) in the Transformer's attention computation are generated through three simple linear transformations, resulting in insufficient representational power and limiting the model's overall feature representation capabilities. Furthermore, frames in a video often exhibit high similarity, and the attention computation in the Transformer further enhances this similarity, leading to rank loss during training.

[0006] In their paper "ActionFormer: Localizing Moments of Actions with Transformers," Chenlin Zhang et al. proposed a Transformer-based single-stage anchor-free video temporal action localization model. This model transforms the temporal action localization problem into a sequence labeling problem, labeling each moment as a triple containing the start time, end time, and action category. The specific steps of the model are as follows: ① First, extract features from the input video using a convolutional network. ② Stack multiple Transformer layers as an encoder to construct a multi-scale feature pyramid, capturing action information at different scales through downsampling operations at different levels. ③ In the Transformer encoder, the model employs a local self-attention mechanism to reduce computational complexity while maintaining effective modeling of long-term dependencies in the video sequence. ④ Use a lightweight convolutional network as a decoder, containing a classification head and a regression head: the classification head is responsible for predicting the action category at each time step, while the regression head estimates the start and end time boundaries of the action. However, frames in a video often exhibit high similarity. During model training, the attention computation in the Transformer further amplifies this similarity. Excessive similarity can lead to rank loss, thus impacting model performance. Furthermore, while the Transformer framework possesses powerful representation capabilities, its initial embedding layer employs a simple linear transformation, limiting the overall modeling power of the Transformer. Moreover, Transformer models typically have a large number of parameters, requiring significant computational resources for training, especially when processing long sequences, where computational complexity increases dramatically, making both training and inference expensive.

[0007] In their paper "TriDet: Temporal Action Detection with Relative Boundary Modeling," Dingfeng Shi et al. proposed TriDet, a single-stage temporal action detection framework aimed at improving the accuracy and efficiency of action detection in videos. The model's specific components are: ① A novel detection head, Trident-head, is designed to model action boundaries by estimating the relative probability distribution of the boundaries rather than directly predicting the offsets. ② A Scalable Granularity Perception (SGP) layer is proposed to replace the self-attention mechanism in the Transformer. This layer employs fully convolutional modules to enhance feature discrimination and capture multi-scale temporal information, thus addressing the rank loss problem encountered by the self-attention mechanism in video feature processing. ③ A relative boundary modeling method is used. By analyzing the relative feature intensity near the boundary, boundary information is effectively utilized to estimate the relative probability distribution of the boundary, thereby accurately locating the start and end times of the action. However, convolutional networks are limited by a fixed receptive field size, and can only extract video features within a local window. They are unable to capture long-distance dependencies between video frames, and perform poorly, especially when dealing with long-term actions or action sequences involving long time spans.

[0008] In summary, the shortcomings of existing technologies include:

[0009] First, video temporal action localization methods based on convolutional networks are limited by a fixed receptive field size, which can only extract video features within a local window. This makes it difficult to capture long-distance dependencies between video frames, especially when dealing with long-term actions or action sequences involving a long time span.

[0010] Second, there is often a high degree of similarity between frames in a video. During model training, the attention calculation in Transformer will further increase this similarity. Excessive similarity may cause rank loss, thereby affecting the performance of the model.

[0011] Third, although the Transformer framework has powerful representation capabilities, its initial embedding layer uses a simple linear transformation, which limits the overall modeling capabilities of the Transformer.

[0012] Fourth, Transformer models typically have a large number of parameters and require significant computational resources during training, especially when dealing with long sequences, where computational complexity increases dramatically, making the training and inference process expensive. Summary of the Invention

[0013] To overcome the shortcomings of the prior art, the present invention aims to provide a video temporal action localization method, system, device, and medium based on surrogate attention and multi-scale Transformer. By combining a hybrid module of convolution and Transformer architectures and a proposed fine-grained surrogate self-attention module, the similarity between video features is reduced, the number of model parameters and computational complexity are reduced, the model's representation ability is improved, and thus the localization accuracy of video actions is improved.

[0014] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0015] A video temporal action localization method based on agent attention and multi-scale Transformer includes the following steps:

[0016] Step 1: Extract features from the input video using a pre-trained feature extractor;

[0017] Step 2: Construct a temporal action localization model based on surrogate self-attention and multi-scale Transformer, stacking seven Transformer modules, of which the first two are multi-scale Transformer modules and the last five are surrogate Transformer modules, used to encode the input video features and extract key information.

[0018] Step 3: Input the different scale features output by the multi-scale Transformer module and the proxy Transformer module constructed in Step 2 into the regression head and the classification head to obtain the localization results of the temporal action localization model based on proxy self-attention and multi-scale Transformer, namely the start time, end time and action category label of the action instance in the uncropped video.

[0019] The specific method for step 2 is as follows:

[0020] Step 2.1: Construct the multi-scale Transformer module, whose structure is as follows:

[0021] Step 2.1.1, for the input video features First, the feature x is processed by Layer Normalization. Then, the processed feature is fed into the Multi-Scale Attention (MSA) module for attention calculation. The MSA module uses a lightweight convolutional structure, and its calculation process is as follows:

[0022]

[0023] Step 2.1.2, process the feature O obtained in step 2.1.1 through the multi-scale attention module MSA. S1 As input, the input features are processed using group normalization. Then, a multilayer perceptron (MLP) is used to extract higher-level feature representations from the video. The MLP consists of two linear transformation layers and an activation function. Finally, a residual structure is used to fuse these features, yielding the final output O of the multi-scale Transformer module. MS The specific calculation process is as follows:

[0024]

[0025] Where W1 and W2 are the weight matrices of the linear transformation layer, b1 and b2 are the basis vectors, and GELU is the activation function of the Gaussian error linear unit, calculated as follows:

[0026]

[0027] Step 2.2, construct the proxy Transformer module, with the following module structure:

[0028] Step 2.2.1, for the input video features First, the feature x is processed by Layer Normalization. Then, the processed feature is fed into the Fine-Grained Agent Self-Attention (FGASA) module for attention calculation, resulting in the output O. S2 The fine-grained proxy self-attention module FGASA uses a quadruple (Q', A, K', V'), where Q', K', and V' are the outputs of the reconstruction and refinement unit, and A is obtained by performing average pooling on Q' over time. The computation process of the fine-grained proxy self-attention module is as follows:

[0029] O S2 =x+FGASA(Q',A,K',V')

[0030] Step 2.2.2, process the feature O after step 2.2.1 through the fine-grained proxy self-attention module FGASA. S2 As input, the features are first processed through a group normalization layer, then a multilayer perceptron is used to extract higher-level feature representations of the video; finally, a residual structure is used to fuse these features to obtain the final output O of the proxy Transformer module. Agent :

[0031]

[0032] The structure of the multi-scale attention module (MSA) in step 2.1.1 is as follows:

[0033] For the input video feature x, the Multi-Scale Attention (MSA) module processes it in two branches. In the multi-scale branch, the input feature is first fed into two one-dimensional convolutional layers with different kernel sizes to extract feature information at different time scales. Then, the feature information at the two different time scales is fused, and then batch normalization and the softmax function are applied sequentially to obtain the output y of the multi-scale branch. multi ;

[0034] In the attention branch, the input features are fed into a one-dimensional convolutional layer with a kernel size of 1 for processing to obtain y. c Simultaneously, the attention branch performs average pooling and max pooling on the input features along the time dimension to extract feature information from different aspects. The results of both pooling operations are then sequentially fed into the same multi-layer perceptron (MLP) to obtain two weights w. avg and w max Next, the two weights w avg and w max The components are fused and processed using the sigmoid function to obtain the final weights w; finally, the weights w are used to adjust the output y of the one-dimensional convolution. c We perform weighted summation to obtain the output y of the attention branch. attn The specific calculation process is as follows:

[0035] y attn =Sigmoid(w avg +w max )y c

[0036]

[0037] Adding the results of the multi-scale branch and the attention branch, we obtain the output y1 of the multi-scale attention module MSA, i.e.:

[0038] y1 = MSA(x)

[0039] =y multi +y attn .

[0040] The structure of the fine-grained proxy self-attention module FGASA in step 2.2.1 is as follows:

[0041] For the input video features x, three different linear transformations are first applied. The input video feature x is transformed into Query, Key, and Value, abbreviated as Q, K, and V, i.e.: Q = xW Q K = xW K V = xW V Next, Q, K, and V are input into the Reconstructive Refinement Unit (RRU) to obtain more refined Query, Key, and Value, namely:

[0042] Q',K',V'=RRU(Q,K,V)

[0043] Then, A is introduced as a proxy for querying Q' to first aggregate information from K' and V', and then propagate the aggregated global information to Q' to obtain the final output of the attention calculation. The specific calculation process is as follows:

[0044]

[0045] Here, A is obtained by performing average pooling on query Q' over the time dimension. Therefore, A contains global information about Q', and the number of tokens is configurable. Additionally, bias terms B1 and B2 are introduced in the two attention calculation processes, respectively. The specific calculation process is as follows:

[0046]

[0047] Where, d a =d q =C; Finally, the result of the input x and the attention calculation is fused through the residual structure as the output O. S2 ,Right now:

[0048]

[0049] The specific structure of the Reconstruction Refinement Unit (RRU) is as follows:

[0050] First, for the input video features, three different linear transformations are applied to obtain three different feature representations, namely matrices Q, K, and V. Next, Q, K, and V are merged along the channel dimension. Then, a cyclic shift operation is performed on Q, K, and V along the merged channel dimension, shifting the first r channels down and the next r channels up, filling the gaps created by the shift with the shifted portions. Finally, Q, K, and V are separated along the channel dimension to restore independent feature matrices. Simultaneously, a shared-parameter convolution operation is performed on each of the cyclically shifted Q, K, and V to obtain new feature representations Q', K', and V', where r represents the shift ratio.

[0051] The value of r is 1 / 3, 1 / 4, 1 / 8, or 1 / 16.

[0052] This invention also provides a video temporal motion localization system based on agent attention and multi-scale Transformer, comprising:

[0053] The video feature extraction module is used to extract features from the input video using a pre-trained feature extractor.

[0054] The temporal action localization model building module based on surrogate self-attention and multi-scale Transformer is used to build a temporal action localization model based on surrogate self-attention and multi-scale Transformer. It stacks seven Transformer modules, of which the first two are multi-scale Transformer modules and the last five are surrogate Transformer modules, which are used to encode the input video features and extract key information.

[0055] The localization result acquisition module is used to input the different scale features output by the multi-scale Transformer module and the surrogate Transformer module into the regression head and the classification head to obtain the localization result of the temporal action localization model based on surrogate self-attention and multi-scale Transformer, namely the start time, end time and action category label of the action instance in the uncropped video.

[0056] The present invention also provides a video temporal motion localization device based on agent attention and multi-scale Transformer, comprising:

[0057] Memory: A computer-readable device that stores the computer program of the above-mentioned video temporal action localization method based on agent attention and multi-scale Transformer;

[0058] Processor: Used to implement the video temporal action localization method based on agent attention and multi-scale Transformer when executing the computer program.

[0059] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the aforementioned video temporal motion localization method based on agent attention and multi-scale Transformer.

[0060] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0061] 1. This invention improves the accuracy of action localization by designing a lighter multi-scale attention module that reduces the similarity between video features during video action localization and effectively captures feature information at different scales in the video. Furthermore, the lightweight design of the multi-scale attention module reduces redundant computation, improves the model's computational efficiency, and enables the model to process video data more efficiently.

[0062] 2. This invention applies agent attention to the video temporal action localization model, which significantly reduces the number of model parameters while maintaining model performance, making the model more lightweight and thus alleviating the problem of large number of parameters and high computational complexity of the current Transformer-based video action localization model.

[0063] 3. This invention, through the design of a reconstruction and refinement unit, enables the model to capture different features of the input data in different data representation subspaces, promoting better adaptation of the model to diverse data distributions. Only a small number of parameters are needed to effectively improve model performance. Simultaneously, by combining the reconstruction and refinement unit with proxy attention, a fine-grained proxy attention module is constructed, further enhancing the model's feature representation capabilities. This allows the model to better capture the relationships between video features, thereby improving the accuracy and precision of action localization.

[0064] In summary, the multi-scale attention module designed in this invention is a more lightweight module that can extract multi-scale feature information while reducing the feature similarity between video frames. This invention also designs a reconstruction refinement unit and, combined with a proxy self-attention mechanism, proposes a fine-grained proxy self-attention mechanism. This reduces the number of model parameters, improves the efficiency of model training and inference, and enhances the model's representation ability, thereby improving the accuracy of video temporal action localization. Attached Figure Description

[0065] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0066] Figure 2 This is a structural diagram of the multi-scale Transformer module of the present invention.

[0067] Figure 3 This is the multi-scale attention structure diagram of the present invention.

[0068] Figure 4 This is a structural diagram of the proxy Transformer module of this invention.

[0069] Figure 5 This is a diagram of the fine-grained proxy self-attention structure of the present invention.

[0070] Figure 6 This is a structural diagram of the reconstructed and refined unit of the present invention. Detailed Implementation

[0071] The technical solution adopted by the present invention will be further described below with reference to the accompanying drawings.

[0072] The implementation of this invention mainly consists of a multi-scale Transformer module, a reconstruction and refinement unit, and a proxy self-attention module. The following is in conjunction with the appendix... Figure 1 Appendix Figure 2 Appendix Figure 3 Appendix Figure 4 Appendix Figure 5 and attached Figure 6 The present invention will be described in detail below.

[0073] like Figure 1 As shown, in step 1, features are extracted from the input video using a pre-trained feature extractor;

[0074] Step 2: Construct a temporal action localization model based on proxy self-attention and multi-scale Transformer. This model consists of seven modules, including two multi-scale Transformer modules and five proxy Transformer modules, which are used to encode the input video features and extract key information.

[0075] The multi-scale Transformer module includes an LN (layer normalization) layer, a multi-scale attention module, a GN (group normalization) layer, and a multilayer perceptron; the surrogate Transformer module includes an LN (layer normalization) layer, a fine-grained surrogate self-attention module, a GN (group normalization) layer, and a multilayer perceptron.

[0076] Specifically, this invention designs a multi-scale attention module to replace the attention module in the original Transformer, thus forming a multi-scale Transformer module. Furthermore, this invention designs a reconstruction refinement unit and, combined with proxy attention, proposes a fine-grained proxy self-attention mechanism. Applying this mechanism to the original Transformer constitutes the proxy Transformer module of this invention.

[0077] like Figure 2 As shown in step 2.1, construct the multi-scale Transformer module. The module structure is as follows:

[0078] Step 2.1.1, for the input video features First, the feature x is processed by Layer Normalization. Then, the processed feature is fed into the Multi-Scale Attention (MSA) module for attention calculation. The MSA module uses a lightweight convolutional structure, which reduces the number of model parameters and reduces the feature similarity between video frames, thereby improving model performance. The calculation process is as follows:

[0079] O S1 =x + MSA(x)

[0080] Step 2.1.2, unlike the traditional Transformer architecture, this invention uses the feature O processed by the multi-scale attention module MSA in step 2.1.1. S1 As input, the input features are processed using group normalization. Then, a multilayer perceptron (MLP) is used to extract higher-level feature representations from the video. The MLP consists of two linear transformation layers and an activation function. Finally, a residual structure is used to fuse these features, yielding the final output O of the multi-scale Transformer module. MS The specific calculation process is as follows:

[0081]

[0082] Where W1 and W2 are the weight matrices of the linear transformation layer, b1 and b2 are the basis vectors, and GELU is the activation function of the Gaussian error linear unit, calculated as follows:

[0083]

[0084] like Figure 3 As shown, the structure of the multi-scale attention module (MSA) in step 2.1.1 is as follows:

[0085] For the input video feature x, the Multi-Scale Attention (MSA) module processes it in two branches. In the multi-scale branch, the input feature is first fed into two one-dimensional convolutional layers with different kernel sizes to extract feature information at different time scales. Then, the feature information at the two different time scales is fused, and then batch normalization and the softmax function are applied sequentially to obtain the output y of the multi-scale branch. multi ;

[0086] In the attention branch, the input features are fed into a one-dimensional convolutional layer with a kernel size of 1 for processing to obtain y. cSimultaneously, the attention branch performs average pooling and max pooling on the input features along the time dimension to extract feature information from different aspects. The results of these two pooling operations are then sequentially fed into the same multi-layer perceptron (MLP) to obtain two weights w. avg and w max Next, the two weights w avg and w max The components are fused and processed using the sigmoid function to obtain the final weights w; finally, the weights w are used to adjust the output y of the one-dimensional convolution. c We perform weighted summation to obtain the output y of the attention branch. attn The specific calculation process is as follows:

[0087] y attn =Sigmoid(w avg +w max )y c

[0088]

[0089] Adding the results of the multi-scale branch and the attention branch, we obtain the output y1 of the multi-scale attention module MSA, i.e.:

[0090] y1 = MSA(x)

[0091] =y multi +y attn ;

[0092] like Figure 4 As shown in step 2.2, construct the proxy Transformer module. The module structure is as follows:

[0093] Step 2.2.1, for the input video features First, the feature x is processed by Layer Normalization. Then, the processed feature is fed into the Fine-Grained Agent Self-Attention (FGASA) module for attention calculation, resulting in the output O. S2The fine-grained proxy self-attention module FGASA differs from the traditional multi-head cross-attention module in that the traditional multi-head cross-attention module uses a triad (Q, K, V) for attention calculation, where Q, K, and V are obtained by performing three different linear transformations on the input features. In contrast, the fine-grained proxy self-attention module in this invention uses a quadruple (Q', A, K', V'), where Q', K', and V' are the outputs of the reconstruction and refinement unit designed in this invention, and A is obtained by performing average pooling on Q' over time. The calculation process of the fine-grained proxy self-attention module is as follows:

[0094]

[0095] Step 2.2.2: Process the features obtained in Step 2.2.1 using the fine-grained proxy self-attention module FGASA. As input, the features are first processed through a group normalization layer, then a multilayer perceptron is used to extract higher-level feature representations from the video; finally, a residual structure is used to fuse these features to obtain the final output O of the proxy Transformer. Agent :

[0096]

[0097] like Figure 5 As shown, the structure of the fine-grained proxy self-attention module FGASA in step 2.2.1 is as follows:

[0098] For the input video features x, three different linear transformations are first applied. The input video feature x is transformed into Query, Key, and Value (Q, K, V for short), i.e.: Q = xW Q K = xW K V = xW V Next, Q, K, and V are input into the Reconstructive Refinement Unit (RRU) to obtain more refined Query, Key, and Value, namely:

[0099] Q',K',V'=RRU(Q,K,V)

[0100] Then, A is introduced as a proxy for querying Q' to first aggregate information from K' and V', and then propagate the aggregated global information to Q' to obtain the final output of the attention calculation. The specific calculation process is as follows:

[0101]

[0102] Here, A is obtained by performing average pooling on query Q' over the time dimension. Therefore, A contains global information about Q', and the number of tokens is configurable. In this invention, it is set to 49, which is much smaller than the number of tokens in Q', thus greatly reducing the number of model parameters and computational complexity. Simultaneously, bias terms B1 and B2 are introduced in the two attention calculation processes, enabling the model to better capture long-distance dependencies in the sequence, thereby improving model performance. The specific calculation process is as follows:

[0103]

[0104] Where, d a =d q =C; Finally, the result of the input x and the attention calculation is fused through the residual structure as the output O. S2 ,Right now:

[0105]

[0106] like Figure 6 As shown, the specific structure of the Reconstructive Refinement Unit (RRU) is as follows:

[0107] First, for the input video features, three different linear transformations are applied to obtain three different feature representations, namely matrices Q, K, and V. Next, Q, K, and V are merged along the channel dimension. Then, a cyclic shift operation is performed on Q, K, and V along the merged channel dimension, allowing the model to capture different features of the input data in different data representation subspaces, thus promoting better adaptation to diverse data distributions. Specifically, the first 1 / 3 of the channels are shifted down, and the last 1 / 3 are shifted up. The shifted portions are then used to fill the gaps created by the shift. Finally, Q, K, and V are separated along the channel dimension, restoring them as independent feature matrices. Simultaneously, to integrate information from different data representation subspaces, a shared-parameter convolution operation is performed on the cyclically shifted Q, K, and V, respectively, resulting in new feature representations Q', K', and V'.

[0108] In summary, by performing cyclic shifts and convolution operations with shared parameters on Q, K, and V, which were originally obtained through linear transformations, the model can better learn the diverse feature representations of the input data with a small number of parameters, thereby improving the model's generalization ability and enabling it to better cope with different types and distributions of input data.

[0109] Step 3: Input the video features output by the multi-scale Transformer module and the proxy Transformer module constructed in Step 2 into the regression head and the classification head to obtain the localization results of the temporal action localization model based on proxy self-attention and multi-scale Transformer, namely the start time, end time and action category label of the action instance in the uncropped video.

[0110] The purpose of this invention is to solve the aforementioned problems in the prior art by reducing the number of model parameters while further exploring the representational capabilities of the Transformer, thereby improving the performance of video temporal action localization models. First, an attention structure centered on a convolutional network is designed, establishing a hybrid module combining convolution and Transformer to reduce the similarity between video features without increasing the number of additional parameters. Second, a fine-grained proxy self-attention mechanism based on multi-head self-attention is proposed to construct a proxy Transformer module, reducing the number of model parameters and computational complexity while maintaining model performance. Furthermore, a more efficient embedding structure is designed to replace the simple linear transformation in the Transformer, effectively improving the performance of Transformer-based video action localization models with only a small increase in parameters.

[0111] This invention aims to address the limited representational capabilities of the attention mechanism in the original Transformer by designing a novel attention structure to enhance the representational power of the original Q, K, and V components, thereby capturing complex relationships in video sequences and improving the overall representational ability of the model. Furthermore, a simple and effective convolutional structure is designed to replace part of the attention computation, appropriately reducing the similarity between video frames, thus alleviating the rank loss problem and improving the accuracy of temporal action localization in videos.

[0112] The multi-scale attention module in this invention employs a simple convolutional structure to capture multi-scale information of the input features, while simultaneously reducing the computational complexity of the model and the similarity between input features. The objective of this invention can also be achieved by replacing the multi-scale attention module with the SGP module in Tridet.

[0113] The reconstruction and refinement unit in this invention reconstructs the Q, K, and V components in the Transformer attention calculation by designing an effective shifting pattern and convolutional design, thereby enhancing the feature representation capabilities of Q, K, and V. If other shifting patterns are used, such as shifting the front 1 / 4 channel upwards and the back 1 / 4 channel downwards along the feature channel dimension; ② shifting the front 1 / 8 channel upwards and the back 1 / 8 channel downwards; ③ shifting the front 1 / 16 channel upwards and the back 1 / 16 channel downwards, and corresponding convolutional structures are designed, the same objective can be achieved. Experiments in this invention show that the model performs best when the feature channel dimension is shifted upwards by 1 / 3 of the front channel and downwards by 1 / 3 of the back channel.

[0114] In this invention, surrogate self-attention can effectively reduce the number of parameters in the model. The surrogate A is obtained by average pooling the query Q. The same objective can be achieved by using other methods, such as treating the surrogate A as a set of learnable parameters.

[0115] This invention also provides a video temporal motion localization system based on agent attention and multi-scale Transformer, comprising:

[0116] The video feature extraction module is used to extract features from the input video using a pre-trained feature extractor in step 1.

[0117] The temporal action localization model construction module based on proxy self-attention and multi-scale Transformer is used to realize the construction of the temporal action localization model based on proxy self-attention and multi-scale Transformer in step 2. It includes two multi-scale Transformer modules and five proxy Transformer modules, which are used to encode the input video features and extract key information.

[0118] The localization result acquisition module is used to input the different scale features output by the multi-scale Transformer module and the proxy Transformer module constructed in step 2 into the regression head and the classification head in step 3, so as to obtain the localization result of the temporal action localization model based on proxy self-attention and multi-scale Transformer, that is, the start time, end time and action category label of the action instance in the uncropped video.

[0119] The present invention also provides a video temporal motion localization device based on agent attention and multi-scale Transformer, comprising:

[0120] Memory: A computer-readable device that stores the computer program of the above-mentioned video temporal action localization method based on agent attention and multi-scale Transformer;

[0121] Processor: Used to implement the video temporal action localization method based on agent attention and multi-scale Transformer when executing the computer program.

[0122] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the aforementioned video temporal motion localization method based on agent attention and multi-scale Transformer.

Claims

1. A video temporal action localization method based on agent attention and multi-scale Transformer, characterized in that, Includes the following steps: Step 1: Extract features from the input video using a pre-trained feature extractor; Step 2: Construct a temporal action localization model based on surrogate self-attention and multi-scale Transformer, stacking seven Transformer modules, of which the first two are multi-scale Transformer modules and the last five are surrogate Transformer modules, used to encode the input video features and extract key information. The specific method for step 2 is as follows: Step 2.1: Construct the multi-scale Transformer module, whose structure is as follows: Step 2.1.1, for the input video features First, the features are normalized using Layer Normalization. The processed features are then fed into the Multi-Scale Attention (MSA) module for attention calculation. The MSA module employs a lightweight convolutional structure, and its calculation process is as follows: Step 2.1.2, process the features obtained in Step 2.1.1 through the Multi-Scale Attention Module (MSA). As input, the input features are processed using group normalization. Then, a multilayer perceptron (MLP) is used to extract higher-level feature representations from the video. The MLP consists of two linear transformation layers and an activation function. Finally, a residual structure is used to fuse these features, yielding the final output of the multi-scale Transformer module. The specific calculation process is as follows: in, and It is the weight matrix of the linear transformation layer. and These are the basis vectors. GELU is the Gaussian error linear unit activation function, calculated as follows: ； Step 2.2, construct the proxy Transformer module, with the following module structure: Step 2.2.1, for the input video features First, the features are normalized using Layer Normalization. The processed features are then fed into the fine-grained surrogate self-attention FGASA module for attention calculation, yielding the output. The fine-grained agent self-attention module FGASA adopts a quadruple configuration. ,in It is the output of the refactoring unit, A is the... The fine-grained proxy self-attention module is calculated by performing average pooling over time. The structure of the fine-grained proxy self-attention module FGASA is as follows: For the input video features First, three different linear transformations are used. , , Input video features Transformed into Query, Key, and Value abbreviations ,Right now: , , Next, The input is fed into the Refinement Unit (RRU), resulting in more refined Query, Key, and Value, namely: Then introduce As a query The agent first from and Aggregate information in the middle, and then propagate the aggregated global information to... The final output of the attention calculation is obtained, and the specific calculation process is as follows: in, For query This is obtained by performing average pooling over the time dimension. Includes It provides global information, and the number of tokens is configurable. Additionally, bias terms are introduced in both attention calculation processes. and The specific calculation process is as follows: in, Finally, the input is fused using the residual structure. The result of attention calculation is used as the output. ,Right now: ； The specific structure of the Reconstruction Refinement Unit (RRU) is as follows: First, for the input video features, three different linear transformations are used to obtain three different feature representations, namely... Three matrices; then on the channel dimension... Perform the merging, and on the merged channel dimension, for Perform a circular shift operation, shifting the previous... r The channel moved down, then r The channel is moved upwards, and the moved-out portion is filled into the gap created by the shift. Finally, the channel is moved along its dimension. Separate and restore the independent feature matrices; simultaneously, perform cyclic shifting... Perform a convolution operation with shared parameters once each to obtain a new feature representation. The r Indicates the proportion of the shift; Step 2.2.2: Process the features obtained in Step 2.2.1 using the fine-grained proxy self-attention module FGASA. As input, the features are first processed through a group normalization layer, then a multilayer perceptron is used to extract higher-level feature representations of the video; finally, a residual structure is used to fuse these features to obtain the final output of the proxy Transformer module. : ； Step 3: Input the different scale features output by the multi-scale Transformer module and the proxy Transformer module constructed in Step 2 into the regression head and the classification head to obtain the localization results of the temporal action localization model based on proxy self-attention and multi-scale Transformer, namely the start time, end time and action category label of the action instance in the uncropped video.

2. The video temporal action localization method based on agent attention and multi-scale Transformer according to claim 1, characterized in that, The structure of the multi-scale attention module (MSA) in step 2.1.1 is as follows: For the input video features The Multi-Scale Attention (MSA) module processes the input features in two branches. In the multi-scale branch, the input features are first fed into two one-dimensional convolutional layers with kernels of different sizes to extract feature information at different time scales. Then, the feature information from the two different time scales is fused, and batch normalization and softmax functions are applied sequentially to obtain the output of the multi-scale branch. ; In the attention branch, the input features are fed into a one-dimensional convolutional layer with a kernel size of 1 for processing. Simultaneously, the attention branch performs average pooling and max pooling on the input features along the time dimension to extract feature information from different aspects. The results of the two pooling operations are then fed sequentially into the same multi-layer perceptron (MLP) to obtain two weights. and Next, the two weights and The weights are then fused and processed using the sigmoid function to obtain the final weights. Finally, use weights. Output of one-dimensional convolution We perform weighted summation to obtain the output of the attention branch. The specific calculation process is as follows: The results of the multi-scale branch and the attention branch are summed to obtain the output of the multi-scale attention module (MSA). ,Right now: 。 3. The video temporal action localization method based on agent attention and multi-scale Transformer according to claim 1, characterized in that, The r The value can be 1 / 3, 1 / 4, 1 / 8, or 1 / 16.

4. A video temporal motion localization system based on surrogate attention and multi-scale Transformer as described in any one of claims 1 to 3, characterized in that, include: The video feature extraction module is used to extract features from the input video using a pre-trained feature extractor. The temporal action localization model building module based on surrogate self-attention and multi-scale Transformer is used to build a temporal action localization model based on surrogate self-attention and multi-scale Transformer. It stacks seven Transformer modules, of which the first two are multi-scale Transformer modules and the last five are surrogate Transformer modules, which are used to encode the input video features and extract key information. The localization result acquisition module is used to input the different scale features output by the multi-scale Transformer module and the surrogate Transformer module into the regression head and the classification head to obtain the localization result of the temporal action localization model based on surrogate self-attention and multi-scale Transformer, namely the start time, end time and action category label of the action instance in the uncropped video.

5. A video temporal motion localization device based on agent attention and multi-scale Transformer, characterized in that, include: The memory is a computer program that stores the video temporal action localization method based on agent attention and multi-scale Transformer as described in any one of claims 1-3, and is a computer-readable device. Processor: Used to implement the video temporal motion localization method based on agent attention and multi-scale Transformer as described in any one of claims 1-3 when executing the computer program.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, can implement the video temporal motion localization method based on agent attention and multi-scale Transformer as described in any one of claims 1-3.

Images