A temporal action localization method based on external attention and feature enhancement

Abstract

The present invention provides a temporal action localization method based on external attention and feature enhancement, comprising extracting video features of uncut videos by using a pre-trained general action classification network, inputting the extracted video features into a Transformer layer equipped with an external attention module to extract into a multi-scale feature pyramid, using a multi-scale feature enhancement module to perform feature enhancement between different scale features and feature fusion of different scale features on the multi-scale feature pyramid, inputting the fused and enhanced features into a detection head including classification and regression components, and obtaining predicted action category labels and action start and end boundaries. The external attention module in the Transformer layer of the method uses two linear layers to replace self-attention, so the external attention has linear complexity, which reduces the algorithm complexity and parameter amount of the entire model, and at the same time, the multi-scale feature enhancement module is used to make up for the inconsistency between the scale features in the feature pyramid to improve the structural effectiveness of the feature pyramid.

Description

Technical Field

[0001] The present invention relates to the technical field of temporal action localization, and in particular to a temporal action localization method based on external attention and feature enhancement. Background Art

[0002] With the development of the Internet, videos have become an indispensable part of people's lives. Violent scenes in videos can seriously affect the physical and mental health of young people, and the sharing and dissemination of videos needs to be reviewed and controlled. For the highlights in the video, intelligent and accurate positioning can improve the speed and quality of editing.

[0003] The existing Transformer-based model achieves end-to-end positioning effect. The model (ActionFormer) uses a 3D convolutional network to extract features from the original video, then uses a Transformer block with local self-attention to build a feature pyramid, and finally implements classification and regression prediction through a convolutional network. ActionFormer not only has a simple model, but also achieves sufficiently good results. However, the inventors of this application have found through research that ActionFormer still has the following problems: although local self-attention can reduce the inherent computational complexity of self-attention by limiting attention to a local window, its essence is still self-attention, so it cannot minimize the computational complexity of the model; in addition, the feature pyramid extracted by the downsampling layer interspersed with the Transformer block contains multi-scale feature information, but the inconsistency between features of different scales will interfere with the gradient training during training and reduce the effectiveness of the feature pyramid structure. Summary of the invention

[0004] With regard to the existing Transformer-based ActionFormer model for video positioning, there is a technical problem that the Transformer block with local self-attention is essentially still self-attention, which cannot reduce the computational complexity of the model to the greatest extent; in addition, the inconsistency between features of different scales in the feature pyramid extracted by the downsampling layer interspersed with the Transformer block will interfere with the gradient training during training and reduce the effectiveness of the feature pyramid structure. The present invention provides a temporal action positioning method based on external attention and feature enhancement.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A temporal action localization method based on external attention and feature enhancement, comprising the following steps:

[0007] S1. Use the pre-trained general action classification network to extract the video features of the uncut video F = {R B ×C×T}; Where B is the number of uncut videos, C is the number of channels, and T is the number of vectors of sequence length;

[0008] S2: Input the extracted video feature F into the Transformer layer equipped with an external attention module, and extract features of n different scales through n downsampling operations, i.e., a multi-scale feature pyramid.

[0009]

[0010] S3, using a multi-scale feature enhancement module to perform feature enhancement between different scale features and feature fusion of different scale features on the multi-scale feature pyramid;

[0011] S4. Input the fused and augmented features into the detection head containing classification and regression components to obtain the predicted action category labels and action start and end boundaries.

[0012] Furthermore, in step S1, the general action classification network uses an I3D or TSP network model.

[0013] Furthermore, in the step S2, the Transformer layer equipped with the external attention module first normalizes the extracted video feature F and then inputs it into the external attention module. in First, it passes through the Query linear layer containing N-heads, and then with the Key memory unit M k Perform matrix multiplication, then normalize to get an N*N attention map, and finally add it to the memory unit M of Value. v Perform matrix multiplication to get the output result F out , and then output the result F out The feature is added to the extracted video feature F. The output after the addition is input into the feedforward neural network after layer normalization. On the other hand, the multi-scale feature pyramid is obtained by directly adding the feature with the output of the feedforward neural network.

[0014] Furthermore, the output result F of the external attention module out The calculation formula is as follows:

[0015]

[0016] Among them, Norm represents the normalization operation.

[0017] Furthermore, in step S3, a multi-scale feature enhancement module is used to perform feature enhancement between features of different scales on the multi-scale feature pyramid, including multi-scale feature scaling and adaptive fusion enhancement. Specifically, the multi-scale features are divided into three parts: upper layer, middle layer and lower layer. Three stages will be encountered during the scaling process. The upper layer area needs to upsample the adjacent K-1 features to match the same scale; the middle layer area needs to downsample the larger scale features and upsample the smaller scale features to achieve a consistent size; the lower layer area needs to downsample all larger scale features.

[0018] Furthermore, in S3, a multi-scale feature enhancement module is used to fuse features of different scales in the multi-scale feature pyramid. Specifically, the existing attention mechanism is called to fuse and enhance the features of different scales in the multi-scale feature pyramid. The fusion enhancement feature of the i-th feature in the feature pyramid is expressed by the following formula:

[0019]

[0020]

[0021] in, represents the features after fusion enhancement, Represents the features after scaling, x i and x j Respectively represent the i-th and j-th features in the feature pyramid, l i represents the scale of the i-th feature, l j represents the scale of the j-th feature, K represents the number of features that need to be fused, and Attention is the attention function.

[0022] Compared with the prior art, the temporal action localization method based on external attention and feature enhancement provided by the present invention first uses a pre-trained general action classification network to extract video features of unedited videos, then uses a Transformer layer equipped with an external attention module to perform multi-scale feature extraction to obtain a multi-scale feature pyramid, then performs feature enhancement and feature fusion between features of different scales on the multi-scale feature pyramid, and finally inputs the enhanced and fused features into a detection head to predict action category labels and action start and end boundaries. The present invention uses a multi-head external attention module in the Transformer layer. The external attention module uses two linear layers to replace self-attention. Therefore, the external attention has linear complexity, which reduces the algorithm complexity and parameter amount of the entire model. It also implicitly considers the correlation between all sample data, thereby solving the shortcoming that self-attention only considers a single sample. At the same time, the present invention also proposes a multi-scale feature enhancement module. In order to compensate for the inconsistency between the various scale features in the feature pyramid, the multi-scale feature enhancement module uses the adjacent K-1 scale features to enhance a certain scale feature, thereby enabling the model to adaptively learn the useful information of each level feature through deep learning, establish a deeper connection between feature maps of different levels, and improve the effectiveness of the feature pyramid structure, so that the model proposed in this application has better performance in positioning accuracy than the existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the overall architecture of the model of the temporal action localization method based on external attention and feature enhancement provided by the present invention.

[0024] Figure 2 It is a schematic diagram of the Transformer layer structure equipped with an external attention module provided by the present invention.

[0025] Figure 3 It is a schematic diagram of the structure of the multi-scale feature enhancement module provided by the present invention.

[0026] Figure 4 This is an example diagram of the application of the temporal action localization method based on external attention and feature enhancement provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below with reference to specific diagrams.

[0028] Please refer to Figure 1 As shown, the present invention provides a temporal action localization method based on external attention and feature enhancement, comprising the following steps:

[0029] S1. Use a pre-trained general action classification network such as I3D or TSP network model to extract video features F = {R B×C×T}; Where B is the number of uncut videos. If four different video features are processed at the same time, B = 4, C is the number of channels, and T is the number of vectors of sequence length;

[0030] S2: Input the extracted video feature F into the Transformer layer equipped with an external attention module, and extract features of n different scales through n downsampling operations, i.e., a multi-scale feature pyramid. Among them, n downsampling is performed in the last layer of the Transformer equipped with an external attention module, and the number of channels C is reduced by half each time, and the number of vectors T of the sequence length remains unchanged, and then n features Z of different scales are obtained;

[0031] S3, using a multi-scale feature enhancement module to perform feature enhancement between different scale features and feature fusion of different scale features on the multi-scale feature pyramid;

[0032] S4. The fused and augmented features are input into the detection head containing classification and regression components to obtain the predicted action category label (classification result) and action start and end (start and end) boundaries.

[0033] As a specific example, please refer to Figure 2 As shown, in the step S2, the Transformer layer equipped with the external attention module first normalizes the extracted video feature F through the layer (LN) and then inputs it into the external attention module. in First, it passes through the Query linear layer containing N-heads, and the data structure remains unchanged. Then, it is combined with the Key memory unit M k Perform matrix multiplication, then normalize to get an N*N attention map, and finally add it to the memory unit M of Value. v Perform matrix multiplication to get the output result F out , and then output the result F outThe feature is added (Concat) with the extracted video feature F. The output after the addition is input into the existing feed-forward neural network (FFN) after layer normalization. On the other hand, the multi-scale feature pyramid is obtained by directly adding the features with the output of the feed-forward neural network. The feed-forward neural network is used to learn deeper features layer by layer through multiple fully connected layers, and back propagation is used during training to help the model training converge.

[0034] In the above specific embodiment, an external attention module is proposed, which can effectively reduce the complexity of the model while also taking into account the potential relationships that may exist in the entire data set and affect the model. The difference between the Transformer layer in this specific embodiment and the Transformer layer in the existing ActionFormer is that the self-attention module is replaced by an external attention module, which uses two different memory units M k and M v , these two memory units are trained in the entire training data, as the key part and the value part respectively. This design aims to explore the relationship between different samples. The memory unit is essentially an independent parameter body, which is used to store parameters as the memory of the entire training, so Key and Value play the role of parameters.

[0035] As a specific embodiment, the output result F of the external attention module out The calculation formula is as follows:

[0036]

[0037] Among them, Norm represents the normalization operation.

[0038] As a specific example, please refer to Figure 3As shown, the multi-scale feature enhancement module in step S3 of the present invention is different from the previous feature enhancement method. This method focuses on mutually enhancing features of different scales, and through experiments on the number of different scales, the inventors of the present application found that when there are two different scales around a feature, the best performance can be achieved. Therefore, according to the aforementioned multi-scale feature scaling strategy, the multi-scale feature enhancement module is used in step S3 to enhance the features between different scales of the multi-scale feature pyramid, including multi-scale feature scaling and adaptive fusion enhancement. Specifically, the multi-scale features are divided into three parts: top, middle and down. Three stages will be encountered during the scaling process. The upper region needs to upsample the adjacent K-1 features to match the same scale; the middle region needs to downsample the larger scale features and upsample the smaller scale features to achieve a consistent size; the lower region needs to downsample all larger scale features. By performing these operations, the multi-scale features in each region will reach the same size.

[0039] As a specific example, please refer to Figure 3 As shown, in S3, the multi-scale feature enhancement module is used to fuse the features of different scales in the multi-scale feature pyramid. Specifically, the existing attention mechanism such as coordinated attention is called to fuse and enhance the different scale features in the multi-scale feature pyramid. The fusion enhancement feature of the i-th feature in the feature pyramid is expressed by the following formula:

[0040]

[0041]

[0042] in, represents the features after fusion enhancement, Represents the features after scaling, x i and x j Respectively represent the i-th and j-th features in the feature pyramid, l i represents the scale of the i-th feature, l j represents the scale of the j-th feature, K represents the number of features that need to be fused, and Attention is the attention function.

[0043] In order to better illustrate the performance of the temporal action localization method based on external attention and feature enhancement provided by the present invention, the following will be compared with the existing methods:

[0044] 1. The comparison of the accuracy (%) of this model and the existing methods on the THUMOS14 dataset is as follows:

[0045] Method 0.3 0.4 0.5 0.6 0.7 Avg TadTR 62.4 57.4 49.2 37.8 26.3 46.6 AFSD 67.3 62.4 55.5 43.7 31.3 52.0 ActionFormer 82.1 77.8 71.0 59.4 43.9 66.8 EAKE-Net(Our) 82.2 78.8 71.8 59.0 45.4 67.5

[0046] 2. Comparison of the accuracy (%) of this model and existing methods on the EPIC-KITCHEN-100 dataset:

[0047]

[0048] 3. Comparison of parameters between this model and existing methods:

[0049] Method Parameters(M) ActionFormer 22.05 EAKE-Net(Our) 12.72

[0050] From the above comparison, it can be seen that the model proposed by the present invention has better performance in terms of accuracy than the existing methods, and the number of parameters is also far less than the existing advanced methods. For an example of sequential action positioning using the sequential action positioning method of the present invention, please refer to Figure 4 shown.

[0051] Compared with the prior art, the temporal action localization method based on external attention and feature enhancement (abbreviated as EAKE-Net) provided by the present invention first uses a pre-trained general action classification network to extract video features of unedited videos, then uses a Transformer layer equipped with an external attention module to perform multi-scale feature extraction to obtain a multi-scale feature pyramid, then performs feature enhancement and feature fusion between features of different scales on the multi-scale feature pyramid, and finally inputs the enhanced and fused features into a detection head to predict action category labels and action start and end boundaries. The present invention uses a multi-head external attention module in the Transformer layer. The external attention module uses two linear layers (linear units Key and Value) to replace self-attention. Therefore, the external attention has linear complexity, which reduces the algorithm complexity and parameter amount of the entire model. It also implicitly considers the correlation between all sample data, thereby solving the shortcoming that self-attention only considers a single sample. At the same time, the present invention also proposes a multi-scale feature enhancement module. In order to compensate for the inconsistency between the various scale features in the feature pyramid, the multi-scale feature enhancement module uses the adjacent K-1 scale features to enhance a certain scale feature, thereby enabling the model to adaptively learn the useful information of each level feature through deep learning, establish a deeper connection between feature maps of different levels, and improve the effectiveness of the feature pyramid structure, so that the model proposed in this application has better performance in positioning accuracy than the existing methods.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A temporal action localization method based on external attention and feature enhancement, characterized in that: The following steps are involved: S1. Use the pre-trained general action classification network to extract the video features of the uncut video F = {R B×C×T }; Where B is the number of uncut videos, C is the number of channels, and T is the number of vectors of sequence length; S2: Input the extracted video feature F into the Transformer layer equipped with an external attention module, and extract features of n different scales through n downsampling operations, i.e., a multi-scale feature pyramid. S3, using a multi-scale feature enhancement module to perform feature enhancement between different scale features and feature fusion of different scale features on the multi-scale feature pyramid; S4, input the fused and augmented features into the detection head containing classification and regression components, so as to obtain the predicted action category label and action start and end boundaries; wherein, In the step S2, the Transformer layer equipped with the external attention module first normalizes the extracted video feature F and then inputs it into the external attention module. in First, it passes through the Query linear layer containing N-heads, and then with the Key memory unit M k Perform matrix multiplication, then normalize to get an N*N attention map, and finally add it to the memory unit M of Value. v Perform matrix multiplication to get the output result F out , and then output the result F out The feature is added to the extracted video feature F. The output of the addition is input into the feedforward neural network after layer normalization. On the other hand, the multi-scale feature pyramid is obtained by directly adding the feature with the output of the feedforward neural network. In step S3, a multi-scale feature enhancement module is used to perform feature enhancement between features of different scales on the multi-scale feature pyramid, including multi-scale feature scaling and adaptive fusion enhancement. Specifically, the multi-scale features are divided into three parts: upper layer, middle layer and lower layer. Three stages will be encountered in the scaling process. The upper layer area needs to upsample the adjacent K-1 features to match the same scale; the middle layer area needs to downsample the larger scale features and upsample the smaller scale features to achieve a consistent size; the lower layer area needs to downsample all larger scale features.

2. The temporal action localization method based on external attention and feature enhancement according to claim 1, characterized in that: In step S1, the general action classification network uses an I3D or TSP network model.

3. The temporal action localization method based on external attention and feature enhancement according to claim 1, characterized in that: The output of the external attention module F out The calculation formula is as follows: Among them, Norm represents the normalization operation.

4. The temporal action localization method based on external attention and feature enhancement according to claim 1, characterized in that: In S3, a multi-scale feature enhancement module is used to fuse features of different scales in the multi-scale feature pyramid. Specifically, the existing attention mechanism is called to fuse and enhance the features of different scales in the multi-scale feature pyramid. The fusion enhancement feature of the i-th feature in the feature pyramid is expressed by the following formula: in, represents the features after fusion enhancement, Represents the features after scaling, x i and x j Respectively represent the i-th and j-th features in the feature pyramid, l i represents the scale of the i-th feature, l j represents the scale of the j-th feature, K represents the number of features that need to be fused, and Attention is the attention function.

Images