Action recognition method and system based on 2D convolutional network with cross attention

Abstract

The present invention discloses a method and system for action recognition based on a 2D convolutional network with cross-attention. The method comprises: reading an RGB video, dividing the video into multiple segments, randomly sampling a frame in each segment, generating an image sequence for action recognition, and processing it to meet the input requirements of the neural network; adding a cross-attention module to the backbone architecture of the 2D-CNN to obtain a cross-attention network, so that the backbone architecture has the ability to process the temporal information in the image sequence, and then training the network using data from the dataset to obtain the parameters of each layer in the cross-attention network; inputting the processed image sequence into the trained cross-attention network to obtain classification features for action recognition, and then outputting the recognition result based on the classification features. The present invention solves the problem of high network computational complexity of the temporal features of videos.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision video understanding, and in particular to an action recognition method and system based on a cross-attention 2D convolutional network. Background Art

[0002] Currently, video action recognition is achieved using RNNs, 3D-CNNs, Transformers, 2D-CNNs, and TSMs. However, temporal features such as video optical flow, as a representation of pixel motion in an image, require extensive computation and often require increased storage and computing resources. Using RNNs reduces the network's parallel capabilities, resulting in slower training and inference. 3D-CNNs have more parameters than 2D-CNNs, making them more difficult to train and computationally more complex. Transformer-based methods require more training data, larger models, and relatively high hardware costs. Consequently, existing methods suffer from the high computational cost of acquiring temporal information about actions. Using a 2D-CNN architecture requires relatively few computational resources, but its temporal information modeling capabilities are relatively weak. Therefore, a video action recognition method is needed that preserves temporal information while reducing the computational complexity of the network. Summary of the Invention

[0003] (1) Technical issues to be solved

[0004] Based on the above problems, the present invention provides an action recognition method and system based on a 2D convolutional network with cross attention, which solves the problem of high network computational complexity of the temporal features of the video, reduces the network computational complexity, and senses the changes in temporal information.

[0005] (2) Technical solution

[0006] Based on the above technical problems, the present invention provides an action recognition method based on a 2D convolutional network with cross attention, comprising:

[0007] S1. Read the RGB video, divide the video into multiple segments, randomly sample a frame in each segment, generate an image sequence for action recognition, and process it to meet the input requirements of the neural network;

[0008] S2. Add the cross-attention module to the backbone architecture of the 2D-CNN to obtain a cross-attention network, including inputting the features after the temporal channel feature interaction output by the TSM time transfer module of the input of the backbone architecture into the forward attention module and the backward attention module in the cross-attention module, obtaining the attention features of the first α frames for the current frame after forward truncation and forward zero-padding, that is, the forward attention features, and at the same time obtaining the attention features of the last α frames for the current frame after backward truncation and backward zero-padding, that is, the backward attention features, weighting the forward attention features and the backward attention features to obtain cross-attention features, adding the cross-attention features and the features after the temporal channel feature interaction and outputting them to the backbone architecture, where α is the set attention step; then use the data in the dataset to train the network to obtain the parameters of each layer in the cross-attention network;

[0009] S3. Input the processed image sequence into the trained cross-attention network to obtain the classification features for action recognition, and then output the recognition result according to the classification features.

[0010] Further, in step S2, the attention features of the first α frames for the current frame obtained after forward truncation and forward zero-padding, that is, the forward attention features, and at the same time the attention features of the last α frames for the current frame obtained after backward truncation and backward zero-padding, that is, the backward attention features, include: first dividing the features after the temporal channel feature interaction of the input into T x's according to the number of frames T, that is, x1, x2,..., x , ,

[0012] , ,

[0011] , , , , , and then respectively performing Conv2d convolution padding on each x to obtain the corresponding a, that is, a1, a2,..., a T , for the forward attention features, when 0 < i ≤ α, when α < i ≤ T, For the backward attention features, when 0 < i ≤ T - α, when T - α < i ≤ T, Obtain Combine each to synthesize the forward attention features or the backward attention features.

[0011] Further, in step S2, the number of cross-attention modules added to each layer of the backbone architecture does not exceed 8.

[0012] Furthermore, in step S2, the adding of the cross attention module to the backbone architecture of the 2D-CNN includes: the backbone architecture of the 2D-CNN is a ResNet50 network, which has 4 layers, each layer includes 4 Bottle Necks, each Bottle Neck includes the input features, which are sequentially subjected to 1×1 convolution, 3×3 convolution, and 1×1 convolution, and then weighted with the input features and output to the next Bottle Neck, and the cross attention module is inserted into the last two Bottle Necks of each layer.

[0013] Furthermore, the α is set to 1 or 2.

[0014] Furthermore, in step S1, the processing includes normalization and regularization processing.

[0015] Furthermore, in step S1, the processing also includes: deforming the sample features (N, T, C, H, W) of a batch to (N*T, C, H, W), where T represents the number of frames sampled from the video sample, C represents the number of video image channels, H and W represent the height and width of the video image respectively, and N represents the number of samples in the batch.

[0016] Furthermore, in step S1, the video is divided into multiple segments, including 8 segments or 16 segments.

[0017] Furthermore, in step S3, outputting the recognition result according to the classification feature includes: obtaining the score of the input image sequence belonging to each action category through a Softmax function, determining the category to which the input image sequence is most likely to belong, and outputting the result.

[0018] The present invention also discloses an action recognition system based on a 2D convolutional network with cross attention, comprising:

[0019] at least one processor; and at least one memory communicatively coupled to the processor, wherein:

[0020] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the action recognition method based on the cross-attention 2D convolutional network.

[0021] (3) Beneficial effects

[0022] The above technical solution of the present invention has the following advantages:

[0023] (1) The present invention is based on a 2D-CNN action recognition network, which introduces the attention mechanism used in images into videos. It learns the cross-attention features of the current frame from adjacent time-series frames and weights the features of the current frame so that the cross-attention features can better integrate the time-series information in the video. 2D-CNN reduces the computational complexity of the deep neural network for action recognition, and the cross-attention features sense the changes in time-series information, thereby improving the accuracy of video action recognition.

[0024] (2) Based on TSM, the cross-attention module of the present invention adds a mechanism to learn spatiotemporal attention from adjacent frames to better perceive the spatial and temporal feature transformation. The cross-attention module moves the learned spatial attention features of the image sequence forward or backward, and improves the module's ability to perceive the temporal sequence range through the attention step, further enhancing the ability of the cross-attention feature to perceive the temporal information in the video;

[0025] (3) The present invention also limits the number of cross-attention modules added to 2D-CNN, and does not add cross-attention modules to all the main frameworks of 2D-CNN. While ensuring the range of temporal information perception, it reduces the additional delay loss caused by the network using the temporal perception module, making the network more lightweight. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0027] Figure 1 Flowchart of an action recognition method based on a 2D convolutional network with cross attention according to an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of the network structure of the cross-attention module of an embodiment of the present invention, including the forward attention (Front Attention) and backward attention (Behind Attention) modules.

[0029] Figure 3 This is a structural diagram of a classification network based on ResNet50 with a cross-attention module added according to an embodiment of the present invention;

[0030] Figure 4 Schematic diagram of the attention step in the cross-attention module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0032] An action recognition method based on 2D convolutional networks with cross attention, such as Figure 1 As shown, the following steps are included:

[0033] S1. Read the RGB video, divide the video into multiple segments, randomly sample a frame in each segment, and generate an image sequence for action recognition;

[0034] In one embodiment of the present invention, step S1 mainly includes the following steps:

[0035] Read a video from the action recognition dataset and divide the video into T segments, where T is usually 8 or 16. Randomly sample one frame from each segment to generate a sequence as the input sample. The characteristic shape of the sample is (T, C, H, W), where T represents the number of frames sampled from the video sample, C represents the number of video image channels, and H and W represent the height and width of the video image, respectively.

[0036] During the training process of a neural network, multiple batches are often read in parallel as input. At this time, the sample feature shape of a batch is (N, T, C, H, W), where N represents the number of samples in the batch. To meet the input requirements of 2D-CNN, the sample features are usually reshaped to (N*T, C, H, W) as the input of the backbone architecture.

[0037] The image sequence data is normalized and regularized to meet the input requirements of the neural network; the sample data is normalized and regularized to enable the neural network to better process the input feature data.

[0038] S2. Add the cross-attention module to the backbone architecture of 2D-CNN to obtain the cross-attention network. The cross-attention module enables the backbone architecture to process the temporal information in the image sequence. Then use the dataset data to train the network to obtain the parameters of each layer in the cross-attention network.

[0039] The 2D-CNN backbone architecture selected for use and the cross-attention module added to the backbone architecture include the following steps:

[0040] S21. After a batch of samples is input into the backbone architecture, the features output by a certain layer of the backbone architecture are input into the TSM time transfer module to obtain features after the interaction of temporal channel features.

[0041] After a batch of samples is input into the backbone architecture, such as ResNet50, the features output at a certain layer have a shape of (N*T, c, h, w). The features are input into the TSM module (Temporal Shift Module) to obtain the features after the interaction of temporal channel features. The shape remains unchanged and this feature is used as the input of the cross-attention module.

[0042] S22. Input the features after the temporal-channel feature interaction into the cross-attention module to obtain forward attention features and backward attention features, and weight the forward attention features and backward attention features to obtain cross-attention features;

[0043] The forward attention feature (Behind Attention) is the attention feature of the first α frames obtained by forward truncation and forward zero-padding for the current frame, and the backward attention feature (Front Attention) is the attention feature of the last α frames obtained by backward truncation and backward zero-padding for the current frame, where α is the set attention step size. Specifically, input the features (N, T, c, h, w) after the temporal-channel feature interaction into the forward attention module and the backward attention module in the cross-attention module respectively. First, divide the input features by the number of frames T into T x's, i.e., x1, x2, …, x T , all in the form of (N, c, h, w). Then, after each x is filled by Conv2d convolution, the corresponding a's are obtained, i.e., a1, a2, …, a T , all in the form of (N, 1, h, w). Multiply the x of the current frame by the corresponding a of the first / last α frames to obtain Keep the x of the current frame unchanged if there are no first / last α frames to obtain That is, for the forward attention feature, when 0 < i ≤ α, when α < i ≤ T, For the backward attention feature, when 0 < i ≤ T - α, when T - α < i ≤ T, Obtain , all in the form of (N, c, h, w). Combine each to synthesize the forward / backward attention feature output in the form of (N, T, c, h, w). Finally, weight the forward attention feature and the backward attention feature to obtain the cross-attention feature.

[0044] The attention step size α can be set to 1 or 2. Taking the step size of 1 as an example, the cross-attention module is as Figure 2As shown in the figure, it contains two convolutional layers with a kernel size of (3,3) and a stride of (1,1). Padding is performed accordingly, keeping the spatial size of the output features constant while reducing the number of channels to 1, resulting in a shape of (N*T,1,h,w). This shape is reshaped to (N,T,1,h,w), and forward truncation and forward zero padding are performed to obtain the attention features of the previous frame with respect to the current frame, called the forward attention features. Corresponding backward truncation and backward zero padding are performed to obtain the attention features of the subsequent frame with respect to the current frame, called the backward attention features. These two attention mechanisms together form the cross attention module. The original features are reshaped to (N,T,c,h,w), and multiplied by the two attention features to obtain the weighted features, namely the cross attention features.

[0045] S23. Add the features after the interaction between the cross-attention feature and the temporal channel feature as the input of the next layer of the backbone architecture.

[0046] After restoring the original shape, it is added to the features obtained after TSM and used as the input of the next layer of the backbone architecture.

[0047] Finally, the cross attention module is added to the backbone architecture to construct the required network. Previous methods, such as TSM, Action Net, STM, TEA and other networks, usually choose to add modules for sensing temporal information in all residual blocks in the original backbone architecture, such as ResNet50. The temporal understanding ability of the module may be redundant when the number of sampling frames is small. When adding the cross attention module, the present invention imposes certain restrictions on the number of modules. For example, for the ResNet50 network, the module is only added to the last two Bottle Necks of each Stage, such as Figure 3 As shown, the 2D-CNN backbone architecture, ResNet50, has four layers, each consisting of four bottle necks. Each bottle neck processes the input features through a 1×1 convolution, a 3×3 convolution, and a 1×1 convolution, before being weighted and output to the next bottle neck. Cross-attention modules are inserted between the last two bottle necks of each layer, reducing the number of cross-attention modules from 16 to 8. This reduces the network's computational complexity while maintaining its temporal perception capabilities. The network was trained on a video recognition dataset to obtain the corresponding parameters.

[0048] When constructing the network, the cross attention module can freely adjust the distance of attention movement, that is, it can learn attention features from farther frame images to guide the learning of current frame features. This distance is called attention stride (α). When the attention stride α is set to 1 or 2, the backward attention module of the cross attention module is as follows Figure 4 As shown in Figure 2. Previous methods often cannot freely adjust this parameter, such as channel shifting, which only allows the channels of two adjacent frames to interact with the features of the current frame. The method proposed in this paper can adjust the attention step size, increasing the attention range of the attention module, directly learning long-range temporal information in a single module, and improving the model's temporal perception ability.

[0049] S3. Input the processed image sequence into the trained cross-attention network to obtain the classification features for action recognition. Then, use the Softmax function to obtain the score of each action category of the input sequence, determine the category to which the input sequence is most likely to belong, and output it.

[0050] In one embodiment of the present invention, step S4 primarily includes the following steps: inputting the processed image sequence from step S2 into the trained network and applying a Softmax function to obtain a final classification feature, which is generally in the shape of (N, S), where S is the number of categories in the dataset. The classification feature corresponding to one of the samples has the shape of (S), and the index corresponding to the maximum value is the category predicted by the network.

[0051] Unlike the self-attention method commonly used in previous action recognition networks, this paper introduces the attention mechanism applied to images into videos, learning the spatial attention features of the current frame from adjacent time-series frames. This allows these attention features to better integrate the temporal information in the video, improving the accuracy of video action recognition. This paper also limits the number of modules added to the 2D-CNN, ensuring the temporal perception range while reducing the additional computational cost when the network uses temporal perception modules, such as the cross-attention module, making the network more lightweight.

[0052] Table 1 Comparison of the proposed method and related methods on four common action recognition datasets

[0053]

[0054]

[0055] Table 2 Comparison of computational complexity with partial correlation method

[0056]

[0057] Table 1 compares the video action recognition accuracy of our proposed method with related methods on four common action recognition datasets. All methods use an 8-frame input sample. Table 2 compares the computational complexity of our proposed method with related methods. All methods use an 8-frame input sample without multi-sampling or cropping. Experiments were conducted on several common video action recognition datasets. For each dataset, we used the following training settings: 50 training cycles, an initial learning rate of 0.001, which was reduced to 0.1 times the original learning rate around the 20th and 40th cycles, a weight decay of 1e-5, and a dropout rate of 0.5. The network was initialized using the parameters of a TSM model pre-trained on the Kinetics400 dataset. To prevent premature overfitting on some small datasets, we froze the batch norm beyond the first layer. During testing, to achieve higher accuracy, we sampled two clips from each video. The final output of each clip was weighted averaged and then passed through the softmax function to obtain the final recognition score. The feasibility of our proposed method was verified on several common datasets. Relevant experiments show that the network proposed in the present invention causes less additional computational loss without sacrificing classification performance.

[0058] Finally, it should be noted that the above-mentioned motion recognition method can be converted into software program instructions, which can be implemented by running a system including a processor and a memory, or by computer instructions stored in a non-transitory computer-readable storage medium. The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0059] In summary, the above-mentioned action recognition method and system based on a 2D convolutional network with cross attention has the following beneficial effects:

[0060] (1) The present invention is based on a 2D-CNN action recognition network, which introduces the attention mechanism used in images into videos. It learns the cross-attention features of the current frame from adjacent time-series frames and weights the features of the current frame so that the cross-attention features can better integrate the time-series information in the video. 2D-CNN reduces the computational complexity of the deep neural network for action recognition, and the cross-attention features sense the changes in time-series information, thereby improving the accuracy of video action recognition.

[0061] (2) Based on TSM, the cross-attention module of the present invention adds a mechanism to learn spatiotemporal attention from adjacent frames to better perceive the spatial and temporal feature transformation. The cross-attention module moves the learned spatial attention features of the image sequence forward or backward, and improves the module's ability to perceive the temporal sequence range through the attention step, further enhancing the ability of the cross-attention feature to perceive the temporal information in the video;

[0062] (3) The present invention also limits the number of cross-attention modules added to 2D-CNN, and does not add cross-attention modules to all the main frameworks of 2D-CNN. While ensuring the range of temporal information perception, it reduces the additional delay loss caused by the network using the temporal perception module, making the network more lightweight.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations shall fall within the scope defined by the appended claims.

Claims

1. An action recognition method based on 2D convolutional network with cross attention, characterized in that include: S1. Read the RGB video, divide the video into multiple segments, randomly sample a frame in each segment, generate an image sequence for action recognition, and process it to meet the input requirements of the neural network; S2. Add the cross attention module to the backbone architecture of 2D-CNN to obtain a cross attention network, including inputting the input of the backbone architecture into the forward attention module and the backward attention module in the cross attention module after the interaction of the temporal channel features output by the TSM time transfer module, and obtaining the attention features of the previous α frame to the current frame by forward interception and forward zero filling, i.e., the forward attention features, and at the same time, obtaining the attention features of the subsequent α frame to the current frame by backward interception and backward zero filling, i.e., the backward attention features, and weighting the forward attention features and the backward attention features to obtain the cross attention features, and adding the features after the interaction of the cross attention features and the temporal channel features and outputting them to the backbone architecture, where α is the set attention step size; and then using the data set data to train the network to obtain the parameters of each layer in the cross attention network; The number of cross attention modules added to each layer of the backbone architecture does not exceed 8; the adding of the cross attention module to the backbone architecture of 2D-CNN includes: the backbone architecture of 2D-CNN is a ResNet50 network with 4 layers, each layer includes 4 bottle necks, each bottle neck includes input features, which are sequentially subjected to 1×1 convolution, 3×3 convolution, and 1×1 convolution, and then weighted with the input features and output to the next bottle neck, and the cross attention module is inserted into the last two bottle necks of each layer; S3. Input the processed image sequence into the trained cross-attention network to obtain classification features for action recognition, and then output recognition results based on the classification features.

2. The action recognition method based on 2D convolutional network with cross attention according to claim 1, characterized in that In step S2, the attention feature of the previous α frame to the current frame obtained by forward interception and forward zero filling, i.e., the forward attention feature, and the attention feature of the subsequent α frame to the current frame obtained by backward interception and backward zero filling, i.e., the backward attention feature, include: first dividing the input features after the interaction of the temporal channel features into T x according to the number of frames T, i.e., x1, x2, ..., x T , and then fill each x with Conv2d convolution to get the corresponding a, i.e. a1, a2, ..., a T , for the forward attention feature, 0<i≤α, When α<i≤T, For the backward attention feature, when 0<i≤T-α, When T-α<i≤T, get Each The forward attention feature or the backward attention feature is synthesized.

3. The action recognition method based on 2D convolutional network with cross attention according to claim 1, characterized in that The α is set to 1 or 2.

4. The action recognition method based on 2D convolutional network with cross attention according to claim 1, characterized in that In step S1, the processing includes normalization and regularization.

5. The action recognition method based on 2D convolutional network with cross attention according to claim 1, characterized in that In step S1, the processing also includes: deforming the sample features (N, T, C, H, W) of a batch to (N*T, C, H, W), where T represents the number of frames sampled from the video sample, C represents the number of video image channels, H and W represent the height and width of the video image respectively, and N represents the number of samples in the batch.

6. The action recognition method based on 2D convolutional network with cross attention according to claim 1, characterized in that In step S1, the video is divided into multiple segments, including 8 segments or 16 segments.

7. The action recognition method based on 2D convolutional network with cross attention according to claim 1, characterized in that In step S3, outputting the recognition result according to the classification feature includes: obtaining the score of the input image sequence belonging to each action category through a Softmax function, determining the category to which the input image sequence is most likely to belong, and outputting the result.

8. An action recognition system based on a 2D convolutional network with cross attention, characterized in that: include: at least one processor; and at least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the action recognition method based on the cross-attention 2D convolutional network as described in any one of claims 1 to 7.

Images