An improved long-term recurrent convolutional network-based video action recognition method

Abstract

The application discloses a kind of based on improved long-term cycle convolution network video action recognition method, comprising the following steps: step 1, using single-head self-attention mechanism to calculate the weighted average of a video frame.Step 2, embedding two-dimensional relative position coding in single-head self-attention mechanism.Step 3, multiple single-head attention is spliced to form multi-head attention to calculate the feature mapping result of a video frame.Step 4, embedding relative position information while maintaining two-dimensional relative self-attention mechanism enhanced convolution operator makes convolution feature map and the feature map result generated by self-attention are spliced so that the model can focus on local features and global context when convolution operation.Finally, compared with the currently popular long-term cycle convolution network, the application combines multi-head self-attention in the convolution part of the network, so that the model can focus on local features and global context when convolution operation, thereby improving the accuracy of the model for video action recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and is a video action recognition method based on an optimized long-term recurrent convolutional network (LRCNs), which is designed to solve the problem of inaccuracy in the video action recognition process. BACKGROUND

[0002] The purpose of human action recognition is to determine the human action category to obtain human behavior information, which is widely used in video monitoring, life assistance, remote medical treatment, robot technology and other fields. Human action recognition in a video aims to classify an input video containing human action into one of the predefined target categories. Currently, a long-term recurrent convolutional network (LRCNs) is proposed, which is a class of architectures for visual recognition and description, combining convolutional layers and long-range temporal recurrence, and is end-to-end trainable. The picture features of each frame of the video are extracted using a convolutional neural network layer, and the visual features of each frame can be used as the input of each long short-term memory network layer (LSTM), and then the LSTM layer learns the temporal relationship of all frames in the video segment to achieve the effect of action recognition.

[0003] Convolutional neural networks have achieved great success in many computer vision applications, especially in image classification. The design of convolutional layers imposes locality through a limited receptive field and translational equivariance through weight sharing. Both of these properties have proven to be crucial inductive biases when designing models that operate on images. However, the local nature of the convolutional kernel prevents it from capturing the global context in an image, which is often necessary for better recognizing objects in the image. Therefore, the extraction of image context features is ignored when the convolutional neural network layer part in the LRCNs network extracts features, thus leading to the problem of inaccurate video action recognition. SUMMARY

[0004] To solve the above problems, the improved long-term recurrent convolutional network is proposed by using two-dimensional relative self-attention mechanism to enhance the convolution operator. The self-attention is an advanced technology for capturing long-range interactions, but it is mainly applied to sequence modeling and generation modeling tasks. The key idea behind self-attention is to produce a weighted average of values calculated from hidden units. Unlike pooling or convolution operators, the weights used in the weighted average operation are dynamically generated by a similarity function between hidden units. Therefore, the interaction between input signals depends on the signals themselves, rather than being predetermined by their relative positions as in convolution. In particular, this enables self-attention to capture long-range interactions without increasing the number of parameters. In addition, two-dimensional position relative encoding is injected into self-attention to maintain translational equivalence when processing video frames. The method enhances the convolution operator by injecting relative position information while maintaining translational equivalence in two-dimensional relative self-attention mechanism during convolution operation in LRCNs, so that the convolution feature extraction can focus on local features and global context, thereby improving the accuracy of the entire model for video action recognition.

[0005] The purpose of the application is to improve the accuracy of the entire model for video action recognition by combining self-attention mechanism during convolution operation in LRCNs to focus on local features and global context during feature extraction in the convolution layer of the model, and embedding two-dimensional relative position encoding in the self-attention mechanism to maintain translational equivalence during video frame recognition.

[0006] The technical solution of the application is: a video action recognition method based on improved long-term recurrent convolutional network, comprising the following steps:

[0007] Step 1, using single-head self-attention mechanism to calculate the weighted average value of a video frame.

[0008] Step 2, embedding two-dimensional relative position encoding in single-head self-attention mechanism.

[0009] Step 3, concatenating multiple single-head attentions to form multi-head attention to calculate the feature mapping result of a video frame.

[0010] Step 4, using two-dimensional relative self-attention mechanism to enhance the convolution operator while maintaining translational equivalence to concatenate the convolution feature map and the feature map result generated by self-attention, so that the model can focus on local features and global context during convolution operation.

[0011] 1. First, define the following naming rules: H, W, F in represent the height, width, and number of input filters of the activation mapping. N h ,d v ,d k represent the number of heads, value depth, query depth, and key depth in multi-head attention, respectively. Assume Nh d v and d k average split, and denotes the value of each attention head, the depth of the query and key. In step 1, given a tensor of shape (H, W, F in (for convenience, the batch dimension is omitted), it is flattened into a matrix, and multi-head attention in the Transformer is performed. The output of single-head self-attention is denoted as:

[0012]

[0013] is a learnable linear transformation that maps the input X to the query Q = XW q , the key k = XW k , and the value v = XW v , respectively. This gives a weighted average of a video frame computed by single-head self-attention. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions.

[0014] 2. Secondly, step 1 would lead to the arrangement of self-attention being equivalent if no positional information is used, which is ineffective for modeling highly structured data such as images. CoordConv has been proposed to introduce sinusoidal wave expansion of positional information and connect the position channel to the activation mapping, but these positional encodings do not help for image classification and object detection, possibly because these positional encodings, while not permutation equivariant, do not satisfy translational equivariance. Translational equivariance is a necessary property when processing images. To this end, the expression in step 1 is embedded into the single-head self-attention by independently adding relevant height information and relevant width information two-dimensional relative position encodings:

[0015]

[0016] is a relative logical matrix along the height and width dimensions, satisfying q i is the i-th row of the Q matrix, and are the learned embeddings of the relevant height j y -i y and the relevant width j x -i x , respectively. The attention logical expression of pixel i = (i x , i y ) to pixel j = (j x , j y ) is:

[0017]

[0018] k j is the j-th row of K matrix. Relative position embedding r H and r W are learned and shared among heads, not between layers.

[0019] 3、Furthermore, step 3 concatenates and projects the outputs of all heads in step 2 to get the result as follows:

[0020] MHA(X)=Concat[O1,…,O Nh ]W O

[0021] are learnable parameters, O1,...,O Nh is the first single-head self-attention computation result to the N h th single-head self-attention computation result, MHA(X) is reshaped to a shape tensor of (H, W, d v ) to match the original spatial dimensions. That is, by obtaining the outputs of multiple single-head self-attention, concatenating their results, and then performing a learnable linear transformation W O , the final video frame feature mapping result is obtained.

[0022] 4、Finally, step 4 forms the final enhanced convolution expression by the features obtained in steps 3 and convolution as follows:

[0023] AACon(X)=Concat[Conv(X),MHA(X)]

[0024] AAConv(X) is the feature mapping result of the video frame obtained by performing convolution on the input tensor X and the feature mapping result of the video frame obtained by concatenating MHA(X) in step 3. For each spatial position (h, w), N h (attention heads) attention maps on the image are calculated by querying and keying. These attention maps are used to calculate the N h weighted average value of the value V. Then the results are connected and reshaped to match the spatial dimensions of the original volume and mixed with the pointwise convolution. Multi-head attention is applied in parallel to the standard convolution operation, and the output is connected.

[0025] Beneficial effects: Compared with the currently popular long-term recurrent convolutional network, the present application combines multi-head self-attention in the convolutional part of the network, can capture long-range interactions, and finally concatenates with the convolutional features, so that the model can pay attention to local features and global context during convolution operation, thereby improving the accuracy of video action recognition of the model. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 Optimized LRCNs model flow chart of the application.

[0027] Figure 2 Self-attention enhanced convolution architecture of the application.

[0028] Figure 3 Improved model of the first layer LRCNs of the application. DETAILED DESCRIPTION

[0029] The application will be further described below with reference to the accompanying drawings.

[0030] The technical idea of the action recognition method based on the long-term recurrent convolutional network optimized by the self-attention enhanced convolution of the application is as follows:

[0031] LRCNs uses a convolutional neural network layer to extract the picture features of each frame of the video, and the visual features of each frame can be used as the input of each long short-term memory network layer (LSTM), and then the LSTM layer learns the time sequence relationship of all frames in the video segment to achieve the effect of action recognition. However, the local nature of the convolution kernel prevents it from capturing the global context in the image, which is often necessary for better recognizing objects in the image. Therefore, the two-dimensional relative self-attention mechanism enhanced convolution operator embedded with relative position information in the convolutional neural network layer part of the LRCNs network maintains translational equivalence, and the convolution feature map and the feature map generated by the self-attention are spliced, so that the model can focus on local features and global context during convolution operation.

[0032] As shown in Figure 1 , the action recognition method based on the long-term recurrent convolutional network optimized by the self-attention enhanced convolution of the application, the specific process is as follows:

[0033] Step 1, using a single-head self-attention mechanism to calculate the weighted average value of a video frame.

[0034] Step 2, embedding two-dimensional relative position coding in the single-head self-attention mechanism.

[0035] Step 3, splicing multiple single-head attentions to form multi-head attention to calculate the feature mapping result of a video frame.

[0036] Step 4, the two-dimensional relative self-attention mechanism enhanced convolution operator embedded with relative position information maintains translational equivalence, and the convolution feature map and the feature map generated by the self-attention are spliced, so that the model can focus on local features and global context during convolution operation.

[0037] 1、Firstly, the following naming rules are defined: H, W, F in represent the height, width, and number of input filters of the activation mapping. Nh ,d v ,d k These represent the number of heads, value depth, query depth, and key depth in a multi-head watch, respectively. Assume N... h d v and d k Divide equally and This represents the value of each attention head, the depth of the query, and the key. Step 1 assumes a tensor (H, W, F) of a given shape. in (For convenience, the batch processing dimension is omitted), and it is flattened into one. The matrix is ​​used to perform multi-head attention in the Transformer. The output of single-head self-attention is represented as:

[0038]

[0039] It is a learnable linear transformation that maps input X to query Q = XW. q Key k = XW k The value v = XW v This yields a weighted average of a video frame calculated using single-head self-attention. Multi-head attention allows the model to jointly focus on information from different representation subspaces at different locations.

[0040] 2. Secondly, if positional information is not used in step 1, the self-attention permutations will be equivalent, which is ineffective for modeling highly structured data such as images. Proposed methods include introducing sinusoidal expansion of positional information and connecting positional channels to activation maps using CoordConv, but these positional encodings are not helpful for image classification and object detection. This may be because, while these positional encodings are not permutationally equivariant, they do not satisfy translational equivariance. Translational equivariance is a necessary property when processing images. Therefore, a two-dimensional relative positional encoding is embedded in the single-head self-attention expression of step 1 by independently adding relevant height and width information:

[0041]

[0042] It is a relative logical matrix along the height and width dimensions, satisfying q i It is the i-th row of the Q matrix. and These are the relevant heights j y -i y and related width j x -i x Learning embeddings. Pixel i = (i x i y) of pixel j = (j x , j y ) is:

[0043]

[0044] k j is the j-th row of K matrix. Relative position embeddings r H and r W are learned and shared across heads, not across layers.

[0045] 3、Furthermore, step 3 concatenates and projects the outputs of all heads in step 2 to get the result as follows:

[0046] MHA(X) = Concat[O1,…,O Nh ]W O

[0047] is a learnable parameter, is the first single-head self-attention calculation result to the N h th single-head self-attention calculation result, MHA(X) is reshaped into a shape tensor of (H, W, d v ) to match the original spatial dimension. That is, by obtaining the outputs of multiple single-head self-attention, concatenating their results, and then performing a learnable linear transformation W O , the final video frame feature mapping result is obtained.

[0048] 4、Finally, step 4 forms the final enhanced convolution expression by the features obtained in steps 3 and convolution as follows:

[0049] AACon(X) = Concat[Conv(X), MHA(X)]

[0050] AAConv(X) is the feature mapping result of the video frame obtained by performing convolution on the input tensor X and the feature mapping result of the video frame obtained by concatenating MHA(X) in step 3. For each spatial position (h, w), N h (heads of multi-head attention) attention maps on the image are calculated by querying and keying. These attention maps are used to calculate the weighted average of N h values V. Then the results are connected and reshaped to match the spatial dimension of the original volume and mixed with pointwise convolution. Multi-head attention is applied in parallel to standard convolution operations, and the outputs are connected.

[0051] Wherein, the two-dimensional relative self-attention mechanism enhanced convolution operator that embeds relative position information while maintaining translational equivalence makes the structure of the convolution feature map and the feature map result produced by self-attention concatenation as Figure 2As shown, the final extracted features can focus on local features and global context.

[0052] The network architecture of the l-th layer of the optimized LRCN is shown in Figure 3 As shown, a frame of video is inputted, and the first five layers of Alexnet are used, in which each layer of convolution uses convolution operation to capture local features, and self-attention is used to capture the global context of the image. The convolution feature map and the feature map generated by the self-attention are spliced and sent to the FC6 layer of the fully connected layer of Alexnet, and then the long short-term network is used to extract the time feature. In order to make a single label prediction for the entire video clip, the label probabilities (the output of the softmax layer of the network) of all frames are averaged, and the most likely label is selected.

Claims

1. An improved long-term recurrent convolutional network-based video action recognition method, characterized in that, The method comprises the following steps: Step 1, using single-head self-attention mechanism to calculate the weighted average of a video frame; Step 2, embedding two-dimensional relative position coding in the single-head self-attention mechanism; Step 3, splicing multiple single-head attentions to form multi-head attention to calculate the feature mapping result of a video frame; Step 4, embedding relative position information while maintaining two-dimensional relative self-attention mechanism that is translationally equivalent, enhancing the convolution operator to splice the convolution feature map and the feature map result generated by self-attention, so that the model can pay attention to local features and global context during convolution operation; The specific process of step 1 is as follows: The naming rules are defined as follows: represent the height, width, and number of input filters that activate the map; respectively represent the number of heads, value depth, query depth, and key depth in multi-head attention; it is assumed will be and split equally, and represent the value, query, and key depth of each attention head; Assume a video frame tensor of shape is flattened into a matrix and multi-headed attention in the Transformer is performed, the output of which is represented as: ； , is a learnable linear transformation that maps input queries keys values to obtain a weighted average of video frame computed by single-head self-attention, multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions; The specific process of step 2 is as follows: If position information is not used, the arrangement of self-attention will be equivalent, which is invalid for modeling highly structured data such as images. CoordConv has been proposed to introduce sinusoidal wave extended position information and connect position channels to activation mapping, but these position encodings are not helpful for image classification and object detection. The possible reason is that these position encodings, although not permutation invariant, do not satisfy translational invariance; translational invariance is a necessary property when processing images. Therefore, by independently adding two-dimensional relative position coding of relevant height information and relevant width information, the expression of step 1 single-head self-attention is embedded as follows: ； is a relative logical matrix along the height and width dimensions, satisfying , , is the row of the matrix, and are the relative height and relative width learning embeddings, respectively, pixel attention logical expression: ; is the row of the matrix, relative position embedding and are learned and shared among heads, not among layers. 2.The method of claim 1, wherein, The specific process of step 3 is as follows: The outputs of all the heads in step 1 are spliced and projected to obtain the following results: ; is a learnable linear transformation, is the first single-head self-attention computation result to the single-head self-attention computation result, is reshaped to shape tensor to match the original spatial dimension, i.e. by concatenating the outputs of multiple single-head self-attention, and then passing them through a learnable linear transformation to get the final video frame feature map result. 3.The method of claim 1, wherein, The specific process of step 4 is as follows: The final enhanced convolution expression obtained by step 3 and convolution is as follows: ； For input tensors The feature mapping results of the video frames obtained by convolution and the results in step 3 The feature mapping results of the video frames obtained by stitching together, for each spatial location Calculate the image by query and key. (Number of heads in multi-head attention) attention maps; these attention maps are used to calculate values. of The weighted average is then calculated; the results are then concatenated, reshaped to match the spatial dimensions of the original volume, and mixed with pointwise convolutions. Multi-head attention is applied in parallel to the standard convolution operation, and the outputs are concatenated.

Images