A video action recognition method based on second-order Transformer modeling

Abstract

This invention discloses a video behavior recognition method based on second-order Transformer modeling, comprising: acquiring a video dataset containing video data and labels; dividing the video dataset into a training set and a test set; and extracting frames from the video dataset; using a pre-trained second-order Transformer architecture as a backbone network, which uses a second-order classification head for data classification; designing two different temporal modeling methods in the backbone network, one being inserting a label shifting module and a spatiotemporal attention module inside the encoder in the middle of the network, and the other being inserting a 3D temporal convolution into the label embedding module at the front end of the network, thereby constructing a video second-order Transformer network; training the video second-order Transformer network using the training set, the training process including forward propagation of network features and backward propagation of errors; updating network parameters in each iteration during model training, initializing the video second-order Transformer network with optimal weights, and evaluating the performance of the network on the test set.

Description

Technical Field

[0001] This invention relates to the field of video behavior recognition, and more specifically to a video behavior recognition method based on second-order Transformer modeling. Background Technology

[0002] Methods based on deep convolutional neural networks (CNNs) have long dominated the field of computer vision. However, the Transformer architecture, currently widely used in natural language processing (NLP), has achieved better performance than CNNs in many visual tasks. Following the design principles of NLP, Transformers can also model the relationships between sequences in a self-attention mechanism, a capability lacking in CNNs due to the biased induction of their convolutions. Benefiting from its global modeling ability regarding long-term dependencies, the Transformer architecture is more suitable than CNNs for video recognition tasks, as video tasks require understanding the long-term intra- and inter-frame context or relationships within video clips. Therefore, for the foundational backbone, video recognition researchers have gradually shifted their attention from CNNs to Transformers, forming a new trend and research hotspot. However, due to the temporal information present in videos, visual Transformers, which have proven effective in image tasks, cannot model the spatiotemporal structure in video recognition. Furthermore, Transformer-based video models only use class labels or visual labels for classification, failing to combine the advantages of both types of labels. Summary of the Invention

[0003] In view of the problems existing in the prior art, the present invention discloses a video behavior recognition method based on second-order Transformer modeling, which specifically includes the following steps:

[0004] Obtain a video dataset containing video data and labels, divide the video dataset into training and testing sets, and extract frames from the video dataset.

[0005] A pre-trained second-order Transformer architecture is used as the backbone network, which uses a second-order classification head to classify data.

[0006] Two different temporal modeling methods are designed in this backbone network. One is to insert a label shifting module and a spatiotemporal attention module inside the encoder in the middle of the network. The other is to insert 3D temporal convolution into the label embedding module at the front end of the network, thereby constructing a second-order Transformer network for video.

[0007] The video second-order Transformer network is trained using the training set. The training process includes forward propagation of network features and backward propagation of errors.

[0008] During model training, the network parameters are updated in each iteration, and the video frames are validated. The optimal weights of the video second-order Transformer network are saved based on the accuracy of the optimal validation set. The video second-order Transformer network is initialized with the optimal weights, and the performance of the network is evaluated on the test set.

[0009] Furthermore, when using a second-order classification head for data classification, a second-order statistical modeling method, svPN, is introduced to collect second-order statistics of visual labels. At the same time, class labels and visual labels are used to form a second-order classification head for video recognition.

[0010] Furthermore, when inserting 3D temporal convolutions into the label embedding module at the front end of the network: for 3D temporal convolutions, video segments are divided into non-overlapping blocks and fed into the label embedding module. 3D convolutions are inserted in a residual manner before each residual block of the module, thereby enhancing feature modeling in the temporal dimension.

[0011] Furthermore, when inserting a marker shifting module and a spatiotemporal attention module inside the encoder in the middle of the network:

[0012] Each video segment has a label matrix consisting of T class labels, c∈R. T×D The marker shifting module first divides the marker matrix into three parts along the channel dimension, as shown below:

[0013] c = [c a ,c b ,c c ],c a ,c b ,c c ∈R T×a ,R T×b ,R T×c

[0014] The content of a frame divided into the head and tail is c a and c c Swap the timestamps before or after it, c b The content remains unchanged. For frame t = 1, 2, ..., t, the swapping across adjacent frames is achieved using the following equation:

[0015] c a (t)=c a (t-1)

[0016] c b (t)=c b (t)

[0017] c c (t)=c c (t+1)

[0018] When inserting the spatiotemporal attention module, the visual label X undergoes a linear transformation operation, which includes three independent convolution operations. The dimension of the visual label X is reduced from D to C = D / 2, resulting in feature vectors Q, K, and V. These feature vectors are then processed by the formula... The attention score matrix Y is obtained through matrix multiplication. This matrix records all pairwise similarities between two visual labels. The similarity matrix is ​​normalized by multiplying by V along the row direction using the softmax function to restore the shape of the attention score matrix Y. A convolution operation is then performed on the attention score matrix Y, and the input visual label X is added using a residual connection. The calculation process is as follows:

[0019]

[0020]

[0021] Q = flatten(YW) Q ),K = flatten(YW K ),V = flatten(YW V ).

[0022] By employing the above-mentioned technical solution, the video behavior recognition method based on second-order Transformer modeling provided by this invention uses the recently introduced Transformer model in the field of vision. The entire network consists of a label embedding module, an encoder layer, and a second-order classifier head. In the early stage, 3D temporal convolution is inserted into the label embedding module through residual method to enhance temporal modeling. In the encoder layer, a label shifting module and spatiotemporal attention are inserted to model temporal relationships and spatiotemporal interactions, respectively. In the later stage, the output visual labels are aggregated through second-order cross-covariance pooling to model the feature statistics of intra-frame correlation and inter-frame cross-correlation. These feature statistics are combined with class labels to form a second-order classifier head for final classification. The network finally obtains a robust high-order representation that can simulate complex spatiotemporal feature distributions. Compared with other methods, this invention achieves competitive results in performance, using fewer parameters and GFLOPs. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 The flowchart of the method of the present invention

[0025] Figure 2 This is a diagram showing the overall structure of the second-order Transformer network for video in this invention.

[0026] Figure 3 The diagram shows the structure of the two time modeling methods in this invention.

[0027] Figure 4 These are 2D and 3D structural diagrams of the mark shifting operation in this invention.

[0028] Figure 5 This is a diagram of the spatiotemporal attention operation structure in this invention. Detailed Implementation

[0029] To make the technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention:

[0030] like Figure 1 The video action recognition method based on second-order Transformer modeling shown includes the following steps:

[0031] S1: Obtain a video dataset containing video data and labels, divide it into a training set and a test set. The training set is used for model training, and the test set is used for model testing. Extract frames from the video dataset.

[0032] S11: Obtain the video dataset Kinetics-400 and its label file. The video data and all data are structured as DataSet = [x1, x1, ... ... N ];

[0033] S12: Divide the video dataset into a training set and a test set. The training set is used for model training, and the test set is used for model testing.

[0034] S13: Perform segment sampling on the above video data;

[0035] S14: Use FFmpeg to extract frames from the sampled data;

[0036] like Figure 2 S2: A pre-trained second-order Transformer architecture is used as the backbone network, which uses a second-order classification head to classify data.

[0037] S21: For one frame, the class label and the aggregated visual label are fed into the softmax classifier, which can be represented as:

[0038]

[0039] Here, FC and Pool represent a fully connected layer and a second-order pooling function, respectively. The pooling function is used to aggregate visual labels and reasonably consider their higher-order statistical information.

[0040] S22: Perform a linear projection operation on the visual markers of each frame:

[0041]

[0042] S23: Using second-order cross-covariance pooling to aggregate visual labels:

[0043]

[0044] S24: Yes Perform singular value power normalization (svPN). Accurate calculation of singular values ​​requires the SVD decomposition algorithm; svPN is given by the following equation:

[0045]

[0046] like Figure 3 , Figure 4 and Figure 5 As shown: S3: Two different temporal modeling methods are designed in this backbone network. One is to insert a marker shifting module and a spatiotemporal attention module inside the encoder in the middle of the network. The other is to insert 3D temporal convolution into the marker embedding module at the front end of the network, thereby constructing a second-order Transformer network for video.

[0047] S31: The video second-order Transformer network consists of three main parts: a label embedding module, an encoder for temporal and spatial encoding of class labels and visual labels, and a second-order classification head for video prediction.

[0048] S32: The label embedding module consists of one convolutional block and three residual blocks. In this stage, for 3D temporal convolution, video segments are divided and mapped into non-overlapping blocks and fed into the label embedding module. 3D convolutions are inserted in a residual manner before each residual block of this module, thereby enhancing feature modeling in the temporal dimension.

[0049] S33: The encoder consists of two parts. The first part is the tag shifting module, multi-head self-attention, and spatiotemporal attention, which can be represented as:

[0050]

[0051]

[0052]

[0053] The second part is layer normalization and multilayer perceptron, namely:

[0054]

[0055]

[0056] S34: When inserting a tag shifting module inside the encoder in the middle of the network, each video segment has T class tags forming a tag matrix c∈R. T×D The marker shifting module first divides the marker matrix into three parts along the channel dimension, as shown below:

[0057] c = [c a ,c b ,c c ],c a ,c b ,c c ∈R T×a ,R T×b ,R T×c

[0058] The content of a frame divided into the head and tail is c a and c c Swap the timestamps before or after it, c b The content remains unchanged. For frame t = 1, 2, ..., t, the swapping across adjacent frames is achieved using the following equation:

[0059] c a (t)=c a (t-1)

[0060] c b (t)=c b (t)

[0061] c c (t)=c c (t+1)

[0062] S35: When inserting the spatiotemporal attention module, the visual label X undergoes a linear transformation operation. This linear transformation includes three independent convolution operations, reducing the dimension of the visual label X from D to C = D / 2. The resulting feature vectors Q, K, and V are then processed by the formula... The attention score matrix Y is obtained through matrix multiplication. This matrix records all pairwise similarities between two visual labels. The similarity matrix is ​​normalized by multiplying by V along the row direction using the softmax function to restore the shape of the attention score matrix Y. A convolution operation is then performed on the attention score matrix Y, and the input visual label X is added using a residual connection. The calculation process is as follows:

[0063]

[0064]

[0065] Q = flatten(XW) Q ), K = flatten(XW K ),V = flatten(XW V )

[0066] S4: Train a second-order Transformer network using the training set. The training process includes forward propagation of network features and backward propagation of errors.

[0067] During implementation, a 14-layer second-order Transformer network for video was used for training, with each frame cropped to a 224×224 shape. It was trained for 45 iterations on the Kinetics-400 dataset with an initial learning rate of 1×e. -4 .

[0068] S5: During model training, update the network parameters in each iteration and validate the model against video frames. Save the optimal weights of the video 2D Transformer network based on the best validation set accuracy. Initialize the video 2D Transformer network using the optimal weights and evaluate its performance on the test set.

[0069] During implementation, tests were conducted using 1 clip × 1 crop and 3 clips × 2 crops strategies, respectively. Top-1 recognition accuracy (%), network parameters, and GFLOPs were used as evaluation criteria.

[0070] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for video action recognition based on second-order Transformer modeling, characterized in that The application relates to a video recognition method based on a second-order Transformer network. The application comprises the following steps: acquiring a video dataset containing video data and labels, dividing the video dataset into a training set and a test set, and frame extraction on the video dataset; using a pre-trained second-order Transformer architecture as a backbone network, and using a second-order classification head for data classification; designing two different time modeling methods in the backbone network, one of which is to insert a label shift module and a space-time attention module in the middle of the network, and the other of which is to insert a 3D time convolution into a label embedding module at the front end of the network, thereby constructing a video second-order Transformer network; training the video second-order Transformer network using the training set, and the training process comprises forward propagation of network features and backward propagation of errors; updating network parameters in each iteration during model training, and verifying video frames, saving the optimal weight of the video second-order Transformer network according to the optimal verification set accuracy, initializing the video second-order Transformer network using the optimal weight, and evaluating the performance of the network on the test set; when the 3D time convolution is inserted into the label embedding module at the front end of the network: for the 3D time convolution, the video segment is divided into non-overlapping blocks and sent into the label embedding module, and the 3D convolution is inserted in a residual manner before each residual block in the module, so as to enhance the feature modeling in the time dimension; Each video clip has T class labels to form a label matrix c e R T×D The label shift module first divides the label matrix into three parts along the channel dimension as follows: c = [c a ,c b ,c c ], c a ,c b ,c c ∈ R T×a ,R T×b ,R T×c The split content c of a frame header and trailer a and c c is exchanged with the timestamp immediately before or after it, respectively, c b The content of c remains unchanged, and the exchange across adjacent frames for the t-th frame t = 1, 2,..., t is achieved by the following equation: c a (t) = c a (t-1) c b (t) = c b (t) c c (t) = c c (t+1) When the spatiotemporal attention module is inserted, the visual mark X is subjected to a linear transformation operation, the linear transformation includes 3 independent convolution operations, the dimension of the visual mark X is reduced from D to C=D / 2, the obtained feature vectors Q, K and V are subjected to matrix multiplication with the formula The attention score matrix Y records all pair similarities between two visual marks, the similarity matrix is subjected to normalization processing along the row direction by using the softmax function, the shape of the attention score matrix Y is restored, the attention score matrix Y is subjected to convolution operation, the input visual mark X is added by using residual connection, and the calculation process is as follows: Q = flatten(YW Q ), K = flatten(YW K ), V = flatten(YW V ).

2. The method of claim 1, wherein: when the label shift module and the space-time attention module are inserted in the middle of the encoder: when the second-order classification head is used for data classification, a second-order statistical modeling method svPN is introduced to collect the second-order statistics of visual labels, and a second-order classification head for video recognition is formed by using the class label and the visual label.

Images