A grouping second-order spatio-temporal feature aggregation method for video behavior recognition

Abstract

The application discloses a grouping second-order spatio-temporal feature aggregation method for video behavior recognition, comprising the following steps: using a pre-trained 2D convolutional neural network as a backbone model, obtaining a data set containing video data and labels and dividing the data set into a training set and a test set, frame extraction is performed on the video and the video is saved locally; the video is sampled in a sparse sampling mode, and the sampled frames are transmitted to an overall grouping second-order aggregation network as input; the grouping second-order aggregation network is trained using the training set, network parameters are updated in each iteration, the test set data is used for verification in each iteration, and the grouping second-order aggregation network weight with the optimal verification accuracy is saved; the grouping second-order aggregation network is initialized using the weight file with the highest accuracy, the test set is tested, the test result is compared with the label data, the number of correct and incorrect results is counted, and the accuracy of the network on the test set is calculated.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video behavior recognition in video understanding, and particularly relates to a grouping second-order spatio-temporal feature aggregation method for video behavior recognition. BACKGROUND

[0002] Video behavior recognition is a basic problem in video understanding, which has a wide range of applications in video review, intelligent monitoring and other application scenarios. With the emergence of deep learning and a series of deep networks, video behavior recognition has made further development. The methods in the field of video behavior recognition can be roughly divided into the following four kinds: dual-stream network, 3D convolutional neural network, computationally efficient network, and Transformer network. However, these methods usually use global average pooling at the end of the network to generate video representation, but global average pooling only simply calculates first-order statistical features in a disordered manner, and has limited ability to capture the complex spatio-temporal dynamic information contained in the video. SUMMARY

[0003] In view of the problems existing in the prior art, the present application discloses a grouping second-order spatio-temporal feature aggregation method for video behavior recognition, which specifically comprises the following steps:

[0004] A pre-trained 2D convolutional neural network is used as a backbone model, and a grouping second-order aggregation module is used to replace the original average pooling layer of the 2D convolutional neural network after the last convolutional layer of the 2D convolutional neural network to obtain a grouping second-order aggregation network.

[0005] A dataset containing video data and labels is obtained and divided into a training set and a test set, the video is frame extracted and saved locally;

[0006] The video is sampled using a sparse sampling method, and the sampled frames are used as input to the overall grouping second-order aggregation network;

[0007] The grouping second-order aggregation network is trained using the training set, the network parameters are updated in each iteration, and the test set data is used to verify in each iteration, and the grouping second-order aggregation network weight with the optimal verification accuracy is saved;

[0008] The grouping second-order aggregation network is initialized using the weight file with the highest verification accuracy, and is tested on the test set, the test results are compared with the label data, the number of correct and incorrect is counted to calculate the accuracy of the network on the test set.

[0009] A pre-trained 2D convolutional neural network ConvNeXt-T is used as a backbone model, and a grouping second-order aggregation module is used to replace the original average pooling layer of ConvNeXt-T after the last convolutional layer of ConvNeXt-T to obtain a grouping second-order aggregation network.

[0010] In the second-order group aggregation module, let the output of the last convolutional layer of ConvNeXt-T be χ ∈ R T×D×H×W , where T, D, H, W are length (number of input video frames), depth (number of channels), width, and height, respectively. Let χ be represented as a set of D-dimensional features X = {x1,..., x N}, where N = T × H × W. Let the entire feature space be C, and C have K subspaces, i.e., C = {c1,..., c K}. Assign feature descriptors x i {i = 1,..., N} to the K subspaces for aggregation.

[0011] Before aggregating the features, the input features are decomposed into M sets of relatively low-dimensional vectors:

[0012]

[0013] When aggregating, feature descriptors x i {i = 1,..., N} are assigned to subspaces using a soft assignment method. First, calculate the soft assignment weight of the feature descriptor to the k-th subspace:

[0014]

[0015]

[0016] α gk (x i ) represents the weight of the feature descriptor assigned to the subspace c k , then multiply the weight with the residual information and apply the attention function between groups:

[0017]

[0018] Attn(x i ) = σ(FC(x i ))

[0019] where FC is a fully connected layer, and σ is a sigmoid activation function. Then, use the summation method to fuse all subspaces:

[0020]

[0021] where represents the k-th subspace, and V F is reshaped to change its shape from to Finally, calculate the covariance to obtain high-order statistical information:

[0022]

[0023] wherein, I and 1 represent n x n identity matrix and all-one matrix respectively. Then the square root of the covariance matrix is approximated by using the iterative matrix square root normalization method to utilize the geometric representation of the covariance matrix.

[0024] After the calculation through the second-order group aggregation module, a layer normalization operation and a fully connected layer for classification are performed.

[0025] The application adopts the above scheme and provides a group second-order spatio-temporal feature aggregation method for video behavior recognition. The method uses a 2D convolutional neural network ConvNeXt-T as a backbone model, has lower calculation overhead than a dual-flow network, a 3D convolutional neural network and a Transformer network, replaces an original average pooling layer in the ConvNeXt-T by using a group second-order aggregation module to obtain a group second-order aggregation network, firstly aggregates video spatio-temporal features into compact video-level expressions by distributing feature descriptors to subspaces in a soft distribution manner in the group second-order aggregation module, then fuses information contained in all subspaces to further reduce the dimension without losing information, and finally calculates a covariance matrix of features fused with all subspaces to capture complex temporal dynamic information and better utilize a geometric representation of the covariance matrix by calculating a square root thereof. The group second-order aggregation network finally obtains a more robust video feature expression, well simulates complex spatio-temporal feature distribution in a video, and achieves competitive results in performance and improves accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0027] Figure 1 is a flowchart of the method of the present application;

[0028] Figure 2 is a structural schematic diagram of the second-order group aggregation network in the present application. DETAILED DESCRIPTION

[0029] In order to make the technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application:

[0030] As Figure 1 shown in a grouping second-order spatio-temporal feature aggregation method for video behavior recognition, specifically comprising the following steps:

[0031] S1: The pre-trained 2D convolutional neural network ConvNeXt-T is used as the backbone model and the last downsampling operation is removed. After the last convolutional layer of ConvNeXt-T, the original global average pooling layer is replaced with a grouping second-order aggregation module to obtain a grouping second-order aggregation network.

[0032] S11: Obtain the video dataset Kinetics-400 and its label. The data contained in the video dataset is represented as Data=[x1,...,x U ], and the video dataset is divided into a training set and a test set;

[0033] S12: Use the FFmpeg tool to frame all data in the video dataset, and store the video frames in RGB format locally;

[0034] S13: Define a single video data in the training set as Data train ∈R (B×T)×C×H×W , where B is the size of the batch, set to 32, T is the number of frames input after sparse sampling, set to 8, C is the number of input channels, since the input is an RGB format frame, it is 3, H and W are the height and width of the input, which are scaled to 224x224, so the size of a single video data input into the network is [32x8,3,224,224];

[0035] S14: Input the data into the grouping second-order aggregation network;

[0036] S15: Train the grouping second-order aggregation network using the training set. In each iteration, update the network parameters through backpropagation, and use the test set data to verify in each iteration, and save the grouping second-order aggregation network weight with the optimal verification accuracy;

[0037] S16: Initialize the grouping second-order aggregation network using the weight file with the highest verified accuracy, test on the test set, compare the test results with the label data, count the number of correct and incorrect to calculate the accuracy of the network on the test set.

[0038] S2: Obtain the video dataset and label, divide the video dataset into a training set and a test set, frame the video dataset and save it;

[0039] In the implementation process, first, obtain the video dataset and label, and the video data and all data constitute Data=[x1,...,x U], where the total number of videos is U, and each video data is x i , i = 1, 2, …, N, U1 videos are extracted from the total number of videos as a training set for model training, and the remaining U-U1 video data is used as a test set for model testing. The existing video data is frame extracted and saved locally.

[0040] As shown in Figure 2 : S3: the calculation process of the grouping second-order aggregation module;

[0041] S31: In the grouping second-order aggregation network, the output feature matrix of the last convolution layer of ConvNeXt-T is X ∈ R (32×8)×768×13×13 , where 32 is the batch size, 8 is the frame number, 768 is the channel number, and 13x13 is the spatial resolution;

[0042] S32: Before the grouping second-order aggregation module, first reshape the feature matrix, and set the number of groups to 8. The shape of the feature matrix changes to:

[0043] [(32x8)x768x13x13]→[32x768x(8x13x13)]→[32x96x8x(8x13x13)]

[0044] S33: Assign the feature descriptors to the subspaces and apply the attention function on the groups:

[0045]

[0046]

[0047]

[0048] Attn(x i )=σ(FC(x i ))

[0049] The shape of the feature matrix changes to: [32x96x8x(8x13x13)]→[32x48x96x(8x13x13)], where 48 is the number of cluster centers, i.e. K in the formula;

[0050] S34: The information contained in all subspaces is fused in an additive manner:

[0051]

[0052] The dimension of the subspace is compressed, and the shape of the feature matrix changes to:

[0053] [32x48x96x(8x13x13)]→[32x96x(8x13x13)]

[0054] S35: reshaping the matrix V F The shape of the feature matrix is changed by a reshaping operation as follows:

[0055] [32x96x(8x13x13)]→[32x96x(8x13)x13]

[0056] S36: calculating the square root of the reshaped matrix V F The covariance matrix is calculated as follows:

[0057]

[0058] The square root of the covariance matrix is then calculated. Using its geometric structure, the final input shape is [32, 4656].

[0059] S4: approximating the square root of the covariance matrix using an iterative matrix square root normalization method;

[0060] S41: dividing the covariance matrix by its trace as follows:

[0061]

[0062] S42: repeatedly approximating the square root of the covariance matrix using a Newton-Schulz iteration method for a fixed number of times;

[0063] S43: multiplying the square root of the covariance matrix trace as follows:

[0064]

[0065] S5: calculating a layer normalization operation and obtaining a prediction probability for each class through a fully connected layer used for classification.

[0066] The application provides a grouping second-order spatiotemporal feature aggregation method for video behavior recognition.

[0067] The above describes only the preferred specific embodiments of the application, but the protection scope of the application is not limited to this. Any person skilled in the art can make equivalent replacements or changes to the technical solutions and the inventive concept of the application within the technical scope disclosed by the application, which should be covered within the protection scope of the application.

Claims

1.A method for group second-order spatio-temporal feature aggregation for video action recognition, characterized in that The application comprises the following steps: A pre-trained 2D convolutional neural network is used as a backbone model, and a grouping second-order aggregation module is used to replace the original average pooling layer of the 2D convolutional neural network after the last convolutional layer of the 2D convolutional neural network to obtain a grouping second-order aggregation network; A data set containing video data and labels is obtained and divided into a training set and a test set, the video is frame extracted and saved locally; The video is sampled in a sparse sampling manner, and the sampled frames are transmitted to the overall grouping second-order aggregation network as input; The grouping second-order aggregation network is trained using the training set, the network parameters are updated in each iteration, and the test set data is used for verification in each iteration, and the grouping second-order aggregation network weight with the optimal verification accuracy is saved; The grouping second-order aggregation network is initialized using the weight file with the highest accuracy, and the test set is tested, the test results are compared with the label data, the number of correct and incorrect results is counted, and the accuracy of the network on the test set is calculated; A ConvNeXt-T in a 2D convolutional neural network is used as a backbone model, and a grouping second-order aggregation module is used to replace the original average pooling layer of the ConvNeXt-T after the last convolutional layer of the ConvNeXt-T to obtain a grouping second-order aggregation network; Let the output of the last convolutional layer of ConvNeXt-T be χ ∈ R T×D×H×W where T, D, H, W are length, depth, width, and height, respectively, and χ is represented as a set of D-dimensional features X = {x1,...,x N}, where N = T x H x W, let the entire feature space be C, C has K subspaces, i.e., C = {c1,...,c K}, and assign the feature descriptors x i {i = 1,...,N} to the K subspaces for aggregation; Before the features are aggregated, the input features are decomposed into M sets of relatively low-dimensional vectors: At the time of aggregation, the feature descriptors x i {i = 1,..., N} are assigned to the subspaces, first the soft assignment weights of the feature descriptors to the k-th subspace are computed: a gk (x i ) denotes a feature descriptor weights assigned to the subspaces c k The weights are multiplied with the residual information and the attention function is applied across the groups: Attn(x i ) = σ(FC(x i )) Where FC is a fully connected layer, and sigma is a sigmoid activation function, and all subspaces are fused in an additive manner: where representing the kth subspace, on V F reshaping operation to transform its shape from to Finally, covariance pooling is performed to obtain higher order statistics: wherein, I and 1 represent an n x n identity matrix and an all-one matrix, respectively, k represents an index of a subspace, c k denotes the kth subspace, and the square root of the covariance matrix is approximated by using an iterative matrix square root normalization method to utilize a geometric representation of the covariance matrix.

Images