A lightweight video action recognition method based on hybrid model

Abstract

The present invention discloses a lightweight video behavior recognition method based on a hybrid model, which belongs to the technical field of computer artificial intelligence disciplines. The present invention constructs a lightweight deep convolutional network for video behavior recognition by combining a lightweight 3D convolution module with a Transformer. While using 3D convolution to extract the advantages of spatiotemporal features, it makes up for the lack of inductive bias in the Transformer, allowing the network to converge faster. The Transformer is used to model long-time series information for tasks such as videos with time dimension information. The two complement each other. By comparing different data sets, the method of the present invention improves the accuracy of the model, reduces the difficulty of training the model, and ensures that the model is still a lightweight network, thereby improving recognition efficiency.

Description

Technical Field

[0001] The present invention relates to a lightweight video behavior recognition method based on a hybrid model, and belongs to the technical field of computer artificial intelligence disciplines. Background Art

[0002] Video behavior recognition is to analyze a video clip to obtain the action behavior of the characters in the video. It has broad application prospects in security monitoring, driver or passenger behavior recognition, short video review, etc.

[0003] 3D convolution can effectively capture information in both temporal and spatial dimensions, but video action recognition networks directly based on 3D convolution (D. Tran, L. Bourdev, R. Fergus, L. Torresani and M. Paluri. Learning Spatiotemporal Features with 3D Convolutional Networks. Proceedings of the IEEE international conference on computer vision (ICCV). 2015: 4489-4497.) often suffer from large parameters and high floating-point computational overhead, making them difficult to apply and popularize directly on edge devices with limited computing power. While 3D structured network models directly derived from lightweight network expansion have smaller parameters and computational overhead, they suffer from significant accuracy loss, which limits the use and popularity of 3D convolution-based video action recognition. Transformer-based video action recognition network models, while offering superior accuracy, often have larger parameters and computational overhead. Compared to models using convolution, they require larger training datasets to achieve the same accuracy, resulting in lower efficiency in building recognition models and performing recognition.

[0004] Therefore, existing video behavior recognition methods are difficult to achieve high-precision recognition effects while maintaining high recognition efficiency. Summary of the Invention

[0005] To address the problem that current video behavior recognition methods cannot guarantee both high accuracy and high efficiency, the present invention provides a lightweight video behavior recognition method and system. The specific technical solutions are as follows:

[0006] A first object of the present invention is to provide a video behavior recognition method, comprising:

[0007] Step 1: Obtain the behavior video to be identified;

[0008] Step 2: Preprocessing the behavior video to be identified to obtain a behavior image sequence;

[0009] Step 3: Input the action image sequence into a lightweight video action recognition network based on a hybrid of Transformer and 3D-CNN;

[0010] Step 4: The lightweight video action recognition network based on the hybrid of Transformer and 3D-CNN calculates the action image sequence and outputs the action recognition result.

[0011] Optionally, the lightweight video action recognition network based on a hybrid of Transformer and 3D-CNN includes: an input layer, a first convolutional layer, a 3D-ShuffleViT network stacked more than once, a second convolutional layer, a pooling layer, a linear classification layer, and an output layer connected in sequence;

[0012] The 3D-ShuffleViT network includes: a 3D-ShuffleNetV2 basic module, a 3D-ShuffleNetV2 downsampling module and a ShuffleViT module;

[0013] The construction process of the 3D-ShuffleViT network includes:

[0014] Step 1: Construct the 3D-ShuffleNetV2 basic module and the 3D-ShuffleNetV2 downsampling module;

[0015] The lightweight ShuffleNetV2 network is expanded to convert its 2D convolution into 3D convolution and 2D pooling into 3D pooling; and the original ReLU activation function is replaced by the HardSwish activation function;

[0016] Step 2: Build the ShuffleViT module;

[0017] Perform Unflod expansion on the feature layer, calculate the self-attention between the patches in the expanded feature layer, and finally fold the expanded feature layer;

[0018] Step 3: Construct a 3D-ShuffleViT network by the 3D-ShuffleNetV2 basic module, the 3D-ShuffleNetV2 downsampling module and the ShuffleViT module.

[0019] Optionally, step 2 includes:

[0020] Step 2.1: First, pass the feature map through a convolution layer with a convolution kernel size of 3x3x3 to perform local feature modeling, and then adjust the number of channels through a convolution layer with a convolution kernel size of 1x1x1;

[0021] Step 2.2: Unflod operation;

[0022] Divide the feature map into patches, and set the patch size to 2x2x2, that is, there are 4 pixels in each patch;

[0023] Step 2.3: Self-attention calculation;

[0024] Each token only performs self-attention calculations with the token at the same position in each patch;

[0025] Step 2.4: Flod operation;

[0026] After the self-attention calculation is completed, the features are folded, and [N, C, T, H, W] is folded into [N, C, T*H*W], where N represents the batch size, C represents the number of channels, T represents the time dimension, and H and W represent the height and width. The features calculated by the self-attention mechanism are folded into the original feature map.

[0027] Optionally, the ShuffleViT network adopts a HardSwish activation function.

[0028] Optionally, the convolution kernel size of the first convolution layer is 3x3x3, and the stride is 1.

[0029] Optionally, the convolution kernel size of the second convolutional layer is 1x1x1.

[0030] Optionally, the process of preprocessing the behavior video to be identified in step 2 includes: normalization, random cropping, and resolution adjustment.

[0031] Optionally, the resolution is adjusted to 224x224.

[0032] A second object of the present invention is to provide a video behavior recognition system, comprising:

[0033] Video acquisition module, used to collect the behavior video to be identified;

[0034] The video preprocessing module is used to process the collected behavior videos and output a behavior image sequence; the above-mentioned 3D-ShuffleViT network is used to calculate the behavior image sequence;

[0035] The output display module is used to output the behavior recognition results based on the calculation results of the 3D-ShuffleViT network.

[0036] Optionally, an alarm device is also included for issuing an alarm signal based on the behavior recognition result.

[0037] The beneficial effects of the present invention are:

[0038] The video behavior recognition method and system of the present invention construct a lightweight video behavior recognition network 3D-ShuffleViT by mixing the Transformer and ShuffleNetV2 upsampling modules and downsampling models, which makes up for the accuracy loss problem of the video behavior recognition network directly expanded based on the lightweight network, significantly improves the network accuracy, ensures that the network remains a lightweight network, and reduces the training difficulty.

[0039] Compared with existing recognition schemes, the present invention not only improves recognition accuracy, but also improves model construction and recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 This is a structural diagram of the 2D-ShuffleNetV2 basic module and the corresponding 3D-ShuffleNetV2 basic module of Example 2 of the present invention.

[0042] Figure 2 This is a structural diagram of the 2D-ShuffleNetV2 downsampling module and the 3D-ShuffleNetV2 downsampling module of the second embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of the Unflod and Flod operations of the 3D-ShuffleViT module in the second embodiment of the present invention.

[0044] Figure 4 This is a structural diagram of a lightweight video behavior recognition network based on a hybrid of Transformer and 3D-CNN according to the second embodiment of the present invention.

[0045] Figure 5 This is a structural diagram of the ShuffleViT module according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0047] First, the basic theoretical knowledge involved in this invention is introduced:

[0048] 1. Transformer

[0049] The Transformer abandons the traditional sequential structure of recurrent neural networks and instead adopts a self-attention mechanism to enable parallel training and capture global information. The Vision Transformer can be considered a graphical version of the Transformer, directly migrating the standard Transformer model to the image domain with minimal modification. To adapt the Transformer model to images, the image is divided into many sub-blocks and these sub-blocks are organized into linear embedding sequences. These linear embedding sequences are then used as input to the Transformer, simulating phrase sequence input in the natural language processing (NLP) field. The Transformer's self-attention mechanism effectively models long-range information.

[0050] 2. ShuffleNetV2 network

[0051] Most current advanced lightweight networks use depthwise separable convolutions or group convolutions to reduce computational overhead. However, neither operation changes the number of channels in a feature, necessitating the use of 1×1 convolutions to facilitate information fusion between channels and transform them to a specified dimension. Consequently, 1×1 convolutions in lightweight networks consume a significant amount of computation and impose constraints between channels, reducing model accuracy to a certain extent. To further reduce computational overhead, ShuffleNetV1 proposed channel shuffling to achieve information fusion between channels. ShuffleNetV2 further addresses this shortcoming by introducing a channel splitting operation, which divides the input features into two parts. One part undergoes a true depthwise separable computation, the result is then concatenated with the other part through channel concatenation. Finally, a channel shuffling operation is performed to achieve information interoperability, reducing the number of parameters and computation while improving model accuracy.

[0052] The structure of the 3D-ShuffleViT of the present invention is mainly improved by combining the ShuffleNetV2 and Transformer models.

[0053] Example 1:

[0054] This embodiment provides a video behavior recognition method, including:

[0055] Step 1: Obtain the behavior video to be identified;

[0056] Step 2: Preprocessing the behavior video to be identified to obtain a behavior image sequence;

[0057] Step 3: Input the action image sequence into a lightweight video action recognition network based on a hybrid of Transformer and 3D-CNN;

[0058] Step 4: The lightweight video action recognition network based on the hybrid of Transformer and 3D-CNN calculates the action image sequence and outputs the action recognition result.

[0059] Example 2:

[0060] This embodiment provides a video behavior recognition method, the method comprising:

[0061] Step 1: Obtain the behavior video to be identified;

[0062] You can use a camera to capture real-time video, or use an existing video that contains the behavior.

[0063] Step 2: Preprocess the behavior video to be identified to obtain a behavior image sequence. The preprocessing process mainly includes: extracting video frames, normalizing the video frames, and performing random cropping operations to obtain the behavior image sequence. In this embodiment, the resolution of the input image sequence is controlled at 224x224.

[0064] Step 3: Input the action image sequence into a lightweight video action recognition network based on a hybrid of Transformer and 3D-CNN;

[0065] The lightweight video action recognition network 3D-ShuffleViT network based on the hybrid of Transformer and 3D-CNN is as follows Figure 4 As shown, it includes the following connected in sequence: input layer, first convolutional layer, 3D-ShuffleViT network stacked more than once, second convolutional layer, pooling layer, linear classification layer and output layer;

[0066] The 3D-ShuffleViT network includes: a 3D-ShuffleNetV2 basic module, a 3D-ShuffleNetV2 downsampling module and a ShuffleViT module;

[0067] The construction process of the 3D-ShuffleViT network includes:

[0068] Step 1: Convert the 2D convolution in the lightweight ShuffleNetV2 network to 3D convolution, and convert the 2D pooling operation to 3D pooling operation to obtain the basic module and downsampling module of the 3D structure ShuffleNetV2. And replace the original ReLU activation function with the HardSwish activation function. The basic module of 3D-ShuffleNetV2 is as follows: Figure 1As shown in (b), the downsampling module is as follows Figure 2 (b) shown.

[0069] Step 2: Build the ShuffleViT module.

[0070] Step 2.1: First, pass the feature map through a convolution layer with a convolution kernel size of 3x3x3 to perform local feature modeling, and then adjust the number of channels through a convolution layer with a convolution kernel size of 1x1x1.

[0071] Step 2.2: Unflod operation. Figure 3 As shown in the figure, the feature map is divided into patches, and the patch size is set to 2x2x2, that is, there are 8 pixels in each patch.

[0072] Step 2.3: Self-attention calculation. Then perform self-attention calculation, each Token (such as Figure 3 Each pixel (that is, each small color block) only performs self-attention calculations with the token at the same position in each patch, thereby reducing the amount of calculation. Since each patch has 4 pixels, the current amount of self-attention calculations is equivalent to 1 / 4 of the original amount.

[0073] Step 2.4: Flood Operation. After the self-attention calculation is completed, the features are folded from [N, C, T, H, W] to [N, C, T * H * W], where N represents the batch size, C represents the number of channels, T represents the time dimension, and H and W represent the height and width. The features calculated by the self-attention mechanism are folded back into the original feature map structure.

[0074] Step 3: Construct the 3D-ShuffleViT network by the 3D-ShuffleNetV2 basic module, 3D-ShuffleNetV2 downsampling module and ShuffleViT module. The specific network structure is as follows Figure 4 shown.

[0075] In this example, the behavioral image sequence input first passes through a convolutional layer with a kernel size of 3x3x3 and a stride of 1. To avoid excessive loss of feature information early on, a stride of 1 is used for the convolution operation. The 3D-ShuffleNetV2 base module, the 3D-ShuffleNetV2 downsampling module, and the ShuffleViT module are then stacked. Finally, a convolutional layer with a kernel size of 1x1x1 is used for final feature extraction. A stride of 1 indicates the 3D-ShuffleNetV2 base module, a stride of 2 indicates the 3D-ShuffleNetV2 downsampling module, and ShuffleViTBlock indicates the ShuffleViT module. Finally, global pooling is applied to the feature layer, followed by a linear classification layer for final classification.

[0076] Step 4: Train and test the lightweight video action recognition network based on the hybrid model.

[0077] Step 4.1: Use the Jester and UCF-101 video behavior datasets to test the effectiveness of the improved model and extract video frames from the video datasets.

[0078] Step 4.2: Perform preprocessing operations on the video frames, such as normalization, random cropping, etc. The resolution of the input image sequence is controlled at 224x 224.

[0079] Step 4.3: Input the preprocessed 16-frame video sequence into the network model for model training. The network's forward computation and loss function are used to calculate the network error. This error is then backpropagated to calculate the error gradient for each layer of the network weights. The weights are then updated, gradually reducing the network error. This process is repeated repeatedly to find the most effective network training parameters. Minimizing the network loss completes the network training process and yields the network model. This process can be simply summarized as parameter optimization.

[0080] Step 4.4: Test and calculate the network model's parameter count, floating-point computational effort, accuracy, and inference latency on both the GPU and CPU.

[0081] Step 4.5: Export the trained model and apply it to a mobile phone device to test the model's inference speed and latency in a real environment.

[0082] To demonstrate the beneficial effects of the present invention, a comparative experiment was conducted to compare the video behavior recognition method of the present invention with a lightweight video behavior recognition method directly extended from a lightweight network. The experimental data are shown in Tables 1 and 2, where Table 1 is the parameter quantity and floating-point calculation amount comparison data of this embodiment, and Table 2 is the recognition accuracy comparison data of this embodiment.

[0083] Table 1 Comparison of parameter quantities and floating-point calculation quantities

[0084]

[0085] Table 2 Comparison of recognition accuracy

[0086]

[0087] As can be seen from the data in Table 1, the lightweight video action recognition method based on the hybrid of Transformer and 3DCNN in this embodiment has a parameter count of 1.7M and a floating-point computational load of 0.6G. Compared with other solutions, this embodiment has the advantage of being lightweight in both parameter count and computational load.

[0088] As can be seen from the data in Table 2, this embodiment has higher recognition accuracy rates in all three data sets compared to other solutions. Combined with Table 1, it can be proved that the present invention can ensure high recognition efficiency while ensuring recognition accuracy.

[0089] In summary, the present invention introduces Transformer to make up for the shortcoming that 3D convolution cannot perform long-distance modeling, and with the help of 3D convolution, it makes up for the lack of inductive bias of Transformer, so that the network has the ability of long-distance time modeling and powerful feature extraction ability, while improving the accuracy of the model and ensuring that the size of the model remains a lightweight network. The lightweight video behavior recognition hybrid model constructed based on Transformer and 3D convolution has the advantages of high precision and high efficiency.

[0090] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0091] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A video behavior recognition method, characterized in that: The video behavior recognition method includes: Step 1: Obtain the behavior video to be identified; Step 2: Preprocessing the behavior video to be identified to obtain a behavior image sequence; Step 3: Input the action image sequence into a lightweight video action recognition network based on a hybrid of Transformer and 3D-CNN; Step 4: The lightweight video action recognition network based on the hybrid of Transformer and 3D-CNN calculates the action image sequence and outputs the action recognition result; The lightweight video action recognition network based on the hybrid of Transformer and 3D-CNN includes: an input layer, a first convolutional layer, a 3D-ShuffleViT network, a second convolutional layer, a pooling layer, a linear classification layer, and an output layer, wherein the 3D-ShuffleViT network is stacked more than once; The 3D-ShuffleViT network includes: a 3D-ShuffleNetV2 basic module, a 3D-ShuffleNetV2 downsampling module and a ShuffleViT module; The construction process of the 3D-ShuffleViT network includes: Step 1: Construct the 3D-ShuffleNetV2 basic module and the 3D-ShuffleNetV2 downsampling module; Convert the 2D convolution in the lightweight ShuffleNetV2 network to 3D convolution, and convert the 2D pooling operation to 3D pooling operation; Step 2: Build the ShuffleViT module; Perform Unflod expansion on the feature layer, calculate the self-attention between the patches in the expanded feature layer, and finally fold the expanded feature layer; Step 3: Construct the 3D-ShuffleViT network by the 3D-ShuffleNetV2 basic module, the 3D-ShuffleNetV2 downsampling module and the ShuffleViT module; The step 2 includes: Step 2.1: First, pass the feature map through a convolution layer with a convolution kernel size of 3x3x3 to perform local feature modeling, and then adjust the number of channels through a convolution layer with a convolution kernel size of 1x1x1; Step 2.2: Unflod operation; Divide the feature map into patches, and set the patch size to 2x2x2, that is, there are 8 pixels in each patch; Step 2.3: Self-attention calculation; Each token only performs self-attention calculations with the token at the same position in each patch; Step 2.4: Flod operation; After the self-attention calculation is completed, the features are folded, and [N, C, T, H, W] is folded into [N, C, T*H*W], where N represents the patch size, C represents the number of channels, T represents the time dimension, and H and W represent the height and width. The features calculated by the self-attention mechanism are folded into the original feature map structure.

2. The video behavior recognition method according to claim 1, characterized in that: The 3D-ShuffleViT network adopts the HardSwish activation function.

3. The video behavior recognition method according to claim 1, characterized in that The convolution kernel size of the first convolutional layer is 3x3x3, and the stride is 1.

4. The video behavior recognition method according to claim 1, characterized in that: The convolution kernel size of the second convolutional layer is 1x1x1.

5. The video behavior recognition method according to claim 1, characterized in that: The process of preprocessing the behavior video to be identified in step 2 includes: normalization, random cropping, and resolution adjustment.

6. The video behavior recognition method according to claim 5, characterized in that: The resolution is adjusted to 224x224.

7. A video behavior recognition system, characterized in that: The video behavior recognition system includes: Video acquisition module, used to collect the behavior video to be identified; Video preprocessing module, used to process the collected behavior video and output behavior image sequence; The lightweight video action recognition network based on a hybrid of Transformer and 3D-CNN according to any one of claims 1 to 6, used to calculate the action image sequence; The output display module is used to output the behavior recognition results according to the calculation results of the lightweight video behavior recognition network based on the hybrid of Transformer and 3D-CNN.

8. The video behavior recognition system according to claim 7, characterized in that: It also includes an alarm device for sending an alarm signal according to the behavior recognition result.

Images