A lightweight video behavior recognition method based on structural reparameterization

Abstract

The application discloses a lightweight video behavior recognition method based on structural reparameterization and belongs to the technical field of computer artificial intelligence. The lightweight video behavior recognition method of the application studies the lightweight problem based on a 3D convolution structure, utilizes the reparameterization of the 3D structure to enable the network to continue using the 3D convolution to extract the advantages of the features on the space-time, and still can achieve the purpose of the lightweight network model after the reparameterization, finally can deploy the model and perform model reasoning with a small parameter amount and a small amount of calculation, that is, trains the network with a large parameter amount, performs reasoning with a small model in reasoning and ensures that the model accuracy does not decrease. The reparameterization design of the 3D structure achieves the purpose of the lightweight 3D network model and keeps the model accuracy from excessively losing. The lightweight network constructed by the application is used for video behavior recognition, which can not only improve the recognition efficiency, but also ensure the recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to a lightweight video behavior recognition method based on structural reparameterization, belonging to the technical field of computer artificial intelligence. BACKGROUND

[0002] Video behavior recognition is to obtain the action behavior of the character in the video by analyzing a video segment, which has wide application prospects in security monitoring, driver or passenger behavior recognition, short video review, etc.

[0003] Using 3D convolution can effectively capture the information of time dimension and space dimension, but the video behavior recognition network based on 3D convolution (D. Tran, L. Bourdev, R. Fergus, L. Torresani and M. Paluri. Learning Spatiotemporal Features with 3D Convolutional Networks. Proceedings of the IEEE international conference on computer vision (ICCV). 2015: 4489-4497.) often causes problems such as large parameter quantity and high floating point calculation quantity, which cannot be directly applied and popularized on edge devices with limited computing power.

[0004] In order to solve the above problems, although the 3D structure network model extended directly based on the lightweight network has smaller parameter quantity and calculation quantity, the precision loss is serious, and the video behavior recognition network based on SlowFast extracts the motion information and background information in the video frame through the fast and slow double branch channels, which reduces the size of the model to a certain extent, but the final model is still relatively bulky, which is difficult to directly deploy and use, which limits the use and popularization of the video behavior recognition based on 3D convolution. SUMMARY

[0005] In order to solve the problem that the current video behavior recognition method cannot guarantee the precision and calculation efficiency at the same time, the present application provides a lightweight video behavior recognition method based on structural reparameterization, and the technical solution is as follows:

[0006] The first purpose of the present application is to provide a lightweight video behavior recognition method, which comprises the following steps:

[0007] Step one: obtaining a behavior video to be recognized;

[0008] Step two: preprocessing the behavior video to be recognized to obtain a behavior image sequence;

[0009] Step three: inputting the behavior image sequence into a lightweight video behavior recognition network based on structural reparameterization;

[0010] Step four: the lightweight video behavior recognition network based on structural reparameterization performs calculation on the behavior image sequence and outputs a behavior recognition result.

[0011] Optionally, the construction process of the lightweight video behavior recognition network based on structural reparameterization comprises:

[0012] Step 1: expanding the DBB module structure into a 3D structure, i.e. replacing the original 2D convolution with 3D convolution, replacing the original 2D pooling operation with 3D pooling operation, and constructing a 3D-DBB module;

[0013] Step 2: constructing a structural reparameterization depthwise convolution, i.e. a 3D-DBB-DepthWise module, using the 3D-DBB module;

[0014] Step 3: constructing a structural reparameterization pointwise convolution, i.e. a 3D-DBB-PointWise module, using the 3D-DBB module;

[0015] Step 4: stacking the 3D-DBB-DepthWise module and the 3D-DBB-PointWise module to construct a structural reparameterization depthwise separable convolution, i.e. a DP3DBB module;

[0016] Step 5: replacing the depthwise separable convolution in a lightweight network with the DP3DBB module based on the lightweight network as a backbone network.

[0017] Optionally, the lightweight network in step 5 comprises a MobileNet series network and / or a ShuffleNet series network.

[0018] Optionally, the replacement manner in step 5 is to regard the basic module and the two branches of the down-sampling model of the ShuffleNet series network as a whole, and directly replace the branch containing the depthwise separable structure with a branch structure containing only one DP3DBB module.

[0019] Optionally, the replacement manner in step 5 is to regard each unit on the two branches of the basic module and the down-sampling model of the ShuffleNet series network as an independent individual, i.e. each unit is replaced by the DP3DBB module.

[0020] Optionally, the activation function of the 3D-DBB module is a HardSwish activation function.

[0021] Optionally, the pre-processing process of the behavior video to be identified in step two comprises normalization, random cropping, and resolution adjustment.

[0022] Optionally, the resolution adjustment is 224x 224.

[0023] A second object of the present application is to provide a video behavior recognition system comprising:

[0024] a video acquisition module configured to collect a behavior video to be identified;

[0025] a video pre-processing module configured to process the collected behavior video and output a behavior image sequence;

[0026] the above-mentioned lightweight video behavior recognition network based on structure re-parameterization is configured to calculate the behavior image sequence;

[0027] an output display module configured to output a behavior recognition result according to the calculation result of the lightweight video behavior recognition network based on structure re-parameterization.

[0028] Optionally, the video behavior recognition system further comprises an alarm device configured to send an alarm signal according to the behavior recognition result.

[0029] The present application has the following advantages:

[0030] The lightweight video behavior recognition method of the present application is based on the lightweight problem of 3D convolution structure, utilizes the re-parameterization of 3D structure to enable the network to continue using 3D convolution to extract the advantages of spatial and temporal features, and still achieves the purpose of lightweight network model after re-parameterization, and finally can deploy the model and perform model inference with a small amount of parameters and calculation, that is, train the network with a large amount of parameters, and perform inference with a small model while ensuring that the model accuracy does not decrease. The re-parameterization design of 3D structure achieves the purpose of lightweight 3D network model and keeps the model accuracy from being excessively lost.

[0031] The lightweight network constructed by the present application for video behavior recognition not only can improve the recognition efficiency, but also can ensure the recognition accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0033] Figure 1is a 3D-DBB module structure diagram of the application.

[0034] Figure 2 is a 3D-DBB-DepthWise module structure diagram of the application.

[0035] Figure 3 is a 3D-DBB-PointWise module structure diagram of the application.

[0036] Figure 4 is a DP3DBB module structure diagram of the application.

[0037] Figure 5 is a structure diagram of two different depth separable convolution replacement methods of the application.

[0038] Figure 6 is a ShuffleDBBv1 basic module structure replacement design diagram of the application.

[0039] Figure 7 is a ShuffleDBBv1 down-sampling module structure replacement design diagram of the application.

[0040] Figure 8 is a ShuffleDBBv2 basic module and down-sampling model structure replacement design diagram of the application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the application more clear, the embodiments of the application will be further described in detail below with reference to the drawings.

[0042] Embodiment one:

[0043] The embodiment provides a lightweight video behavior recognition method, comprising:

[0044] Step one: obtaining a behavior video to be recognized;

[0045] Step two: preprocessing the behavior video to be recognized to obtain a behavior image sequence;

[0046] Step three: inputting the behavior image sequence into a lightweight video behavior recognition network based on structural reparameterization;

[0047] Step four: the lightweight video behavior recognition network based on structural reparameterization performs calculation on the behavior image sequence and outputs a behavior recognition result.

[0048] Embodiment two: taking 3D-ShuffleNetV2 as a backbone network and using the first replacement method to replace the depth separable structure

[0049] Step one: obtain the behavior video to be identified;

[0050] Real-time video can be collected by a camera, or an existing video containing behavior can be used.

[0051] Step two: pre-process the behavior video to be identified to obtain a behavior image sequence, the pre-processing process mainly includes: extracting video frames, normalizing video frames, random cropping, etc., to obtain a behavior image sequence, and the resolution of the input image sequence is controlled at 224x 224 in this embodiment.

[0052] Step three: input the behavior image sequence into a lightweight video behavior recognition network based on structural reparameterization;

[0053] The construction process of the recognition network is as follows:

[0054] Step 1: In order to be suitable for video behavior recognition, which is data with a time sequence dimension, this embodiment uses a DBB (Diverse Branch Block) structure through expansion, that is, uses a 3D convolution to replace the original 2D convolution, and uses a 3D pooling operation to replace the original 2D pooling operation, so as to extract features in the space-time dimension, so that it can be suitable for three-dimensional convolution-based tasks, and the 3D-DBB module structure obtained after expansion is as shown in Figure 1 .

[0055] Step 2: use the 3D-DBB module to construct a depthwise convolution with structural reparameterization, and name it as a 3D-DBB-DepthWise module, the module structure is as shown in Figure 2 , group each convolution in the standard 3D-DBB module, so as to simulate the core convolution of the standard depthwise convolution, so as to reduce the parameter amount and the calculation amount.

[0056] Step 3: use the 3D-DBB module proposed in step 1 to construct a pointwise convolution with structural reparameterization, and name it as a 3D-DBB-PointWise module, the module structure is as shown in Figure 3 , that is, all 3x3x3 convolutions in the standard 3D-DBB module are replaced by 1x1x1 convolutions, so as to simulate the effect of the pointwise convolution of the standard depthwise convolution.

[0057] Step 4: stack the 3D-DBB-DepthWise module and the 3D-DBB-PointWise module of step two to construct a depthwise separable convolution with structural reparameterization, that is, a DP3DBB module, the DP3DBB structure is as shown in Figure 4 .

[0058] Step 5: Using the 3D-ShuffleNetV2 as the backbone network, the structure of the lightweight ShuffleDBBv1 network is constructed using replacement method one. As shown in Figure 5 , the two branches of the basic module and the down-sampling module of the 3D-ShuffleNetV2 are each regarded as a whole, and the branch path containing the depth separable structure is directly replaced by a branch structure containing only a DP3DBB structure. The basic module and the down-sampling of the initial 3D-ShuffleNetV2 are shown in Figure 6 (a) and Figure 7 (a), respectively. The convolution structure containing the depth separable structure on the original branch path is directly replaced by the DP3DBB module using method one. The structure of the basic module and the down-sampling module after replacement are shown in Figure 6 (b) and Figure 7 (b), respectively. After structure re-parameterization, the basic module and the down-sampling module of the final inference structure are shown in Figure 6 (c) and Figure 7 (c), respectively.

[0059] Step 6: Training and testing the network model based on the combination of structure re-parameterization and lightweight network. During training, the network before structure re-parameterization is used for training, and during inference testing, the network after structure re-parameterization is used for inference testing.

[0060] Step 6.1: Test the effectiveness of the improved model using the EgoGesture, Jester, and UCF-101 video behavior data sets. Perform video frame extraction operations on the video data sets of the data sets.

[0061] Step 6.2: Perform preprocessing operations on the video frames, such as normalization, random cropping, etc. Control the resolution of the input image sequence to 224x 224.

[0062] Step 6.3: Input the preprocessed sequence of 16 consecutive video frames into the network model for model training. Calculate the network error through forward calculation of the network and the loss function, and propagate the error back to calculate the error gradient of each layer weight of the network and update the weight value, gradually reducing the network error value. Continuously execute the above process to find the most effective network training parameters, so that the network loss is reduced to a minimum, which completes the network training process and obtains the network model. This process can be simply summarized as parameter optimization.

[0063] Step 6.4: Test and calculate the parameter quantity, floating point calculation quantity, and inference delay on GPU and CPU before and after the network model re-parameterization.

[0064] Step four: the lightweight video behavior recognition network based on structural reparameterization is used to calculate the behavior image sequence and output a behavior recognition result.

[0065] Embodiment three: using the second replacement method to replace the depth separable structure with 3D-ShuffleNetV2 as the backbone network

[0066] The implementation steps are basically the same as those of embodiment two, except that the replacement method in step 5 is different. The specific replacement method in step 5 of embodiment three is as follows:

[0067] Using the second replacement method to build a lightweight ShuffleDBBv2 network with structural reparameterization, the second replacement method is as shown in Figure 5 Each unit on the two branches of the basic module and the down-sampling model of ShuffleNetV2 is regarded as an independent individual, that is, each unit is replaced by the corresponding 3D-DBB module structure. The replacement process is as shown in Figure 8 , wherein Figure 8 (a) is the basic module of the original ShuffleNet, (b) is the structure reparameterization after the replacement of the second method, (c) is the down-sampling model of the original ShuffleNet, and (d) is the down-sampling module after the structure reparameterization after the replacement of the second method.

[0068] Embodiment four: using 3D-MobileNetV1 as the backbone network

[0069] The embodiment provides a lightweight video behavior recognition method, which comprises the following steps:

[0070] The implementation steps are basically the same as those of embodiment two, except that the replacement method in step 5 is different. The specific replacement method in step 5 of embodiment three is as follows:

[0071] Using the DP3DBB module to replace the depth separable structure in the 3D-MobileNetV1 network with 3D-MobileNetV1 as the backbone network. The specific network structure is shown in Table 1.

[0072] Table 1 MobileV1DBB network structure

[0073]

[0074] In order to prove the beneficial effects of the present application, a comprehensive comparative experiment was carried out, in which the video behavior method of the present application was compared with a lightweight video behavior recognition method directly extended based on a lightweight network and an advanced video behavior recognition network, and the experimental data is shown in Table 2 and Table 3. Table 1 is the parameter amount, floating point calculation amount, accuracy on EgoGesture, Jester and UCF-101 data sets, and inference delay on CPU and GPU of the network model of the present embodiment and the classical video behavior recognition network and the network model directly extended based on the lightweight network. Table 2 is the accuracy on the EgoGesture data set after using the pre-trained model of the present embodiment and the advanced video behavior recognition network.

[0075] Table 2 Parameter amount, floating point calculation amount, accuracy on EgoGesture, Jester and UCF-101 data sets, and inference delay on CPU and GPU of each video behavior recognition model

[0076]

[0077] Table 3 Comparison of results on the EgoGesture data set after using the pre-trained model

[0078]

[0079] As can be seen from the data in Table 2, the accuracy of the present application is significantly improved compared with the classical video behavior recognition network and the network directly extended based on the lightweight network, and after reparameterization, the speed can be comparable to the video behavior recognition network directly extended based on the lightweight network. As can be seen from the data in Table 3, the accuracy of the present embodiment only decreases by 0.42% when the parameter amount is only 1 / 50 of the ResNeXt-101 network.

[0080] In summary, the lightweight video behavior recognition method of the present application is based on the lightweight problem of the 3D convolution structure, and the reparameterization of the 3D structure enables the network to continue to use the advantage of 3D convolution to extract the features in space and time, and after reparameterization, the purpose of the lightweight network model can still be achieved, and finally the model can be deployed and the model inference can be performed with small parameter amount and calculation amount, that is, the network is trained with large parameter amount, and the model is inferred with small model and the model accuracy is not reduced. The reparameterization design of the 3D structure achieves the purpose of lightweight 3D network model and keeps the model accuracy from being excessively lost.

[0081] Some steps in the embodiments of the present application can be implemented by software, and the corresponding software program can be stored in a readable storage medium such as an optical disc or a hard disk.

[0082] The above merely describes preferred embodiments of the present application and is not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A lightweight video action recognition method, characterized in that, The method comprises: Step 1: obtaining a behavior video to be identified; Step 2: pre-processing the behavior video to be identified to obtain a behavior image sequence; Step 3: inputting the behavior image sequence into a lightweight video behavior recognition network based on structural reparameterization; Step 4: the lightweight video behavior recognition network based on structural reparameterization performs calculation on the behavior image sequence and outputs a behavior recognition result; The construction process of the lightweight video behavior recognition network based on structural reparameterization comprises: Step 1: expanding the DBB module structure into a 3D structure, i.e. replacing the original 2D convolution with a 3D convolution, and replacing the original 2D pooling operation with a 3D pooling operation, to construct a 3D-DBB module; Step 2: constructing a depth-wise convolution based on structural reparameterization using the 3D-DBB module, i.e. a 3D-DBB-DepthWise module; Step 3: constructing a point-wise convolution based on structural reparameterization using the 3D-DBB module, i.e. a 3D-DBB-PointWise module; Step 4: stacking the 3D-DBB-DepthWise module and the 3D-DBB-PointWise module to construct a depth separable convolution based on structural reparameterization, i.e. a DP3DBB module; Step 5: replacing the depth separable convolution module in a lightweight network with the DP3DBB module based on the lightweight network as the backbone network.

2. The lightweight video action recognition method of claim 1, wherein, The lightweight network in step 5 comprises: a MobileNet series network and / or a ShuffleNet series network.

3. The lightweight video action recognition method of claim 2, wherein, The replacement mode in step 5 is: regarding the basic module and the two branches of the down-sampling model of the ShuffleNet series network as a whole, and directly replacing the branch containing the depth separable structure with a branch structure containing only one DP3DBB module.

4. The lightweight video action recognition method of claim 2, wherein, The replacement mode in step 5 is: regarding each unit on the two branches of the basic module and the down-sampling model of the ShuffleNet series network as an independent individual, i.e. replacing each unit with the DP3DBB module.

5. The lightweight video action recognition method of claim 1, wherein, The activation function of the 3D-DBB module is a HardSwish activation function.

6. The lightweight video action recognition method of claim 1, wherein, The pre-processing process of the behavior video to be identified in step 2 comprises: normalization, random cropping, and resolution adjustment.

7. The lightweight video action recognition method of claim 6, wherein, The resolution adjustment is 224x 224.

8. A video behavior recognition system, characterized by, The video behavior recognition system comprises: A video acquisition module for collecting a behavior video to be identified; A video preprocessing module for processing the collected behavior video and outputting a behavior image sequence; A lightweight video behavior recognition network based on structural reparameterization for calculating the behavior image sequence, the construction process of the lightweight video behavior recognition network based on structural reparameterization being as described in any one of claims 1-7; An output display module for outputting a behavior recognition result according to the calculation result of the lightweight video behavior recognition network based on structural reparameterization.

9. The video behavior recognition system of claim 8, wherein, Further comprising an alarm device for sending an alarm signal according to the behavior recognition result.

Images