Multi-feature fusion video motion recognition method based on MNv3-MFAE

Abstract

The invention discloses a fusion multi-feature video action recognition method based on MNv3-MFAE (MNv3-MFAE). According to the scheme, the method comprises the following steps: (1) designing a redundancy elimination space-time modeling module (STEV); (2) designing a long-distance time aggregation module (LTA); (3) designing a fusion multi-feature video action recognition network (MNv3-MFAE); (4) replacing the SE module of the channel extrusion attention module with a TiedSE module; and (5) training and testing the improved network. The problems that video action recognition is insufficient in time modeling, the symmetric action classification effect is poor, and a video action recognition model is large are solved. According to the method, the recognition accuracy of video action recognition is improved, the training time of the network is greatly shortened, and compared with the prior art, breakthroughs are made in the aspects of reducing the model complexity, improving the recognition accuracy and enhancing the time modeling capability.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a MNv3-MFAE-based fusion multi-feature video action recognition method. Background Art

[0002] Video action recognition is one of the research hotspots in the field of computer vision and is widely used in video surveillance, intelligent transportation, sports analysis, virtual reality and other fields.

[0003] Through video action recognition technology, specific behavior types can be identified, which is of great significance for improving the intelligence and accuracy of video surveillance systems and achieving a more intelligent interactive experience.

[0004] Video action recognition usually involves a large amount of computing and storage resources for image processing, feature extraction, classification, and requires saving video frames, feature vectors, classifier parameters, etc. Therefore, it is a common problem that it takes up a lot of resources. Reducing the amount of computation and parameters of the model is an urgent challenge that needs to be solved and is of great significance in practical application deployment.

[0005] Among the existing video action recognition methods based on deep learning: the heavy computation of 3D convolutional networks limits their application, the 2D convolutional networks have weak feature extraction capabilities, and the parameter scale of dual-stream networks and ViT networks is very large.

[0006] To solve the above problems, the present invention proposes a fusion multi-feature video action recognition method based on MNv3-MFAE, and proposes an improved network model based on the 2D convolutional network, comprehensively considering feature extraction and computational cost, overcoming the limitation of the 2D convolutional network's weak feature extraction ability, improving the accuracy while maintaining a low computational burden. The lightest MobilenetV3 in the MobileNet series of classification networks is used as the backbone network, and a sparse time sampling strategy is adopted to introduce it into the field of video recognition. Summary of the invention

[0007] 1. Purpose of the present invention

[0008] The present invention aims to improve the recognition accuracy of symmetrical actions without increasing the number of parameters, and therefore provides a video action recognition method with a 2D convolution as the backbone network.

[0009] 2. Technical solution adopted by the present invention

[0010] The present invention proposes a method for fusion multi-feature video action recognition based on MNv3-MFAE, which is implemented through the following steps:

[0011] (1) Design of redundant spatiotemporal modeling module (STEV):

[0012] (1-1) Definition: The redundant spatiotemporal modeling module uses 3D convolution to stimulate the spatiotemporal information of video sequences and uses the channel and spatial reconstruction attention module to reduce the inherent redundancy of the module;

[0013] (1-2) 3D convolution splitting: In view of the high cost of 3D convolution, this paper uses depth-separable (2+1)D convolution to split the 3D convolution;

[0014] (1-3) Redundancy removal: The spatial reconstruction redundancy removal branch and the channel reconstruction redundancy removal branch are compressed by half from the channel dimension through 1×1 convolution to reduce the amount of network calculation;

[0015] (1-4) Information fusion: First, restore the extracted information to its original channel, and then use 1×1 convolution to perform information interaction to obtain the overall module output;

[0016] (2) Design of long-distance temporal aggregation modeling module (LTA):

[0017] (2-1) Definition: Long-distance temporal aggregation modeling module, which uses 1D convolution to stimulate the temporal information changes of video sequences and adopts a multi-fragment residual structure connection mechanism to capture both short-term and long-term temporal dependencies;

[0018] (2-2) Channel segmentation: The input features are divided into four segments along the channel dimension with the size of [N, T, C / 4, H, W]. Each segment is adjusted by average pooling to adapt to local convolution;

[0019] (2-3) Serial variable cascade structure: In order to capture the temporal features in detail, the module is converted into a hierarchical cascade structure. The last three segments use cascaded temporal sub-convolutional layers to gradually expand the receptive field to capture a wider range of temporal contexts.

[0020] (2-4) Residual-like structure: avoids gradient vanishing, enhances model stability and learning ability, and adopts residual-like structure connection mechanism to promote information flow across segments;

[0021] (3) Design of a multi-feature video action recognition network (MNv3-MFAE):

[0022] (3-1) Introducing shift operations: Shift operations can realize different ways of moving input data. This operation can be used for certain specific requirements in the model, such as time-sensitive feature learning, without increasing the computational cost.

[0023] (3-2) Introducing the motion information extraction module (ME): Based on feature-level modeling, the difference between video frames is used to analyze the motion characteristics in the time dimension. For mirror actions, their spatial information is similar but the motion information is opposite, so accurate extraction of motion information is particularly important for distinguishing symmetrical actions;

[0024] (3-3) Design a plug-and-play multi-feature action extraction module (MFAE): It uses feature fusion to efficiently model the relationship between frames and automatically focus on key frames. The module can be migrated to any 2D network.

[0025] (3-4) Fusion Multi-Feature Video Action Recognition Network (MNv3-MFAE): Insert the plug-and-play multi-feature action extraction module into the Block of MobileNetV3 to form the MFAE-Block structure;

[0026] (4) Replace the squeeze-excitation module SE module with the TiedSE module:

[0027] (4-1) Definition: The squeeze-excitation module (SE module) is an attention module based on channel weight adjustment. The structure of the TiedSE module is similar to that of the S module. The difference is that the idea of ​​bound block convolution is applied to the fully connected layer to form a TB-FC structure. The TiedSE module can effectively reduce the number of parameters thanks to bound block convolution.

[0028] (4-2) Parameter reduction: The bound block convolution only requires one half-size filter, while the ordinary convolution requires two full-size filters, so the parameters are 1 / 4 of the ordinary convolution;

[0029] (5) Train and test the improved network:

[0030] The training was performed on the Jester dataset. When the overall network tended to converge, the improved network model MNv3-MFAE was obtained. Part of the test set was input into the network model for testing. Evaluation indicators such as recognition accuracy, model parameter quantity, frame rate, etc. were obtained. The network model was evaluated and recognized, and finally summarized and analyzed.

[0031] Preferably, the de-redundancy spatiotemporal modeling module in step (1) is composed of a spatiotemporal feature extraction part and a de-redundancy part. In the spatiotemporal feature extraction part, the 3D convolution is split into a 1×3×3 spatial convolution, a 3×1×1 depth convolution and a 1×1×1 depth convolution. The de-redundancy part is divided into channel de-redundancy and spatial de-redundancy. The parallel input method is adopted. When entering the de-redundancy operation, the input size should be [N, T, C / 2, H, W]. After the de-redundancy operation, the output should be restored to the original number of channels [N, T, C, H, W]. Finally, a 1×1 convolution is applied for information interaction.

[0032] Preferably, in step (2), the long-distance temporal aggregation modeling module divides the input size [N, T, C, H, W] into four segments N1 to N4, that is, each segment is [N, T, C / 4, H, W], and the last three segments use cascaded temporal sub-convolutional layers. Different segments in the long-distance temporal aggregation modeling module can have receptive fields of different sizes. Since the first segment directly retains the information of the input segment N1, its receptive field is kept within the minimum range of 1×1×1; and as the information is transmitted and aggregated between segments, the equivalent receptive field of N4 is significantly expanded to three times the original receptive field. The entire module has a residual structure to avoid gradient disappearance.

[0033] Preferably, in step (3), a multi-feature video action recognition network is integrated, and a multi-feature action extraction module MFAE is used to extract rich spatiotemporal information. The multi-feature action extraction module is composed of a multi-branch structure: first, the information is introduced into the time information and the long-distance time aggregation modeling module through the shift operation Shift; then, the information is extracted through the de-redundant spatiotemporal modeling module and the motion information extraction module; finally, the processed features are fused and output. The multi-feature action extraction module is designed as a plug-and-play feature extraction block and inserted into the MNv3 backbone network.

[0034] Preferably, in step (4), all squeeze-excitation modules SE modules are replaced by TiedSE modules, and tied block convolution is applied to the 2D convolution before the feature extraction block and the 2D convolution after the block.

[0035] 3. Beneficial effects of the present invention

[0036] (1) The present invention uses the public dataset Jester to train and test the neural network involved; designs a redundant spatiotemporal modeling module to model the spatiotemporal characteristics of video sequences, and adopts a method of parallel removal of spatial redundancy and channel redundancy to reduce the number of model parameters.

[0037] (2) The present invention designs a long-distance temporal aggregation modeling module so that the output features contain richer and more comprehensive spatiotemporal representations, thus overcoming the limitation of traditional temporal feature extraction methods that rely only on a single local convolution.

[0038] (3) The present invention designs a fusion multi-feature video action recognition network, which combines the module designed in step (1), the module designed in step (2), the shift module and the motion feature extraction module through a multi-branch method to extract different extracted features. This makes up for the difference in 2D in extracting features from video sequences and improves the recognition accuracy of the model.

[0039] (4) The present invention adopts the TiedSE module, which effectively reduces the occupation of parameter resources while maintaining network performance, thereby improving model efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to clearly explain the technical details of the present invention, the following is a brief introduction to the drawings involved in the embodiments. The drawings described below are only some examples recorded in the present invention. For those skilled in the art, other structural forms of drawings can be derived from these drawings without additional innovative efforts.

[0041] Figure 1 It is the overall algorithm flow chart of the present invention;

[0042] Figure 2 It is a detailed structural diagram of the redundant space-time modeling module of the present invention;

[0043] Figure 3 It is a detailed structural diagram of the long-distance time aggregation modeling module of the present invention;

[0044] Figure 4 It is a detailed structural diagram of the multi-feature extraction module of the present invention;

[0045] Figure 5 This is a structural diagram of a fusion multi-feature video action recognition network of the present invention; DETAILED DESCRIPTION

[0046] Typical embodiments of the present invention are described below in conjunction with the accompanying drawings. For the sake of clarity and brevity, not all details of the actual implementation are described in detail in the specification. It should be understood that in the process of developing any such actual implementation, many decisions closely related to the implementation need to be made to achieve the developer's specific goals. For example, these decisions need to comply with various system and business-related constraints, which may vary depending on the implementation.

[0047] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings. Specific implementation method one:

[0049] This embodiment is a fusion multi-feature video action recognition method based on MNv3-MFAE, combined with Figure 1 , the method comprises the following steps:

[0050] Step 1: Design a redundant spatiotemporal modeling module;

[0051] Step 2: Design a long-distance time aggregation modeling module;

[0052] Step 3: Design a multi-feature video action recognition network;

[0053] Step 4: Replace the extrusion-excitation module SE module with the TiedSE module;

[0054] Step 5: Train and test the improved network. Specific implementation method 2:

[0056] Different from the specific implementation method 1, this implementation method is a fusion multi-feature video action recognition method based on MNv3-MFAE, combined with Figure 2 The specific method for designing a redundant spatiotemporal modeling module in step 1 is as follows:

[0057] The de-redundancy spatiotemporal modeling module consists of a spatiotemporal feature extraction part and a de-redundancy part. In the spatiotemporal feature extraction part, the 3D convolution is split into a 1×3×3 spatial convolution, a 3×1×1 depth convolution, and a 1×1×1 depth convolution. The de-redundancy part is divided into channel de-redundancy and spatial de-redundancy. The parallel input method is adopted. When entering the de-redundancy operation, the input size should be [N, T, C / 2, H, W]. After the de-redundancy operation, the output should be restored to the original number of channels [N, T, C, H, W]. Finally, a 1×1 convolution is applied for information interaction. Specific implementation method three:

[0059] Different from the specific implementation methods 1 and 2, the present implementation method is a fusion multi-feature video action recognition method based on MNv3-MFAE, combined with Figure 3 The specific method for designing the long-distance time aggregation modeling module in step 2 is:

[0060] The input size [N, T, C, H, W] is divided into four segments N1~N4, that is, each segment is [N, T, C / 4, H, W], and the last three segments use cascaded temporal sub-convolutional layers. Different segments in the long-distance temporal aggregation modeling module can have receptive fields of different sizes. Since the first segment directly retains the information of the input segment N1, its receptive field remains in the minimum 1×1×1 range; as the information is transmitted and aggregated between segments, the equivalent receptive field of N4 is significantly expanded to three times the original receptive field. The entire module has a residual structure to avoid gradient disappearance. Specific implementation method four:

[0062] Different from the specific implementation methods 1, 2, and 3, the present implementation method is a fusion multi-feature video action recognition method based on MNv3-MFAE, combined with Figure 4 The specific method of designing a fusion multi-feature video action recognition network in step 3 is:

[0063] The multi-feature action extraction module MFAE is used to extract rich spatiotemporal information. The multi-feature action extraction module is composed of a multi-branch structure: first, the information is introduced into the time information and the long-distance time aggregation modeling module through the shift operation Shift; then, the information is extracted through the de-redundant spatiotemporal modeling module and the motion information extraction module; finally, the processed features are fused and output. The multi-feature action extraction module is designed as a plug-and-play feature extraction block and inserted into the MobileNetV3 backbone network. The insertion position is the 1st, 5th, 6th, 8th, 10th and 11th layer blocks with jump connections in the original MobileNetV3. Specific implementation method five:

[0065] Different from the specific implementation methods 1, 2, 3, and 4, the present implementation method is a fusion multi-feature video action recognition method based on MNv3-MFAE, combined with Figure 5 In step 4, the specific process of replacing the extrusion-excitation module SE module with the TiedSE module is as follows:

[0066] All SE modules are replaced by TiedSE modules, and tied block convolution is applied to the 2D convolution before the block in the feature extraction part and the 2D convolution after the block. Specific implementation method six:

[0068] Different from the specific implementation methods 1, 2, 3, 4, and 5, the present implementation method is a fusion multi-feature video action recognition method based on MNv3-MFAE. The specific process of training and testing the network in step 5 is as follows:

[0069] The training is based on the public video action dataset Jester, and the performance of the loss function during the network training process is analyzed. If the loss function does not show a convergence trend, the network parameters are continuously updated and the network architecture is adjusted, and the training process is re-executed. Once the network training tends to converge, the action recognition network model can be obtained. Then, the model is tested to determine the classification accuracy. Through comparison and evaluation, the classification results of the video actions are obtained, and finally analyzed and summarized.

[0070] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may modify or supplement the specific embodiments described or replace them in a similar manner without departing from the spirit of the present invention or exceeding the scope defined by the appended claims.

[0071] Although this document uses more terms such as MNv3, MFAE, Shift, STEV, LTA, ME, SE, TiedSE, etc., it does not exclude the possibility of using other terms. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.

[0072] The above is only the technical solution of the present invention. The actual implementation may not be limited to the above form. The technical personnel of this patent may adopt other equivalent modes without departing from the existing technical mode. Therefore, all equivalent implementation modes should be included in the scope of the present invention.

Claims

1. A method for fusion multi-feature video action recognition based on MNv3-MFAE, characterized by: The method is implemented by the following steps: (1) Design of redundant spatiotemporal modeling module (STEV): (1-1) Definition: The redundant spatiotemporal modeling module uses 3D convolution to stimulate the spatiotemporal information of video sequences and uses the channel and spatial reconstruction attention module to reduce the inherent redundancy of the module; (1-2) 3D convolution splitting: In view of the high cost of 3D convolution, this paper uses depth-separable (2+1)D convolution to split the 3D convolution, splitting the 3D convolution into 1×3×3 spatial convolution, 3×1×1 depth convolution and 1×1×1 depth convolution; (1-3) Redundancy removal: The spatial reconstruction redundancy removal branch and the channel reconstruction redundancy removal branch are compressed by half from the channel dimension through 1×1 convolution to reduce the amount of network calculation; (1-4) Information fusion: First, restore the extracted information to its original channel, and then use 1×1 convolution to perform information interaction to obtain the overall module output; (2) Design of long-distance temporal aggregation modeling module (LTA): (2-1) Definition: Long-distance temporal aggregation modeling module, which uses 1D convolution to stimulate the temporal information changes of video sequences and adopts a multi-fragment residual structure connection mechanism to capture both short-term and long-term temporal dependencies; (2-2) Channel segmentation: The input features are divided into four segments along the channel dimension with the size of [N, T, C, H, W]. Each segment is adjusted by average pooling to adapt to local convolution; (2-3) Serial variable cascade structure: In order to capture the temporal features in detail, the module is converted into a hierarchical cascade structure. The last three segments use cascaded temporal sub-convolutional layers to gradually expand the receptive field to capture a wider range of temporal contexts. (2-4) Residual-like structure: avoids gradient vanishing, enhances model stability and learning ability, and adopts a residual-like structure connection mechanism to promote information flow across segments. Different segments in the long-distance temporal aggregation modeling module can have receptive fields of different sizes. Since the first segment directly retains the information of the input segment N1, its receptive field remains within the minimum 1×1×1 range; and as information is transmitted and aggregated between segments, the equivalent receptive field of N4 is significantly expanded, reaching three times the original receptive field; (3) Design of a multi-feature video action recognition network (MNv3-MFAE): (3-1) Introducing shift operations: Shift operations can realize different ways of moving input data. This operation can be used for certain specific requirements in the model, such as time-sensitive feature learning, without increasing the computational cost. (3-2) Introducing the motion information extraction module (ME): Based on feature-level modeling, the difference between video frames is used to analyze the motion characteristics in the time dimension. For mirror actions, their spatial information is similar but the motion information is opposite, so accurate extraction of motion information is particularly important for distinguishing symmetrical actions; (3-3) Design a plug-and-play multi-feature action extraction module (MFAE): It uses feature fusion to efficiently model the relationship between frames and automatically focus on key frames. The module can be migrated to any 2D network. (3-4) Fusion Multi-Feature Video Action Recognition Network (MNv3-MFAE): Insert the plug-and-play multi-feature action extraction module into the Block of MobileNetV3 to form the MFAE-Block structure; (4) Replace the squeeze-excitation module SE module with the TiedSE module: (4-1) Definition: The squeeze-excitation module (SE module) is an attention module based on channel weight adjustment. The structure of the TiedSE module is similar to that of the S module. The difference is that the idea of ​​bound block convolution is applied to the fully connected layer to form a TB-FC structure. The TiedSE module can effectively reduce the number of parameters thanks to bound block convolution. (4-2) Parameter reduction: bound block convolution only requires one half-size filter, while ordinary convolution requires two full-size filters, so the parameters are 1 / 4 of ordinary convolution; (5) Train and test the improved network: The training was performed on the Jester dataset. When the overall network tended to converge, the improved network model MNv3-MFAE was obtained. Part of the test set was input into the network model for testing. Evaluation indicators such as recognition accuracy, model parameter quantity, frame rate, etc. were obtained. The network model was evaluated and recognized, and finally summarized and analyzed.

2. According to claim 1, a method for fusion multi-feature video action recognition based on MNv3-MFAE, characterized in that: The multi-branch method is used to combine the shift operation, redundant spatiotemporal feature extraction module, motion information extraction module, and long-distance time aggregation modeling module into a plug-and-play multi-feature extraction module. The video action dataset is input into the network model, and 8 frames of action images are fed into a batch. The improved network model MNv3-MFAE is used for feature extraction to obtain a richer and more comprehensive spatiotemporal representation. Finally, after processing with the softmax classifier, the accurate probabilities of the video action Top-1 and Top-5 are obtained, thereby predicting the video action.

3. According to claim 1, a method for fusion multi-feature video action recognition based on MNv3-MFAE, characterized in that: In the fusion multi-feature video action recognition network, the MFAE-Block structure replaces the 1st, 5th, 6th, 8th, 10th and 11th layer blocks with skip connections in the original MobileNetV3.

4. The method for fusion multi-feature video action recognition based on MNv3-MFAE according to claim 1, characterized in that: The TiedSE module is used to improve model performance while reducing the number of network parameters.

Images