A video behavior recognition method based on improved MobileNet

Abstract

The present invention provides a video behavior recognition method based on improved MobileNet, which includes the following steps: first, multiple continuous behavior frames in the behavior video to be recognized are input into the proposed recognition network to mine the motion trend features between the previous and next frames in the behavior video. Then, in the weighted point-by-point convolution process, a random fade-in factor is added to the time axis to provide different weights for each related frame to more effectively utilize the motion trend relationship between behavior frames at different moments. The behavior recognition method provided by the present invention adopts a lightweight network architecture based on multi-frame MobileNet, and introduces multiple continuous behavior frames to describe the internal differences of similar behaviors, thereby realizing fine-grained behavior recognition, detection and evaluation, and improving the accuracy of video-based behavior recognition.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a video behavior recognition method based on improved MobileNet. Background Art

[0002] Video behavior recognition is an important research direction in the field of computer vision, and has great application potential in intelligent video surveillance, sports behavior assessment, gait recognition, etc. Simple video behavior recognition, namely action classification, only needs to correctly classify a given video into several known action categories, while complex behavior recognition usually includes a set of actions with a strict temporal order relationship, rather than just one action category in the video. In addition, compared with static images, video content and background are more complex and changeable. Different behaviors may have similarities, while the same behavior may behave differently in different environments.

[0003] Thanks to the rapid development of deep learning and image processing technology, researchers have proposed many models and methods that can be applied to behavior recognition, such as convolutional neural network (CNN), recurrent neural network (RNN), residual network (ResNet), attention mechanism, etc. CNN is a deep feedforward neural network containing convolution operations, which has the ability of representation learning and can classify input information according to its hierarchical structure. RNN is a recursive neural network that takes sequence data as input and recurs along the direction of sequence evolution. It has memory, parameter sharing and Turing completeness. ResNet is a convolutional neural network proposed in 2015. It was first applied to image classification and object recognition. It is easy to optimize and can improve accuracy by increasing considerable depth. The attention mechanism in computer vision can give the neural network the ability to focus on a subset of its input, so that the model can ignore irrelevant information and focus on key information. However, most of these models are designed for simple action classification of static images and cannot be used for complex behavior recognition and evaluation with strong temporal relationships. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies of the prior art and provide a video behavior recognition method based on an improved MobileNet, which can better utilize the relationship information between consecutive frames in the video to be recognized and achieve a better recognition accuracy in the recognition and evaluation of complex behaviors with strong temporal relationships. The specific technical solution is as follows:

[0005] A video behavior recognition method based on an improved MobileNet, wherein the improved MobileNet includes a quasi-3D convolution layer, a weighted point-by-point convolution layer, a two-step convolution module, a maximum pooling layer, a fully connected layer, and a SoftMax layer; the method specifically includes the following steps:

[0006] Step 1: Use the quasi-3D convolution layer to perform a 3D convolution operation on multiple frames of continuous images in the input video to generate a feature map corresponding to each frame of the image;

[0007] Step 2: Perform weighted point-by-point convolution on the feature map output in step 1 along the time axis to generate an energy trend map;

[0008] Step 3: Perform N series of two-step convolution operations on all feature graphs in the energy trend graph output in step 2 to extract features, and obtain the feature vector of the behavior to be identified;

[0009] The two-step convolution is used to first increase the number of layers of feature maps and then reduce the dimension of each layer of feature maps;

[0010] Step 4: Based on the feature vector obtained in step 3, use the maximum pooling layer and the fully connected layer to integrate information, and use the SoftMax layer for classification to obtain the behavior recognition result.

[0011] Furthermore, step 1 is implemented through the following sub-steps:

[0012] Step 1.1: Decode the input video to obtain a series of ordered color images;

[0013] Step 1.2: Take the M consecutive color images decoded in step 1.1 as the input of the quasi-3D convolutional layer, and the size of each frame is [W, H, 3];

[0014] Step 1.3: Use a convolution kernel of size 3x3x3 to perform a 3D convolution operation on the input M frame image, and output a feature map of size [W,H,M];

[0015] Step 1.4: Repeat steps 1.2 and 1.3 until all images obtained in step 1.1 are processed and a set of feature maps of size [W, H, M] are obtained.

[0016] Furthermore, step 2 is implemented through the following sub-steps:

[0017] Step 2.1: For each feature map outputted in step 1, construct 32 random vectors of fade-in factors of the same size, with a minimum value of 0.0, a maximum value of 1.0, and a vector dimension of M;

[0018] Step 2.2: Generate 32 convolution kernels of size [1x1xM] based on the 32 random vectors obtained in step 2.1;

[0019] Step 2.3: Perform point-by-point convolution on the current feature map based on the convolution kernel obtained in step 2.2, and output a feature map of size [W, H, 32];

[0020] Step 2.4: Repeat steps 2.1 to 2.3 until all feature maps output by step 1 are processed to obtain a set of feature maps of size [W, H, 32] to form an energy trend map.

[0021] Furthermore, the first step of the two-step convolution operation in step 3 is used to obtain a feature map with more layers, and the second step is used to reduce the dimension of the input feature map;

[0022] The first step of the two-step convolution operation specifically includes:

[0023] (1) Each input feature map is processed using a depthwise convolution with a step size of 1. This step does not change the size of the input feature map.

[0024] (2) performing point-by-point convolution on the feature map processed in step (1) using a convolution kernel having a higher dimension than the feature map processed in step (1);

[0025] The second step of the two-step convolution operation specifically includes:

[0026] (3) Process the output of step (2) using a depthwise convolution with a step size greater than 1 to reduce the width and height of the input feature map;

[0027] (4) Processing the feature map processed in step (3) using a convolution kernel having the same depth as the feature map processed in step (3).

[0028] The beneficial effects of the present invention are as follows:

[0029] The video behavior recognition method based on improved MobileNet proposed in the present invention realizes fine-grained behavior recognition, detection and evaluation by introducing multiple continuous behavior frames to analyze and mine the internal differences of similar behaviors, and provides different weights for each related frame by adding a fade-in factor on the timeline, thereby better utilizing the progressive relationship between behavior frames at different times. It can be used to identify and evaluate complex behavior recognition with strong temporal relationship, and has high recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the implementation process of the method of the present invention;

[0031] Figure 2 It is a schematic diagram of 3D convolution;

[0032] Figure 3 It is a schematic diagram of the synthesis of energy trend graph;

[0033] Figure 4 It is a two-step convolutional module architecture. DETAILED DESCRIPTION

[0034] The present invention will be described in detail below based on the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more clear. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0035] like Figure 1 As shown, the improved MobileNet model for video behavior recognition in the present invention includes a quasi-3D convolution layer, a weighted point-by-point convolution layer, a two-step convolution module, a maximum pooling layer, a fully connected layer and a SoftMax layer.

[0036] The video behavior recognition method based on improved MobileNet of the present invention specifically includes the following steps:

[0037] 1. Use the quasi-3D convolution layer to perform 3D convolution operation on multiple consecutive frames of images in the input video to generate a feature map corresponding to each frame of the image, including the following sub-steps:

[0038] 1.1 Decode the input video to obtain a series of ordered color images;

[0039] 1.2 Take the M consecutive color images decoded in step 1.1 as the input of the quasi-3D convolutional layer. The size of each image frame is [W, H, 3], where W and H are the bandwidth and height of the image to be processed, respectively, and 3 represents the three color channels of red, green and blue;

[0040] 1.3 Use a convolution kernel of size 3x3x3 to perform a 3D convolution operation on the input M frame images, and output a feature map of size [W,H,M]. Figure 2 Schematic diagram of the 5-layer feature map obtained from 5 frames of continuous images when M is 5.

[0041] 1.4 Repeat steps 1.2 and 1.3 until all images obtained in step 1.1 are processed and a set of feature maps of size [W, H, M] are obtained.

[0042] 2. Perform weighted point-by-point convolution on the feature map output in step 1.4 along the time axis to generate an energy trend map, including the following sub-steps:

[0043] 2.1 For each feature map output in step 1, construct 32 random vectors of fade-in factors of the same size, with a minimum value of 0.0, a maximum value of 1.0, and a vector dimension of M. The 32 here is to extract feature information of more scales and determine the number of layers of the generated feature map. Experimental results show that compared with the value of 16, 64 or other values, in most cases, the value of 32 works better;

[0044] 2.2 Generate 32 convolution kernels of size [1x1xM] based on the 32 random vectors obtained in step 2.1;

[0045] 2.3 Based on the convolution kernel obtained in step 2.2, perform point-by-point convolution on the current feature map, and output a feature map of size [W, H, 32]. The point-by-point convolution here is to perform a convolution operation with a convolution kernel size of 1x1 on the feature map to be processed to achieve feature extraction at a single pixel point;

[0046] 2.4. Repeat steps 2.1 to 2.3 until all feature maps output from step 1 are processed, and a set of feature maps with a size of [W, H, 32] are obtained to form an energy trend map, such as Figure 3 shown.

[0047] 3. Perform N series of two-step convolution operations on all feature maps in the energy trend map output in step 2 to extract features and obtain the feature vector of the behavior to be identified; the two-step convolution is used to first increase the number of layers of feature maps and then reduce the dimension of each layer of feature maps. Figure 4 As shown in Figure 2, the specific implementation process of the two-step convolution operation is given.

[0048] The first step in the two-step convolution operation is used to increase the number of layers of the feature map, including:

[0049] (1) Each input feature map is processed using a depthwise convolution with a step size of 1. This step does not change the size of the input feature map.

[0050] (2) Performing point-by-point convolution on the feature map processed in step (1) using a convolution kernel having a higher dimension than the feature map processed in step (1).

[0051] The second step of the two-step convolution operation specifically includes:

[0052] (3) Process the output of step (2) using a depthwise convolution with a step size greater than 1 to reduce the width and height of the input feature map;

[0053] (4) Processing the feature map processed in step (3) using a convolution kernel having the same depth as the feature map processed in step (3).

[0054] 4. Based on the feature vector obtained in step 3, use the maximum pooling layer and the fully connected layer to integrate information, and use the SoftMax layer for classification to obtain the behavior recognition result.

[0055] Here, the role of the maximum pooling layer and the fully connected layer is to fuse the features extracted by the previous layers, while the SoftMax layer maps multiple inputs to the (0,1) interval, which can be understood as probabilities, thereby achieving the output of multi-classification results.

[0056] The following is a comparison of the effects of the method of the present invention and the existing method on two data sets.

[0057] Example 1: Behavior recognition test on the UCF101 database

[0058] UCF101 is an action recognition dataset of realistic action videos, collected from YouTube, providing 13,320 videos from 101 action categories. The videos in the 101 action categories are divided into 25 groups, each of which can contain 4-7 videos of an action. UCF101 provides the greatest diversity in action, and has great variations in camera motion, object appearance and posture, object scale, viewpoint, cluttered background, lighting conditions, etc. The behavior recognition results of this embodiment are compared with VGG16, VGG19, ResNet101, ResNet152, ShuffleNet, SqueezeNet, and MobileNet methods. The comparative experimental results of the method proposed in the present invention and seven existing methods are shown in Table 1. As can be seen from Table 1, compared with the existing behavior recognition methods, the method of the present invention has obvious advantages in accuracy (precision) and recall (recall).

[0059] Table 1 Comparative test results on the UCF101 dataset

[0060] VGG16 VGG19 ResNet101 ResNet152 ShuffleNet SqueezeNet MobileNet Method of the present invention Accuracy 0.65 0.68 0.80 0.83 0.69 0.75 0.85 0.91 Recall 0.80 0.82 0.83 0.85 0.84 0.79 0.82 0.89

[0061] Example 2: Behavior recognition test on the HMDB51 dataset

[0062] HMDB51 contains 51 types of actions, with a total of 6849 videos. Each action contains at least 51 videos, and the sample resolution is 320*240. Each annotation is verified by at least two people to ensure consistency. The recognition results of this embodiment were compared with the VGG16, VGG19, ResNet101, ResNet152, ShuffleNet, SqueezeNet, and MobileNet methods. The comparative experimental results of the method proposed in the present invention and the seven existing methods are shown in Table 2. It can be seen from Table 2 that compared with the existing behavior recognition scheme, the method of the present invention has significant advantages in terms of overall prediction accuracy.

[0063] Table 2 Comparative test results on the HMDB51 dataset

[0064] VGG16 VGG19 ResNet101 ResNet152 ShuffleNet SqueezeNet MobileNet Method of the present invention Accuracy 0.68 0.71 0.85 0.88 0.73 0.76 0.87 0.93 Recall 0.81 0.83 0.85 0.88 0.86 0.81 0.84 0.91

[0065] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A video behavior recognition method based on improved MobileNet, characterized in that: The improved MobileNet includes a quasi-3D convolution layer, a weighted point-by-point convolution layer, a two-step convolution module, a maximum pooling layer, a fully connected layer and a SoftMax layer; the method specifically includes the following steps: Step 1: Use the quasi-3D convolution layer to perform a 3D convolution operation on multiple frames of continuous images in the input video to generate a feature map corresponding to each frame of the image; Step 2: Perform weighted point-by-point convolution on the feature map output in step 1 along the time axis to generate an energy trend map; step 2 is implemented through the following sub-steps: Step 2.1: For each feature map output in step 1, construct 32 random vectors of fade factors of the same size, with a minimum value of 0.0 and a maximum value of 1.

0. The vector dimension is M ; Step 2.2: Generate 32 random vectors of size [1x1x M ]’s convolution kernel; Step 2.3: Perform point-by-point convolution on the current feature map based on the convolution kernel obtained in step 2.2, and the output size is [ W,H, 32 ] feature map; Step 2.4: Repeat steps 2.1 to 2.3 until all feature maps output from step 1 are processed and a set of features with a size of [ W,H,32 ] feature graph, forming an energy trend graph; Step 3: All feature graphs in the energy trend graph output in step 2 are processed sequentially N A two-step convolution operation is performed in series to extract features and obtain the feature vector of the behavior to be identified; The two-step convolution is used to first increase the number of layers of feature maps and then reduce the dimension of each layer of feature maps; Step 4: Based on the feature vector obtained in step 3, use the maximum pooling layer and the fully connected layer to integrate information, and use the SoftMax layer for classification to obtain the behavior recognition result.

2. The video behavior recognition method based on improved MobileNet according to claim 1, characterized in that: Step 1 is implemented through the following sub-steps: Step 1.1: Decode the input video to obtain a series of ordered color images; Step 1.2: Take the continuous M The frame color image is used as the input of the 3D convolutional layer, and the size of each frame image is [ W,H ,3]; Step 1.3: Use a convolution of size 3x3x3 to filter the input M The frame image is subjected to 3D convolution operation, and the output size is [ W,H,M ] feature map; Step 1.4: Repeat steps 1.2 and 1.3 until all images obtained in step 1.1 are processed and a set of images with a size of [ W,H,M ]’s feature map.

3. The video behavior recognition method based on improved MobileNet according to claim 1, characterized in that: The first step of the two-step convolution operation in step 3 is used to obtain a feature map with more layers, and the second step is used to reduce the dimension of the input feature map; The first step of the two-step convolution operation specifically includes: (1) Each input feature map is processed using a depthwise convolution with a step size of 1. This step does not change the size of the input feature map. (2) performing point-by-point convolution on the feature map processed in step (1) using a convolution kernel having a higher dimension than the feature map processed in step (1); The second step of the two-step convolution operation specifically includes: (3) Use a depthwise convolution with a step size greater than 1 to process the output of step (2) to reduce the width and height of the input feature map; (4) Process the feature map processed in step (3) using a convolution kernel having the same depth as the feature map processed in step (3).

Images