Modeling Method for Video Feature Extraction Model Based on Spatiotemporal Separable Convolution and Asymmetric Convolution

Abstract

The present invention discloses a modeling method for a video feature extraction model based on spatio-temporal separable convolution and asymmetric convolution, belonging to the technical fields of pattern recognition, machine vision, and video processing. The method includes: building a C3D network model; converting the first four convolutional layers of the C3D network into spatio-temporal separable convolutions, and explicitly decomposing the 3D convolution into a continuous operation of a 2D spatial convolution and a 1D temporal convolution; improving the last four convolutional layers of the C3D network to reduce model parameters, and improving the 3D convolution to an asymmetric convolution; constructing a video feature extraction model based on spatio-temporal separable convolution and asymmetric convolution. Experiments show that the advantages of the present invention are that the improved model has better effects in video feature extraction, and can also effectively compress model parameters, having high application value and promotion value.

Description

Technical Field

[0001] The present invention belongs to the technical fields of pattern recognition, machine vision, and video processing, and relates to human behavior recognition and video feature extraction network model technology. In particular, it relates to a method for a video feature extraction network model, which constructs a video feature extraction network model based on spatio-temporal separable convolution and asymmetric convolution, effectively improves the human behavior recognition effect, and proves that its video feature extraction quality is good. Background Art

[0002] Multimedia on the Internet is growing rapidly, resulting in more and more videos being shared every minute on the Internet. To cope with the information explosion, it is very necessary to understand and analyze the information of these videos. The purposes of understanding and analyzing video information are very diverse, such as facilitating users to search for videos, recommending favorite videos to users, ranking the popularity of videos of the same category, etc. For decades, a large number of professionals have been committed to the analysis of video information and have solved various problems, such as video classification, video understanding, and object detection in videos. Considerable progress has been made in these problems by adopting different specific solutions. However, there is an increasing need in our country for a general video feature extraction method to detect video descriptors to help people solve large-scale video information analysis tasks in the same way.

[0003] The task of extracting features that can characterize video features in a video is called video feature extraction. Inspired by the breakthroughs in deep learning in the field of images, feature learning has made rapid progress in the past few years, and various pre-trained convolutional network models can be used to extract image features. These image features are obtained through the activation of the last few fully connected layers of the network model, and the fully connected layers perform well in transfer learning tasks. However, due to the lack of motion modeling, this image-based deep feature is not directly applicable to videos. Nowadays, there are several major challenges in developing effective video feature extraction models.

[0004] In terms of the dataset: First of all, it is very difficult to define the label space for training the video feature extraction model. This is because the dataset for video feature extraction is usually clipped videos. To train the model, it is necessary to annotate the videos in the dataset. However, annotating videos requires a lot of time and effort because all video frames need to be watched, and at the same time, the action range for annotating videos is also very vague because it is difficult to determine the exact start and end moments of most actions. Then, some popular benchmark datasets only release video links for users to download instead of the actual videos. This situation will result in different evaluations of different data even though the same dataset is being processed. Therefore, it is very difficult to make a fair comparison of various models and draw a persuasive conclusion even when using the same dataset.

[0005] In terms of modeling: First, the data in the dataset are all clipped videos capturing human actions, and extracting the action features of videos has significant intra-class and inter-class differences. People can perform the same action at different speeds in different scenarios. In addition, some action categories have similar motion patterns, making it very difficult to distinguish them successfully. Second, extracting the action features of videos requires simultaneously understanding the short-term action-specific motion information and long-term temporal information, so a single convolutional neural network cannot be used, and a complex model is needed to handle different perspectives. Finally, the computational costs of model training and testing are both very high, which greatly hinders the development and deployment of video feature extraction models.

[0006] However, with the development of the times, the popularity of deep learning is getting higher and higher. In the aspect of video feature extraction, deep learning-based models can extract video action features. Many people tried to apply convolutional neural networks to video feature extraction in the early days, and the deep three-dimensional convolutional network (C3D) is one of them. (Tran D, Bourdev L, Fergus R, et al. Learning Spatiotemporal Features with 3D Convolutional Networks[J]. IEEE, 2015.) The C3D network is a good feature learning method that can simultaneously simulate the appearance and motion of objects in a video. Since the C3D network can directly extract spatio-temporal features and its structure is simply stacked by 3D convolutional layers, 3D pooling, and fully connected layers, the structure is relatively simple and is widely used in human behavior recognition research of videos and video feature extraction tasks. The original C3D model has a total of 8 convolutional layers with a convolutional kernel of 3×3×3. Each convolutional layer is followed by a 3D pooling layer, and finally there are two fully connected layers and a softmax classifier. Although the C3D model has achieved good results in human behavior recognition tasks and feature extraction tasks of videos, problems still occur in actual applications. Due to the relatively shallow network structure of the original C3D model, the accuracy needs to be improved, the feature extraction is not sufficient enough, and the number of parameters is relatively large, which is not conducive to model migration in actual applications. Summary of the Invention

[0007] The object of the present invention is to realize a modeling method for a video feature extraction model based on spatio-temporal separable convolution and asymmetric convolution.

[0008] The technical solution provided by the present invention is as follows:

[0009] As shown in the attached Figure 1 figure, the present invention provides a modeling method for a video feature extraction model based on spatio-temporal separable convolution and asymmetric convolution, and its steps include:

[0010] 1) Build the original C3D network model, which includes eight convolutional layers Conv1a, Conv2a, Conv3a, Conv3b, Conv4a, Conv4b, Conv5a, and Conv5b, five pooling layers, and two fully connected layers;

[0011] 2) Convert the first four 3D convolutional layers of the C3D network into spatio-temporally separable convolutions. Decompose each 3D convolutional layer into a 2D spatial convolution and a 1D temporal convolution. After conversion, Conv1a, Conv2a, Conv3a, and Conv3b become a total of eight convolutional layers Conv1a, Conv1b, Conv2a, Conv2b, Conv3a, Conv3b, Conv3c, and Conv3d. Among them, the 2D spatial convolutional layers are Conv1a, Conv2a, Conv3a, and Conv3c, and the 1D temporal convolutional layers are Conv1b, Conv2b, Conv3b, and Conv3d. After conversion, the convolutional kernel of the 2D spatial convolutional layer changes from the original 3×3×3 to 1×3×3, and the convolutional kernel of the 1D temporal convolutional layer changes from the original 3×3×3 to 3×1×1, completing the video feature extraction in the first stage;

[0012] 3) Modify the last four convolutional layers Conv4a, Conv4b, Conv5a, and Conv5b of the C3D network into asymmetric convolutions. Among them, the convolutional kernels of Conv4a and Conv4b are respectively converted from the original 3×3×3 to 3×1×5 and 3×5×1, and the convolutional kernels of Conv5a and Conv5b are respectively converted from the original 3×3×3 to 3×1×7 and 3×7×1, forming a new video feature extraction model;

[0013] 4) Introduce the cross-entropy loss function to measure the difference between classification probability distributions. The calculation process in the model is as follows:

[0014] First, the input is of size (minibatch, C), where minibatch is the batch size of classes and C is the total number of classes. After inputting the size, the loss function is calculated according to formula (1):

[0015]

[0016] In formula (1), x is an array of loss functions corresponding to each class, loss is the loss value corresponding to the class, j is the number of classes, class is the label value. The label value class in formula (1) does not participate in the direct calculation but serves as an index, and the index object is the actual class.

[0017] Further, after obtaining the loss function, the cross-entropy value is calculated through formula (2), where p is the expected output of the probability distribution and q is the actual output of the probability distribution:

[0018]

[0019] The cross-entropy values of each category are calculated therefrom. The smaller the cross-entropy value, the closer the two probability distributions are, and then the classification task of the input video features is realized.

[0020] Experiments of the present invention show that its advantage is that the improved model has better effects in video feature extraction, and the model parameters can also be effectively compressed. Description of the Drawings

[0021] Figure 1 : Flowchart of the video feature extraction model based on spatio-temporal separable convolution and asymmetric convolution of the present invention;

[0022] Figure 2 : C3D network structure;

[0023] Figure 3 : Model based on spatio-temporal separable convolution and asymmetric convolution;

[0024] Figure 4 : Schematic diagram of decomposing 3D convolution into (2+1)D convolution;

[0025] Figure 5 : Schematic diagram of the combination and decomposition of convolutions; wherein (a) is decomposed into smaller convolutions, and a 5×5 convolution kernel can be decomposed into two 3×3 convolution kernels, and (b) is decomposed into asymmetric convolutions, and a 3×3 convolution kernel can be decomposed into a 1×3 convolution kernel and a 3×1 convolution kernel, and a symmetric convolution can be split into two asymmetric convolutions;

[0026] Figure 6 : Loss and accuracy change curves on the UCF101 dataset using the present invention; wherein, (a) is a comparison of the loss change curves of each round of training of the present invention and the original C3D model on the UCF101 dataset, and (b) is a comparison of the accuracy change curves of each round of training of the present invention and the original C3D model on the UCF101 dataset;

[0027] Figure 7 : Loss and accuracy change curves on the HMDB51 dataset using the present invention; wherein, (a) is a comparison of the loss change curves of each round of training of the present invention and the original C3D model on the HMDB51 dataset, and (b) is a comparison of the accuracy change curves of each round of training of the present invention and the original C3D model on the HMDB51 dataset. Detailed Embodiments

[0028] Next, in combination with the accompanying drawings, experiments are conducted using the UCF101 dataset and the HMDB51 dataset for common human behavior recognition. The Adam optimization parameters are adopted, and a BN layer is added after each convolutional layer to accelerate the training of the network while preventing gradient explosion and disappearance, and the present invention will be further described.

[0029] Specifically, the method of the present invention includes the following steps:

[0030] A. Build the original C3D network, as Figure 2 shown:

[0031] The specific structure of the original C3D network is as follows;

[0032] The first convolutional layer Conv1a, with a three-dimensional convolutional kernel size of 3×3×3, a stride of 1, 64 in number, a padding of 1×1×1, a channel input of 3, and a channel output of 64;

[0033] The first pooling layer 3DMaxpool, with a three-dimensional pooling kernel size of 1×2×2 and a stride of 1×2×2;

[0034] The second convolutional layer Conv2a, with a three-dimensional convolutional kernel size of 3×3×3, a stride of 1, 128 in number, a padding of 1×1×1, a channel input of 64, and a channel output of 128;

[0035] The second pooling layer 3DMaxpool, with a three-dimensional pooling kernel size of 2×2×2 and a stride of 2×2×2;

[0036] The third convolutional layer Conv3a, with a three-dimensional convolutional kernel size of 3×3×3, a stride of 1, 256 in number, a padding of 1×1×1, a channel input of 128, and a channel output of 256;

[0037] The fourth convolutional layer Conv3b, with a three-dimensional convolutional kernel size of 3×3×3, a stride of 1, 256 in number, a padding of 1×1×1, a channel input of 256, and a channel output of 256;

[0038] The third pooling layer 3DMaxpool, with a three-dimensional pooling kernel size of 2×2×2 and a stride of 2×2×2;

[0039] The fifth convolutional layer Conv4a, with a three-dimensional convolutional kernel size of 3×3×3, a stride of 1, 512 in number, a padding of 1×1×1, a channel input of 256, and a channel output of 512;

[0040] The sixth convolutional layer Conv4b has a 3D convolutional kernel size of 3×3×3, a stride of 1, 512 kernels, a padding of 1×1×1, a channel input of 512, and a channel output of 512;

[0041] The fourth pooling layer 3DMaxpool has a 3D pooling kernel size of 2×2×2 and a stride of 2×2×2;

[0042] The seventh convolutional layer Conv5a has a 3D convolutional kernel size of 3×3×3, a stride of 1, 512 kernels, a padding of 1×1×1, a channel input of 512, and a channel output of 512;

[0043] The eighth convolutional layer Conv5b has a 3D convolutional kernel size of 3×3×3, a stride of 1, 512 kernels, a padding of 1×1×1, a channel input of 512, and a channel output of 512;

[0044] The fifth pooling layer 3DMaxpool has a 3D pooling kernel size of 2×2×2 and a stride of 2×2×2;

[0045] The first fully connected layer FC6-4608 has 4096 neurons;

[0046] The second fully connected layer FC7-4608 has 4096 neurons;

[0047] B. Based on the C3D network, the first four convolutional layers of the C3D network are improved. The first four 3D convolutions are replaced with spatio-temporal separable convolution methods, and the 3D convolution is decomposed into separate spatial and temporal components to improve the model accuracy; as Figure 3 shown, the specific implementation is as follows:

[0048] B1. Each 3D convolutional layer can be split into a 2D spatial convolution and a 1D temporal convolution. Select the appropriate number of 2D and 1D convolutions so that the number of parameters in the (2+1)D block matches the number of parameters in the complete 3D convolutional block. The decomposition of the 3D convolution into the (2+1)D convolution process is as Figure 4 shown;

[0049] B2. First, transform the first convolutional layer Conv1a into a spatio-temporal separable convolution. Among them, the spatio-temporal separable convolution can be written as a (2+1)D convolution, which clearly decomposes the 3D convolution into two separate consecutive operations, a 2D spatial convolution and a 1D temporal convolution;

[0050] B2.1 The first convolutional layer Conv1a has 64 3×3×3 convolutional kernels. According to formula (1), it can be known that replacing N i convolutional kernels of size N i-1The 3D convolution kernel of ×t×d×d is a (2+1)D convolution module, and this (2+1)D convolution module is respectively composed of M i pieces of 2D convolution kernels with a size of N i-1 ×1×d×d and N i pieces of temporal convolution kernels with a size of M i ×t×1×1. In order to make the number of parameters in the (2+1)D block approximately equal to the number of parameters of the 3D convolution, the convolution kernel of the first convolutional layer Conv1a is transformed into a spatio-temporal separable convolution, that is, these 64 3×3×3 convolution kernels are transformed into M i pieces of 2D spatial dimension convolution kernels of 1×3×3 and N i pieces of temporal dimension convolution kernels with a size of 3×1×1. Among them, N i is 64, N i-1 is 3, t is 3, d is 3. According to formula (3), M i can be obtained as 23. Therefore, according to the principle of spatio-temporal separable convolution, the 64 3×3×3 convolution kernels of the first convolutional layer Conv1a can be replaced by 23 2D spatial dimension convolution kernels of 1×3×3 and 64 temporal dimension convolution kernels of 3×1×1, so as to ensure that the number of parameters in the replaced (2+1)D block is approximately equal to the 3D convolution parameters of the first convolutional layer Conv1a;

[0051]

[0052] B2.2 After the replacement of the first convolutional layer Conv1a, it becomes two convolutional layers Conv1a and Conv1b, which are respectively:

[0053] The first convolutional layer Conv1a, with a three-dimensional convolution kernel size of 1×3×3, a stride of 1, 23 in number, padding of 1×1×1, a channel input of 3, and a channel output of 23;

[0054] The second convolutional layer Conv1b, with a three-dimensional convolution kernel size of 3×1×1, a stride of 1, 64 in number, padding of 1×1×1, a channel input of 23, and a channel output of 64;

[0055] B3. Then, the second convolutional layer Conv2a is transformed into a spatio-temporal separable convolution;

[0056] The second convolutional layer Conv2a in B3.1 has 128 convolutional kernels of size 3×3×3. According to formula (1), the 128 convolutional kernels of size 3×3×3 in the second convolutional layer Conv2a can be replaced by 230 convolutional kernels of size 1×3×3 in the 2D spatial dimension and 128 convolutional kernels of size 3×1×1 in the temporal dimension, so as to ensure that the number of parameters in the replaced (2+1)D block is approximately equal to the number of 3D convolutional parameters in the second convolutional layer Conv2a;

[0057] After replacement, the second convolutional layer Conv2a becomes two convolutional layers Conv2a and Conv2b, which are respectively:

[0058] The third convolutional layer Conv2a, with a three-dimensional convolutional kernel size of 1×3×3, a stride of 1, 230 kernels, a padding of 1×1×1, a channel input of 64, and a channel output of 230;

[0059] The fourth convolutional layer Conv2b, with a three-dimensional convolutional kernel size of 3×1×1, a stride of 1, 128 kernels, a padding of 1×1×1, a channel input of 230, and a channel output of 128;

[0060] B4. Then, the third convolutional layer Conv3a is transformed into a spatio-temporally separable convolution;

[0061] The third convolutional layer Conv3a in B4.1 has 256 convolutional kernels of size 3×3×3. According to formula (1), the 256 convolutional kernels of size 3×3×3 in the third convolutional layer Conv3a can be replaced by 460 convolutional kernels of size 1×3×3 in the 2D spatial dimension and 256 convolutional kernels of size 3×1×1 in the temporal dimension, so as to ensure that the number of parameters in the replaced (2+1)D block is approximately equal to the number of 3D convolutional parameters in the third convolutional layer Conv3a;

[0062] After replacement, the third convolutional layer Conv3a becomes two convolutional layers Conv3a and Conv3b, which are respectively:

[0063] The fifth convolutional layer Conv3a, with a three-dimensional convolutional kernel size of 1×3×3, a stride of 1, 460 kernels, a padding of 1×1×1, a channel input of 128, and a channel output of 460;

[0064] The sixth convolutional layer Conv3b, with a three-dimensional convolutional kernel size of 3×1×1, a stride of 1, 256 kernels, a padding of 1×1×1, a channel input of 460, and a channel output of 256;

[0065] B5. Then, the fourth convolutional layer Conv3b is transformed into a spatio-temporally separable convolution;

[0066] The fourth convolutional layer Conv3b in B5.1 has 256 convolutional kernels of 3×3×3. According to formula (1), the 256 convolutional kernels of 3×3×3 in the fourth convolutional layer Conv3b can be replaced with 576 convolutional kernels of 1×3×3 in the 2D spatial dimension and 256 convolutional kernels of 3×1×1 in the temporal dimension, so as to ensure that the number of parameters in the replaced (2+1)D block is approximately equal to the number of 3D convolutional parameters in the fourth convolutional layer Conv3b;

[0067] After replacement, the fourth convolutional layer Conv3b in B5.2 becomes two convolutional layers Conv3c and Conv3d, which are respectively:

[0068] The seventh convolutional layer Conv3c, with a three-dimensional convolutional kernel size of 1×3×3, a stride of 1, 576 in number, a padding of 1×1×1, a channel input of 256, and a channel output of 576;

[0069] The eighth convolutional layer Conv3d, with a three-dimensional convolutional kernel size of 3×1×1, a stride of 1, 256 in number, a padding of 1×1×1, a channel input of 576, and a channel output of 256;

[0070] After completing the spatio-temporal separable convolution transformation of the first four convolutional layers Conv1a, Conv2a, Conv3a, and Conv3b, the video feature extraction in the first stage is completed, and the improvement of the video feature extraction model based on the principle of spatio-temporal separable convolution is realized. The model includes twelve convolutional layers, five pooling layers, and two fully connected layers, which are respectively:

[0071] The first convolutional layer Conv1a, with a three-dimensional convolutional kernel size of 1×3×3, a stride of 1, 23 in number, a padding of 1×1×1, a channel input of 3, and a channel output of 23;

[0072] The second convolutional layer Conv1b, with a three-dimensional convolutional kernel size of 3×1×1, a stride of 1, 64 in number, a padding of 1×1×1, a channel input of 23, and a channel output of 64;

[0073] The first pooling layer 3DMaxpool, with a three-dimensional pooling kernel size of 1×2×2 and a stride of 1×2×2;

[0074] The third convolutional layer Conv2a, with a three-dimensional convolutional kernel size of 1×3×3, a stride of 1, 230 in number, a padding of 1×1×1, a channel input of 64, and a channel output of 230;

[0075] The fourth convolutional layer Conv2b has a 3D convolutional kernel size of 3×1×1, a stride of 1, 128 kernels, a padding of 1×1×1, a channel input of 230, and a channel output of 128;

[0076] The second pooling layer 3DMaxpool has a 3D pooling kernel size of 2×2×2 and a stride of 2×2×2;

[0077] The fifth convolutional layer Conv3a has a 3D convolutional kernel size of 1×3×3, a stride of 1, 460 kernels, a padding of 1×1×1, a channel input of 128, and a channel output of 460;

[0078] The sixth convolutional layer Conv3b has a 3D convolutional kernel size of 3×1×1, a stride of 1, 256 kernels, a padding of 1×1×1, a channel input of 460, and a channel output of 256;

[0079] The seventh convolutional layer Conv3c has a 3D convolutional kernel size of 1×3×3, a stride of 1, 576 kernels, a padding of 1×1×1, a channel input of 256, and a channel output of 576;

[0080] The eighth convolutional layer Conv3d has a 3D convolutional kernel size of 3×1×1, a stride of 1, 256 kernels, a padding of 1×1×1, a channel input of 576, and a channel output of 256;

[0081] The third pooling layer 3DMaxpool has a 3D pooling kernel size of 2×2×2 and a stride of 2×2×2;

[0082] The ninth convolutional layer Conv4a has a 3D convolutional kernel size of 3×3×3, a stride of 1, 512 kernels, a padding of 1×1×1, a channel input of 256, and a channel output of 512;

[0083] The tenth convolutional layer Conv4b has a 3D convolutional kernel size of 3×3×3, a stride of 1, 512 kernels, a padding of 1×1×1, a channel input of 512, and a channel output of 512;

[0084] The fourth pooling layer 3DMaxpool has a 3D pooling kernel size of 2×2×2 and a stride of 2×2×2;

[0085] The eleventh convolutional layer Conv5a has a 3D convolutional kernel size of 3×3×3, a stride of 1, 512 kernels, a padding of 1×1×1, a channel input of 512, and a channel output of 512;

[0086] The twelfth convolutional layer Conv5b has a 3D convolutional kernel size of 3×3×3, a stride of 1, 512 kernels, padding of 1×1×1, 512 input channels, and 512 output channels;

[0087] The fifth pooling layer 3DMaxpool has a 3D pooling kernel size of 2×2×2 and a stride of 2×2×2;

[0088] The first fully connected layer FC6-4608 has 4096 neurons;

[0089] The second fully connected layer FC7-4608 has 4096 neurons;

[0090] C. Based on the video feature extraction model using spatio-temporal separable convolution, the last four convolutional layers of the model are improved to reduce the model parameters, and the 3D convolution is improved to an asymmetric convolution. The combination and decomposition of the convolution are as Figure 5 shown,

[0091] C1. Through the combination and decomposition of the convolution, the 512 3×3×3 convolutional kernels of the ninth convolutional layer Conv4a and the 512 3×3×3 convolutional kernels of the tenth convolutional layer Conv4b in the video feature extraction model using spatio-temporal separable convolution are combined and decomposed into 512 3×1×5 convolutional kernels and 512 3×5×1 convolutional kernels, thus forming new ninth and tenth convolutional layers Conv4a and Conv4b, respectively:

[0092] The ninth convolutional layer Conv4a has a 3D convolutional kernel size of 3×1×5, a stride of 1, 512 kernels, padding of 1×1×1, 256 input channels, and 512 output channels;

[0093] The tenth convolutional layer Conv4b has a 3D convolutional kernel size of 3×5×1, a stride of 1, 512 kernels, padding of 1×1×1, 512 input channels, and 512 output channels;

[0094] C2. Through the combination and decomposition of the convolution, the 512 3×3×3 convolutional kernels of the eleventh convolutional layer Conv5a and the 512 3×3×3 convolutional kernels of the twelfth convolutional layer Conv5b in the video feature extraction model using spatio-temporal separable convolution are combined and decomposed into 512 3×1×7 convolutional kernels and 512 3×7×1 convolutional kernels, thus forming new eleventh and twelfth convolutional layers Conv5a and Conv5b, respectively:

[0095] The eleventh convolutional layer Conv5a has a three-dimensional convolutional kernel size of 3×1×7, a stride of 1, 512 kernels, padding of 1×1×1, a channel input of 512, and a channel output of 512;

[0096] The twelfth convolutional layer Conv5b has a three-dimensional convolutional kernel size of 3×7×1, a stride of 1, 512 kernels, padding of 1×1×1, a channel input of 512, and a channel output of 512;

[0097] C3. After completing the merging and decomposition of the last four convolutional layers Conv4a, Conv4b, Conv5a, and Conv5b, the video feature extraction in the second stage is completed, realizing the improvement of the video feature extraction model based on spatio-temporal separable convolution and asymmetric convolution. The model includes twelve convolutional layers, five pooling layers, and two fully connected layers, which are respectively:

[0098] The first convolutional layer Conv1a has a three-dimensional convolutional kernel size of 1×3×3, a stride of 1, 23 kernels, padding of 1×1×1, a channel input of 3, and a channel output of 23;

[0099] The second convolutional layer Conv1b has a three-dimensional convolutional kernel size of 3×1×1, a stride of 1, 64 kernels, padding of 1×1×1, a channel input of 23, and a channel output of 64;

[0100] The first pooling layer 3DMaxpool has a three-dimensional pooling kernel size of 1×2×2 and a stride of 1×2×2;

[0101] The third convolutional layer Conv2a has a three-dimensional convolutional kernel size of 1×3×3, a stride of 1, 230 kernels, padding of 1×1×1, a channel input of 64, and a channel output of 230;

[0102] The fourth convolutional layer Conv2b has a three-dimensional convolutional kernel size of 3×1×1, a stride of 1, 128 kernels, padding of 1×1×1, a channel input of 230, and a channel output of 128;

[0103] The second pooling layer 3DMaxpool has a three-dimensional pooling kernel size of 2×2×2 and a stride of 2×2×2;

[0104] The fifth convolutional layer Conv3a has a three-dimensional convolutional kernel size of 1×3×3, a stride of 1, 460 kernels, padding of 1×1×1, a channel input of 128, and a channel output of 460;

[0105] The sixth convolutional layer Conv3b has a three-dimensional convolutional kernel size of 3×1×1, a stride of 1, 256 kernels, a padding of 1×1×1, a channel input of 460, and a channel output of 256;

[0106] The seventh convolutional layer Conv3c has a three-dimensional convolutional kernel size of 1×3×3, a stride of 1, 576 kernels, a padding of 1×1×1, a channel input of 256, and a channel output of 576;

[0107] The eighth convolutional layer Conv3d has a three-dimensional convolutional kernel size of 3×1×1, a stride of 1, 256 kernels, a padding of 1×1×1, a channel input of 576, and a channel output of 256;

[0108] The third pooling layer 3DMaxpool has a three-dimensional pooling kernel size of 2×2×2 and a stride of 2×2×2;

[0109] The ninth convolutional layer Conv4a has a three-dimensional convolutional kernel size of 3×1×5, a stride of 1, 512 kernels, a padding of 1×1×1, a channel input of 256, and a channel output of 512;

[0110] The tenth convolutional layer Conv4b has a three-dimensional convolutional kernel size of 3×5×1, a stride of 1, 512 kernels, a padding of 1×1×1, a channel input of 512, and a channel output of 512;

[0111] The fourth pooling layer 3DMaxpool has a three-dimensional pooling kernel size of 2×2×2 and a stride of 2×2×2;

[0112] The eleventh convolutional layer Conv5a has a three-dimensional convolutional kernel size of 3×1×7, a stride of 1, 512 kernels, a padding of 1×1×1, a channel input of 512, and a channel output of 512;

[0113] The twelfth convolutional layer Conv5b has a three-dimensional convolutional kernel size of 3×7×1, a stride of 1, 512 kernels, a padding of 1×1×1, a channel input of 512, and a channel output of 512;

[0114] The fifth pooling layer 3DMaxpool has a three-dimensional pooling kernel size of 2×2×2 and a stride of 2×2×2;

[0115] The first fully connected layer FC6-4608 has 4096 neurons;

[0116] The second fully connected layer FC7-4608 has 4096 neurons;

[0117] D. The present invention uses C3D as the baseline model, adds a spatio-temporal separable convolution (2+1)D module and the combination and decomposition of convolutional kernels to construct a video feature extraction model based on spatio-temporal separable convolution and asymmetric convolution. The input dataset is used, and the cross-entropy loss function is used. The calculation process in the model is as follows:

[0118] First, the input has a size of (minibatch, C), where minibatch is the batch size of classes and C is the total number of classes. The loss function is calculated according to formula (2);

[0119]

[0120] In formula (2), x is an array of loss functions corresponding to each class, loss is the loss value corresponding to the class, j is the number of classes, class is the label value. The label value class in formula (2) does not participate in the direct calculation but serves as an index, and the index object is the actual class.

[0121] To prove the effectiveness of the feature extraction of the improved video feature extraction network, it is applied to the human action recognition task for fair evaluation;

[0122] D1. The model training uses Adam to optimize the parameters, and a BN layer is added after each convolutional layer to accelerate the training of the network and prevent gradient explosion and disappearance. The parameter settings of the Adam algorithm are as follows:

[0123] D1.1 The learning rate Ir takes the default value of 1×e -3 ;

[0124] D1.2 The coefficients for calculating the running averages of the gradient and the squared gradient are betas. The coefficient for calculating the running average of the gradient is set to 0.9, and the coefficient for calculating the running average of the squared gradient is set to 0.999;

[0125] D1.3 The term eps added to the denominator to increase the numerical calculation stability is set to 1×e -8 ;

[0126] D1.4 The weight decay weight_decay is set to 5×e -4 ;

[0127] D2. Before training, the dataset needs to be processed:

[0128] D2.1 Perform preprocessing operations on the video dataset. On the public dataset, retain 1 frame every 4 frames. This uniform sampling method can not only retain video features but also reduce the overall computational amount;

[0129] D2.2 The entire dataset is divided into a training set, a validation set, and a test set according to a ratio of 6:2:2;

[0130] D2.3 The input image frames are uniformly cropped into pixels of size 112×112. A sliding window is used to select 16 frames of network input video frames, so the network input size is 3×16×112×112;

[0131] The present invention will be further described below in combination with specific experiments:

[0132] To prove the effectiveness of the feature extraction of the improved video feature extraction network, it is applied to the human action recognition task. Experiments are carried out using the UCF101 dataset and the HMDB51 dataset for public human action recognition for fair evaluation. Among them, HMDB51 was launched in 2011. It is mainly collected from movies and a small part comes from public datasets such as the Prelinger Archive, YouTube, and Google Videos. The dataset contains 6,849 clips, divided into 51 action categories, and each action category contains at least 101 clips. UCF101 was launched in 2012 and is an extension of the previous UCF50 dataset. It contains 13,320 videos from YouTube, covering 101 human behaviors;

[0133] Adam optimization parameters are used for model training, and other experimental settings are shown in Table 1, and the experimental environment is shown in Table 2;

[0134] Table 1 Experimental parameter settings

[0135]

[0136] Table 2 Experimental environment

[0137]

[0138] Process the dataset:

[0139] 1. Preprocess the video dataset. On the public dataset, keep 1 frame every 4 frames. This uniform sampling method can not only retain video features but also reduce the overall computational load;

[0140] 2. The entire dataset is divided into a training set, a validation set, and a test set according to a ratio of 6:2:2;

[0141] 3. The input image frames are uniformly cropped into pixels of size 112×112. A sliding window is used to select 16 frames of network input video frames, so the network input size is 3×16×112×112;

[0142] The model of the present invention was trained on the UCF101 dataset and the HMDB51 dataset respectively, and the results are as follows:

[0143] When training on the UCF101 dataset, the total training duration was approximately 10.5h. Table 3 shows the mAP results on the UCF101 dataset, and its loss rate change curve and accuracy change curve are as Figure 6 shown: The accuracy of the original C3D network model on the UCF101 dataset was 78.09%, and the number of parameters was 78.41×10 6 ; The recognition accuracy of our improved network on the UCF101 dataset reached 82.55%, and the number of parameters decreased to 62.74×10 6 ;

[0144] Table 3 Experimental results on UCF101

[0145]

[0146] When training on the HMDB51 dataset, the total training duration was approximately 4.5h. Table 4 shows the mAP results on the HMDB51 dataset, and its loss rate change curve and accuracy change curve are as Figure 7 shown: The accuracy of the original C3D network model on the HMDB51 dataset was 48.28%, and the number of parameters was 78.20×10 6 ; The recognition accuracy of our improved network on the HMDB51 dataset reached 51.11%, and the number of parameters decreased to 62.51×10 6 ;

[0147] Table 4 Experimental results on HMDB51

[0148]

[0149] The performance has been improved on both public datasets, with the accuracy increased by approximately 4%, which proves that the improved model has a better effect in video feature extraction and can also effectively compress the model parameters.

[0150] Finally, it should be noted that the purpose of publishing the embodiments is to help further understand the present invention. However, those skilled in the art can understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection claimed by the present invention shall be defined by the scope of the claims.

Claims

1. A modeling method for a video feature extraction model based on spatio-temporal separable convolution and asymmetric convolution, the steps of which include: 1) Build a C3D network model, which includes eight convolutional layers Conv1a, Conv2a, Conv3a, Conv3b, Conv4a, Conv4b, Conv5a and Conv5b, five pooling layers and two fully connected layers; 2) Convert the first four 3D convolutional layers of the C3D network into spatio-temporal separable convolutions, that is, decompose each 3D convolutional layer into a 2D spatial convolution and a 1D temporal convolution. After conversion, Conv1a, Conv2a, Conv3a and Conv3b become a total of eight convolutional layers Conv1a, Conv1b, Conv2a, Conv2b, Conv3a, Conv3b, Conv3c and Conv3d. Among them, the 2D spatial convolutional layers are Conv1a, Conv2a, Conv3a and Conv3c, and the 1D temporal convolutional layers are Conv1b, Conv2b, Conv3b and Conv3d. After conversion, the convolutional kernel of the 2D spatial convolutional layer changes from the original 3×3×3 to 1×3×3, and the convolutional kernel of the 1D temporal convolutional layer changes from the original 3×3×3 to 3×1×1, completing the video feature extraction in the first stage; 3) Modify the last four convolutional layers Conv4a, Conv4b, Conv5a and Conv5b of the C3D network into asymmetric convolutions. Among them, the convolutional kernels of Conv4a and Conv4b are respectively converted from the original 3×3×3 to 3×1×5 and 3×5×1, and the convolutional kernels of Conv5a and Conv5b are respectively converted from the original 3×3×3 to 3×1×7 and 3×7×1, constituting a new video feature extraction model; where the model is trained using Adam to optimize the parameters, and a BN layer is added after each convolutional layer. The parameter settings of the Adam algorithm are as follows: 3-1) The learning rate Ir takes the default value of 1×e -3 ; 3-2) The coefficient for calculating the running averages of the gradient and the squared gradient is betas. The coefficient for calculating the running average of the gradient is set to 0.9, and the coefficient for calculating the running average of the squared gradient is set to 0.999; 3-3) The term eps added to the denominator to increase the stability of numerical calculations is set to 1×e -8 ; 3 - 4) The weight decay weight_decay is set to 5×e -4 ; 4) Input the video data set, and use the cross-entropy loss function to measure the difference between the classification probability distributions. The calculation process in the model is as follows: First, the input is of size (minibatch, C), where minibatch is the batch size of the class and C is the total number of classes. After inputting the size, the loss function is calculated according to formula (1): In formula (1), x is an array of loss functions corresponding to each class, loss is the loss value corresponding to the class, j is the number of classes, class is the label value. The label value class in formula (1) does not participate in the direct calculation, but serves as an index, and the index object is the actual class.

2. The modeling method according to claim 1, characterized in that, After obtaining the loss function, the value of the cross-entropy is calculated through formula (2), where p is the expected output of the probability distribution and q is the actual output of the probability distribution: The cross-entropy values of each category are calculated accordingly. The smaller the cross-entropy value, the closer the two probability distributions are, thereby realizing the classification task of the input video features.

3. The modeling method according to claim 1, wherein In step 2), the convolutional layer Conv1a has N i 3D convolutional kernels of size N i-1 ×t×d×d, and the (2+1)D convolutional module consists of M i 2D convolutional kernels of size N i-1 ×1×d×d and N i temporal convolutional kernels of size M i ×t×1×1, where 4. The modeling method according to claim 1, wherein In step 4), the video dataset is processed as follows: D2.1 Perform preprocessing operations on the video dataset in the form of retaining 1 frame every 4 frames on the public dataset; D2.2 The entire dataset is divided into a training set, a validation set, and a test set according to the ratio of 6:2:2; D2.3 Uniformly crop the input image frames into pixels of size 112×112. Use a sliding window to select 16 network input video frames, and the network input size is 3×16×112×112.

Images