Behavior recognition method based on p2cs_3dnet

Abstract

The application discloses a behavior recognition method based on P2CS_3Dnet. Including steps: reading data set video action, obtaining video label; through the way of equal difference sampling, video frame extraction is carried out on each video, and size normalization processing is carried out on the extracted video frame; the video frame is input into P2C_1Net network to extract low-level features of the video action, and effective extraction of time information is realized; then, the video frame is input into P2C_2Net network to extract high-level features, effectively extract the features on the space, strengthen the features, reduce the parameter quantity and speed up the running time; the extracted features are input into CSENet attention model, feature compression and feature excitation are completed, and the features are gradually convolved, the local features are paid attention to; finally, the residual module is fused to fuse the low-level feature information and the high-level feature information of the video, and after the fused features are output by the output layer, the action classification result of the video is obtained.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology and relates to a method for recognizing the movement of people or objects, specifically a behavior recognition method based on P2CS_3DNet (P2CNet+CSENet+Residual Module). Background Technology

[0002] With the continuous development of deep learning technology and the increasing application scenarios of action recognition, deep learning-based action recognition technology is gradually becoming a key field in the field of action recognition.

[0003] Human actions such as jumping, striking, and spinning are ubiquitous in daily life, and action recognition has broad application prospects in society. Action recognition technology is an important research area in computer vision, and it has gradually become a research hotspot in the field in recent years. Action recognition research methods can be broadly divided into two types: image-based static action recognition and video-based dynamic action recognition.

[0004] In video action recognition, compared to 2D images, temporal information is added. Therefore, deep learning action recognition methods can learn not only spatial sequence information but also recognize temporal sequence features. For example, Simonyan first proposed the classic two-stream CNN, which uses spatial and temporal flow networks to learn spatial and temporal features respectively, and finally fuses the two. This method compensates for the loss of temporal feature information in traditional machine learning methods. Another example is the C3D model proposed by Tran et al. to extract spatiotemporal features from videos; however, this method suffers from excessive parameters and high computational cost. Summary of the Invention

[0005] To address the problems of complex feature extraction difficulties, large C3D computational parameters, and inability to focus on specific information extraction in the aforementioned action recognition methods, this invention proposes a more comprehensive and effective action recognition method, P2CS_3DNet. This method decouples 3D convolution into temporal and spatial convolutions, enabling action recognition to extract local temporal and spatial information. This allows the extracted action features to more accurately represent action videos, while also incorporating multiple residual modules to reduce the number of parameters. This invention utilizes P2C_1Net to extract low-order feature information from motion videos, adding a CBAM attention mechanism after temporal convolution to better capture information in the temporal dimension, while simultaneously fusing it with the initial features. P2C_2Net then extracts high-order feature information from the video, adding a CBAM attention mechanism after spatial convolution to better capture information in the spatial dimension, and fusing it with the low-order features from P2C_1Net. This information is then input into CSENet for local information extraction and fusion, enhancing the extraction of local spatial and temporal information. Finally, a Residual Module is added to connect the initial input video features and the features output by CSENet, completing the initial feature extraction and fusion, which can reduce overfitting and strengthen the connections between multi-layer feature information.

[0006] The technical solution of the present invention is as follows, including the following steps:

[0007] 1) Divide the video dataset containing actions into training set, test set and validation set, read the actions of each video in the training set, and obtain the behavior label corresponding to the action of each video;

[0008] 2) Extract video frames for each video using arithmetic progression sampling, and then normalize the size of the extracted video frames.

[0009] 3) Input the normalized video frames into the P2C_1Net (P3D Module+CBAM) module to extract low-level features of video actions, effectively extracting and enhancing temporal information;

[0010] 4) Input the extracted low-level features into the P2C_2Net module to extract high-level features of video actions, effectively extracting and enhancing spatial information;

[0011] 5) Input the extracted features into the CSENet (Conv-SENet) attention module to complete feature compression and feature activation, while also performing progressive convolution on the features to achieve attention to local features.

[0012] 6) Finally, the input video features and the feature information output in step 5 are fused through the residual module, which can reduce overfitting and strengthen the connection between multi-layer feature information.

[0013] 7) The fused features are output through the output layer to obtain the final classification result.

[0014] Step 2) specifically refers to:

[0015] 2.1) Read the number of video frames for each video using OpenCV;

[0016] 2.2) Select video frames with specified parameters by arithmetic progression. Specifically, the specified parameter -depth is 10 video frames.

[0017] 2.3) Normalize the size of the selected video frames. Specifically, the size of the video frames is unified to (128, 128, 3).

[0018] Step 3) is as follows:

[0019] The P2CS_3DNet network is mainly an improvement on the C3D network model, addressing the problem that the C3D model cannot extract key feature information. P2CS_1Net is proposed to extract temporal features by decoupling the 3D convolution into a 1D convolution in the temporal dimension and a 2D convolution in the spatial dimension. CBAM channel attention is then added after the 1D temporal convolution to extract temporal features.

[0020] 3.1) Take the specified 10 video frames (128, 128, 3) as input to P2C_1Net, and input the first temporal convolution with 32 kernels (3×1×1) and the first spatial convolution with 1×3×3 to obtain temporal and spatial information respectively.

[0021] 3.2) To obtain important information from the temporal convolution, the CBAM attention mechanism is applied after the 3×1×1 temporal convolution, and the temporal and spatial features are fused by the Add operation. Then, a max pooling layer with a stride of 2×2×2 is added to compress the fused features.

[0022] 3.3) The obtained features are then input into a 3×1×1 temporal convolutional layer with a kernel size of 64 to further enhance the temporal information. The features are then connected to a 1×3×3 spatial convolutional layer with a kernel size of 64, and then a second max pooling operation with a stride of 2×2×2 is performed for secondary feature compression.

[0023] 3.4 Finally, the Residual Module is used to connect the initial input features (128, 128, 3) and the features after the second max pooling operation to fuse them, thus completing the extraction and combination of the initial video features.

[0024] Step 4) is as follows:

[0025] P2C_2Net is proposed to extract spatial features. It decouples the three-dimensional convolution into a one-dimensional convolution in the temporal dimension and a two-dimensional convolution in the spatial dimension, and adds CBAM channel attention after the spatial two-dimensional convolution to extract temporal features.

[0026] 4.1) The video features output by P2C_2Net are input into a 1×3×3 third spatial convolution and a 3×1×1 third temporal convolution with a kernel of 128, respectively, to obtain spatial and temporal information.

[0027] 4.2) After the 1×3×3 third spatial convolutional layer, CBAM channel attention is added to enhance the extraction of spatial information. Then, the Add operation is used to fuse spatial and temporal features, and a max pooling layer with a stride of 2×2×2 is added to compress the fused features.

[0028] 4.3) The obtained features are then input into a 1×3×3 fourth spatial convolution with a kernel of 256 to further enhance the temporal information. The features are then input into a 3×1×1 fourth temporal convolution with a kernel of 256, and then a second max pooling operation with a stride of 2×2×2 is performed for secondary feature compression.

[0029] 4.5) Finally, use the Residual Module to fuse the features output in step 2 and the features output in step 3.

[0030] Step 5) is as follows:

[0031] The feature information from step 4 is input into CSE Net. CSE Net enhances important features to improve the accuracy of the feature information. CSE Net mainly consists of two branches, each of which includes a feature compression module, a local feature extraction module, and a feature activation module.

[0032] The feature compression Squeeze module includes a global average pooling layer and a global max pooling layer. The feature tensor containing high- and low-order features is input into the global average pooling process and outputs a feature compression tensor. The feature compression tensor retains only the channel features in its width and height, and is transformed into a one-dimensional feature vector.

[0033] The local feature extraction Conv module includes a 1×1×1 3D convolutional module with a channel / ratio kernel and a ReLU activation function. The convolutional module performs progressive convolution on the feature tensor, highlighting important information within the features. Simultaneously, it scales a one-dimensional vector to the channel / ratio of the input one-dimensional vector, thus achieving the purpose of filtering out secondary features and selecting primary ones.

[0034] The feature excitation module consists of fully connected layers and activation layers. The output parameters of the fully connected layers have the same size as the feature tensor after feature compression. Finally, the sigmoid function is used to generate the feature weights for each channel.

[0035] 5.1) Using the feature information output from step 4 as the input to the first branch, the attention channel undergoes the GlobalAveragePooling3D operation to reduce the number of parameters and utilize global information; the number of channels is set to the number of channels of the features output from step 4, and the reduction ratio is set to 16; a 1×1×1 Conv3D operation with a kernel of channel / ratio is used, and the activation function is ReLU to extract local information; finally, a fully connected layer with the number of channels is added, and the activation function is sigmoid.

[0036] 5.2) Simultaneously, the feature information output in step 4 is used as the input of the second branch. The attention channel undergoes GlobalMaxPooling3D and Reshape operations. A 1×1×1 Conv3D operation with a convolution kernel of channel / ratio and a ReLU activation function is used to extract local information. A fully connected layer with the number of channels is added and the activation function is sigmoid.

[0037] 5.3) Use the Multiply layer to fuse steps 5.1) and 5.2) to complete feature fusion. Finally, use the residual module to fuse the multiplied and fused features with the output information of step 4.

[0038] Step 6) is as follows:

[0039] The Residual Module fuses the initial low-level feature information (128, 128, 3) with the high-level feature information output in step 5) using the Add operation; this can be achieved through the following formula.

[0040]

[0041] Where H(x) is the output function of the residual module; F(x) is the residual mapping function; G(x) is the identity mapping function; and x is the input feature of the residual module.

[0042] Step 7) is as follows:

[0043] The feature map obtained in step 6) above is transformed into a one-dimensional feature map through the Flatten operation, and then passed through three Dense layers:

[0044] The first Dense layer has 1024 neural unit nodes and uses the ReLU activation function. Then, a Dropout layer is added with a dropout rate of 0.5.

[0045] The second Dense layer is designed with 512 neural unit nodes, using the ReLU activation function, followed by a Dropout layer with a dropout rate of 0.5.

[0046] The third Dense layer contains neurons equal to the number of action categories (action labels). A SoftMax classifier is used for classification, along with an Adam optimizer and a learning rate of 0.0003. The final action classification results are obtained from the P2CS_3DNet network.

[0047] The beneficial effects of this invention are:

[0048] 1) This invention designs the overall structure of the P2CS_3DNet network, which can not only acquire local information, but also realize the fusion of low-level and high-level information.

[0049] 2) This invention proposes the P2C_1Net network to extract low-level temporal feature information, complete the extraction of local temporal information, and fuse initial feature information and low-level feature information;

[0050] 3) This invention proposes the P2C_2Net network to extract high-level spatial feature information, complete the extraction of local spatial information, and fuse low-level feature information and high-level feature information;

[0051] 4) This invention proposes the CSENet network to enhance feature information and achieves the focus and extraction of local information of the enhanced information; Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the technical solution of the method of the present invention.

[0053] Figure 2 and Figure 3 This is a portion of the dataset used in the method of this invention.

[0054] Figure 4 This is a diagram of the P2C_1Net network structure of the present invention.

[0055] Figure 5 This is a diagram of the P2C_2Net network structure of the present invention.

[0056] Figure 6 This is a diagram of the CSENet network structure of the present invention.

[0057] Figure 7This is a graph showing the accuracy of the test set and the accuracy of the validation set in this invention. acc is the accuracy of the test set, and val_acc is the accuracy of the validation set.

[0058] Figure 8 This is a graph showing the test set loss rate and validation set loss rate of the present invention. loss is the test set loss rate, and val_loss is the validation set loss rate. Detailed Implementation

[0059] The invention will be further described below with reference to the accompanying drawings and embodiments.

[0060] The technical solution process of this invention is as follows: Figure 1 As shown.

[0061] This invention uses the UCF101 open-source dataset, which includes 101 categories of videos related to human-object interaction, human movement, human-human interaction, musical instrument performance, and sports, for motion recognition. (Some datasets are shown below.) Figure 2 , 3 As shown.

[0062] like Figure 1 As shown, the process begins by reading videos of various categories to obtain category labels. Then, each video undergoes preprocessing, using arithmetic progression sampling to extract video frames with specified parameters, followed by size normalization. The selected video frame sequence is used as input to the P2CS_3DNet network. First, it is fed into the P2C_1Net (P3D Module + CBAM) network for low-level feature extraction of video actions, enhancing temporal local information. Then, it is fed into the P2C_2Net network for high-level feature extraction, enhancing spatial local information. The extracted features are then fed into the CSENet (Conv-SENet) attention model to focus on local information. Finally, a Residual Module is integrated to fuse low-level and high-level video feature information. The output features pass through two fully connected (FC) layers and are finally classified using a SoftMax classifier to obtain the classification result.

[0063] The specific steps of this invention are as follows:

[0064] Step 1) Read videos of each category, obtain category labels, and then perform preprocessing operations on each video. Obtain the number of frames for each video action using OpenCV; use an arithmetic progression method to obtain video frames with specified parameters; and perform size normalization processing on the extracted video frames.

[0065] Step 2) Input the obtained video sequence into the P2C_1Net network to extract low-level feature information. Decouple the 3D convolution into a 1D convolution in the temporal dimension and a 2D convolution in the spatial dimension. Then, add CBAM channel attention after the 1D temporal convolution to extract temporal features. Figure 4 As shown, the input is first fed into a 3×1×1 temporal convolutional layer with a kernel of 32 and a 1×3×3 spatial convolutional layer to obtain temporal and spatial information respectively. CBAM channel attention is then applied after the 3×1×1 temporal convolutional layer to enhance temporal information extraction. Finally, an Add operation is used to fuse spatial and temporal features, followed by a 2×2×2 max-pooling layer to compress the fused features. The obtained features are then fed into a 3×1×1 temporal convolutional layer with a kernel of 64 to further enhance temporal information, followed by a 1×3×3 spatial convolutional layer with a kernel of 64, and then a second max-pooling operation with a stride of 2×2×2 for secondary feature compression. Finally, a Residual Module is used to connect the features of the initial input (128, 128, 3) and the features after the second max-pooling operation for fusion, completing the extraction and combination of the initial video features. This can be expressed by the following formula.

[0066]

[0067] Where, f(x) 3×1×1 ) is a 3×1×1 temporal convolutional layer; f(x) 1×3×3 It is a 1×3×3 spatial convolutional layer; For attention mechanisms;

[0068] Step 3) as Figure 5 As shown, the video sequence is input into the P2C_2Net network to extract high-level feature information. The 3D convolution is decoupled into a 1D convolution in the temporal dimension and a 2D convolution in the spatial dimension. CBAM channel attention is added after the spatial 2D convolution to extract temporal features. First, the data is input into 1×3×3 spatial and 3×1×1 temporal convolutional layers with kernels of 128 to obtain spatial and temporal information respectively. CBAM channel attention is then added after the 1×3×3 spatial convolutional layer to enhance spatial information extraction. Finally, an Add operation is used to fuse spatial and temporal features, followed by a 2×2×2 max pooling layer to compress the fused features. The obtained features are then input into a 1×3×3 spatial convolutional layer with kernels of 256 to further enhance temporal information, followed by a 3×1×1 temporal convolutional layer with kernels of 256, and then a second max pooling operation with a stride of 2×2×2 is performed for secondary feature compression. Finally, the Residual Module is used to connect the features output from step 2 and the features output from step 3 for fusion, completing the extraction and combination of the initial video features. This can be expressed by the following formula.

[0069]

[0070]

[0071]

[0072] Where, f(y) 3×1×1 f(y) is a 3×1×1 temporal convolutional layer; 1×3×3 It is a 1×3×3 spatial convolutional layer; This represents the attention mechanism; H(x) is the output feature of the previous step.

[0073] Step 4) Figure 6 As shown, the feature information is input into CSE Net, and CSE Net enhances important features to improve the accuracy of the feature information.

[0074] 4-1) Using feature information as input, the attention channel undergoes GlobalAveragePooling3D and Reshape operations to reduce the number of parameters in the fully connected function (fc) and also utilize global information; the number of channels is set to the number of channels of the output features in step 3), and the reduction ratio is set to 16; a 1×1×1 Conv3D operation with a kernel of channel / ratio is used, and the activation function is ReLU to extract local information; finally, a fully connected layer with the number of channels is added, and the activation function is sigmoid.

[0075] 4-2) Using step feature information as input, the attention channel undergoes GlobalMaxPooling3D and Reshape operations; a 1×1×1 Conv3D operation with a convolution kernel of channel / ratio and ReLU activation function is used to extract local information; a fully connected layer with the number of channels is added and sigmoid activation function is used.

[0076] The feature fusion is completed by fusing steps 4-1 and 4-2 using the Multiply layer. Finally, the feature fusion after multiplication and fusion is fused with the output information of step 4 using the residual module.

[0077] The feature compression Squeeze module includes a global average pooling layer and a global max pooling layer. A feature tensor containing both high- and low-order features is input to the global average pooling layer, and the output is a compressed feature tensor. The compressed feature tensor retains only channel features in its width and height, transforming it into a one-dimensional feature vector. This is set using the following formula:

[0078]

[0079] Among them, F sq(x) is the Squeeze function; GAP(*) is the Global Average Pooling function; GMP(*) is the Global Max Pooling function; x is the feature tensor containing high- and low-order features;

[0080] The local feature extraction Conv module consists of a 1×1×1 three-dimensional convolutional module with a channel / ratio kernel and the ReLU activation function. The convolutional module performs progressive convolution on the feature tensor, highlighting important information within the features. Simultaneously, it scales a one-dimensional vector to the channel / ratio of the input one-dimensional vector, thus achieving the purpose of filtering out secondary features and selecting primary ones.

[0081] The feature excitation module consists of fully connected layers and activation layers. The output parameters of the fully connected layers have the same size as the feature tensor after feature compression. Finally, the sigmoid function is used to generate the feature weights for each channel. This is set using the following formula:

[0082]

[0083] Among them, F ex (x) is the Excitation function; Dense(x) is the fully connected function; x is the feature tensor after feature compression; It is the ReLU activation function; Use the Sigmoid activation function;

[0084] Step 5) The Residual Module fuses the initial low-level feature information (128, 128, 3) with the high-level feature information output from Step 5 using the Add operation; this can be achieved using the following formula.

[0085]

[0086] Where H(x) is the output function of the residual module; F(x) is the residual mapping function; G(x) is the identity mapping function; and x is the input feature of the residual module.

[0087] Step 6) The obtained feature map is flattened from a multidimensional feature map to a single dimension using the Flatten operation. This is followed by three Dense layers: the first Dense layer has 1024 neural units, the second Dense layer has 512 neural units, and to prevent overfitting, a Dropout layer is added after the first two Dense layers with a dropout rate of 0.5 and a ReLU activation function. The number of neurons in the third Dense layer corresponds to the number of action categories in the video. A SoftMax classifier is used for classification, and an Adam optimizer is employed with a learning rate of 0.0003. Finally, the action classification results of the P2CS_3DNet network are obtained.

[0088] Depend on Figure 7 It can be seen that the recognition accuracy of the model of the present invention reaches 90.6%, which is achieved by... Figure 8 It can be seen that the recognition loss value of the model of the present invention is low.

[0089] The experimental results of LSTM(RGB), C3D, IDT, Two-Stream, I3D(RGB), P3D ResNet, TDD and the present invention are compared as shown in the table below. The results show that the recognition accuracy of the present invention is higher than that of the above models.

[0090] Table 1 Comparison of Experimental Results

[0091] METHOD Datasets Accuracy LSTM(RGB) UCF101 81.0 C3D UCF101 82.3 IDT UCF101 86.4 Two-Stream UCF101 88 I3D(RGB) UCF101 84.5 P3D ResNet UCF101 88.6 TDD UCF101 90.3 ours UCF101 90.6

Claims

1. A behavior recognition method based on P2CS_3DNet network, characterized in that, Includes the following steps: Step 1) Divide the video dataset containing actions into training set, test set and validation set, read the actions of each video in the training set and obtain the behavior label corresponding to the action of each video; Step 2) Extract video frames for each video using arithmetic progression sampling, and normalize the size of the extracted video frames. Step 3) Input the video frames extracted in Step 2) into the P2CS_3DNet network for video action recognition. The P2CS_3DNet network includes the P2C_1Net module, the P2C_2Net module, and the CSENet attention module. Step 3.1) Input the video frames extracted in Step 2) into the P2C_1Net module to extract low-level features of video actions, effectively extracting and enhancing temporal information; Step 3.1) specifically involves: The P2C_1Net network is constructed based on the P3D Module. The P2C_1Net network includes a first temporal convolution, a first spatial convolution, a second temporal convolution with 64 kernels (3×1×1), a second spatial convolution (1×3×3), and a first max pooling layer. Video frames input to the P2C_1Net network are respectively input to the first temporal convolution and the first spatial convolution. The output of the first temporal convolution is processed by the CBAM attention mechanism to output temporal features, and the first spatial convolution outputs spatial features. The temporal features and temporal features are fused by the Add operation and then sequentially input to the second temporal convolution, the second spatial convolution, and the first max pooling layer. Finally, the residual module fuses the video features input to the P2C_1Net network with the features compressed by the first max pooling layer to complete the low-level feature extraction. Step 3.2) Input the low-level features extracted in Step 3.1) into the P2C_2Net module to extract high-level features of video actions, effectively extracting and enhancing spatial information; Step 3.2) specifically involves: The P2C_2Net network is constructed based on the P3D Module. The P2C_2Net network includes a third spatial convolution, a third temporal convolution, a fourth spatial convolution, a fourth temporal convolution, and a max pooling layer. The low-level features extracted in step 3.1) are input into the third spatial convolution and the third temporal convolution, respectively. The output of the third spatial convolution is processed by the CBAM attention mechanism to output spatial features, and the third temporal convolution outputs temporal features. The temporal and spatial features are fused through the Add operation and then output sequentially as the fourth spatial convolution, the fourth temporal convolution, and the second max pooling layer. Finally, the low-level features extracted in step 3.1) and the features compressed by the second max pooling layer are fused through the residual module to complete the extraction of high-level features. Step 3.3) Input the features extracted in Step 3.2) into the CSENet attention module to perform feature compression and feature activation while performing progressive convolution on the features to achieve attention to local features; Step 3.3) specifically involves: The CSENet attention model is built based on SENet. The CSENet attention model includes two branches. The first branch includes a global max pooling, a 1×1×1 convolution, and a fully connected layer connected in sequence. The second branch includes a global average pooling, a 1×1×1 convolution, and a fully connected layer connected in sequence. The high-level features extracted in step 3.2) are input into the first branch and the second branch, respectively. The features are compressed by the global max pooling and global average pooling of the first and second branches, respectively. Then, local features are extracted by a 1×1×1 convolution, and finally, global features are extracted by a fully connected layer. Multiply is used to multiply and fuse the features output from the first and second branches. Finally, the residual module is used to fuse the multiplied and fused features with the high-level features extracted in step 3.2) to complete the extraction of local feature information. Step 3.4) Finally, the video features extracted in step 2) and the feature information output in step 3.3) are fused through the residual module. The fused features are then output through the output layer to obtain the video action classification result.

2. The behavior recognition method based on P2CS_3DNet network according to claim 1, characterized in that: Step 2) specifically refers to: Step 2.1) Read the number of video frames for each video using OpenCV; Step 2.2) Select video frames with specified parameters using an arithmetic progression method; Step 2.3) Normalize the size of the selected video frames.

3. The behavior recognition method based on P2CS_3DNet network according to claim 1, characterized in that: The output layer of step 3.4) includes three Dense layers, and the number of neurons in the third Dense layer is equal to the number of video action categories.

Images