Video behavior recognition method based on dual-path mixed attention network

Abstract

The invention discloses a video behavior recognition method based on a double-path mixed attention network, and the method comprises the steps: firstly, setting a low-speed path and a high-speed path respectively, adding a channel attention mechanism and a space attention mechanism into the low-speed path and the high-speed path respectively, enabling features to cover more parts of an object to be recognized, and enabling the features to be higher than the features of the object to be recognized; different connection weights can be dynamically generated by highlighting the characteristics and effects of the surrounding environment; secondly, the training data set is preprocessed, and independently placed action video frames are put into corresponding action folders; a series of experiments prove that the method can better pay attention to feature information, ignore other information and highlight the features of the person executing the action, so that the action is more accurately marked, and the accuracy of behavior recognition is improved.

Description

Technical Field

[0001] The present invention belongs to the field of video behavior recognition, and in particular to a video behavior recognition method based on a dual-path hybrid attention network. Background Art

[0002] Computers automatically analyze human actions to better detect objects and improve the efficiency of video action recognition, which has become a research hotspot in the field of computer vision. The analysis and research of human actions in videos mainly include data set selection, human target detection, action feature extraction, action classification, action understanding, etc. Among them, the research on human action recognition in videos has made great achievements, but there are still many challenges such as overly precise positioning, precise tracking, extreme sensitivity to camera viewpoints, noise, occlusion, etc.

[0003] At present, motion recognition algorithms based on deep learning, including recurrent neural networks (RNNs) and convolutional neural networks (CNNs), can be classified according to different input types, and are mainly divided into two categories: one is classified according to the type of input signal, and the other is classified according to the number of input signal streams. The number of input signal streams can be divided into single-stream, dual-stream and multi-stream motion recognition models. The main difference between these three models lies in the number of input data. The above methods generally ignore the importance of the features themselves and the correlation between them. The attention mechanism can recognize from multiple different angles by learning different parts of the human body, but it still has the disadvantage of not being able to capture position information. Its capture ability and performance on small data sets are not as good as RNN and CNN. The above problems existing in the existing technology lead to low accuracy of behavior recognition. Summary of the invention

[0004] The present invention aims to solve the problem of low accuracy of existing behavior recognition based on deep learning, and proposes a video behavior recognition method based on a dual-path hybrid attention network.

[0005] The technical solution of the present invention is: a video behavior recognition method based on a dual-path hybrid attention network, which inputs a video into a model for recognition, wherein the model includes a low frame rate slow path and a high frame rate fast path, and the outputs of the low frame rate slow path and the high frame rate fast path are connected to a prediction module, wherein the low frame rate slow path and the high frame rate fast path are both composed of a video acquisition module, a channel attention mechanism module and a spatial attention mechanism module, and are constructed according to the following method:

[0006] Step 1. Classify the video frames according to the action names, and name the video frames representing an action with the action;

[0007] Step 2. Feed the video into the low frame rate slow path for the following processing:

[0008] Step 2.1 The video acquisition module performs low frame rate sampling according to the set sampling frequency parameter τ, and acquires a total of L low frame feature maps S L ∈R C×H×W , where C is the number of channels, H×W is the resolution of the feature, denoted as {S 1 ,S 2 ,...,S L};

[0009] Step 2.2: Transform the low-frame feature map S L The recalibrated slow feature map MS is obtained by sequentially passing through the channel attention mechanism module SCAM and the spatial attention mechanism module SSAM. L ;

[0010] Step 2.2.1 Channel Attention Mechanism Module SCAM first generates an average pooling feature representation vector by average pooling and maximum pooling on the low-frame feature map. And the maximum pooling feature representation vector is Then, these two spatial feature representation vectors are put into the shared neural network, and the output is summed element by element to obtain the slow channel attention map

[0011] Step 2.2.2 Slow Spatial Attention Mechanism Module SSAM takes the slow channel attention graph Multiply by the low frame feature map S L As input, we first focus on the graph based on the slow channel The width and height of are subjected to global maximum pooling and global average pooling in turn, and the global maximum pooling slow feature maps are obtained respectively. And global average pooling slow feature map Then, two slow feature maps are concatenated based on the channel, and the slow feature dimension is changed from H×W to 1×1; next, the channel dimension is reduced and restored to the original dimension through a convolution layer with a convolution kernel of 7×7 and a Relu activation function; finally, a slow spatial attention map is generated through a Sigmoid activation function

[0012] Step 2.2.3 Focus the slow channel on the graph and slow spatial attention graph After normalization, fusion is performed to obtain the slow feature map MS recalibrated in spatial and channel dimensions. L ;

[0013] Step 3. Feed the video into the high frame rate fast path for the following processing:

[0014] Step 3.1 The video acquisition module performs high frame rate sampling according to the set sampling frequency parameter 2τ, and acquires a total of N high-frame feature maps FN ∈R C×H×W , where C is the number of channels, H×W is the resolution of the feature, denoted as {F 1 ,F 2 ,...,F N};

[0015] Step 3.2: Transform the high-frame feature map F N The recalibrated fast feature map MF is obtained by sequentially passing through the channel attention mechanism module FCAM and the spatial attention mechanism module FSAM. N ;

[0016] Step 3.2.1 Channel Attention Mechanism Module FCAM first generates an average pooling feature representation vector by average pooling and maximum pooling on the high-frame feature map. And the maximum pooling feature representation vector is Then, these two spatial feature representation vectors are put into the shared neural network, and the output is summed element by element to obtain the fast channel attention map

[0017] Step 3.2.2 Fast Spatial Attention Mechanism Module FSAM will fast track attention graph Multiply by the high frame feature map F N As input, we first focus on the fast-path graph The width and height of are subjected to global maximum pooling and global average pooling in turn, and the global maximum pooling fast feature maps are obtained respectively. and global average pooling fast feature map Then, two fast feature maps are concatenated based on the channel, and the fast feature dimension is changed from H×W to 1×1; next, the channel dimension is reduced and restored to the original dimension through a convolution layer with a convolution kernel of 7×7 and a Relu activation function; finally, a fast spatial attention map is generated through a Sigmoid activation function

[0018] Step 3.2.3: Add the fast track focus graph and fast spatial attention graph After normalization, fusion is performed to obtain a fast feature map MF that is recalibrated in spatial and channel dimensions. N ;

[0019] Step 4. Recalibrate the slow feature map MS L and recalibrated fast feature map MF N Input to the prediction module to predict human behavior;

[0020] Step 5. Train the network. The specific steps are as follows:

[0021] Step 5.1: Randomly initialize network parameters;

[0022] Step 5.2: Start training the entire network and update parameters; the agreed variable epoch records the number of current network training iterations, and the epoch is initially 0;

[0023] Step 5.3: Calculate the gradient in this iteration, perform back propagation, and re-update the network parameters through the Adam optimization algorithm;

[0024] Step 5.4: This round of iteration ends, set epoch = epoch + 1; when the epoch reaches the set number of times, the network stops training, saves the training model and parameters, and the model construction is completed; otherwise, enter step 5.2 again and repeat the training of the network.

[0025] The present invention first sets up a slow path and a fast path respectively, and adds a channel attention mechanism and a space attention mechanism to the slow path and the fast path respectively, so that the features cover more parts of the object to be identified, so as to highlight the features and functions of the surrounding environment, and dynamically generate different connection weights; secondly, the training data set is preprocessed, and the separately placed action video frames are placed in the corresponding action folder. Through a series of experiments, it is proved that the present invention can better focus on feature information, ignore other information, highlight the characteristics of the person performing the action, thereby more accurately marking the action and improving the accuracy of behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is the overall network structure diagram of the present invention. DETAILED DESCRIPTION

[0027] A video behavior recognition method based on a dual-path hybrid attention network of the present invention is as follows Figure 1 As shown, the video is input into the model for recognition, and the model includes a low frame rate slow path and a high frame rate fast path. The outputs of the low frame rate slow path and the high frame rate fast path are connected to the prediction module. The low frame rate slow path and the high frame rate fast path are both composed of a video acquisition module, a channel attention mechanism module and a spatial attention mechanism module.

[0028] The slow path is designed to capture slow-moving, slowly-refreshing images or a small amount of sparse low-frame videos; the fast path is responsible for capturing fast-changing motion, high-frame videos with fast refresh rates and high temporal resolution; both the fast and slow paths use channel attention (CAM) and spatial attention mechanism (SAM). CAM can accurately capture the key information in the image by evaluating the importance of each channel, thereby improving the accuracy of image classification and detection; while SAM can better capture the contextual information of different regions in the image, thereby improving the accuracy of the model;

[0029] Build it as follows:

[0030] Step 1. Classify the video frames according to the action names, and name the video frames representing an action with the action;

[0031] The embodiment of the present invention uses Charades, a large-scale indoor video dataset for behavior recognition, which has 157 categories and includes 9848 video clips, including 9.8k training videos and 1.8k verification videos, as well as action and text description labels. In order to improve the recognition of features such as actions and environments, the dataset is preprocessed: the video frames are classified according to the action names, and the video frames representing an action are named with the action, that is, there are 157 types of action names in 7895 videos;

[0032] Step 2. Feed the video into the low frame rate slow path for the following processing:

[0033] Step 2.1 The video acquisition module performs low frame rate sampling according to the set sampling frequency parameter τ, and acquires a total of L low frame feature maps S L ∈R C×H×W , where C is the number of channels, H×W is the resolution of the feature, denoted as {S 1 ,S 2 ,...,S L};

[0034] Step 2.2: Transform the low-frame feature map S L The recalibrated slow feature map MS is obtained by sequentially passing through the channel attention mechanism module SCAM and the spatial attention mechanism module SSAM. L ;

[0035] Step 2.2.1 Channel Attention Mechanism Module SCAM needs to compress the feature map at low temporal resolution and convert it into a one-dimensional vector. First, the low-frame feature map is sequentially processed through average pooling and maximum pooling to generate an average pooling feature representation vector And the maximum pooling feature representation vector is Then, these two spatial feature representation vectors are put into a shared neural network (multi-layer perceptron MLP), and the output is summed element by element to obtain the slow channel attention map.

[0036] Step 2.2.2 Slow Spatial Attention Mechanism Module SSAM takes the slow channel attention graph Multiply by the low frame feature map S L As input, in order to obtain the attention features in the spatial dimension, we first focus on the slow channel map The width and height of are subjected to global maximum pooling and global average pooling in turn, and the global maximum pooling slow feature maps are obtained respectively. And global average pooling slow feature map Then, two slow feature maps are concatenated based on the channel, and the slow feature dimension is changed from H×W to 1×1; next, the channel dimension is reduced and restored to the original dimension through a convolution layer with a convolution kernel of 7×7 and a Relu activation function; finally, a slow spatial attention map is generated through a Sigmoid activation function

[0037] Step 2.2.3 Focus the slow channel on the graph and slow spatial attention graph After normalization, fusion is performed to obtain the slow feature map MS recalibrated in spatial and channel dimensions. L ;

[0038] Step 3. Feed the video into the high frame rate fast path for the following processing:

[0039] Step 3.1 The video acquisition module performs high frame rate sampling according to the set sampling frequency parameter 2τ, and acquires a total of N high-frame feature maps F N ∈R C×H×W , where C is the number of channels, H×W is the resolution of the feature, denoted as {F 1 ,F 2 ,...,F N};

[0040] Step 3.2: Transform the high-frame feature map F N The recalibrated fast feature map MF is obtained by sequentially passing through the channel attention mechanism module FCAM and the spatial attention mechanism module FSAM. N ;

[0041] Step 3.2.1 The channel attention mechanism module FCAM needs to compress the high-frame feature map and obtain a one-dimensional vector. First, the high-frame feature map is sequentially subjected to average pooling and maximum pooling to generate an average pooling feature representation vector. And the maximum pooling feature representation vector is Then, these two spatial feature representation vectors are put into a shared neural network (multi-layer perceptron MLP), and the output is summed element by element to obtain the fast channel attention map.

[0042] Step 3.2.2 Fast Spatial Attention Mechanism Module FSAM will fast track attention graph Multiply by the high frame feature map F N As input, in order to obtain the attention features in the spatial dimension, we first focus on the fast channel map The width and height of are subjected to global maximum pooling and global average pooling in turn, and the global maximum pooling fast feature maps are obtained respectively. and global average pooling fast feature map Then, two fast feature maps are concatenated based on the channel, and the fast feature dimension is changed from H×W to 1×1; next, the channel dimension is reduced and restored to the original dimension through a convolution layer with a convolution kernel of 7×7 and a Relu activation function; finally, a fast spatial attention map is generated through a Sigmoid activation function

[0043] Step 3.2.3: Add the fast track focus graph and fast spatial attention graph After normalization, fusion is performed to obtain a fast feature map MF that is recalibrated in spatial and channel dimensions. N ;

[0044] Step 4. Recalibrate the slow feature map MS L and recalibrated fast feature map MF N Input to the prediction module to predict human behavior;

[0045] Step 5. Train the network. The specific steps are as follows:

[0046] Step 5.1: Randomly initialize network parameters;

[0047] Step 5.2: Start training the entire network and update parameters; the agreed variable epoch records the number of current network training iterations, and the epoch is initially 0;

[0048] Step 5.3: The learning rate used is initially 1e-2, the weight decay rate is 1e-4, and the batch size is 8. The gradient in this round of iteration is calculated, back-propagation is performed, and the network parameters are updated again through the Adam optimization algorithm;

[0049] Step 5.4: This round of iteration ends, set epoch = epoch + 1; when the epoch reaches the set number of times, the network stops training, saves the training model and parameters, and the model construction is completed; otherwise, enter step 5.2 again and repeat the training of the network.

[0050] Input the Charades test set into the trained model to obtain the accuracy results (all experiments were performed on a PC with Intel Xeon (R) 4210R CPU and NVIDIA RTX 3090 GPU, based on the PyTorch platform). The specific steps are as follows:

[0051] The trained model and parameter files are imported into the test code. The present invention is compared with the current mainstream method using the Charades dataset, and the mAP is calculated. The results are shown in Table 1.

[0052] Table 1 Evaluation results of the proposed method and current mainstream methods on the Charades dataset

[0053]

[0054] The results show that the mAP of the present invention is higher than that of other methods, indicating that the present invention has a higher accuracy rate in video behavior recognition.

Claims

1. A video behavior recognition method based on a dual-path hybrid attention network is to input the video into the model for recognition, characterized in that: The model includes a low frame rate slow path and a high frame rate fast path, the outputs of the low frame rate slow path and the high frame rate fast path are connected to the prediction module, and the low frame rate slow path and the high frame rate fast path are both composed of a video acquisition module, a channel attention mechanism module and a spatial attention mechanism module, and are constructed according to the following method: Step 1. Classify the video frames according to the action names, and name the video frames representing an action with the action; Step 2. Feed the video into the low frame rate slow path for the following processing: Step 2.1 The video acquisition module performs low frame rate sampling according to the set sampling frequency parameter τ, and acquires a total of L low frame feature maps S L ∈R C×H×W , where C is the number of channels, H×W is the resolution of the feature, denoted as {S1,S2,...,S L }; Step 2.2: Transform the low-frame feature map S L The recalibrated slow feature map MS is obtained by sequentially passing through the channel attention mechanism module SCAM and the spatial attention mechanism module SSAM. L ; Step 2.2.1 Channel Attention Mechanism Module SCAM first generates an average pooling feature representation vector by average pooling and maximum pooling on the low-frame feature map. And the maximum pooling feature representation vector is Then, these two spatial feature representation vectors are put into the shared neural network, and the output is summed element by element to obtain the slow channel attention map Step 2.2.2 Slow Spatial Attention Mechanism Module SSAM takes the slow channel attention graph Multiply by the low frame feature map S L As input, we first focus on the graph based on the slow channel The width and height of are subjected to global maximum pooling and global average pooling in turn, and the global maximum pooling slow feature maps are obtained respectively. And global average pooling slow feature map Then, two slow feature maps are concatenated based on the channel, and the slow feature dimension is changed from H×W to 1×1; Next, the channel dimension is reduced by a convolution layer with a 7×7 kernel and a Relu activation function and then restored to the original dimension; finally, a slow spatial attention map is generated by a Sigmoid activation function. Step 2.2.3 Focus the slow channel on the graph and slow spatial attention graph After normalization, fusion is performed to obtain the slow feature map MS recalibrated in spatial and channel dimensions. L ; Step 3. Feed the video into the high frame rate fast path for the following processing: Step 3.1 The video acquisition module performs high frame rate sampling according to the set sampling frequency parameter 2τ, and acquires a total of N high-frame feature maps F N ∈R C×H×W , where C is the number of channels, H×W is the resolution of the feature, denoted as {F1,F2,...,F N }; Step 3.2: Transform the high-frame feature map F N The recalibrated fast feature map MF is obtained by sequentially passing through the channel attention mechanism module FCAM and the spatial attention mechanism module FSAM. N ; Step 3.2.1 Channel Attention Mechanism Module FCAM first generates an average pooling feature representation vector by average pooling and maximum pooling on the high-frame feature map. And the maximum pooling feature representation vector is Then, these two spatial feature representation vectors are put into the shared neural network, and the output is summed element by element to obtain the fast channel attention map Step 3.2.2 Fast Spatial Attention Mechanism Module FSAM will fast track attention graph Multiply by the high frame feature map F N As input, we first focus on the fast-path graph The width and height of are subjected to global maximum pooling and global average pooling in turn, and the global maximum pooling fast feature maps are obtained respectively. and global average pooling fast feature map Then, two fast feature maps are concatenated based on the channel, and the fast feature dimension is changed from H×W to 1×1; Next, the channel dimension is reduced by a convolution layer with a 7×7 kernel and a ReLU activation function and then restored to the original dimension; finally, a fast spatial attention map is generated by a Sigmoid activation function. Step 3.2.3: Add the fast track focus graph and fast spatial attention graph After normalization, fusion is performed to obtain a fast feature map MF that is recalibrated in spatial and channel dimensions. N ; Step 4. Recalibrate the slow feature map MS L and recalibrated fast feature map MF N Input to the prediction module to predict human behavior; Step 5. Train the network. The specific steps are as follows: Step 5.1: Randomly initialize network parameters; Step 5.2: Start training the entire network and update parameters; the agreed variable epoch records the number of current network training iterations, and the epoch is initially 0; Step 5.3: Calculate the gradient in this iteration, perform back propagation, and re-update the network parameters through the Adam optimization algorithm; Step 5.4: This round of iteration ends, set epoch = epoch + 1; when the epoch reaches the set number of times, the network stops training, saves the training model and parameters, and the model construction is completed; otherwise, enter step 5.2 again and repeat the training of the network.

Images