Attention-based multi-level spatio-temporal feature fusion behavior recognition method and system

Abstract

The application discloses a multi-level spatio-temporal feature fusion behavior recognition method and system combined with attention, and comprises the following steps: (1) acquiring a video to be recognized, and performing a pretreatment operation on the video to obtain an image; (2) inputting the pretreated image into a human body behavior recognition network combined with multi-level spatio-temporal feature fusion to perform convolutional neural network training; and (3) outputting a corresponding human body action classification result. The application innovates on the basis of a traditional three-dimensional convolution method in behavior recognition, and realizes the effects of reducing a certain parameter quantity and effectively improving the recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and relates to the design and application of a human behavior recognition method. BACKGROUND

[0002] With the rapid development of computer and network technology, multimedia information is growing explosively. As a carrier of information, video contains a large amount of key information. In daily life, there is a kind of video content about human behavior and activities. The technology of understanding and classifying the action in the video is called human behavior recognition technology. Behavior recognition technology is a key technology for machines to understand the world and human behavior.

[0003] In recent years, with the development of artificial intelligence technology, people have also widely applied deep learning to the field of behavior recognition. At the beginning, DeepVideo proposed to use a 2D CNN model independently on each video frame, but 2D convolution ignores the time correlation between video frames in the video feature extraction process, resulting in the loss of time dimension features; in order to better capture time information, there is a C3D model to directly extract time and spatial dimension features, which not only achieves good recognition effect, but also has strong generalization ability. Later, 3D convolution kernel was used to replace 2D Resnet, and the residual structure was successfully introduced into the field of human behavior recognition. However, 3D convolution has a very large weight parameter quantity and calculation amount, which leads to very low operation efficiency, and there is also a lot of redundant information. After that, many behavior recognition network models have been committed to reducing the calculation parameter quantity to improve the calculation efficiency of 3DCNN. For example, R3D uses a 3D residual structure to prevent gradient explosion in a deep network, P3D and R(2+1)D try to decompose the 3D convolution kernel into 1D time convolution and 2D spatial convolution in cascade to reduce the parameter quantity. Some methods also increase the attention mechanism to reduce information redundancy. However, these methods have single convolution kernel size, lack hierarchical analysis of different scale spatio-temporal features, and can only extract local short-distance time sequence information, which affects the recognition accuracy of the model. TPN uses a feature pyramid structure on the time frame to study the speed of the frame rate, and realizes the multi-scale feature fusion of the time dimension. However, the feature pyramid structure is complex and difficult to optimize in the training stage, so the multi-level spatio-temporal feature extraction method still needs to be improved.

[0004] Therefore, the existing method cannot reduce information redundancy and realize multi-level spatio-temporal feature extraction while ensuring a low parameter quantity, so as to achieve higher recognition accuracy. SUMMARY

[0005] Inventive purpose: In view of the problems of single space-time feature scale extraction, information redundancy and insufficient extraction of channel mid-frequency domain information in three-dimensional convolutional neural network, a multi-level space-time feature fusion network model combined with attention is proposed. First, multi-level space-time feature separation, splicing and fusion are used to realize the fusion of different scale spatial receptive field and short, medium and long time sequence information under the premise of reducing network parameters. Then, the GAP preprocessing operation in channel attention is expanded to the frequency domain to further reduce feature redundancy. The invention innovates the traditional three-dimensional convolution method in behavior recognition, realizes the effect of reducing certain parameter quantity and effectively improving the recognition accuracy.

[0006] The technical scheme is as follows: a multi-level space-time feature fusion behavior recognition method combined with attention, comprising the following steps:

[0007] (1) Obtain the video to be recognized, and perform preprocessing operation on the video to obtain images;

[0008] (2) input the preprocessed image into the multi-level space-time feature fusion human behavior recognition network combined with attention for convolutional neural network training;

[0009] (3) output the corresponding human action classification result.

[0010] Further, the preprocessing operation comprises:

[0011] The video is decomposed into images frame by frame without changing the structure, and then interval video frames are randomly extracted;

[0012] The picture is scaled and randomly cropped to a specified size as input;

[0013] Further, the multi-level space-time feature fusion human behavior recognition network combined with attention comprises an input layer, a 3D convolution layer, a continuously stacked multi-level space-time feature fusion basic module (MSF), a multi-frequency channel and a space-time attention module (FSAM), a pooling layer, a classification layer and an output layer.

[0014] Further, the construction process of the multi-level space-time feature fusion human behavior recognition network combined with attention comprises:

[0015] The multi-level space-time feature fusion basic module, the multi-frequency channel and the space-time attention module are constructed;

[0016] The multi-level space-time feature fusion basic module is composed of a three-dimensional convolution layer and a multi-level space-time feature decomposition layer. First, two cascaded 3x3 two-dimensional convolution kernels are expanded to two 3x3x3 three-dimensional convolution kernels in series, and then the second 3x3x3 convolution kernel is decomposed into multi-level space-time features;

[0017] The multi-frequency channel and space-time attention module includes two parts: a multi-frequency channel attention module and a space-time attention module.

[0018] The multi-frequency channel attention module and the space-time attention module make the feature map sequentially perform an attention mechanism in the channel and space-time dimensions, and finally output a result after optimization of the features.

[0019] Further, the multi-level space-time feature decomposition process includes:

[0020] The C channels of the input feature X are evenly split into S feature subsets, and the S feature subsets are denoted as x i , wherein i is an element in the set {1, 2,..., S}, and the channel number C' is that is, wherein the x1 subset is not transformed and is directly output as y1; the other subsets x i Each branch corresponds to a class residual structure, and each branch has a 3D convolution with a 3x3x3 convolution kernel, denoted as f 3×3×3 ; x i The feature map generated after the 3D convolution is denoted as y i , y i The expression of y i is:

[0021]

[0022] In the multi-level space-time feature decomposition process, y i is again separated into two groups of sub-feature maps y i1 and y i2 , wherein y i1 is directly spliced to the final output, and y i2 is directly spliced to the next group of feature maps x i+1 ; the expression of the space-time feature output by the multi-level space-time feature decomposition layer is: Y = cat[x1, y 22 , y 32 ,..., y (i-1)2 , y i ]; and the total output of the multi-level space-time feature fusion base module is: Output = X + Y + Input.

[0023] Further, the multi-frequency channel and space-time attention module is divided into two parts: a multi-frequency channel attention module and a space-time attention module.

[0024] The multi-frequency channel attention module specifically includes: first, the input feature map is divided into n parts along the channel dimension, and each part of the feature map can be represented as [X 0 , X 1 ,..., X n-1 ], wherein The final multi-spectrum pre-processing vector can be obtained by splicing:

[0025] Freq=cat([Freq 0 ,Freq 1 ,···,Freq n-1 ]), (3)

[0026] Wherein The formula of the multi-frequency channel attention framework is:

[0027] Fc_att=sigmoid(fc(Freq)) (4)

[0028] The spatial-temporal attention module specifically comprises: for an input feature First, respectively passing through the maximum pooling and average pooling in the channel dimension to generate two 3D features And Then passing through a 3D convolution to generate a 3D spatial-temporal attention feature,

[0029] ST_att=sigmoid(f 7×7×7 (cat[MaxPool(F),AvgPool(F)])) (5)

[0030] The formula of the spatial-temporal attention framework is the above formula.

[0031] Further, given a feature As the input of the multi-frequency channel and spatial-temporal attention module, the corresponding multi-frequency channel attention feature is output, and then broadcasted along the time and space dimensions, and the final attention feature is output through the spatial-temporal attention module,

[0032]

[0033] The output of the entire attention module is the above formula.

[0034] Further, the picture of the continuous frame is input into the behavior recognition network, and after passing through several multi-level spatial-temporal feature fusion basic modules and multi-frequency channel and spatial-temporal attention modules, the last 3D average pooling layer is passed, and the final behavior category is output by the softmax.

[0035] The multi-level spatial-temporal feature fusion behavior recognition system combined with attention, characterized in that, comprising a preprocessing module, a network construction module, a network training module and an output module;

[0036] The preprocessing module comprises obtaining a video to be recognized, and performing a preprocessing operation on the video to obtain an image.

[0037] The network construction module comprises a human behavior recognition network of multi-level spatio-temporal feature fusion combined with attention;

[0038] The network training module comprises inputting the preprocessed image into the human behavior recognition network of multi-level spatio-temporal feature fusion combined with attention for convolutional neural network training;

[0039] The output module comprises outputting a corresponding human action classification result.

[0040] Further, the human behavior recognition network of multi-level spatio-temporal feature fusion combined with attention comprises an input layer, a 3D convolution layer, a continuously stacked multi-level spatio-temporal feature fusion basic module, a multi-frequency channel and spatio-temporal attention module, a pooling layer, a classification layer and an output layer; the construction process of the human behavior recognition network of multi-level spatio-temporal feature fusion combined with attention comprises:

[0041] The multi-level spatio-temporal feature fusion basic module and the multi-frequency channel and spatio-temporal attention module are constructed;

[0042] The multi-level spatio-temporal feature fusion basic module is composed of a three-dimensional convolution layer and a multi-level spatio-temporal feature decomposition layer, two cascaded 3*3 two-dimensional convolution kernels are first expanded in three dimensions to two 3*3*3 three-dimensional convolution kernels in cascade, and then the second 3*3*3 convolution kernel is decomposed in multi-level spatio-temporal features;

[0043] The multi-frequency channel and spatio-temporal attention module comprises two parts: a multi-frequency channel attention module and a spatio-temporal attention module;

[0044] The multi-frequency channel attention module and the spatio-temporal attention module enable the feature map to execute attention mechanism in the channel and spatio-temporal dimensions in turn, and finally output the result after feature optimization.

[0045] Beneficial effects: compared with the prior art, the present application has the following beneficial effects:

[0046] 1. A new multi-level spatio-temporal feature fusion (MSF) module is designed, which can not only expand the receptive field of the feature map in the spatial dimension on the basis of reducing the model parameters, but also realize the fusion of short, medium and long time sequences of time information;

[0047] 2. The pre-processing GAP operation in SENet is expanded to the frequency domain, the spatial attention is expanded to the spatio-temporal dimension, a new multi-frequency channel and spatio-temporal attention module (FSAM) is proposed, so that the model can capture key action information in multiple frequency channels;

[0048] By a brand-new multi-level spatio-temporal feature fusion (MSF) module and a multi-frequency channel and spatio-temporal attention module (FSAM), the existing behavior recognition network model is compared. Under the premise of reducing the parameter quantity, higher recognition accuracy is realized, and the efficiency of behavior recognition is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a detailed structure diagram of the multi-level spatio-temporal feature fusion (MSF) module

[0050] Figure 2 is a detailed structure diagram of the multi-frequency channel attention (FCA)

[0051] Figure 3 is a detailed structure diagram of the spatio-temporal attention (STA)

[0052] Figure 4 is a whole structure diagram of the multi-frequency channel and spatio-temporal attention (FSAM)

[0053] Figure 5 is a whole network model diagram of the multi-level spatio-temporal feature behavior recognition method combined with attention DETAILED DESCRIPTION

[0054] The technical solutions of the present application will be further described below in combination with the drawings.

[0055] In order to solve the problems of large parameter quantity, information redundancy, low recognition efficiency and low recognition accuracy in the field of human behavior recognition at present, the present application proposes a behavior recognition method combined with attention and multi-level spatio-temporal feature fusion, and the specific technical solutions are as follows:

[0056] Obtain the video to be recognized;

[0057] Preprocess the video;

[0058] The preprocessing operation of the video includes:

[0059] The video is decomposed into images frame by frame without changing the structure, and then 8 frames of video frames are randomly extracted at intervals;

[0060] The picture is scaled to 171*128 size;

[0061] The picture is randomly cropped to 112*112 size as input;

[0062] The 8 continuous pictures after preprocessing are input into the human behavior recognition network combined with attention and multi-level spatio-temporal feature fusion for convolutional neural network training, and the corresponding human action classification result is output.

[0063] The attention combined multi-level spatio-temporal feature fusion behavior recognition network mainly comprises an input layer, a 3D convolution layer, a continuously stacked multi-level spatio-temporal feature fusion structure (MSF), a multi-frequency channel and a spatio-temporal attention module (FSAM), a continuously stacked MSF, an FSAM, a pooling layer, a classification layer and an output layer.

[0064] The construction process of the attention combined multi-level spatio-temporal feature fusion behavior recognition network comprises:

[0065] The multi-level spatio-temporal feature fusion basic module (MSF) and the multi-frequency channel and spatio-temporal attention module (FSAM) are constructed.

[0066] The multi-level spatio-temporal feature fusion basic module (MSF) is composed of a three-dimensional convolution layer and a multi-level spatio-temporal feature decomposition layer (MSL). Two cascaded 3x3 two-dimensional convolution kernels are first expanded in three dimensions to two 3x3x3 three-dimensional convolution kernels in cascade, and then the second 3x3x3 convolution kernel is decomposed in multiple levels of spatio-temporal features.

[0067] The multi-level spatio-temporal feature decomposition process is as shown in Figure 1 , specifically:

[0068] The C channels of the input feature X are evenly split into S feature subsets (the split scale parameter S = 4 in the figure), and the S feature subsets are denoted as x i , wherein i ∈ {1, 2,..., S}, and the channel number C' is , that is, wherein the x1 subset is not transformed and directly output as y1; the other subsets x i each correspond to a branch of a class residual structure, and each branch has a 3D convolution with a 3x3x3 convolution kernel, denoted as f 3×3×3 ; x i The feature map generated after 3D convolution is denoted as y i , and the expression of y i is:

[0069]

[0070] In the multi-level spatio-temporal feature separation process, in order to increase the information fusion between the semantics of each branch and the different frame rates, y i is separated into two groups of feature maps y i1 and y i2 by channel, wherein y i1 is directly spliced to the final output, and y i2 is directly spliced to the next group of feature maps x i+1This is because the neural network will get a lot of similar feature maps when extracting features, and the feature map redundancy is serious. Therefore, by feature reuse, the hierarchical connection of the features after convolution and the features before convolution is expanded, the spatio-temporal receptive field range of the convolution layer is expanded, and the model parameter amount is further reduced. The spatio-temporal feature expression output by the MSL is: Y = cat [x1, y 22 ,y 32 ,…,y (i-1)2 ,y i ]. The total output of the multi-level spatio-temporal feature fusion base module (MSF) is: Output = X + Y + Input.

[0071] The multi-frequency channel and spatio-temporal attention module (FSAM) is divided into two parts: multi-frequency channel attention and spatio-temporal attention.

[0072] The process of the multi-frequency channel attention is as shown in Figure 2 , specifically, first, the input feature map is divided into n parts along the channel dimension, and each part of the feature map can be represented as [X 0 ,X 1 ,···,X n-1 ], wherein A two-dimensional frequency component is allocated to each part, and the final multi-frequency spectrum preprocessing vector can be obtained by splicing:

[0073] Freq = cat ([Freq 0 , Freq 1 ,···, Freq n-1 ]), (3) Wherein The multi-frequency channel attention framework formula is:

[0074] Fc_att = sigmoid (fc (Freq)) (4)

[0075] The spatio-temporal attention is as shown in Figure 3 . Specifically, for an input feature , first, respectively, the maximum pooling and average pooling along the channel dimension are generated two 3D features and , and then a 3D convolution is performed to generate a 3D spatio-temporal attention feature, and the spatio-temporal attention framework formula is:

[0076] ST_att = sigmoid (f 7×7×7 (cat [MaxPool (F), AvgPool (F)])) (5)

[0077] Multi-frequency channel attention and space-time attention module (FSAM). The feature map is sequentially executed in the channel and space-time dimensions. Finally, the optimized feature is output. Specifically, given the feature As the input of the FSAM, the corresponding multi-frequency channel attention feature is output, and then broadcasted along the time and space dimensions. Finally, the last attention feature is output through the space-time attention module. The process is as shown in Figure 4 As shown in the figure, the whole attention module can be summarized as:

[0078]

[0079] The continuous frame picture is input into the behavior recognition network, as shown in Figure 5 After a plurality of multi-level space-time feature fusion basic modules (MSF) and multi-frequency channel and space-time attention modules (FSAM), the last 3D average pooling layer is used, and then the softmax is used to output the final behavior category.

[0080] Specific test part:

[0081] I. UCF101 data set

[0082] The UCF101 behavior recognition data set is selected for experimental verification. The UCF101 data set contains 101 types of human behavior actions, including applying lipstick, applying eye makeup, long jump, swimming, playing golf, high jump, ice dancing, diving, shaving, etc. Each type of action in the data set is divided into 25 groups, each group containing 4-7 videos of an action, totaling 13320 videos. The actions involve human-object interaction, pure limb movement, human-human interaction, instrument playing, and sports.

[0083] The UCF101 adopts the test01 division method provided by the official website, and all data are trained from scratch without using any pre-trained weight parameters.

[0084] II. Behavior recognition R3D network

[0085] The R3D behavior recognition network is a deep learning network model based on the three-dimensional expansion of resnet2D. All 3x3 convolution kernels in the residual structure are changed to 3x3x3 convolution. Except that the conv1 uses a 1x2x2 step, all other down-sampling convolution layers use a 2x2x2 step, and the first maximum pooling layer is deleted.

[0086] III. Behavior recognition C3D network structure

[0087] C3D as a classic network model in 3D CNN, due to its simple, compact, easy to train and use the characteristics, is widely used in behavior recognition, video similarity analysis, dynamic expression recognition and other fields. Its network structure has eight convolutional layers, five pooling layers, two fully connected layers and a output softmax layer. Among them, the convolutional layer uses a 3D convolution kernel with a size of 3x3x3 to extract the spatio-temporal features of the video sequence, overcoming the limitations of 2D convolution which can only learn features in space; The pooling layer uses a 3D pooling kernel with a size of 2x2x2 to compress feature information and remove redundant information.

[0088] Embodiment one:

[0089] A multi-level spatio-temporal feature fusion behavior recognition method (MSF-R3D+FSAM) based on R3D combined with attention, including the following steps:

[0090] 1. Obtain the behavior video to be identified;

[0091] Real-time video can be captured using a camera, or existing video dataset containing behaviors can be used: for example, the UCF101 dataset used in this case.

[0092] 2. Preprocess the video to be identified to obtain the image sequence of the behavior. The preprocessing process includes: randomly extracting 16 consecutive frames from the video, discarding even frames, and only keeping odd frames to form a new image sequence, and then performing normalization, random cropping and other operations on the image sequence. In this implementation case, the resolution of the input image sequence is controlled at 112x112.

[0093] 3. Input the image sequence into the behavior recognition network for experiment, which includes two stages of training and testing:

[0094] Training stage: select 8 video frames as a batch, randomly initialize network parameters and start training the network from scratch, train with mini-batch, batch size is 16; use Adam optimization algorithm to train model parameters, learning rate uses a segmented constant decay strategy, the first epoch is set to 0.0001, then every 10 epochs decay to 1 / 10 of the original, until the network converges; Use horizontal flip, contrast enhancement and other operations to increase the number of training samples to prevent overfitting during network training.

[0095] Test stage: randomly select odd 8 frames from a continuous 16 frames of a video in the test set as network input, output 101 behavior classification scores through forward propagation, and the class with the highest score is the prediction result.

[0096] Embodiment two:

[0097] A multi-level spatio-temporal feature fusion behavior recognition method (MSF-C3D+FSAM) based on C3D, comprising the following steps:

[0098] 1. Obtain a video to be recognized;

[0099] Real-time video can be collected using a camera, or an existing video dataset containing behaviors can be used, such as the UCF101 dataset used in this case.

[0100] 2. Preprocess the video to be recognized to obtain an image sequence of the behavior. The preprocessing process includes: randomly extracting 16 consecutive frames of pictures from the video to form a new image sequence, and then performing normalization, random cropping, etc. on the image sequence. In this implementation case, the resolution of the input image sequence is controlled at 112x112.

[0101] 3. Input the image sequence into the behavior recognition network for experiments, including two stages of training and testing:

[0102] Training stage: take 16 consecutive video frames as a batch, randomly initialize network parameters, and train the network from scratch. Mini-batch data is used for training, and the batch size is 16. The Adam optimization algorithm is used to train the model parameters, and the learning rate uses a segmented constant decay strategy. The first epoch is set to 0.0001, and then every 10 epochs are decayed to 1 / 10 of the original, until the network converges. Horizontal flip, contrast enhancement, etc. are used to increase the number of training samples to prevent overfitting during training of the network.

[0103] Test stage: In the test stage, 16 consecutive frames of a video are randomly cut in the test set as network input, and 101 behavior classification scores are output through forward propagation. The class with the highest score is the prediction result.

[0104] Result analysis:

[0105] Table 1 is a comparison of the performance of the method used in this invention and other recognition methods on the UCF101 dataset.

[0106] Table 1 Comparison of the method used in this invention and other methods

[0107]

[0108] From the table we can see that compared with the classic network and the relatively new network, our method shows better recognition performance. Among them, whether it is MSF or FSAM module, it can improve the network performance separately. After applying the method in this paper to the R3D network structure, not only the parameter quantity is reduced by 34%, but also the recognition accuracy on the UCF101 dataset is improved by 7.2%. In order to further improve the persuasiveness of the experimental results, we apply the method in this paper to the C3D network model for generalization ability test, and the test results show that the recognition accuracy is improved by 8.9%, which fully embodies the universality of the method in this paper. It can be speculated that our method can be flexibly embedded into any 3DCNN architecture to reduce the parameter quantity while improving the recognition accuracy.

[0109] While embodiments of the present application have been shown and described with reference to particular embodiments thereof, it will be understood by those skilled in the art that various changes in form and details can be made therein without departing from the spirit and scope of the application. The scope of the application is defined by the appended claims and their equivalents.

Claims

1. A method for action recognition based on multi-level spatio-temporal feature fusion with attention, characterized in that, The method comprises the following steps: (1) obtaining a video to be recognized, and performing a preprocessing operation on the video to obtain an image; (2) inputting the preprocessed image into a human behavior recognition network combining attention-based multi-level spatio-temporal feature fusion for convolutional neural network training; (3) outputting a corresponding human action classification result; The human behavior recognition network combining attention-based multi-level spatio-temporal feature fusion comprises an input layer, a 3D convolutional layer, a continuously stacked multi-level spatio-temporal feature fusion basic module, a multi-frequency channel and spatio-temporal attention module, a pooling layer, a classification layer and an output layer; The construction process of the human behavior recognition network combining attention-based multi-level spatio-temporal feature fusion comprises: constructing the multi-level spatio-temporal feature fusion basic module and the multi-frequency channel and spatio-temporal attention module; The multi-level spatio-temporal feature fusion basic module is composed of a three-dimensional convolutional layer and a multi-level spatio-temporal feature decomposition layer, two 3*3 two-dimensional convolutional kernels are first expanded in three dimensions to two 3*3*3 three-dimensional convolutional kernels, and then the second 3*3*3 convolutional kernel is decomposed in multiple spatio-temporal features; The multi-frequency channel and spatio-temporal attention module comprises two parts: a multi-frequency channel attention module and a spatio-temporal attention module; The multi-frequency channel attention module and the spatio-temporal attention module enable the feature map to execute attention mechanisms in the channel and spatio-temporal dimensions in turn, and finally output a result after feature optimization; The multi-frequency channel and spatio-temporal attention module is divided into two parts: a multi-frequency channel attention module and a spatio-temporal attention module. The multi-frequency channel attention module specifically comprises: firstly, input feature maps are divided into n parts along the channel dimension, and the feature maps of each part can be represented as [X 0 , X 1 , ···,X n-1 ], wherein A two-dimensional frequency component is allocated to each part, and finally, a multi-frequency spectrum preprocessing vector can be obtained by splicing. Freq = cat([Freq 0 , Freq 1 ,..., Freq n-1 ]), wherein The multi-frequency channel attention framework formula is: Fc_att = sigmoid(fc(Freq)) The spatio-temporal attention module specifically comprises: for an input feature First, two 3D features are generated through maximum pooling and average pooling in the channel dimension respectively And Then, a 3D convolution is performed to generate a 3D spatio-temporal attention feature, ST_att = sigmoid(f 7×7×7 (cat[MaxPool(F), AvgPool(F)]) The formula of the spatio-temporal attention framework is as shown in the above formula.

2. The multi-level spatio-temporal feature fusion behavior recognition method with attention according to claim 1, characterized in that, The preprocessing operation comprises: frame-by-frame decomposition of the video into images while keeping the structure unchanged, and then random extraction of interval video frames; scaling the pictures and randomly cropping the pictures to a specified size as input.

3. The method of claim 1, wherein the method of recognizing behaviors by fusing multi-level spatio-temporal features with attention comprises: The process of the multi-level spatio-temporal feature decomposition comprises: Splitting the C channels of input feature X evenly into S feature subsets, denoted as x i where i∈{1, 2, …, S}, the channel number C' is the input feature channel number That is where x1subset is not transformed and directly output as y1; other subsets x i Each branch corresponds to a class residual structure, and each branch has a 3D convolution with a 3x3x3 convolution kernel, denoted as f 3×3×3 () i The generated feature map after 3D convolution is denoted as y i , y i The expression is: In the process of multi-level spatio-temporal feature decomposition, y i will be separated into two groups of sub-feature maps y i1 and y i2 again by channel, where y i1 is directly spliced to the final output, y i2 is directly spliced to the next group of feature maps x i+1 ; the spatio-temporal feature expression of the last output of the multi-level spatio-temporal feature decomposition layer is: Y = cat [x1, y 22 , y 32 , …, y (i-1)2 , y i ]; and the total output of the multi-level spatio-temporal feature fusion basic module is: Output = X + Y + Input.

4. The multi-level spatio-temporal feature fusion behavior recognition method with attention according to claim 1, characterized in that, Given features As the input of the multi-frequency channel and spatial-temporal attention module, the corresponding multi-frequency channel attention features are output, and then broadcast along the time and spatial dimensions. Finally, the attention features are output through the spatial-temporal attention module. The output of the entire attention module is as shown in the above formula.

5. The multi-level spatio-temporal feature fusion behavior recognition method with attention according to claim 1, characterized in that, The pictures of the continuous frames are input into the behavior recognition network, and after passing through a plurality of multi-level spatio-temporal feature fusion basic modules and multi-frequency channel and spatio-temporal attention modules, the last 3D average pooling layer, and then the softmax outputs the final behavior category.

6. The multi-level spatio-temporal feature fusion behavior recognition system that binds attention, characterized in that, The method comprises a preprocessing module, a network construction module, a network training module and an output module; The preprocessing module comprises obtaining a video to be recognized, and performing a preprocessing operation on the video to obtain an image; The network construction module comprises constructing a human behavior recognition network combining attention-based multi-level spatio-temporal feature fusion; The network training module comprises inputting the preprocessed image into the human behavior recognition network combining attention-based multi-level spatio-temporal feature fusion for convolutional neural network training; The output module comprises outputting a corresponding human action classification result; The human behavior recognition network combining attention-based multi-level spatio-temporal feature fusion comprises an input layer, a 3D convolutional layer, a continuously stacked multi-level spatio-temporal feature fusion basic module, a multi-frequency channel and spatio-temporal attention module, a pooling layer, a classification layer and an output layer; and the construction process of the human behavior recognition network combining attention-based multi-level spatio-temporal feature fusion comprises: The multi-level space-time feature fusion basic module and the multi-frequency channel and space-time attention module are constructed. The multi-level space-time feature fusion basic module is composed of a three-dimensional convolution layer and a multi-level space-time feature decomposition layer, two parts, first, two 3*3 two-dimensional convolution kernels are expanded in three dimensions, converted into two 3*3*3 three-dimensional convolution kernels in cascade, and then the second 3*3*3 convolution kernel is decomposed in multiple space-time features; The multi-frequency channel and space-time attention module includes two parts: a multi-frequency channel attention module and a space-time attention module. The multi-frequency channel attention module and the space-time attention module make the feature map execute the attention mechanism in the channel and space-time dimensions in turn, and finally output the result after feature optimization. The multi-frequency channel and space-time attention module is divided into two parts: a multi-frequency channel attention module and a space-time attention module. The multi-frequency channel attention module specifically comprises: firstly, input feature maps are divided into n parts along the channel dimension, and the feature maps of each part can be represented as [X 0 , X 1 , ... , X n-1 ], wherein A two-dimensional frequency component is allocated to each part, and finally, a multi-frequency spectrum preprocessing vector can be obtained by splicing. Freq = cat([Freq 0 , Freq 1 ,..., Freq n-1 ]), wherein The multi-frequency channel attention framework formula is: Fc_att=sigmoid(fc(Freq)) The spatio-temporal attention module specifically comprises: for an input feature First, two 3D features are generated through maximum pooling and average pooling in the channel dimension respectively And Then, a 3D convolution is performed to generate a 3D spatio-temporal attention feature, ST_att = sigmoid(f 7×7×7 (cat[MaxPool(F), AvgPool(F)]) The space-time attention framework formula is the above formula.

Images