A human behavior recognition method based on an improved deep residual network

Abstract

The application discloses a human behavior recognition method based on an improved deep residual network. The method comprises a training stage and a testing stage. In the training stage, an image dataset is acquired by using a camera, original frames of a training video are extracted by using a sparse sampling strategy of segmented sampling, and the extracted original frames are sent to an improved deep residual network with a channel attention mechanism for training. In the testing stage, original frames of a testing video are extracted, the original frames are sent to the improved deep residual network model obtained through training, and the final behavior category is determined by using a softmax classifier. According to the method, important features can be enhanced and unimportant features can be inhibited according to the importance of feature channels, so that the feature extraction capability of the model for input data is improved. The network has high running speed, high behavior recognition accuracy and low network complexity, and can effectively extract features and has good performance in some complex actions and difficult-to-recognize actions.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer, especially the technical field of behavior recognition, and relates to a method for recognizing human behavior in a video, in particular a behavior recognition method based on an improved deep residual network. BACKGROUND

[0002] Behavior recognition is a basic direction in video understanding technology, and its core technology is to enable the computer to classify the behavior of the target in the video by learning the feature information of the sequence frame image, so as to achieve the purpose of recognition, and is commonly used in human-computer interaction and intelligent robot monitoring. The behavior recognition of the target in the video data usually has time dependence, and not only includes the spatial information in each frame image, but also contains the time information between frames, such as the behaviors of the old people falling down and carrying objects.

[0003] In recent years, the mainstream method of behavior recognition technology is still based on two-dimensional convolution network model and three-dimensional convolution network model. Simonyan K, et al. (Simonyan K, Zisserman A. Two-stream convolutional networks for action recognition in videos[J]. Advances in neural information processing systems, 2014, 27.) uses two-dimensional convolution to construct a double-flow network for behavior recognition using optical flow and RGB pictures. Tran D, et al. (Tran D, Bourdev L, Fergus R, et al. Learning spatiotemporal features with 3d convolutional networks[C] / / Proceedings of the IEEE international conference on computer vision. 2015: 4489-4497.) expands two-dimensional convolution into three-dimensional convolution in the time dimension to construct a C3D network for behavior recognition. Pan T, et al. (Pan T, Song Y, Yang T, et al. Video moco: Contrastive video representation learning with temporally adversarial examples[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 11205-11214.) re-constructs a two-dimensional residual network using three-dimensional convolution into a 3D-Resnet18 network for behavior recognition. Pan T, et al. (Pan T, Song Y, Yang T, et al. Video moco: Contrastive video representation learning with temporally adversarial examples[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 11205-11214.) replaces three-dimensional convolution with one-dimensional time convolution and two-dimensional spatial convolution to construct an R(2+1)D network for behavior recognition.Qiu Z et al (Qiu Z, Yao T, Mei T. Learning spatio-temporal representation with pseudo-3d residual networks [C] / / proceedings of the IEEE International Conference on Computer Vision. 2017:5533-5541.) explored several different combinations of one-dimensional time convolution and two-dimensional space convolution instead of three-dimensional convolution, and constructed a P3D network for behavior recognition. Kopuklu O et al (Kopuklu O, Kose N, Gunduz A, et al. Resource efficient 3d convolutional neural networks [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision Workshops. 2019:0-0.) converted various well-known resource-efficient two-dimensional neural networks into three-dimensional neural networks and constructed a 3D-SqueezeNet network for behavior recognition.

[0004] At present, most behavior recognition technologies are limited to practical scenarios due to their inherent heavy computational load, which slows down the model parameter derivation speed. Therefore, there is an urgent need to propose a behavior recognition scheme that can improve accuracy and reduce computational cost, improve the feature expression ability of the model to sequence frame images, and enhance the performance of behavior recognition. SUMMARY

[0005] The purpose of the present application is to provide a human behavior recognition method based on an improved deep residual network to better express features and improve the accuracy of behavior recognition.

[0006] A human behavior recognition method based on an improved deep residual network, characterized in that it comprises the following steps:

[0007] Step 1: Use a camera to obtain a human behavior video, build a video label into a folder, and place the corresponding human behavior video under the corresponding label folder to build a human behavior recognition dataset;

[0008] Step 2: Divide the human behavior recognition dataset obtained in step 1 into a training dataset, a test dataset and a validation dataset; divide the video into T segments, denoted as S={S1, S2,..., ST}, assuming the video length is A, then the length of each segment is T} denotes rounding down; if A / T is not an integer, first divide the video into A\T segments, \ denotes remainder, randomly drop one frame for each of the A\T segments, then divide the remaining video frames into T segments, sample one video frame for each segment, and normalize the image size to MxN, M∈(224, 1080) and N∈(224, 1080) represent the height and width of the image respectively, to form a training set, a test set and a validation set;

[0009] Step 3: Constructing a feature fusion module based on an attention mechanism, the feature fusion module based on the attention mechanism includes the following structure:

[0010] First, use global average pooling to reduce the dimensionality of the two features that need to be fused, directly add the reduced features to form an overall feature, then use one-dimensional convolution with an adaptive convolution kernel for feature extraction, and finally use a Sigmoid activation function layer to learn two parameters a0 and a1 = 1-a0, multiply a0 and a1 with the two input features respectively and then concatenate them in the channel dimension to form the fused feature; the input of this module is two features of video samples X0 and X1 with n frames f, f∈(1, 64], X0 and X1 are both cxf a×b matrices, a∈(224, 1080), b∈(224, 1080), c is the number of channels of the video image, c∈{1, 3}; use global average pooling for dimensionality reduction to concentrate global information while reducing computational complexity, and use one-dimensional convolution with an adaptive selection of convolution kernel size to realize local interaction of channels, the formula for adaptive selection of convolution kernel size is:

[0011]

[0012] where C is the number of feature channels, K is the size of the adaptive selection of convolution kernel, the features after one-dimensional convolution are used to generate weight parameters a0 and a1 using the Sigmod activation function, and the weight parameter calculation formula is:

[0013] a0=σ(C1D k (GAP(X1)+GAP(X0)))

[0014] a1=1-a0

[0015] where σ(·) is the Sigmod activation function, C1D k represents one-dimensional convolution operation with adaptive convolution kernel, GAP is global average pooling operation, the weight parameters are multiplied with the input features X0 and X1 respectively and then concatenated in the channel dimension to obtain the output feature H, and the formula of the output H is:

[0016] H=Concate(X0⊙a0,X1⊙a1)

[0017] Wherein represents the channel level multiplication, H is the fused feature, Concate represents the splicing operation in the channel dimension, and the operation formula of Concate is as follows:

[0018]

[0019] In the formula, X and Y are feature maps of the channel input and X=Y, k represents the number of input feature maps X, d is the number of channels of the input feature map Y, i [1, k], j [1, d];

[0020] Step 4: using step 3 to construct the feature fusion module based on attention mechanism to improve two kinds of deep residual module Identity Block and Conv Block, Identity Block input dimension and output dimension are the same, can be connected in series, the effect is to deepen the network, Conv Block input and output channel number is different, cannot be connected in series, the effect is to change the dimension of the network;

[0021] (1) the improved deep residual module Identity Block includes the following structure:

[0022] The first layer contains a 3*3*3 convolution kernel, the channel number of this layer is half of the input feature channel number, the step is 1*1*1, the zero padding width is 1*1*1, the batch normalization BN processing and the linear unit Relu processing;

[0023] The second layer contains a 3*3*3 group convolution kernel, the channel number of this layer is half of the input feature channel number, the group number is half of the input feature channel number, the step is 1*1*1, the zero padding width is 1*1*1, the batch normalization BN processing;

[0024] The second branch:

[0025] The first layer contains a 3*3*3 convolution kernel, the channel number of this layer is half of the input feature channel number, the step is 1*1*1, the zero padding width is 1*1*1, the batch normalization BN processing;

[0026] The output features of the first branch and the second branch are input into the feature fusion module based on attention mechanism to obtain the output features;

[0027] (2) the improved deep residual module Conv Block includes the following structure:

[0028] The first branch:

[0029] The first layer contains a 3*3*3 convolution kernel, the channel number of this layer is half of the input feature channel number, the step is 2*2*2, the zero padding width is 1*1*1, the batch normalization BN processing and the linear unit Relu processing;

[0030] The second layer comprises a 3x3x3 grouped convolution kernel, the number of channels of the layer is half of the number of input feature channels, the number of groups is half of the number of input feature channels, the step is 1x1x1, the zero padding width is 1x1x1, and batch normalization (BN) processing is performed;

[0031] The second branch comprises:

[0032] The first layer comprises a 3x3x3 convolution kernel, the number of channels of the layer is half of the number of input feature channels, the step is 2x2x2, the zero padding width is 1x1x1, and batch normalization (BN) processing is performed;

[0033] The output features of the first branch and the second branch are input into an attention mechanism-based feature fusion module to obtain output features;

[0034] Step 5: constructing an improved deep residual network, wherein the improved deep residual network comprises the following structure:

[0035] (1) The first layer comprises a 3x7x7 convolution kernel, the number of channels of the layer is 64, the step is 1x2x2, the zero padding width is 1x3x3, batch normalization (BN) processing, linear unit (Relu) processing, and a maximum pooling layer, the size of the kernel of the maximum pooling layer is 3x3x3, the step is 2x2x2, and the zero padding width is 1x1x1;

[0036] (2) The second layer comprises two identical improved residual modules (Identity Block), and the two Identity Blocks are connected in series, and the number of input channels and the number of output channels are both 64;

[0037] (3) The third layer comprises an improved Conv Block and an Identity Block, the Conv Block and the Identity Block are connected in series, the number of input channels is 64, and the number of output channels is 128;

[0038] (4) The fourth layer comprises an improved Conv Block and an Identity Block, the number of input channels is 128, and the number of output channels is 256;

[0039] (5) The fifth layer comprises an improved Conv Block and an Identity Block, the number of input channels is 256, and the number of output channels is 512;

[0040] (6) The fifth layer is connected with an average pooling layer, after the average pooling layer, a fully connected layer with a channel number of 512 is used, and a softmax activation function layer is connected;

[0041] Step 6: The training set and test set obtained in step 2 are input into the improved deep residual network constructed in step 5 for training, the training input is a video sample X3 of n frames and f, f e (1, 64], X3 is a a x b matrix of c x f, a e (224, 1080), b e (224, 1080), c is the number of channels of the video image, c e {1, 3}; the learning rate is set to L, L e (0, 1), the learning rate changes to 1 / O of the original value every E training period, the batch size is set to J, E, O and J are positive integers, the cross entropy loss function is used to evaluate the difference between the predicted value and the true value, the stochastic gradient descent algorithm SGD is used to optimize the parameters, the impulse is set to S, S e (0, 1), the weight decay index is A, A e (0, 1); the validation set is used to verify the training process, and the accuracy Acc of the validation set is calculated i , when the accuracy Acc i stops rising, Acc i e (0, 1), i represents the training round, i e (1, N], N is the maximum training round, which is a positive integer; the following formula is used to evaluate the model recognition accuracy Acc i :

[0042]

[0043] In the formula, TP is the number of correct positive samples predicted by the model, TN is the number of correct negative samples predicted by the model, FP is the number of incorrect positive samples predicted by the model, FN is the number of incorrect negative samples predicted by the model, the parameter model with the highest accuracy is saved as P, P i represents the network parameters at the i-th training, and Acc represents the highest accuracy; the update of P is according to the following formula:

[0044]

[0045] Step 7: The deployment parameter P saved in step 6 is used as the network parameter of the improved deep residual network, and the improved deep residual network with network parameter P is deployed on the recognition terminal, and the terminal performs human behavior recognition on the input video to obtain the score V of the video belonging to each type of human behavior, V is a q x s matrix, V is predicted by the maxout function to obtain the model recognition result W, W is a vector of length q, and the calculation formula of W is as follows:

[0046]

[0047] In the formula, W n represents the n-th element of the model recognition result W, x qs represents q x s elements of the prediction matrix V, x qs e (0, 1], j nj represents the column number of the largest element in the nth row of the prediction matrix V n s represents the number of human behavior categories in the video sample in the training sample X3, W n The nth element of the recognition result W represents the human behavior classification.

[0048] The innovation of the present application is:

[0049] (1) Compared with the 3D-Resnet18 method, the residual module of the above method uses element addition to directly perform feature fusion, the present application uses a channel splicing method instead of an element addition method to perform feature fusion, and a feature fusion module based on an attention mechanism is used to supervise feature fusion, thereby reducing network complexity while improving network running speed and recognition accuracy.

[0050] (2) Compared with the I3D, P3D, C3D and other classic abnormal behavior recognition methods, the three neural networks all use three-dimensional convolution as a feature extractor, the present application also uses three-dimensional convolution as a feature extractor, but has lower network complexity, faster running speed and higher recognition accuracy.

[0051] Advantages:

[0052] Compared with the prior art, the present application has the following advantages:

[0053] Based on the channel attention mechanism, the improved residual module using the feature fusion module based on the attention mechanism overcomes the shortcomings of insufficient feature fusion of the residual module, overcomes the shortcomings of only considering single features without considering overall features by introducing the attention mechanism, reduces the complexity of the network, improves the running speed and recognition accuracy of the network, and has wide application value. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 The flowchart for human behavior recognition is composed of four parts of original video data set, data preprocessing, the algorithm of the present application and behavior recognition classification, represents the processing flow of human behavior recognition, and the present application improves the behavior recognition network;

[0055] Figure 2 The feature fusion module based on the attention mechanism represents the weight proportion allocated by the feature fusion;

[0056] Figure 3 The improved deep residual module Identity Block has the same input dimension and output dimension, can be connected in series, and is used for deepening the network;

[0057] Figure 4The improved deep residual module Conv Block has different input and output dimensions and cannot be concatenated; it is used to change the dimension of the network output features.

[0058] Figure 5 To improve deep residual networks for human behavior recognition; Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0060] Example:

[0061] This example uses the UCF101 dataset, a motion recognition dataset for real-world motion videos, collected from YouTube, and provides 101 motion categories.

[0062] 1. A method for human behavior recognition based on an improved deep residual network, characterized by the following steps:

[0063] Step 1: Use a camera to acquire videos of human behavior, create folders by tagging the videos, and place the corresponding videos of human behavior in the corresponding tag folders to build a human behavior recognition dataset;

[0064] Step 2: Divide the human behavior recognition dataset obtained in Step 1 into training, testing, and validation datasets in a 16:4:5 ratio; divide the video into K = 16 segments, denoted as S = {S1, S2, ..., S...} K Assuming the video length is A, then the length of each segment is... The expression indicates rounding down; if A / K is not an integer, the video is first divided into A\K segments, where \ indicates modulo, and one frame is randomly lost from each of the A\K segments. Then, the remaining video frames are divided into K segments, and one video frame is sampled from each segment. The image size is normalized to M×N, where M=224 and N=224 represent the height and width of the image, respectively, forming the training set, test set, and validation set.

[0065] Step 3: Construct an attention-based feature fusion module, which includes the following structure:

[0066] First, the two features that need to be fused are reduced in dimension using global average pooling, and the reduced features are directly added to form an overall feature, then one-dimensional convolution with adaptive convolution kernel is used for feature extraction, finally sigmoid activation function layer is used to learn two parameters a0 and a1=1-a0, multiply a0 and a1 with two input features respectively and then concatenate in the channel dimension to form the fused feature; the input of the module is two features of video samples X0 and X1 with n frames f, f∈(1, 64], X0 and X1 are both c×f a×b matrices, a∈(224, 1080), b∈(224, 1080), c is the channel number of video image, c∈{1,3}; global average pooling is used for dimension reduction to reduce the amount of calculation while concentrating global information, one-dimensional convolution with adaptive selection of convolution kernel size is used to realize the local interaction of channels, the formula for adaptive selection of convolution kernel size is:

[0067]

[0068] where C is the number of feature channels, K is the size of the adaptive selection of convolution kernel, the features after one-dimensional convolution are used to generate weight parameters a0 and a1 using Sigmod activation function, the weight parameter calculation formula is:

[0069] a0=σ(C1D k (GAP(X1)+GAP(X0)))

[0070] a1=1-a0

[0071] where σ(·) is the Sigmod activation function, C1D k represents one-dimensional convolution operation with adaptive convolution kernel, GAP is global average pooling operation, the weight parameters are multiplied with input features X0 and X1 respectively and then concatenated in the channel dimension to obtain output feature H, the formula of output H is:

[0072] H=Concate(X0⊙a0,X1⊙a1)

[0073] where ⊙ represents channel-level multiplication, H is the fused feature, Concate represents concatenation operation in the channel dimension, the operation formula of Concate is as follows:

[0074]

[0075] where X and Y are feature maps input in the channel, X=Y, k represents the number of channels of input feature map X, d is the number of channels of input feature map Y, i∈[1,k], j∈[1,d];

[0076] Step 4: Two deep residual modules Identity Block and Conv Block are improved by using step 3 to construct an attention mechanism-based feature fusion module. Identity Block has the same input and output dimensions and can be connected in series. Its role is to deepen the network. Conv Block has different input and output channel numbers and cannot be connected in series. Its role is to change the dimension of the network.

[0077] (1) The improved deep residual module Identity Block includes the following structure:

[0078] The first layer contains a 3x3x3 convolution kernel. The channel number of this layer is half of the input feature channel number, the step is 1x1x1, the zero padding width is 1x1x1, and the batch normalization BN processing and linear unit Relu processing are performed.

[0079] The second layer contains a 3x3x3 grouped convolution kernel. The channel number of this layer is half of the input feature channel number, the group number is half of the input feature channel number, the step is 1x1x1, the zero padding width is 1x1x1, and the batch normalization BN processing is performed.

[0080] Second branch:

[0081] The first layer contains a 3x3x3 convolution kernel. The channel number of this layer is half of the input feature channel number, the step is 1x1x1, the zero padding width is 1x1x1, and the batch normalization BN processing is performed.

[0082] The output features of the first branch and the second branch are input into the attention mechanism-based feature fusion module to obtain the output features.

[0083] (3) The improved deep residual module Conv Block includes the following structure:

[0084] First branch:

[0085] The first layer contains a 3x3x3 convolution kernel. The channel number of this layer is half of the input feature channel number, the step is 2x2x2, the zero padding width is 1x1x1, and the batch normalization BN processing and linear unit Relu processing are performed.

[0086] The second layer contains a 3x3x3 grouped convolution kernel. The channel number of this layer is half of the input feature channel number, the group number is half of the input feature channel number, the step is 1x1x1, the zero padding width is 1x1x1, and the batch normalization BN processing is performed.

[0087] Second branch:

[0088] The first layer contains a 3x3x3 convolution kernel. The channel number of this layer is half of the input feature channel number, the step is 2x2x2, the zero padding width is 1x1x1, and the batch normalization BN processing is performed.

[0089] The output features of the first branch and the second branch are input into an attention mechanism-based feature fusion module to obtain output features;

[0090] Step 5: constructing an improved deep residual network, which comprises the following structure:

[0091] (1) The first layer contains a 3x7x7 convolution kernel, the channel number of which is 64, the step is 1x2x2, the zero padding width is 1x3x3, batch normalization (BN) processing, linear unit (Relu) processing and a maximum pooling layer, the size of the kernel of the maximum pooling layer is 3x3x3, the step is 2x2x2, and the zero padding width is 1x1x1;

[0092] (2) The second layer contains two identical improved residual modules Identity Block, and the two Identity Blocks are connected in series, with an input channel number and an output channel number of 64;

[0093] (3) The third layer contains an improved Conv Block and an Identity Block, and the Conv Block and the Identity Block are connected in series, with an input channel number of 64 and an output channel number of 128;

[0094] (4) The fourth layer contains an improved Conv Block and an Identity Block, with an input channel number of 128 and an output channel number of 256;

[0095] (5) The fifth layer contains an improved Conv Block and an Identity Block, with an input channel number of 256 and an output channel number of 512;

[0096] (6) The fifth layer is followed by an average pooling layer, and after the average pooling layer, a fully connected layer with a channel number of 512 is used, followed by a softmax activation function layer;

[0097] Step 6: The training set and the test set obtained in step 2 are input into the improved deep residual network constructed in step 5 for training, and 8-frame video samples X3 with a frame number of 16 are input for training; X3 is 3x16 224x224 matrices; the learning rate is set to lr=0.01, the learning rate is changed to 1 / L of the original value every H training period, the batch size is set to B, H=10, L=10, B=8, the cross-entropy loss function is used to evaluate the difference between the predicted value and the true value, the stochastic gradient descent (SGD) algorithm is used to optimize the parameters, the impulse is set to M=0.9, the weight decay index is set to A=0.0005, the validation set is used to verify the training process, and the accuracy Acc of the validation set is calculated i, training to the accuracy Acc i When no longer rising, Acc i ∈(0, 1), i represents the training round, i∈(1, N]; N is the maximum training round, N=50, the model recognition accuracy Acc is evaluated using the following formula i :

[0098]

[0099] In the formula, TP is the number of correct predictions of positive samples by the model, TN is the number of correct predictions of negative samples by the model, FP is the number of incorrect predictions of positive samples by the model, FN is the number of incorrect predictions of negative samples by the model, the parameter model with the highest accuracy is saved as P, P i represents the network parameters at the i-th training round, Acc represents the highest accuracy, and the update of P is according to the following formula:

[0100]

[0101] Step 7: using the deployment parameters P saved in step 6 as the network parameters of the improved deep residual network, and deploying the improved deep residual network with network parameters P to the recognition terminal, while the terminal performs human behavior recognition on an input video to obtain the score V of the video belonging to each type of human behavior, V is a 1x101 matrix, and the highest score in V is the behavior recognition classification of the sample.

[0102] The following table shows the accuracy of the network model proposed in the patent compared with other network models.

[0103] Table 1 Comparison of recognition accuracy of the method of the present application and different models

[0104] Tab.1 Our method improves the recognition rate of various models

[0105] Model UCF101 dataset accuracy (%) Computational volume (GFloat) Parameter volume (Mbit) C3D 82.3 64.23G 78.41M 3D-Resnet18 74.1 45.89G 33.18M The algorithm of the present application 90.74 7.42G 9.88M R(2+1)D 78.7 40.51G 33.3M P3D 88.6 28.6G 98M 3D-SqueezeNet 74.94 18.4G 2.15M

Claims

1. A human action recognition method based on an improved deep residual network, characterized in that Comprising the following steps: Step 1: acquire human behavior video using camera, build video label into folder, place corresponding human behavior video under corresponding label folder, and build human behavior recognition dataset; Step 2: divide the human behavior recognition dataset obtained in step 1 into a training dataset, a test dataset and a validation dataset; divide the video into T segments, denoted as S = {S1, S2,..., ST}, assuming that the length of the video is A, then the length of each segment is T represents rounding down; if A / T is not an integer, first divide the video into A\T segments, \ represents the remainder, randomly lose a frame for each of the A\T segments, then divide the remaining video frames into T segments, sample one video frame for each segment, and normalize the image size to MxN, M∈(224, 1080) and N∈(224, 1080) represent the height and width of the image respectively, to form the training set, the test set and the validation set;​ Step 3: build a feature fusion module based on attention mechanism, which comprises the following structure: First, use global average pooling to reduce the dimension of the two features that need to be fused, add the reduced features directly to form an overall feature, then use one-dimensional convolution with adaptive convolution kernel for feature extraction, and finally use the Sigmoid activation function layer to learn two parameters a0 and a1 = 1-a0, multiply a0 and a1 with the two input features respectively, and then concatenate them in the channel dimension to form the fused feature; The input of this module is two features of video samples X0 and X1 with n frames f, f∈(1,64], X0 and X1 are both c×f a×b matrices, a∈(224,1080), b∈(224,1080), c is the number of channels of video images, c∈{1,3}; Use global average pooling to reduce dimension and concentrate global information while reducing computation, and use one-dimensional convolution with adaptive selection of convolution kernel size to realize local interaction of channels, the formula for adaptive selection of convolution kernel size is: Where C is the number of feature channels, K is the size of the adaptive selection of convolution kernel, the features after one-dimensional convolution are used to generate weight parameters a0 and a1 using Sigmod activation function, and the weight parameter calculation formula is: a0 = σ(C1D k (GAP(X1)+GAP(X0))) a1=1-a0 where σ(·) is a Sigmod activation function, C1D k represents a one-dimensional convolution operation with adaptive convolution kernel, GAP is a global average pooling operation, the weight parameters are multiplied with the input features X0 and X1 respectively, and then the output features H are obtained by concatenating in the channel dimension, and the formula of the output H is: H=Concate(X0⊙a0,X1⊙a1) Where represents channel-level multiplication, H is the fused feature, and Concate represents the concatenation operation in the channel dimension, and the operation formula of Concate is as follows: Where X and Y are feature maps input to the channel, X=Y, k represents the number of input feature map X channels, d is the number of input feature map Y channels, i∈[1,k], j∈[1,d]; Step 4: use the feature fusion module based on attention mechanism built in step 3 to improve two deep residual modules Identity Block and Conv Block, Identity Block has the same input and output dimensions and can be concatenated, which deepens the network, Conv Block has different input and output channel numbers and cannot be concatenated continuously, which changes the dimension of the network; (1) The improved deep residual module Identity Block comprises the following structure: First branch: The first layer contains a 3x3x3 convolution kernel, the channel number of this layer is half of the input feature channel number, the step is 1x1x1, the zero padding width is 1x1x1, the batch normalization BN processing and linear unit Relu processing; The second layer contains a 3x3x3 grouped convolution kernel, the channel number of this layer is half of the input feature channel number, the group number is half of the input feature channel number, the step is 1x1x1, the zero padding width is 1x1x1, and the batch normalization BN processing is performed; Second branch: The first layer comprises a 3*3*3 convolution kernel, the number of channels of the layer is half of the number of input feature channels, the step is 1*1*1, the zero padding width is 1*1*1, and batch normalization (BN) processing is performed; The output features of the first branch and the second branch are input into a feature fusion module based on an attention mechanism to obtain output features; (2) The improved deep residual module Conv Block comprises the following structure: The first branch: The first layer comprises a 3*3*3 convolution kernel, the number of channels of the layer is half of the number of input feature channels, the step is 2*2*2, the zero padding width is 1*1*1, batch normalization (BN) processing and linear unit (Relu) processing are performed; The second layer comprises a 3*3*3 grouped convolution kernel, the number of channels of the layer is half of the number of input feature channels, the number of groups is half of the number of input feature channels, the step is 1*1*1, the zero padding width is 1*1*1, and batch normalization (BN) processing is performed; The second branch: The first layer comprises a 3*3*3 convolution kernel, the number of channels of the layer is half of the number of input feature channels, the step is 2*2*2, the zero padding width is 1*1*1, and batch normalization (BN) processing is performed; The output features of the first branch and the second branch are input into a feature fusion module based on an attention mechanism to obtain output features; Step 5: constructing an improved deep residual network: the improved deep residual network comprises the following structure: (1) The first layer comprises a 3*7*7 convolution kernel, the number of channels of the layer is 64, the step is 1*2*2, the zero padding width is 1*3*3, batch normalization (BN) processing, linear unit (Relu) processing and a maximum pooling layer, the size of the kernel of the maximum pooling layer is 3*3*3, the step is 2*2*2, and the zero padding width is 1*1*1; (2) The second layer comprises two identical improved residual modules Identity Block, and the two Identity Blocks are connected in series, and the input channel number and the output channel number are both 64; (3) The third layer comprises an improved Conv Block and an Identity Block, and the Conv Block and the Identity Block are connected in series, the input channel number is 64, and the output channel number is 128; (4) The fourth layer comprises an improved Conv Block and an Identity Block, and the Conv Block and the Identity Block are connected in series, the input channel number is 128, and the output channel number is 256; (5) The fifth layer comprises an improved Conv Block and an Identity Block, and the Conv Block and the Identity Block are connected in series, the input channel number is 256, and the output channel number is 512; (6) The fifth layer is connected with an average pooling layer, after the average pooling layer, a fully connected layer with a channel number of 512 is used, and a softmax activation function layer is connected after the fully connected layer; Step 6: The training set and test set obtained in step 2 are input into the improved deep residual network constructed in step 5 for training, the training input is a video sample X3 of n frames f, f ∈ (1, 64], X3 is a a x b matrix of c x f, a ∈ (224, 1080), b ∈ (224, 1080), c is the number of channels of the video image, c ∈ {1, 3}; the learning rate is set to L, L ∈ (0, 1), the learning rate changes to 1 / O of the original value every E training period, the batch size is set to J, E, O and J are all positive integers, the cross-entropy loss function is used to evaluate the difference between the predicted value and the true value, the stochastic gradient descent algorithm SGD is used to optimize the parameters, the impulse is set to S, S ∈ (0, 1), the weight decay index is t, t ∈ (0, 1); the validation set is used to verify the training process, and the accuracy Acc of the validation set is calculated i , when the accuracy Acc i does not rise any more, Acc i ∈ (0, 1), i represents the training round, i ∈ (1, N], N is the maximum training round, which is a positive integer; The model recognition accuracy Acc is evaluated using the following formula i : In the formula, TP is the number of correct positive samples predicted by the model, TN is the number of correct negative samples predicted by the model, FP is the number of incorrect positive samples predicted by the model, FN is the number of incorrect negative samples predicted by the model, the parameter model with the highest accuracy is saved as P, P i represents the network parameters at the i-th round of training, Acc represents the highest accuracy saved; the update of P is according to the following formula: Step 7: using the deployment parameters P saved in step 6 as network parameters of the improved deep residual network, and deploying the improved deep residual network with network parameters P to the recognition terminal, while the terminal performs human behavior recognition on the input video to obtain the score V of the video belonging to each type of human behavior, V is a qxs matrix, and the model recognition result W is obtained by predicting V through a maxout function, W is a vector with a length of q, and the calculation formula of W is as follows: where x qs represents the q x s elements of the prediction matrix V, x qs ∈ (0, 1], j n represents the column number of the largest element in the nth row of the prediction matrix V, j n ∈ [1, s], s represents the number of human behavior categories in the video samples in the training sample X3, W n represents the human behavior category of the nth element of the recognition result W.

Images