A feature fusion behavior recognition method and system based on deformable deep convolution and attention mechanism

Abstract

The present invention relates to a feature fusion behavior recognition method and system based on deformable deep convolution and attention mechanism; comprising: pre-processing the training set of the human behavior video data set and inputting it into the neural network model for training, pre-processing the test set and inputting it into the trained neural network model for behavior detection, and obtaining the detection accuracy of the neural network model; inputting the human behavior video data to be identified into the tested neural network model for behavior recognition. Compared with the traditional neural network model based on 3D convolution, it has high computational efficiency and low hardware requirements, can better focus on the key features of the target, make more full use of the shallow and deep information of the network, and greatly improve the accuracy of various behavior recognition. The accuracy rate on the HMDB51 data set reaches 80.59%, and the accuracy rate on the UCF101 data set reaches 97.52%.

Description

Technical Field

[0001] The present invention relates to a feature fusion behavior recognition method and system based on deformable deep convolution and attention mechanism, belonging to the field of artificial intelligence and behavior recognition. Background Art

[0002] With the development of science and technology and the progress of society, people's living standards have gradually improved, and security issues have become the focus of people's attention. Monitoring scenarios are constantly expanding and covering more and more areas. For example, hospitals, schools, parking lots, shopping malls, banks, intersections, and mines are equipped with monitoring equipment to record what happens, but the monitoring system will not actively detect and identify behaviors. Traditional methods require manual monitoring of surveillance videos to detect abnormal behaviors in a timely manner. Traditional manual identification methods require a lot of manpower and financial resources. People are prone to fatigue when they stare at surveillance videos for a long time, and there are a lot of missed detections. Therefore, there is an urgent need for a method and system that can automatically analyze and capture human behavior based on surveillance videos and continuously detect behavior all day long, so as to avoid dangerous incidents or other abnormal situations.

[0003] In recent years, with the rapid development of artificial intelligence and neural networks, such as the emergence and improvement of algorithms such as recurrent neural networks (RNN), convolutional neural networks (CNN), and residual networks (ResNet), intelligent detection of human behavior has gradually been applied to people's daily lives, such as real-time detection of behaviors such as fighting, falling, falling out of bed, and abnormal sleep. On the basis of 2D convolution, 3D convolution considers the continuity in time and processes video data by adding the time dimension. The convolution kernel contains convolution operations in the time and space dimensions, which can extract spatiotemporal information and improve the accuracy of behavior detection. Therefore, human behavior recognition algorithms based on 3D convolution have been widely used. However, 3D convolution has high computational complexity and large memory consumption, which affects the network inference speed and has high hardware requirements.

[0004] The channel separation method used in the field of 2D image recognition can effectively save computational overhead. Based on this method, scholars proposed deep convolution. The core idea is to set the number of groups of 3×3×3 3D grouped convolution to the number of channels, and based on this, replace the 3×3×3 3D convolution in the 3D-RESNET network with 3×3×3 3D deep convolution, which greatly reduces the amount of calculation while achieving higher accuracy. Deformable convolution can be closer to the shape and size of the human body when sampling, has higher robustness, and has certain regularization ability and better generalization. The attention mechanism can imitate the visual and cognitive characteristics of humans. When processing video image data, it can focus on more important areas and reduce attention to non-important areas. Applying the attention mechanism to neural networks can assign higher weights to key features, thereby improving the accuracy of the model. Feature fusion is an important technology in neural networks, which can fully fuse the features of each layer, eliminate redundant features, obtain differential features, and reduce overfitting. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a feature fusion behavior recognition method based on deformable deep convolution and attention mechanism.

[0006] The present invention detects and identifies various behaviors by analyzing video data sets containing various human behaviors, which will help to promote the subsequent application of surveillance video behavior detection in key places, improve detection efficiency and accuracy, avoid illegal behaviors or other abnormal situations, and maintain the personal safety of personnel and the stability of the place.

[0007] In order to reduce the model calculation cost, close to the human body contour, focus on mining key channel features, and make full use of the features extracted from each layer, the present invention constructs a feature fusion neural network based on the combination of deformable deep convolution and attention mechanism, constructs a bottleneck block based on deep convolution and attention mechanism and a bottleneck block based on deformable deep convolution and attention mechanism, forms a feature extraction structure on this basis, and performs feature fusion on the features extracted from the bottleneck block to achieve full judgment and accurate identification of various behaviors. Compared with the traditional 3D convolutional network, deep convolution can reduce the model calculation amount and improve the detection speed while ensuring the network performance. Deformable convolution makes the network more adaptable to the human body shape and improves the network feature extraction ability. The attention mechanism makes the network more focused during the detection process and attaches importance to important channel features. Feature fusion can make full use of multi-dimensional features, realize information complementarity, and further improve the model accuracy.

[0008] The present invention also provides a feature fusion behavior recognition system based on deformable deep convolution and attention mechanism.

[0009] Terminology explanation:

[0010] The CSN (channel-separated convolutional network) model is a channel-separated convolutional neural network. The core idea of ​​this model is to replace the 3×3×3 3D conventional convolution in each bottleneck block with 3×3×3 3D deep convolution based on the 3D-RESNET network model, which greatly reduces the amount of calculation and effectively saves computing overhead.

[0011] The technical solution of the present invention is as follows:

[0012] A feature fusion behavior recognition method based on deformable deep convolution and attention mechanism, comprising: preprocessing a training set of a human behavior video data set and inputting it into a neural network model for training, preprocessing a test set and inputting it into the trained neural network model for behavior detection, and obtaining the detection accuracy of the neural network model; inputting the human behavior video data to be identified into the tested neural network model for behavior recognition.

[0013] Preferably, according to the present invention, the neural network model includes a 3×3×3 convolutional layer connected in sequence, a maximum pooling layer, three feature extraction layers stage1, stage2 and stage3 formed by stacking bottleneck blocks based on deep convolution and attention mechanism, a feature extraction layer stage4 formed by stacking bottleneck blocks based on deformable deep convolution and attention mechanism, a feature fusion module, an average pooling layer, a fully connected layer, and a softmax function layer.

[0014] Preferably, according to the present invention, the bottleneck block based on the deep convolution and attention mechanism includes a 1×1×1 convolution, a 3×3×3 deep convolution, a 1×1×1 convolution, and a SE module connected in sequence;

[0015] The bottleneck block based on deformable deep convolution and attention mechanism includes 1×1×1 convolution, 3×3×3 deformable deep convolution, 1×1×1 convolution, and SE module connected in sequence; among them, the deformable deep convolution is a deformable convolution operation based on the deep convolution;

[0016] For any point p0 on the input feature map, the convolution operation and depth convolution operation are shown in formula (1):

[0017]

[0018] In formula (1), p n Represents the relative position of each element in the convolution kernel, w(p n ) represents the weight of the corresponding position of the convolution kernel, x(p0+p n ) represents p0+p on the input feature map nThe pixel value at position y(p0) represents the pixel value at position p0 on the output feature map;

[0019] The deformable convolution operation is shown in formula (2):

[0020]

[0021] In formula (2), Δp n is the offset, p n Represents the relative position of each element in the convolution kernel, w(p n ) represents the weight of the corresponding position of the convolution kernel, x(p0+p n +Δp n ) represents p0+p on the input feature map n +Δp n The pixel value at position y′(p0) represents the pixel value at position p0 on the output feature map, which is obtained by convolving the weight with the input feature map.

[0022] Preferably, according to the present invention, the bottleneck block based on the deep convolution and attention mechanism and the bottleneck block based on the deformable deep convolution and attention mechanism both include a residual unit, and the residual unit directly connects the input and output of the bottleneck block structure in a skip connection manner to form an overall output; the calculation of the residual unit is shown in formula (3):

[0023] x t+1 =h(x t )+F(x t )#(3)

[0024] In formula (3), x t represents the input of the residual unit, x t+1 represents the output of the residual unit; h(x t )=x t represents the identity mapping relationship; F is the nonlinear residual function.

[0025] Preferably, in the SE module, the SE attention mechanism obtains the attention degree of each channel in the feature map through operation processing, and configures an attention weight for each feature channel according to the attention degree. The SE attention mechanism includes a compression operation and an excitation operation:

[0026] In the compression operation, the input feature map is compressed into a vector through global average pooling;

[0027] In the excitation operation, the bottleneck structure of two fully connected layers is passed in turn, and the correlation between channels is constructed through two fully connected layers. Among them, the first fully connected layer reduces the channel dimension and uses the ReLU activation function. The second fully connected layer increases the channel dimension and restores it to the original dimension. The Sigmoid activation function is used to obtain the weight of each channel. Finally, the output weight is multiplied by the original input channel by channel to obtain a feature map with channel weights.

[0028] Preferably, according to the present invention, the feature fusion module performs 3×3×3 3D convolution on the feature maps before stage1, after stage1, after stage2, after stage3, and after stage4, respectively, extracts features, and outputs feature maps as A, B, C, D, and E, respectively. A and B are spliced ​​in the channel dimension in turn, and maximum pooling and average pooling are performed and added at the same time. The result is then spliced ​​with C in the channel dimension, and maximum pooling and average pooling are performed and added at the same time. The result is then spliced ​​with D in the channel dimension, and maximum pooling and average pooling are performed and added at the same time. The result is then spliced ​​with E in the channel dimension, and finally 1×1×1 convolution is performed to adjust the number of channels to obtain the feature fusion result.

[0029] Preferably, according to the present invention, the training process of the neural network model is as follows:

[0030] Step 1: Data download and preprocessing: Download the HMDB51 dataset and the UCF101 dataset, as well as the partitioning files, and divide the datasets into training sets and test sets according to the partitioning files, and divide and annotate all video clips;

[0031] Step 2: Input the data in the training set obtained in step 1 into the neural network model for supervised training to obtain a trained neural network model.

[0032] According to the preferred embodiment of the present invention, the preprocessing specifically includes:

[0033] Divide, frame and annotate public data sets according to official classification files;

[0034] One frame is taken every two frames, and 32 frames are taken each time as a video clip, which is input into the neural network model for training.

[0035] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a feature fusion behavior recognition method based on deformable deep convolution and attention mechanism are implemented.

[0036] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a feature fusion behavior recognition method based on deformable deep convolution and attention mechanism.

[0037] A feature fusion behavior recognition system based on deformable deep convolution and attention mechanism, including:

[0038] The neural network model construction and training module is configured to: pre-process the training set of the human behavior video data set and input it into the neural network model for training, pre-process the test set and input it into the trained neural network model for behavior detection, and obtain the detection accuracy of the neural network model;

[0039] The behavior recognition module is configured to: input the human behavior video data to be recognized into the tested neural network model for behavior recognition.

[0040] The beneficial effects of the present invention are:

[0041] 1. The present invention provides a deformable deep convolution, which replaces the 3×3×3 3D conventional convolution in the last three bottleneck blocks of the 3D-RESNET network with a 3×3×3 3D deformable deep convolution, and replaces the 3×3×3 3D conventional convolution in the remaining bottleneck blocks with a 3×3×3 3D deep convolution. Deep convolution can greatly reduce the computational cost. Deformable convolution based on deep convolution can adaptively adjust the shape of the convolution kernel, thereby capturing more detailed information, making the receptive field more flexible, and improving model performance. Compared with the traditional neural network model based on 3D convolution, this method has low hardware requirements, high computational efficiency, and high detection accuracy.

[0042] 2. The present invention provides a 3D attention mechanism and introduces a channel attention mechanism (SE module) in the channel separation convolutional network to assign weights to feature maps to distinguish the importance of different channels, help the model better focus on the features of important channels, and improve the generalization ability of the model.

[0043] 3. The present invention provides a feature fusion module, which is introduced into the overall network architecture. Different features captured by different layers are used to obtain more shallow detail information and deep semantic information, so that the fused features are more comprehensive and the overfitting of the model is reduced.

[0044] 4. The present invention provides a feature fusion behavior recognition method based on deformable deep convolution and attention mechanism. Compared with the traditional neural network model based on 3D convolution, it has low computational cost and low hardware requirements. It can better focus on the key features of the target and make more comprehensive judgments by making full use of the deep and shallow information of the network, thus greatly improving the accuracy of various behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a structural schematic diagram of the neural network model of the present invention;

[0046] Figure 2 Schematic diagram of the deep convolution structure of the present invention, namely DWconv structure;

[0047] Figure 3 Schematic diagram of the deformable deep convolution structure of the present invention, namely DDWconv structure;

[0048] Figure 4 This is a schematic diagram of the SE module structure of the present invention;

[0049] Figure 5 Schematic diagram of compression operation and excitation operation in the SE module of the present invention;

[0050] Figure 6 This is a schematic diagram of the bottleneck block structure based on deep convolution and attention mechanism of the present invention. Figure 1 Bottleneck (DWconv&SE) in

[0051] Figure 7 This is a schematic diagram of the bottleneck block structure based on deep convolution and attention mechanism of the present invention. Figure 1 Schematic diagram of Bottleneck (DDWconv&SE) in . DETAILED DESCRIPTION

[0052] The present invention will be further defined below in conjunction with the accompanying drawings and embodiments, but is not limited thereto.

[0053] Example 1

[0054] A feature fusion behavior recognition method based on deformable deep convolution and attention mechanism, comprising: preprocessing a training set of a human behavior video data set and inputting it into a neural network model for training, preprocessing a test set and inputting it into the trained neural network model for behavior detection, and obtaining the detection accuracy of the neural network model; inputting the human behavior video data to be identified into the tested neural network model for behavior recognition.

[0055] Example 2

[0056] The feature fusion behavior recognition method based on deformable deep convolution and attention mechanism described in Example 1 is different in that:

[0057] like Figure 1As shown, the neural network model includes a 3×3×3 convolution layer, a maximum pooling layer, three bottleneck blocks Bottleneck (DWconv&SE) stacked based on deep convolution and attention mechanism to form feature extraction layers stage1, stage2 and stage3, a feature extraction layer stage4 formed by stacking bottleneck blocks Bottleneck (DDWconv&SE) based on deformable deep convolution and attention mechanism, a feature fusion module, an average pooling layer, a fully connected layer, and a softmax function layer. Among them, three bottleneck blocks based on deep convolution and SE attention mechanism are stacked to form feature extraction layer stage1, four bottleneck blocks based on deep convolution and SE attention mechanism are stacked to form feature extraction layer stage2, six bottleneck blocks based on deep convolution and SE attention mechanism are stacked to form feature extraction layer stage3, and three bottleneck blocks based on deformable deep convolution and SE attention mechanism are stacked to form feature extraction layer stage4;

[0058] The features extracted by the shallow network layer have more fine-grained pixel detail information, and the features extracted by the deep network layer have more coarse-grained abstract semantic features. The fusion of multiple features can improve the robustness and stability of the algorithm, making the algorithm adaptable and fault-tolerant to different data. Therefore, the present invention designs a feature fusion module, and performs 3×3×3 3D convolution on the feature maps before stage1, after stage1, after stage2, after stage3, and after stage4 to extract features, so as to avoid the loss of key information by adjusting the size of the feature map in subsequent downsampling, and the output feature maps are recorded as A, B, C, D, and E respectively. Downsampling uses parallel maximum pooling and average pooling to reduce the size, which can retain more feature information and the most significant feature information. Concatenate A and B in the channel dimension in turn, perform maximum pooling and average pooling for size transformation and addition at the same time, which can retain more feature information and the most significant feature information. The result is then concatenated with C in the channel dimension, and maximum pooling and average pooling are performed for size transformation and addition. The result is then concatenated with D in the channel dimension, and maximum pooling and average pooling are performed for size transformation and addition. The result is then concatenated with E in the channel dimension. Finally, 1×1×1 convolution is used to adjust the number of channels to obtain the feature fusion result.

[0059] The bottleneck block based on deep convolution and attention mechanism includes 1×1×1 convolution, 3×3×3 deep convolution, 1×1×1 convolution, and SE module connected in sequence; among them, deep convolution is a kind of grouped convolution. Grouped convolution is to divide the input feature map into g groups according to the channel, and perform convolution on each group. After grouping, the number of channels of each group of input feature map is The number of output channels of grouped convolution is at least g. Under the same input and output size, the number of parameters and operations is reduced to the original The depth convolution sets the number of groups and output channels in the grouped convolution to be consistent with the number of input channels, that is, g = C in =C out ;

[0060] The bottleneck block based on deformable deep convolution and attention mechanism includes 1×1×1 convolution, 3×3×3 deformable deep convolution, 1×1×1 convolution, and SE module connected in sequence; among them, deformable deep convolution is a deformable convolution operation based on deep convolution; a convolution layer learns the offset offset so that the sampling points of the convolution kernel on the input feature map are offset and focused on the area or target of interest. The number of offset channels is 3N (N=27), where N is the number of elements of the convolution kernel. Because of the offset in space, the three direction values ​​of H, W, and T need to be changed. According to the obtained coordinate value with offset, the pixel point is taken, and then trilinear interpolation is performed to obtain the final convolution kernel for subsequent convolution operations.

[0061] like Figure 2 As shown in the figure, deep convolution is a type of grouped convolution. Grouped convolution divides the input feature map into g groups by channel, and then performs regular convolution on each group. Therefore, each group requires at least one convolution kernel, that is, the number of output channels of grouped convolution is at least g. If the number of convolution kernels in each group is n, then the number of output channels is n×g, so the number of output channels is an integer multiple of the number of groups. Deep convolution means the number of input channels = the number of groups = the number of output channels.

[0062] like Figure 3 As shown in the figure, the deformable deep convolution adds an offset to the sampling position in the deep convolution operation, so that the convolution kernel can be extended to a large range. The convolution kernel offset of the deformable deep convolution is generated by 3×3×3 convolution. The channel dimension of the offset field in the offset space is 3N (where N is 27), which represents the offset of the corresponding 3×3×3 convolution kernel in height H, width W and length T. When the convolution kernel is adjusted using the offset, the trilinear interpolation method is used to obtain the corresponding pixel value. Then, the convolution kernel adjusted by the offset is used for the deep convolution operation.

[0063] For any point p0 on the input feature map, the convolution operation and depth convolution operation are shown in formula (1):

[0064]

[0065] In formula (1), p n Represents the relative position of each element in the convolution kernel, w(p n) represents the weight of the corresponding position of the convolution kernel, x(p0+p n ) represents p0+p on the input feature map n The pixel value at position y(p0) represents the pixel value at position p0 on the output feature map; it is obtained by convolving the weights with the input feature map.

[0066] Deformable convolution introduces an offset for each point based on the above formula. The offset is generated by performing a convolution operation on the input feature map and is usually a decimal. The deformable convolution operation is shown in formula (2):

[0067]

[0068] In formula (2), Δp n is the offset, p n Represents the relative position of each element in the convolution kernel, w(p n ) represents the weight of the corresponding position of the convolution kernel, x(p0+p n +Δp n ) represents p0+p on the input feature map n +Δp n The pixel value at position y′(p0) represents the pixel value at position p0 on the output feature map, which is obtained by convolving the weight with the input feature map.

[0069] The bottleneck block based on deep convolution and attention mechanism and the bottleneck block based on deformable deep convolution and attention mechanism both include residual units, which directly connect the input and output of the bottleneck block structure in a skip connection manner to form the overall output; this method allows data to flow easily between convolutional layers without hindering the learning ability of the deep learning model. During forward propagation, the input signal can be directly transmitted from any low layer to the high layer; during backward propagation, the error signal can be directly transmitted to the low layer without any intermediate weight matrix transformation, which can alleviate the gradient diffusion problem to a certain extent (even if the weight of the intermediate layer matrix is ​​very small, the gradient will basically not disappear). Therefore, the residual network converges faster with the same number of layers.

[0070] Residual units such as Figure 3 As shown in Figure 2, the residual unit can alleviate the gradient vanishing problem. During forward propagation, the input signal jumps from the shallow layer to the deep layer; during backward propagation, the gradient can be more easily transferred to the shallow layer. The calculation of the residual unit is shown in formula (3):

[0071] x t+1 =h(x t )+F(x t )#(3)

[0072] In formula (3), x trepresents the input of the residual unit, x t+1 represents the output of the residual unit; h(x t )=x t represents the identity mapping relationship; F is the nonlinear residual function.

[0073] The present invention also introduces the SE channel attention module (SE module), which assigns corresponding weights to different feature channels, focuses on key channels, and improves the accuracy of network detection. The key operations of this attention mechanism are as follows: Figure 4 As shown in the figure, in the SE module, the SE attention mechanism obtains the attention level of each channel in the feature map after operation processing, and configures an attention weight for each feature channel according to this attention level, so that the convolution network pays more attention to the feature channel, thereby realizing the channels of the feature map that are useful for the current task and suppressing the feature channels that are not very useful for the current task. The SE attention mechanism includes compression operation (Squeeze) and excitation operation (Excitation):

[0074] In the compression operation, the input feature map is compressed into a vector through global average pooling. Specifically, a global average pooling is used to compress the length, width and height of the input feature map, and D×H×W×C is compressed into a feature vector of size 1×1×1×C.

[0075] In the excitation operation, the bottleneck structure of two fully connected layers is passed through in sequence, and the correlation between channels is constructed through two fully connected layers. Using two fully connected layers has more nonlinearity than directly using one fully connected layer, which can better fit the complex correlation between channels. The number of output weight values ​​is the same as the number of channels in the input feature map. Among them, the first fully connected layer performs a dimensionality reduction operation on the channel dimension to reduce the number of channels to The ReLU activation function is used in the second fully connected layer to restore the channel dimension to the original dimension C, and the Sigmoid activation function is used to obtain the weight of each channel. Finally, the output weight is multiplied by the original input channel by channel to obtain a feature map with channel weights.

[0076] like Figure 5 The figure shows the overall schematic diagram of SE module. tr It is a normal convolution operation, which convolves the input feature map. sq (·) is the compression operation, F ex (·,W) is the excitation operation, F scale (·,·) is multiplied with the original input. The entire process of the SE module is: input feature map for basic convolution operation, and then compress F sq (·) and the excitation F ex(·,W) two steps to get a 1×1×1×C weight vector, and multiply the weight vector by the input feature map F scale (·,·), thereby assigning weight values ​​to each channel.

[0077] like Figure 6 As shown in the figure, the bottleneck block based on deep convolution and SE attention mechanism connects 1×1×1 convolution, 3×3×3 deep convolution, 1×1×1 convolution and SE module in sequence, and includes residual unit to jump connect the input and output.

[0078] like Figure 7 As shown in the figure, the bottleneck block based on deformable deep convolution and SE attention mechanism connects 1×1×1 convolution, 3×3×3 deformable deep convolution, 1×1×1 convolution and SE module in sequence, and includes residual unit to jump connect the input and output.

[0079] The feature fusion module performs 3×3×3 3D convolution on the feature maps before stage1, after stage1, after stage2, after stage3, and after stage4 to extract features and avoid the loss of key information in the feature maps in subsequent downsampling operations. The output feature maps are recorded as A, B, C, D, and E, respectively. A and B are spliced ​​in the channel dimension in turn, and maximum pooling and average pooling are performed and added at the same time to keep the size of the output feature map consistent with the deep feature map when spliced. Parallel downsampling can better capture the overall information in the feature map, smooth the image and reduce noise, while emphasizing the most prominent features of the target. The result is then spliced ​​with C in the channel dimension, and maximum pooling and average pooling are performed and added at the same time. The result is then spliced ​​with D in the channel dimension, and maximum pooling and average pooling are performed and added at the same time. The result is then spliced ​​with E in the channel dimension. Finally, a 1×1×1 convolution is performed to adjust the number of channels to obtain the feature fusion result.

[0080] The training process of the neural network model is as follows:

[0081] Step 1: Data download and preprocessing: Official download of HMDB51 dataset and UCF101 dataset (HMDB51 dataset was released by Brown University in 2011, containing 51 types of actions, a total of 6849 videos, each action contains at least 100 videos, the actions mainly include: common facial actions (smiling, chewing, talking), complex facial actions (smoking, eating, drinking water), common body movements (climbing, diving, jumping), complex body movements (combing, grabbing, sword drawing), multi-person interactive body movements (hugging, kissing, shaking hands), etc. The UCF101 dataset was released by the Computer Vision Research Center of the University of Central Florida in 2021 and contains 101 types of actions There are 13,320 videos in total, and each action contains at least 100 videos. The actions mainly include: human-object interaction (applying eye makeup, applying eye makeup, brushing teeth), simple body movements (hammering, push-ups, walking), human-to-human interaction (sumo, military parade, wrestling), playing musical instruments (playing guitar, playing piano, playing violin), sports (skiing, shot put, parachuting), etc.), as well as partitioning files, and dividing the data set according to the partitioning files to obtain training sets and test sets (the ratio of the number of samples in the training set and the test set is 7:3), frame segmentation and annotation processing, taking 1 frame every 2 frames, and intercepting 32 frames from each video clip, inputting them into the network for training, and obtaining the best action recognition accuracy results.

[0082] Step 2: Input the data in the training set obtained in step 1 into the neural network model for supervised training to obtain a trained neural network model. The specific implementation process is:

[0083] Before training the neural network model, the network parameters of the neural network model were loaded using the CSN model pre-training parameters. Kaiming initialization was performed on the parameters without pre-training. The learning rate was initialized to 0.000125, and the learning rate was warmed up. A total of 58 rounds of training were performed. In each training round, all samples in the training set were divided into S sets, each of which contained the same number of 3 samples. The neural network model calculated the output value generated by the sample according to the forward propagation algorithm, and calculated the loss function value by comparing it with the label value. The average value of all loss function values ​​in the sample set was calculated, and the back propagation algorithm was used to dynamically adjust the parameters of the neural network model. During the back propagation process, SGD stochastic gradient descent was used to gradually converge to the optimal value. After multiple rounds of training, the accuracy of the neural network model for each behavior detection was tested using the test set after each round of training. The hyperparameters of the neural network model, including the learning rate and regularization parameter, were adjusted according to the accuracy of the test set, and multiple rounds of training were further performed to obtain the optimal accuracy of the neural network model detection.

[0084] The structural configuration of the neural network model is shown in Table 1:

[0085] Table 1

[0086]

[0087]

[0088] Preprocessing, the specific implementation process includes:

[0089] The public dataset is divided according to the official division file (the ratio of the number of samples in the training set to the test set is 7:3), framed, and annotated;

[0090] One frame is taken every two frames, and 32 frames are taken each time as a video clip, which is input into the neural network model for training.

[0091] The experimental results are shown in Table 2:

[0092] Table 2

[0093] Model HMDB51 UCF101 CSN 78.82％ 96.56％ CSN+DDW 79.22％ 96.91％ CSN+DDW+SE 80.26％ 97.46％ CSN+DDW+SE+feature fusion 80.59％ 97.52％

[0094] Kaiming initialization is performed on the parameters without pre-training, the learning rate is initialized to 0.000125, and the learning rate is warmed up. A total of 58 rounds of training are performed. In each training round, all samples of the training sets of the two data sets are divided into S sets (S = total number of training sets / 3), and each set contains the same number of 3 samples. Finally, the feature fusion model based on deformable deep convolution and attention mechanism of the present invention improves the accuracy of the CSN model on the HMDB51 data set by 1.77%, and the accuracy on the UCF101 data set by 0.96%.

[0095] Example 3

[0096] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the feature fusion behavior recognition method based on deformable deep convolution and attention mechanism described in Example 1 or 2 are implemented.

[0097] Example 4

[0098] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the feature fusion behavior recognition method based on deformable deep convolution and attention mechanism of embodiment 1 or 2.

[0099] Example 5

[0100] A feature fusion behavior recognition system based on deformable deep convolution and attention mechanism, including:

[0101] The neural network model construction and training module is configured to: pre-process the training set of the human behavior video data set and input it into the neural network model for training, pre-process the test set and input it into the trained neural network model for behavior detection, and obtain the detection accuracy of the neural network model;

[0102] The behavior recognition module is configured to: input the human behavior video data to be recognized into the tested neural network model for behavior recognition.

Claims

1. A feature fusion behavior recognition method based on deformable deep convolution and attention mechanism, characterized in that: include: The training set of the human behavior video data set is preprocessed and input into the neural network model for training, and the test set is preprocessed and input into the trained neural network model for behavior detection to obtain the detection accuracy of the neural network model; the human behavior video data to be identified is input into the tested neural network model for behavior recognition; The neural network model includes a 3×3×3 convolutional layer connected in sequence, a maximum pooling layer, three feature extraction layers stage1, stage2 and stage3 formed by stacking bottleneck blocks based on deep convolution and attention mechanism, a feature extraction layer stage4 formed by stacking bottleneck blocks based on deformable deep convolution and attention mechanism, a feature fusion module, an average pooling layer, a fully connected layer, and a softmax function layer; The bottleneck block based on deformable deep convolution and attention mechanism includes 1×1×1 convolution, 3×3×3 deformable deep convolution, 1×1×1 convolution, and SE module connected in sequence; among them, the deformable deep convolution is a deformable convolution operation based on the deep convolution; For any point p0 on the input feature map, the convolution operation and depth convolution operation are shown in formula (1): In formula (1), p n Represents the relative position of each element in the convolution kernel, w(p n ) represents the weight of the corresponding position of the convolution kernel, x(p0+p n ) represents p0+p on the input feature map n The pixel value at position y(p0) represents the pixel value at position p0 on the output feature map; The deformable convolution operation is shown in formula (2): In formula (2), Δp n is the offset, p n Represents the relative position of each element in the convolution kernel, w(p n ) represents the weight of the corresponding position of the convolution kernel, x(p0+p n +Δp n ) represents p0+p on the input feature map n +Δp n The pixel value at position y′(p0) represents the pixel value at position p0 on the output feature map, which is obtained by convolving the weight with the input feature map; In the SE module, the SE attention mechanism obtains the attention degree of each channel in the feature map through operation processing, and configures an attention weight for each feature channel according to this attention degree. The SE attention mechanism includes compression operation and excitation operation: In the compression operation, the input feature map is compressed into a vector through global average pooling; In the excitation operation, the bottleneck structure of the two fully connected layers is passed in turn, and the correlation between channels is constructed through two fully connected layers. Among them, the first fully connected layer reduces the channel dimension and uses the ReLU activation function. The second fully connected layer restores the channel dimension to the original dimension and uses the Sigmoid activation function to obtain the weight of each channel. Finally, the output weight is multiplied by the original input channel by channel to obtain a feature map with channel weights. The feature fusion module performs 3×3×3 3D convolution on the feature maps before stage1, after stage1, after stage2, after stage3, and after stage4 to extract features. The output feature maps are recorded as A, B, C, D, and E, respectively. A and B are concatenated in the channel dimension in turn, and maximum pooling and average pooling are performed and added. The result is then concatenated with C in the channel dimension, and maximum pooling and average pooling are performed and added. The result is then concatenated with D in the channel dimension, and maximum pooling and average pooling are performed and added. The result is then concatenated with E in the channel dimension. Finally, a 1×1×1 convolution is performed to adjust the number of channels to obtain the feature fusion result.

2. According to claim 1, a feature fusion behavior recognition method based on deformable deep convolution and attention mechanism is characterized in that: The bottleneck block based on deep convolution and attention mechanism and the bottleneck block based on deformable deep convolution and attention mechanism both include residual units, which directly connect the input and output of the bottleneck block structure in a skip connection manner to form the overall output; The calculation of the residual unit is shown in formula (3): x t+1 =h(x t )+F(x t )#(3) In formula (3), x t represents the input of the residual unit, x t+1 represents the output of the residual unit; h(x t )=x t represents the identity mapping relationship; F is the nonlinear residual function.

3. According to claim 1, a feature fusion behavior recognition method based on deformable deep convolution and attention mechanism is characterized in that: The training process of the neural network model is as follows: Step 1: Data download and preprocessing: Download the HMDB51 dataset and the UCF101 dataset, as well as the partitioning files, and divide the datasets into training sets and test sets according to the partitioning files, and divide and annotate all video clips; Step 2: Input the data in the training set obtained in step 1 into the neural network model for supervised training to obtain a trained neural network model.

4. According to claim 1, a feature fusion behavior recognition method based on deformable deep convolution and attention mechanism is characterized in that: Preprocessing, the specific implementation process includes: Divide, frame and annotate public data sets according to official classification files; One frame is taken every two frames, and 32 frames are taken each time as a video clip, which is input into the neural network model for training.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the feature fusion behavior recognition method based on deformable deep convolution and attention mechanism described in any one of claims 1-4 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the feature fusion behavior recognition method based on deformable deep convolution and attention mechanism described in any one of claims 1-4 are implemented.

7. A feature fusion behavior recognition system based on deformable deep convolution and attention mechanism, characterized in that: include: The neural network model construction and training module is configured to: pre-process the training set of the human behavior video data set and input it into the neural network model for training, pre-process the test set and input it into the trained neural network model for behavior detection, and obtain the detection accuracy of the neural network model; The behavior recognition module is configured to: input the video data of human behavior to be recognized into the tested neural network model for behavior recognition; The neural network model includes a 3×3×3 convolutional layer connected in sequence, a maximum pooling layer, three feature extraction layers stage1, stage2 and stage3 formed by stacking bottleneck blocks based on deep convolution and attention mechanism, a feature extraction layer stage4 formed by stacking bottleneck blocks based on deformable deep convolution and attention mechanism, a feature fusion module, an average pooling layer, a fully connected layer, and a softmax function layer; The bottleneck block based on deformable deep convolution and attention mechanism includes 1×1×1 convolution, 3×3×3 deformable deep convolution, 1×1×1 convolution, and SE module connected in sequence; among them, the deformable deep convolution is a deformable convolution operation based on the deep convolution; For any point p0 on the input feature map, the convolution operation and depth convolution operation are shown in formula (1): In formula (1), p n Represents the relative position of each element in the convolution kernel, w(p n ) represents the weight of the corresponding position of the convolution kernel, x(p0+p n ) represents p0+p on the input feature map n The pixel value at position y(p0) represents the pixel value at position p0 on the output feature map; The deformable convolution operation is shown in formula (2): In formula (2), Δp n is the offset, p n Represents the relative position of each element in the convolution kernel, w(p n ) represents the weight of the corresponding position of the convolution kernel, x(p0+p n +Δp n ) represents p0+p on the input feature map n +Δp n The pixel value at position y′(p0) represents the pixel value at position p0 on the output feature map, which is obtained by convolving the weight with the input feature map; In the SE module, the SE attention mechanism obtains the attention degree of each channel in the feature map through operation processing, and configures an attention weight for each feature channel according to this attention degree. The SE attention mechanism includes compression operation and excitation operation: In the compression operation, the input feature map is compressed into a vector through global average pooling; In the excitation operation, the bottleneck structure of the two fully connected layers is passed in turn, and the correlation between channels is constructed through two fully connected layers. Among them, the first fully connected layer reduces the channel dimension and uses the ReLU activation function. The second fully connected layer restores the channel dimension to the original dimension and uses the Sigmoid activation function to obtain the weight of each channel. Finally, the output weight is multiplied by the original input channel by channel to obtain a feature map with channel weights. The feature fusion module performs 3×3×3 3D convolution on the feature maps before stage1, after stage1, after stage2, after stage3, and after stage4 to extract features. The output feature maps are recorded as A, B, C, D, and E, respectively. A and B are concatenated in the channel dimension in turn, and maximum pooling and average pooling are performed and added. The result is then concatenated with C in the channel dimension, and maximum pooling and average pooling are performed and added. The result is then concatenated with D in the channel dimension, and maximum pooling and average pooling are performed and added. The result is then concatenated with E in the channel dimension. Finally, a 1×1×1 convolution is performed to adjust the number of channels to obtain the feature fusion result.

Images