A video abnormal behavior recognition method based on attention feature gating

Abstract

This invention discloses a video anomaly behavior recognition method based on attention feature gating. This method performs feature dimensionality reduction during multi-dimensional training to construct a more efficient recognition network. The specific implementation method is as follows: First, consecutive video images are randomly extracted from the entire dataset and preprocessed as training samples. Then, they are fed into the proposed network model AFG-NET for training. The model then performs channel separation on the frame rate dimension of the training samples, extracts features from multiple channels using two-dimensional convolution, and then uses a channel feature sparsification function to scale and sum the features, multiplying them with the global features of the training samples to obtain the output. The subsequent part of the model then extracts and recognizes the features, finally obtaining the prediction result of the network model. Finally, a joint loss combining weighted sparsity penalty and cross-entropy loss is used, and stochastic gradient descent is used to backpropagate the joint loss of the model to optimize the parameters of the network model to achieve global optimum.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and video understanding, specifically to a method for identifying abnormal video behavior based on attention feature gating. Background Technology

[0002] Current mainstream video action recognition research mainly focuses on ultra-large-scale datasets. However, for datasets with smaller data volume and simpler scenes, the background of the video image is large and the proportion of the subject to be recognized is small, causing the model to rely more on the recognition of the background. This results in poor recognition of different actions in the same scene and poor generalization performance of the model. "Diba A, Fayyaz M, Sharma V, et al. Temporal 3d convnets: new architecture and transfer learning for video classification[J]arXiv preprint arXiv:1711.08200, 2017" extends the idea of ​​Inception to 3D convolution from different levels and proposes T3D network. It captures rich spatiotemporal information through convolution at different scales, but its parameter volume is large, which easily leads to a lot of redundancy in model parameters. "Du T, Wang H, Torresani L, et al. A Closer Look at Spatiotemporal Convolutions for ActionRecognition[C] / / 2018IEEE / CVF Conference on Computer Vision and PatternRecognition (CVPR). IEEE, 2018. Combining the idea of ​​spatiotemporal segmentation, the structure of classic 3D convolution is decomposed into temporal convolution and spatial convolution, which effectively reduces the number of network parameters and increases the nonlinear relationship within the convolution, improving the network's ability to extract spatiotemporal features. However, its convolution size is fixed and lacks analysis of the hierarchical structure of spatiotemporal features at different scales, resulting in a single spatiotemporal receptive field and poor generalization for diverse human behavior recognition. Therefore, this patent proposes the MSFP pooling method to filter and fuse multi-scale information from video streams, making the model more focused on moving subjects in video images. Building upon this foundation, an AFG layer is proposed, enabling the model to purposefully select optimal channels. Combined with an attention mechanism, it further fuses information from different dimensions. This patented method not only reduces the feature dimensionality of training samples, allowing 2D convolutional networks to adapt to 3D data training, but also demonstrates excellent model learning capabilities. Finally, this patented method optimizes the MSFP structure, resulting in a miniMSFP more suitable for a simplified network, further optimizing the network structure. Experimental results on the Tobacco factory and HMDB51 datasets show that this patented method significantly improves the model's recognition accuracy, proving its effectiveness compared to other mainstream methods. Summary of the Invention

[0003] The present invention mainly overcomes the shortcomings of the prior art. The purpose of the present invention is to provide a method based on deep learning theory and to efficiently deploy a simplified neural network model on edge devices with limited computing power.

[0004] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0005] 1. A video abnormal behavior recognition method based on attention feature gating, characterized by the following steps:

[0006] Step 1: Create training samples X. The specific steps are as follows:

[0007] (1) Randomly select n video samples X′ with f frames from the dataset, f∈(1,64]; X′ is c×f a×b matrices, a∈(224,1080); b∈(224,1080); c is the number of channels of the video image, c∈{1,3};

[0008] (2) Resize the n video samples X′ with f frames to obtain the resized video samples X″, where X″ is c×f w×h matrices; w and h are fixed values ​​of 224; then perform maximum and minimum value normalization on X″ to obtain the normalized video samples X″′, where X″′ is c×f w×h matrices; the calculation formula is as follows:

[0009]

[0010] In the formula x′ w " h Represents a single element in matrix X″′, x′ w " h ∈(-1, 1); x′ w ′ h Represents a single element in matrix X″′, x′ w ′ h ∈(0, 255); x min x represents the minimum value of normalization. min =-1; x max x represents the maximum value of the normalized value. max =1;

[0011] (3) Finally, the n normalized video samples X″′ are combined into a set X as training samples, X={X″′1,X″′2,X″′3,…,X″′ n The training samples X consist of n×c×f w×h matrices; n is the number of normalized video samples X″′ in the training samples X, n∈N;

[0012] Step 2: Construct the AFG-NET network model, feed the training sample X into the AFG-NET network model for training, and obtain the model recognition result Y. The specific steps are as follows:

[0013] (1) Construct the video abnormal behavior recognition model AFG-NET. AFG-NET consists of AFG, miniMSFP, Inception and two-dimensional convolution. The calculation formula is as follows:

[0014] AFG-NET(X)=Inception(miniMSFP(Inception(Conv 3×3 (Conv 1×1 (Conv 7×7 (AFG(X)))))))

[0015] In the formula, Conv 1×1 (·) represents a 1×1 convolution, Conv 3×3 (·) represents a 3×3 convolution, Conv 7×7 (·) represents a 7×7 convolution, AFG represents an attention feature gate layer, and AFG consists of a global pooling GAP(·) and a multi-channel feature sparsification function sp(·), calculated as follows:

[0016] AFG(X) = GAP(X) × sp(X)

[0017] miniMSFP is a small multi-scale fusion block. It is composed of multi-scale average pooling and 1×1 convolution, and the calculation formula is as follows:

[0018]

[0019] In the formula, GAP(·) represents global pooling, and Conv 1×1 (·) represents a 1×1 kernel convolution, N is the number of scales in a miniMSFP, N=3; AP k×k (·) represents average pooling with a kernel of k×k, where k∈{3, 5, 7};

[0020] Inception represents a multi-scale feature extraction block. Inception consists of a 3×3 convolutional Conv... 3×3 (·) and 1×1 convolution Conv 1×1 The (·) component is calculated using the following formula:

[0021] Inception(X) = Concat(Conv 1×1 (X), Conv 3×3 (Conv 1×1 (X)), Conv 3×3 (Conv1×1 (X)))

[0022] In the formula, Concat(X1, X2, X3) represents arranging the matrix features of features X1, X2, and X3 in order to obtain X. Cat X1, X2, and X3 are n×c×f w×h matrices; X Cat There are n×c×3×f w×h matrices;

[0023] (2) Input the training sample X into the AFG-NET network model;

[0024] (3) The AFG layer in the AFG-NET network model performs global feature calculation on the training sample X to obtain the global pooling feature X. GAP Global pooling feature X GAP Let X be an n×c matrix. GAP ∈(-1, 1); the calculation formula is as follows:

[0025] X GAP =GAP(X)=X n×c ·mean(X f×w×h )

[0026] In the formula, GAP(·) represents global pooling of the f, w, and h dimensions of the training sample X, where X represents the training sample. n×c This is an n×c matrix formed by the n and c dimensions of the training samples X; mean(·) represents taking the mean, X f×w×h f w×h matrices representing the f, w, and h dimensions of the training sample X;

[0027] (4) The input layer AFG layer in the AFG-NET network model performs multi-channel feature sparsification on the training sample X to obtain the sparsified features X. SP , sparsification feature X SP It is a set of n×c w×h matrices; the calculation formula is as follows:

[0028]

[0029] In the formula, X represents the training samples, sp(·) is the multi-channel feature sparsification function, and x i It is the feature matrix of the training sample X expanded along the third dimension for f channels, and the feature matrix x is... i There are n×c w×h matrices; a i Represents the characteristic matrix x i The inhibitory factor, a i The calculation formula is as follows:

[0030]

[0031] In the formula, λ is the channel suppression ratio, λ∈{10} n}, n∈{-3,-4,-5};γ i γ is the weight scaling ratio of the BN layer of the i-th channel in the multi-channel feature sparsification function sp(·). i ∈(0,1); γ i The calculation formula is as follows:

[0032]

[0033] Conv 1×1 (x i ) represents the input of the i-th channel BN layer in the multi-channel feature sparsification function sp(·), Conv 1×1 (x i ) represents n×c w×h matrices; BN(x i ) represents the output of the i-th channel BN layer in the function sp(·), BN(x) i ) represents n×c w×h matrices; γ i The scaling weight of the BN layer for the i-th channel in the representation function F(X), γ i ∈(0,1); β i The bias weight of the i-th channel BN layer in the representative function sp(·) is β. i ∈(0,1);

[0034] (5) Using global pooling feature X GAP The n×c elements and the sparsity feature X SP The output matrix of the AFG layer is obtained by multiplying the n×c w×h matrices respectively. Output matrix There are n×c w×h matrices; the calculation formula is as follows:

[0035]

[0036] The special symbol ⊙ indicates that the elements of the previous matrix are multiplied by the corresponding matrices in the next set;

[0037] (6) Output of AFG The data is fed into AFG-NET for feature extraction, and finally the model prediction matrix X is obtained. out The calculation formula is as follows:

[0038]

[0039] Model prediction matrix X out Let X be an n×s matrix, and let X be the model prediction matrix. outThe model's recognition result Y is obtained by predicting using the maxout function. Y is a vector of length n, representing the recognition result of AFG-NET for n video samples in the training sample X. The calculation formula is as follows:

[0040]

[0041] In the formula Y n The nth element of the model recognition result Y is represented by Y. n ∈N, Y n ∈[1, n]; x ns Represents the prediction matrix X out n×s elements, x ns ∈(0,1),j n Represents the prediction matrix X out The column number of the largest element in the nth row, j n ∈[1, s]; s represents the number of abnormal behavior categories in the video samples of the training sample X;

[0042] Step 3: Use backpropagation to optimize and update the parameters in the AFG-NET network model, and save the network parameter P that achieves the highest recognition accuracy (Acc) in all training rounds. t The specific steps are as follows:

[0043] (1) Backpropagation of error is adopted, and the cross-entropy loss function is used to measure the true result. The distance to the model's recognition result Y, and the L1 norm are used to sparsify the parameters γ of AFG. The above two losses form a joint loss function, and the training loss L, L∈(0,+∞), is calculated. The formula for calculating the joint loss function is as follows:

[0044]

[0045] In the formula Let λg(γ) be the cross-entropy loss function, and let λg(γ) be the sparsity penalty for the scaling factor γ of the BN layer in the multi-channel feature sparsification function sp(·). λ is the scaling factor of the penalty term used to balance these two losses, and λ∈{10} n}, n∈{-3,-4,-5}; g(·) is the L1 norm penalty formula, calculated as follows:

[0046]

[0047] In the formula, f represents the number of channels in the multi-channel feature sparsification function sp(·), and γ i The scaling weight of the BN layer for the i-th channel in the function sp(·);

[0048] (2) Through each training session of the training sample X, the stochastic gradient descent function SGD is used to backpropagate the loss value L in the AFG-NET network model, thereby improving the model parameters P of the i-th training round of AFG-NET. i Randomly varying the model parameters in the direction of the negative gradient to optimize the network, the formula for updating the model parameters using SGD is as follows:

[0049]

[0050] In the formula, P i P represents the model parameters in the i-th round of training of AFG-NET. i-1 η represents the model parameters in the (i-1)th round of training of AFG-NET, and η represents the optimization step size of SGD, where η∈(0,1); The training loss L represents the training parameters P of the model in the (i-1)th round of training of AFG-NET. i-1 Perform differentiation;

[0051] (3) Calculate the current training round e i Accuracy of AFG-NET model in Chinese i Acc i ∈(0,1), and store the training round e. i AFG-NET model P i e i e represents the current i-th training round. i ∈(1, e m ]; e m For the maximum number of training rounds, e m ∈N;

[0052]

[0053] In the formula, TP is the number of positive samples correctly predicted by the model, TN is the number of negative samples correctly predicted by the model, FP is the number of positive samples incorrectly predicted by the model, and FN is the number of negative samples incorrectly predicted by the model.

[0054] (4) Compare the model recognition accuracy Acc in each training round. i And save Acc i The highest model parameter P i And used as deployment parameter P t The calculation is as follows:

[0055]

[0056] Step 4: Use the deployment parameters P saved in Step 3. t As the network parameters of the AFG-NET model, and with the network parameters set as P tThe AFG-NET model is deployed on the recognition terminal. At the same time, the terminal performs video abnormal behavior recognition on the input video and obtains the video abnormal behavior recognition result y, where y is a positive integer representing the abnormal behavior category of the video input, and y∈[1,s].

[0057] The innovation of this invention is reflected in:

[0058] (1) Compared with the Network Slimming method, the above methods can only prune the neural network after the model is trained. The method of reducing network complexity in this invention is feature dimensionality reduction, which can be carried out in real time during network training.

[0059] (2) Compared with classic abnormal behavior recognition methods such as I3D, S3D, and T3D, all three neural networks use three-dimensional convolution as feature extractors. Although this invention uses three-dimensional data as input, it performs dimensionality reduction inside the network, so that all feature extractors use two-dimensional convolution, thus reducing complexity.

[0060] Beneficial effects:

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. By utilizing the AFG layer proposed in this invention, 3D data is transformed into 2D data, resulting in feature reduction. Furthermore, the feature extractor is converted from 3D convolution to 2D convolution, which greatly reduces the complexity of the model and makes it more suitable for the application needs of edge devices.

[0063] 2. By utilizing the miniMSFP layer proposed in this invention, the multi-scale feature fusion capability of the method in this patent is enhanced. Furthermore, due to the lightweight design of this patent, the complexity of this method is lower and the performance is better than other similar algorithms. Attached Figure Description

[0064] Figure 1 This is a structural diagram of the AFG-NET network model proposed in this patent. The AFG-NET network consists of AFG, inception block, and miniMSFP.

[0065] Figure 2 This is a structural diagram of the AFG layer proposed in this patent. The AFG layer is composed of multiple channels connected in parallel. In the final output stage, the channels with small scaling sparsity are suppressed.

[0066] Figure 3 This is a structural diagram of the miniMSFP block proposed in this patent. The miniMSFP block is composed of pooling layers and convolutional layers of different sizes.

[0067] Figure 4This is a structural diagram of the 2D inception block used in this patent. The 2D inception block is composed of a combination of 3×3 convolution, 1×1 convolution and BN layer.

[0068] Figure 5 This is a structural diagram of the MSFP block proposed in this patent. The MSFP block is composed of a combination of 3×3×3 convolutions, 1×1×1 convolutions, and BN layers. Figure 3 The miniMSFP block shown is an improvement upon this; Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0070] Example:

[0071] A video abnormal behavior recognition method based on attention feature gating, the implementation steps of which are as follows:

[0072] Step 1: Create training samples X. The specific steps are as follows:

[0073] (1) Randomly select n video samples X′ with f frames from the dataset, f∈(1,64]; X′ is c×f a×b matrices, a∈(224,1080); b∈(224,1080); c is the number of channels of the video image, c∈{1,3};

[0074] (2) Resize the n video samples X′ with f frames to obtain the resized video samples X″, where X″ is c×f w×h matrices; w and h are fixed values ​​of 224; then perform maximum and minimum value normalization on X″ to obtain the normalized video samples X″′, where X″′ is c×f w×h matrices; the calculation formula is as follows:

[0075]

[0076] In the formula x′ w " h Represents a single element in matrix X″′, x′ w " h ∈(-1, 1); x′ w ′ h Represents a single element in matrix X″′, x′ w ′ h ∈(0, 255); x min x represents the minimum value of normalization. min =-1; x max x represents the maximum value of the normalized value. max=1;

[0077] (3) Finally, the n normalized video samples X″′ are combined into a set X as training samples, X={X1″, X2″′, X3″′, …, X′ n "}; The training sample X consists of n×c×f w×h matrices; n is the number of normalized video samples X″′ in the training sample X, n∈N;

[0078] Step 2: Construct the AFG-NET network model, feed the training sample X into the AFG-NET network model for training, and obtain the model recognition result Y. The specific steps are as follows:

[0079] (1) Construct the video abnormal behavior recognition model AFG-NET. AFG-NET consists of AFG, miniMSFP, Inception and two-dimensional convolution. The calculation formula is as follows:

[0080] AFG-NET(X)=Inception(miniMSFP(Inception(Conv 3×3 (Conv 1×1 (Conv 7×7 (AFG(X)))))))

[0081] In the formula, Conv 1×1 (·) represents a 1×1 convolution, Conv 3×3 (·) represents a 3×3 convolution, Conv 7×7 (·) represents a 7×7 convolution, AFG represents an attention feature gate layer, and AFG consists of a global pooling GAP(·) and a multi-channel feature sparsification function sp(·), calculated as follows:

[0082] AFG(X) = GAP(X) × sp(X)

[0083] miniMSFP is a small multi-scale fusion block. It is composed of multi-scale average pooling and 1×1 convolution, and the calculation formula is as follows:

[0084]

[0085] In the formula, GAP(·) represents global pooling, and Conv 1×1 (·) represents a 1×1 kernel convolution, N is the number of scales in a miniMSFP, N=3; AP k×k (·) represents average pooling with a kernel of k×k, where k∈{3, 5, 7};

[0086] Inception represents a multi-scale feature extraction block. Inception consists of a 3×3 convolutional Conv... 3×3(·) and 1×1 convolution Conv 1×1 The (·) component is calculated using the following formula:

[0087] Inception(X) = Concat(Conv 1×1 (X), Conv 3×3 (Conv 1×1 (X)), Conv 3×3 (Conv 1×1 (X)))

[0088] In the formula, Concat(X1, X2, X3) represents arranging the matrix features of features X1, X2, and X3 in order to obtain X. Cat X1, X2, and X3 are n×c×f w×h matrices; X Cat There are n×c×3×f w×h matrices;

[0089] (2) Input the training sample X into the AFG-NET network model;

[0090] (3) The AFG layer in the AFG-NET network model performs global feature calculation on the training sample X to obtain the global pooling feature X. GAP Global pooling feature X GAP Let X be an n×c matrix. GAP ∈(-1, 1); the calculation formula is as follows:

[0091] X GAP =GAP(X)=X n×c ·mean(X f×w×h )

[0092] In the formula, GAP(·) represents global pooling of the f, w, and h dimensions of the training sample X, where X represents the training sample. n×c This is an n×c matrix formed by the n and c dimensions of the training samples X; mean(·) represents taking the mean, X f×w×h f w×h matrices representing the f, w, and h dimensions of the training sample X;

[0093] (4) The input layer AFG layer in the AFG-NET network model performs multi-channel feature sparsification on the training sample X to obtain the sparsified features X. SP , sparsification feature X SP It is a set of n×c w×h matrices; the calculation formula is as follows:

[0094]

[0095] In the formula, X represents the training samples, sp(·) is the multi-channel feature sparsification function, and x iIt is the feature matrix of the training sample X expanded along the third dimension for f channels, and the feature matrix x is... i There are n×c w×h matrices; a i Represents the characteristic matrix x i The inhibitory factor, a i The calculation formula is as follows:

[0096]

[0097] In the formula, λ is the channel suppression ratio, λ∈{10n}, n∈{-3,-4,-5}; γ i γ is the weight scaling ratio of the BN layer of the i-th channel in the multi-channel feature sparsification function sp(·). i ∈(0,1); γ i The calculation formula is as follows:

[0098]

[0099] Conv 1×1 (x i ) represents the input of the i-th channel BN layer in the multi-channel feature sparsification function sp(·), Conv 1×1 (x i ) represents n×c w×h matrices; BN(x i ) represents the output of the i-th channel BN layer in the function sp(·), BN(x) i ) represents n×c w×h matrices; γ i The scaling weight of the BN layer for the i-th channel in the representation function F(X), γ i ∈(0,1); β i The bias weight of the i-th channel BN layer in the representative function sp(·) is β. i ∈(0,1);

[0100] (5) Using global pooling feature X GAP The n×c elements and the sparsity feature X SP The output matrix of the AFG layer is obtained by multiplying the n×c w×h matrices respectively. Output matrix There are n×c w×h matrices; the calculation formula is as follows:

[0101]

[0102] The special symbol ⊙ indicates that the elements of the previous matrix are multiplied by the corresponding matrices in the next set;

[0103] (6) Output of AFG The data is fed into AFG-NET for feature extraction, and finally the model prediction matrix X is obtained. outThe calculation formula is as follows:

[0104]

[0105] Model prediction matrix X out Let X be an n×s matrix, and let X be the model prediction matrix. out The model's recognition result Y is obtained by predicting using the maxout function. Y is a vector of length n, representing the recognition result of AFG-NET for n video samples in the training sample X. The calculation formula is as follows:

[0106]

[0107] In the formula Y n The nth element of the model recognition result Y is represented by Y. n ∈N, Y n ∈[1, n]; x ns Represents the prediction matrix X out n×s elements, x ns ∈(0,1),j n Represents the prediction matrix X out The column number of the largest element in the nth row, j n ∈[1, s]; s represents the number of abnormal behavior categories in the video samples of the training sample X;

[0108] Step 3: Use backpropagation to optimize and update the parameters in the AFG-NET network model, and save the network parameter P that achieves the highest recognition accuracy (Acc) in all training rounds. t The specific steps are as follows:

[0109] (1) Backpropagation of error is adopted, and the cross-entropy loss function is used to measure the true result. The distance to the model's recognition result Y, and the L1 norm are used to sparsify the parameters γ of AFG. The above two losses form a joint loss function, and the training loss L, L∈(0,+∞), is calculated. The formula for calculating the joint loss function is as follows:

[0110]

[0111] In the formula Let λg(γ) be the cross-entropy loss function, where λg(γ) is the sparsity penalty for the scaling factor γ of the BN layer in the multi-channel feature sparsification function sp(·), and λ is the scaling factor of the penalty term used to balance these two losses, λ∈{10n}, n∈{-3,-4,-5}; g(·) is the L1 norm penalty formula, calculated as follows:

[0112]

[0113] In the formula, f represents the number of channels in the multi-channel feature sparsification function sp(·), and γ i The scaling weight of the BN layer for the i-th channel in the function sp(·);

[0114] (2) Through each training session of the training sample X, the stochastic gradient descent function SGD is used to backpropagate the loss value L in the AFG-NET network model, thereby improving the model parameters P of the i-th training round of AFG-NET. i Randomly varying the model parameters in the direction of the negative gradient to optimize the network, the formula for updating the model parameters using SGD is as follows:

[0115]

[0116] In the formula, P i P represents the model parameters in the i-th round of training of AFG-NET. i-1 η represents the model parameters in the (i-1)th round of training of AFG-NET, and η represents the optimization step size of SGD, where η∈(0,1); The training loss L represents the training parameters P of the model in the (i-1)th round of training of AFG-NET. i-1 Perform differentiation;

[0117] (3) Calculate the current training round e i Accuracy of AFG-NET model in Chinese i Acc i ∈(0,1), and store the training round e. i AFG-NET model P i e i e represents the current i-th training round. i ∈(1, e m ]; e m For the maximum number of training rounds, e m ∈N;

[0118]

[0119] In the formula, TP is the number of positive samples correctly predicted by the model, TN is the number of negative samples correctly predicted by the model, FP is the number of positive samples incorrectly predicted by the model, and FN is the number of negative samples incorrectly predicted by the model.

[0120] (4) Compare the model recognition accuracy Acc in each training round. i And save Acc i The highest model parameter P i And used as deployment parameter P t The calculation is as follows:

[0121]

[0122] Step 4: Use the deployment parameters P saved in Step 3. t As the network parameters of the AFG-NET model, and with the network parameters set as P t The AFG-NET model is deployed on the recognition terminal. At the same time, the terminal performs video abnormal behavior recognition on the input video and obtains the video abnormal behavior recognition result y, where y is a positive integer representing the abnormal behavior category of the video input, y∈[1,s]; The table below shows the accuracy of the network model proposed in this patent compared with other network models.

[0123] Table 1 Comparison of recognition accuracy between the method of the present invention and different models

[0124] Tab.1Our method improves the recognition rate ofvarious models

[0125]

[0126] [1]He K, Zhang X, Ren S, et al.Deep Residual Learning for ImageRecognition[J]. IEEE, 2016.

[0127] [2] Sandler M, Howard A, Zhu M, et al. Inverted Residuals and LinearBottlenecks: Mobile Networks for Classification, Detection and Segmentation[J]. 2018.

[0128] [3]Huang G, Liu Z, Laurens V, et al. Densely Connected Convolutional Networks[J]. IEEE Computer Society, 2016.

[0129] [4]Tan M,Le Q V.EfficientNet:Rethinking Model Scaling forConvolutional Neural Networks[J].2019.

[0130] [5]Szegedy C,Ioffe S,Vanhoucke V,et al.Inception-v4,inception-ResNetand the impact ofresidual connections on learning.2017.

[0131] [6]Iandola F N,Han S,Moskewicz M W,et al.SqueezeNet:AlexNet-levelaccuracy with 50x fewerparameters and<0.5MB model size[J].2016。

Claims

1. A video abnormal behavior recognition method based on attention feature gating, characterized in that... Includes the following steps: Step 1: Create training samples X. The specific steps are as follows: (1) Randomly select n video samples X′ with f frames from the dataset, f∈(1,64]; X′ is c×f a×b matrices, a∈(224,1080); b∈(224,1080); c is the number of channels of the video image, c∈{1,3}; (2) Resize the n video samples X′ with f frames to obtain the resized video samples X″, where X″ is c×f w×h matrices; w and h are fixed values ​​of 224; then perform maximum and minimum value normalization on X″ to obtain the normalized video samples X″′, where X″′ is c×f w×h matrices; the calculation formula is as follows: In the formula x″′ wh Represents a single element in matrix X″′, x″′ wh ∈(-1, 1); x″ wh Represents a single element in matrix X″′, x″ wh ∈(0, 255); x min x represents the minimum value of normalization. min =-1; x max x represents the maximum value of the normalized value. max =1; (3) Finally, the n normalized video samples X″′ are combined into a set X as training samples, X={X″′1,X″′2,X″′3,…,X″′ n The training samples X consist of n×c×f w×h matrices; n is the number of normalized video samples X″′ in the training samples X, n∈N; Step 2: Construct the AFG-NET network model, feed the training sample X into the AFG-NET network model for training, and obtain the model recognition result Y. The specific steps are as follows: (1) Construct the video abnormal behavior recognition model AFG-NET. AFG-NET consists of AFG, miniMSFP, Inception and two-dimensional convolution. The calculation formula is as follows: AFG-NET(X)=Inception(miniMSFP(Inception(Conv 3×3 (Conv 1×1 (Conv 7×7 (AFG(X))))))) In the formula, Conv 1×1 (·) represents a 1×1 convolution, Conv 3×3 (·) represents a 3×3 convolution, Conv 7×7 (·) represents a 7×7 convolution, AFG represents an attention feature gate layer, and AFG consists of a global pooling GAP(·) and a multi-channel feature sparsification function sp(·), calculated as follows: AFG(X) = GAP(X) × sp(X) miniMSFP is a small multi-scale fusion block. It is composed of multi-scale average pooling and 1×1 convolution, and the calculation formula is as follows: In the formula, GAP(·) represents global pooling, and Conv 1×1 (·) represents a convolution with a kernel of 1×1, and N is the number of scales in the miniMSFP, where N = 3; AP k×k (·) represents average pooling with a kernel of k×k, where k∈{3, 5, 7}; Inception represents a multi-scale feature extraction block. Inception consists of a 3×3 convolutional Conv... 3×3 (·) and 1×1 convolution Conv 1×1 The (·) component is calculated using the following formula: Inception(X)=Concat(Conv 1×1 (X),Conv 3×3 (Conv 1×1 (X)),Conv 3×3 (Conv 1×1 (X))) In the formula, Concat(X1, X2, X3) represents arranging the matrix features of features X1, X2, and X3 in order to obtain X. Cat X1, X2, and X3 are n×c×f w×h matrices; X Cat There are n×c×3×f w×h matrices; (2) Input the training sample X into the AFG-NET network model; (3) The AFG layer in the AFG-NET network model performs global feature calculation on the training sample X to obtain the global pooling feature X. GAP Global pooling feature X GAP Let X be an n×c matrix. GAP ∈(-1, 1); the calculation formula is as follows: X GAP =GAP(X)=X n×c ·mean(X f×w×h ) In the formula, GAP(·) represents global pooling of the f, w, and h dimensions of the training sample X, where X represents the training sample. n×c This is an n×c matrix formed by the n and c dimensions of the training samples X; mean(·) represents taking the mean, X f×w×h f w×h matrices representing the f, w, and h dimensions of the training sample X; (4) The input layer AFG layer in the AFG-NET network model performs multi-channel feature sparsification on the training sample X to obtain the sparsified features X. SP , sparsification feature X SP It is a set of n×c w×h matrices; the calculation formula is as follows: In the formula, X represents the training sample, sp(·) is the multi-channel feature sparsification function, and x i It is the feature matrix of the training sample X expanded along the third dimension for f channels, and the feature matrix x is... i There are n×c w×h matrices; a i Represents the characteristic matrix x i The inhibitory factor, a i The calculation formula is as follows: In the formula, λ is the channel suppression ratio, λ∈{10} n }, n∈{-3,-4,-5};γ i γ is the weight scaling ratio of the BN layer of the i-th channel in the multi-channel feature sparsification function sp(·). i ∈(0,1); γ i The calculation formula is as follows: Conv 1×1 (x i ) represents the input of the i-th channel BN layer in the multi-channel feature sparsification function sp(·), Conv 1×1 (x i ) represents n×c w×h matrices; BN(x i ) represents the output of the i-th channel BN layer in the function sp(·), BN(x) i ) represents n×c w×h matrices; γ i The scaling weight of the BN layer for the i-th channel in the representation function F(X), γ i ∈(0,1); β i The bias weight of the i-th channel BN layer in the representative function sp(·) is β. i ∈(0,1); (5) Using global pooling feature X GAP The n×c elements and the sparsity feature X SP The output matrix of the AFG layer is obtained by multiplying the n×c w×h matrices respectively. Output matrix There are n×c w×h matrices; the calculation formula is as follows: The special symbol ⊙ indicates that the elements of the previous matrix are multiplied by the corresponding matrices in the next set; (6) Output of AFG The data is fed into AFG-NET for feature extraction, and finally the model prediction matrix X is obtained. out The calculation formula is as follows: Model prediction matrix X out Let X be an n×s matrix, and let X be the model prediction matrix. out The model's recognition result Y is obtained by predicting using the maxout function. Y is a vector of length n, representing the recognition result of AFG-NET for n video samples in the training sample X. The calculation formula is as follows: In the formula Y n The nth element of the model recognition result Y is represented by Y. n ∈N, Y n ∈[1, n]; x ns Represents the prediction matrix X out n×s elements, x ns ∈(0,1),j n Represents the prediction matrix X out The column number of the largest element in the nth row, j n ∈[1, s]; s represents the number of abnormal behavior categories in the video samples of the training sample X; Step 3: Use backpropagation to optimize and update the parameters in the AFG-NET network model, and save the network parameter P that achieves the highest recognition accuracy (Acc) in all training rounds. t The specific steps are as follows: (1) Backpropagation of error is adopted, and the cross-entropy loss function is used to measure the true result. The distance to the model's recognition result Y, and the L1 norm are used to sparsify the parameters γ of AFG. The above two losses form a joint loss function, and the training loss L, L∈(0,+∞), is calculated. The formula for calculating the joint loss function is as follows: In the formula Let λg(γ) be the cross-entropy loss function, and let λg(γ) be the sparsity penalty for the scaling factor γ of the BN layer in the multi-channel feature sparsification function sp(·). λ is the scaling factor of the penalty term used to balance these two losses, and λ∈{10} n }, n∈{-3,-4,-5}; g(·) is the L1 norm penalty formula, calculated as follows: In the formula, f represents the number of channels in the multi-channel feature sparsification function sp(·), and γ i The scaling weight of the BN layer for the i-th channel in the function sp(·); (2) Through each training session of the training sample X, the stochastic gradient descent function SGD is used to backpropagate the loss value L in the AFG-NET network model, thereby improving the model parameters P of the i-th training round of AFG-NET. i Randomly varying the model parameters in the direction of the negative gradient to optimize the network, the formula for updating the model parameters using SGD is as follows: In the formula, P i P represents the model parameters in the i-th round of training of AFG-NET. i-1 η represents the model parameters in the (i-1)th round of training of AFG-NET, and η represents the optimization step size of SGD, where η∈(0,1). The training loss L represents the training parameters P of the model in the (i-1)th round of training of AFG-NET. i-1 Perform differentiation; (3) Calculate the current training round e i Accuracy of AFG-NET model in Chinese i Acc i ∈(0,1), and store the training round e. i AFG-NET model P i e i Represents the current i-th training round, e i ∈(1, e m ]; e m For the maximum number of training rounds, e m ∈N; In the formula, TP is the number of positive samples correctly predicted by the model, TN is the number of negative samples correctly predicted by the model, FP is the number of positive samples incorrectly predicted by the model, and FN is the number of negative samples incorrectly predicted by the model. (4) Compare the model recognition accuracy Acc in each training round. i And save Acc i The highest model parameter P i And used as deployment parameter P t The calculation is as follows: Step 4: Use the deployment parameters P saved in Step 3. t As the network parameters of the AFG-NET model, and with the network parameters set as P t The AFG-NET model is deployed on the recognition terminal. At the same time, the terminal performs video abnormal behavior recognition on the input video and obtains the video abnormal behavior recognition result y, where y is a positive integer representing the abnormal behavior category of the video input, and y∈[1,s].

Images