A lightweight violent behavior recognition method under a monitoring scene

Abstract

The application relates to the field of video analysis and behavior recognition, and discloses a light-weight violent behavior recognition method in a monitoring scene, which comprises the following steps: step 1, acquiring video data in the monitoring scene, and pre-processing and data enhancement of the video; step 2, taking the data obtained in step 1 as input, and extracting data space-time features; step 3, performing feature fusion on the double-flow features; and step 4, performing category judgment of violent / non-violent behaviors through a classifier composed of full connection layers. The application solves the problem that violent behaviors cannot be timely and effectively recognized in a video monitoring scene, and further improves the accuracy of violent behavior recognition by using a light-weight deep learning network model and combining various excitation modules under the condition of reducing the model parameter quantity and the calculation amount.

Description

Technical Field

[0001] This invention relates to the field of video analysis and behavior recognition, and more particularly to a lightweight method for recognizing violent behavior in surveillance scenarios. Background Technology

[0002] Public safety has always been a focus of social concern. Timely and effective detection of violence can prevent conflicts from escalating. Surveillance systems are an important means of protecting the personal safety of the people and maintaining social peace and stability. With advancements in hardware technology and reduced costs, surveillance equipment is now ubiquitous. Shopping malls, banks, intersections, squares, stations, and other public places are equipped with numerous cameras. Even many households have installed multiple surveillance systems to form smart home systems for fire prevention, theft prevention, and monitoring of the elderly.

[0003] Traditional surveillance systems rely on manual filtering of video information. A single person can only monitor a limited number of scenes simultaneously, and much of the information in the video is useless. Prolonged observation by the human eye can easily lead to visual fatigue and decreased attention, resulting in missed alarms. Due to its time-consuming, labor-intensive, and inefficient nature, and the tendency to miss alarms, traditional surveillance systems are increasingly unable to meet actual monitoring needs.

[0004] Intelligent video surveillance systems incorporate a video analytics module. This module leverages machine learning and computer vision technologies to fully utilize the powerful data processing capabilities of computers, replacing manual methods for filtering useful and distracting information. Intelligent video surveillance systems can achieve fully automated, 24 / 7, and real-time monitoring. They can issue alarms or take immediate action upon detecting any anomalies in the monitored footage. Compared to traditional monitoring systems, they not only have a lower rate of missed detections but also save on manpower and resources.

[0005] In recent years, the development and application of deep learning technology have greatly promoted the rapid development of computer vision, natural language processing, and other related fields, ushering in a new era of artificial intelligence. Deep learning technology has also achieved excellent results in the research of violent behavior detection, providing new research ideas and solutions for intelligent video surveillance technology. On the one hand, with the support of high-performance GPU devices, deep learning technology can effectively solve the problems of numerous model parameters and large computational loads. Advances in hardware technology provide a fundamental guarantee for the real-time detection and processing of video data in surveillance systems. On the other hand, deep learning models require a sufficient number of training samples during the learning process, and massive amounts of video data can meet the sample requirements for model training. In conclusion, in the context of video surveillance, relying on deep learning technology for violent behavior detection is an important means of rationally allocating resources and efficiently executing security and prevention tasks, possessing significant theoretical and practical value.

[0006] Currently, deep learning-based methods for violent behavior recognition can be broadly categorized into two types: those based on three-dimensional convolutional neural networks (3DCNN) and those based on two-dimensional convolutional neural networks (2DCNN). 3D CNN-based methods can achieve better recognition accuracy, but their model parameters are often excessively large, involving significant computation, requiring high-end equipment, and exhibiting poor real-time performance. 2D CNN-based methods have significantly reduced model parameters and computational cost compared to 3D CNN, but they lose the ability to model temporal information, resulting in lower recognition accuracy.

[0007] Therefore, how to achieve a good balance between model recognition accuracy, number of parameters, and computational cost has become a problem worth exploring. Summary of the Invention

[0008] This invention provides a lightweight method for identifying violent behavior in surveillance scenarios. This method combines multiple attention mechanisms and uses a 2D CNN as the backbone network to construct a lightweight two-stream network model. The model includes a multi-scale spatiotemporal excitation module with regularization constraints for extracting multi-scale spatiotemporal features; a temporal excitation module with regularization constraints, using the output of a cropped 2D CNN as input for temporal feature extraction; and a fusion excitation module with regularization constraints to adaptively calibrate the feature responses of different branch dimensions of the two-stream model. The specific technical solution is as follows:

[0009] A lightweight method for identifying violent behavior in surveillance scenarios includes the following steps:

[0010] Step 1: Acquire video data from the monitoring scene, and perform preprocessing and data augmentation on the video data;

[0011] Step 2: Using the data obtained in Step 1 as input, extract the spatiotemporal features of the data;

[0012] Step 3: Perform feature fusion on the dual-stream features;

[0013] Step 4: Use a classifier composed of fully connected layers to classify violent / non-violent behaviors.

[0014] Further, step 1 is as follows: After acquiring video data from the monitoring scene using a video acquisition tool, the video frame sequence is obtained using OpenCV and converted into an easy-to-process data format using the NumPy library. Then, uniform sampling is used to sample the video sequence to remove redundant information. Image horizontal / vertical flipping, rotation, color perturbation, and light perturbation operations are performed. Subsequently, inter-frame difference and background subtraction operations are performed on the processed video frame sequence, which are used as inputs to the inter-frame difference branch and background subtraction branch of the two-stream network, respectively.

[0015] Furthermore, step 2 is detailed below:

[0016] Step 2.1: Construct a multi-scale spatiotemporal activation module using multiple convolution operators with different kernel sizes to obtain multi-scale spatiotemporal features, and improve the model's resistance to overfitting through regularization constraints.

[0017] Step 2.2: The data processed in step 2.1 is used as input to the cropped lightweight 2D CNN for further spatial feature extraction.

[0018] Step 2.3 uses the data processed in Step 2.2 as input to the time excitation module that applies regularization constraints, explicitly models the interdependencies in the time dimension, and improves the model's resistance to overfitting through regularization constraints.

[0019] Furthermore, step 2.1 is as follows: Input B, T, H, and W represent batch size, time, height of a single video frame, and width of a single video frame, respectively. The channel dimension is 3, and the average value across the channel dimensions is taken to obtain a spatiotemporal feature map containing global channel information.

[0020]

[0021] “:” indicates all possible values ​​in this dimension;

[0022] This feature map is input into 3D convolutional layers K1, K2, and K3 with different kernel sizes to obtain feature maps respectively. The features are integrated using channel splicing to obtain... The process is described as follows:

[0023] F1 = K1 * F

[0024] F2=K2*F

[0025] F3 = K3 * F

[0026] Concat[F1,F2,F3] (2)

[0027] Wherein, Concat represents F i (i = 1, 2, 3) are concatenated along the channel dimensions. A spatiotemporal feature attention map is obtained after passing through a Sigmoid activation layer. Right now

[0028] M=σ(F c (3)

[0029] Where σ represents the Sigmoid activation function. The final output Y can be expressed as:

[0030]

[0031] Furthermore, step 2.3 is as follows: First, the input to this module... Perform average pooling along the spatial dimension to obtain

[0032]

[0033] The number of channels in F is compressed by using a fully connected layer (FC), represented as:

[0034] F r =g1(F) (6)

[0035] in g1 represents the fully connected (FC) layer used for channel compression. Subsequently, global pooling along the channel dimension is performed, encoding the channel features at each time point into global features, resulting in... Represented as

[0036]

[0037] By learning the nonlinear relationships between different time points, a weight matrix graph in the time dimension is finally obtained.

[0038]

[0039] g2 represents two fully connected (FC) layers. The first FC layer is used for dimensionality reduction, and the second FC layer is used for decompression of the time dimension; σ represents the operation of the sigmoid activation layer.

[0040] Feature maps of intermediate layers and the final output It can be represented as

[0041]

[0042]

[0043] Furthermore, step 3 is as follows: First, the output of the background subtraction stream... Inter-frame differential stream output Stacking, represented as

[0044] F = Stack(X) bs ,X diff (11)

[0045] in,

[0046] Then, by repeating the same operations as equations (8) and (9), the intermediate layer output is obtained as follows:

[0047]

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] Advantage 1: The backbone network of this invention uses a 2D CNN, which has lower model parameter count and computational cost compared to methods using 3D CNNs. It also offers better real-time performance, is easier to deploy, and achieves a more ideal recognition accuracy. A good balance is achieved between parameter count, computational cost, real-time performance, and recognition accuracy.

[0050] Advantage 2: This invention employs a variety of lightweight excitation modules, which can effectively improve the model's accuracy in recognizing violent behavior while introducing only a small number of parameters;

[0051] In summary, the technical solution of this invention solves the problem of timely and effective identification of violent behavior in video surveillance scenarios. While reducing the number of model parameters and computational load, it utilizes a lightweight deep learning network model and combines multiple incentive modules to further improve the accuracy of violent behavior identification. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart of the lightweight violent behavior recognition method in the monitoring scenario of the present invention;

[0054] Figure 2 A schematic diagram of the multi-scale spatiotemporal excitation module structure for applying regular constraints according to the present invention;

[0055] Figure 3 A schematic diagram of the time excitation module structure for applying regular constraints in this invention;

[0056] Figure 4 A schematic diagram of the fusion excitation module structure for applying regular constraints to the present invention. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort should fall within the scope of protection of the present invention.

[0058] like Figure 1 As shown, the lightweight violent behavior recognition method in the monitoring scenario of the present invention includes the following steps:

[0059] Step 1: Acquire video data from the monitoring scene, and perform preprocessing and data augmentation on the video data.

[0060] After acquiring video data from the monitoring scene using video capture tools, the video frame sequence is obtained using OpenCV and then converted into an easily processed data format using the NumPy library. Uniform sampling is then applied to the video sequence to remove redundant information. Data enhancement operations such as horizontal / vertical flipping, rotation, color perturbation, and lighting perturbation are then performed. Finally, inter-frame differencing and background subtraction operations are performed on the processed video frame sequence, serving as inputs to the inter-frame differencing branch and the background subtraction branch of the two-stream network, respectively.

[0061] Step 2: Use the data obtained in Step 1 as input to extract the spatiotemporal features of the data.

[0062] Step 2.1 involves constructing a multi-scale spatiotemporal activation module using multiple convolution operators with different kernel sizes to obtain multi-scale spatiotemporal features. Regularization constraints are then used to improve the model's resistance to overfitting.

[0063] like Figure 2 As shown, input B, T, H, and W represent batch size, time, height of a single video frame, and width of a single video frame, respectively. The channel dimension is 3, and the average value across the channel dimensions is taken to obtain a spatiotemporal feature map containing global channel information.

[0064]

[0065] “:” indicates all possible values ​​in this dimension;

[0066] Subsequently, this feature map is input into 3D convolutional layers K1, K2, and K3 with different kernel sizes to obtain feature maps respectively. The features are then integrated using channel concatenation to obtain... This process can be described as follows:

[0067] F1 = K1 * F

[0068] F2=K2*F

[0069] F3 = K3 * F

[0070] Concat[F1,F2,F3] (2)

[0071] Wherein, Concat represents F i (i = 1, 2, 3) are concatenated along the channel dimensions. A spatiotemporal feature attention map is obtained after passing through a Sigmoid activation layer. Right now

[0072] M=σ(F c (3)

[0073] Where σ represents the Sigmoid activation function. The final output Y can be expressed as:

[0074]

[0075] Step 2.2 involves using the data processed in Step 2.1 as input to the pruned, lightweight 2D CNN backbone network for further spatial feature extraction. Pruning is performed for two main reasons: first, to reduce the number of model parameters; and second, to ensure that the dimensionality of the output data matches the dimensionality of the input data to the subsequent time-based stimulus module with regularization constraints.

[0076] In this embodiment, a MobileNetV2 network model with the last 30 layers removed is used as the backbone network for spatial feature extraction. The output feature map shape is (B, T, H, W, C).

[0077] Step 2.3 uses the data processed in Step 2.2 as input to the time-based stimulus module with regularization constraints. This explicitly models the interdependencies in the time dimension, making the model focus more on image frames that correspond to the target event, thus overcoming the limitation of 2D CNNs in extracting temporal features. Regularization constraints are also used to improve the model's resistance to overfitting.

[0078] like Figure 3 As shown, the input to this module is first... Perform average pooling along the spatial dimension to obtain

[0079]

[0080] Next, the number of channels in F is compressed using a fully connected (FC) layer with a compression ratio of 5, which can be expressed as:

[0081] F r =g1(F) (6)

[0082] in g1 represents the fully connected (FC) layer used for channel compression. Subsequently, global pooling along the channel dimension is performed, encoding the channel features at each time point into global features, resulting in... Represented as

[0083]

[0084] Subsequently, the nonlinear relationships between different time points are learned, ultimately yielding a weight matrix diagram along the time dimension.

[0085]

[0086] g2 represents two fully connected (FC) layers. The first FC layer is used for dimensionality reduction, compressing the time dimension to half its original size. The second FC layer decompresses the time dimension. σ represents the operation of the sigmoid activation layer.

[0087] Feature maps of intermediate layers and the final output It can be represented as

[0088]

[0089]

[0090] Step 3: Perform feature fusion on the dual-stream features.

[0091] like Figure 4 As shown, the output of the background subtraction stream is first processed. Inter-frame differential stream output Stacking is represented as

[0092] F = Stack(X) bs ,X diff (11)

[0093] in, Then, the compression and decompression strategies in equation (8) and equation (9) were used again to obtain the intermediate layer output as follows. At this point, the model has learned the importance of different tributaries.

[0094] Since the output tensor will be input into a classifier composed of FC layers for class determination, and FC layers can only process two-dimensional feature maps, it is necessary to process the output tensor Y. tempPerform dimensional reshaping to meet the requirements of the fully connected (FC) layer and obtain the output.

[0095] Step 4: Use a classifier composed of fully connected layers to classify violent / non-violent behaviors.

[0096] The above embodiments of the present invention are merely for illustrative purposes and do not represent the superiority or inferiority of the embodiments.

[0097] In the above embodiments of the present invention, the descriptions of each embodiment have their own focus. For parts that are not described in detail in a certain example, please refer to the relevant descriptions of other embodiments.

[0098] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring lightweight violent behavior recognition in a scene, characterized in that The method comprises the following steps: Step 1, obtaining video data in a monitoring scene, pre-processing and data enhancement are performed on the video data; Step 2, taking the data obtained in step 1 as input, data spatio-temporal feature extraction is performed; Step 3, feature fusion is performed on the double-flow features; Step 4, a classifier composed of full connection layers is used to make a category decision on violent / non-violent behavior; Step 2.1, a multi-scale spatio-temporal excitation module is constructed by using a plurality of convolution operators with different kernel sizes to obtain multi-scale spatio-temporal features, and a regular constraint is used to improve the anti-overfitting ability of the model; Step 2.2, taking the data processed in step 2.1 as the input of the cropped lightweight 2D CNN to further extract spatial features; Step 2.3, taking the data processed in step 2.2 as the input of the time excitation module with regular constraint, explicitly modeling the mutual dependence relationship in the time dimension, and improving the anti-overfitting ability of the model through regular constraint.

2. The method of claim 1, wherein the method of monitoring lightweight violent behavior recognition in a scene is characterized by: Step 1 is as follows: after obtaining the video data in the monitoring scene by using a video acquisition tool, the video frame sequence is obtained by using the OpenCV tool, and the video frame sequence is converted into a data format easy to process by using the numpy library, then the video sequence is sampled to remove the redundant information in the video sequence by using uniform sampling; image horizontal / vertical flipping, rotation, color disturbance and light disturbance operations are performed; then, inter-frame difference operation and background subtraction operation are performed on the video frame sequence processed by the above operations, respectively, to serve as the input of the inter-frame difference branch flow and the background subtraction branch flow of the double-flow network. 3.The method of claim 1, wherein the method further comprises: Step 2.1 is specified as follows: input B, T, H, W denote batch size, time, height of individual video frame picture, width of individual video frame picture, channel dimension is 3, average over channel dimension to get spatio-temporal feature map containing global channel information " indicates all values in this dimension; The feature map is input to 3D convolution layers K1, K2, and K3 with different kernel sizes, to obtain feature maps and the features are integrated in a channel splicing manner to obtain The process is described as F1 = K1 * F F2 = K2 * F F3 = K3 * F Wherein, Concat represents the channel dimension splicing of After the Sigmoid activation layer, the spatiotemporal feature attention map is obtained That is M = σ (F c ) (3) Where sigma represents the Sigmoid activation function; the final output Y can be represented as 4. The method of claim 3, wherein the method of monitoring lightweight violent behavior in a scene is characterized by: Step 2.3 is specified as follows: First, average pooling of spatial dimensions is performed on the input to the module to obtain The channel number of F is compressed by the full connection layer FC, which is represented as F r = g1(F) (6) wherein g1 represents FC layer for channel compression; then global pooling in channel dimension is performed on it, and the channel features at the time node are encoded into global features, obtaining is represented as learning the nonlinear relationship between different time nodes, and finally obtaining a weight matrix graph of the time dimension g2 represents two FC layers, the first FC layer is used for dimension reduction operation, and the second FC layer is used for decompression of the time dimension; sigma represents the operation of the Sigmoid activation layer; Feature maps of the intermediate layers And the final output Can be represented as 5. The method of claim 4, wherein the method of monitoring lightweight violent behavior in a scene is characterized by: Step 3 is specified as follows: First, the output of the background subtraction stream is the output of the interframe difference stream Stacking is performed, denoted as F = Stack(X bs , X diff ) (11) wherein After that, the same operation of equation (8) and equation (9) is adopted again, and the output of the intermediate layer is obtained as

Images