A method for analyzing abnormal behavior of personnel based on deep learning

Abstract

The present invention discloses a method for analyzing abnormal behavior of personnel based on deep learning, which belongs to the field of image data processing technology, and includes the steps of: obtaining a data set D for detecting abnormal behavior of personnel; constructing an improved YOLOv5 network based on a spatial feature enhancement module; training the improved YOLOv5 network to obtain a target detection model; training the SlowFast network to obtain a SlowFast model based on the loss function of the improved YOLOv5 network, the DeepSORT algorithm and the improved SlowFast network; and performing abnormal behavior recognition on the target in the video sample to be recognized based on the target detection model, the DeepSORT algorithm and the SlowFast model. The present invention uses adaptive weights to solve the problem of category imbalance, and uses time information to improve the accuracy of video action recognition, thereby improving the robustness and accuracy of the model for complex scenes, and also reduces the demand for computing resources through lightweight design, providing greater flexibility and efficiency for practical applications.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a method for analyzing abnormal behavior of personnel based on deep learning. Background Art

[0002] With the acceleration of urbanization and the growth of social security needs, security monitoring systems in public places are receiving more and more attention. Traditional security monitoring mainly relies on manual monitoring, which not only consumes a lot of human resources, but also makes it difficult to achieve 24-hour uninterrupted and efficient monitoring due to the limited attention of people. In addition, when an incident occurs, it is often difficult to prevent and stop the occurrence of adverse events in time by reviewing the video footage afterwards. In recent years, with the development of computer vision technology and artificial intelligence, automated personnel behavior analysis has become possible. Existing technical solutions mainly include:

[0003] Rule-based approach: Determine whether an abnormal behavior is detected by presetting a specific behavior pattern. For example, if a person stays in a certain area for more than a certain period of time, it may be considered suspicious behavior. This method is simple and direct, and is suitable for known and fixed behavior types, such as intrusion detection and abandoned item monitoring, but it cannot flexibly respond to new or changing behaviors, lacks flexibility for unknown or non-standard behavior patterns, and is difficult to handle complex scene changes.

[0004] Methods based on traditional machine learning: rely on manually designed features (such as HOG, SIFT, etc.) to describe image content, and then use classifiers (such as SVM, random forest, etc.) to classify behaviors. They can adapt to different behavior patterns to a certain extent, and the models are relatively simple to understand and deploy. However, they require a lot of domain knowledge for feature selection and engineering, which increases development costs and reduces the scalability and efficiency of the system; and these methods have limited generalization capabilities for new environments, especially in complex backgrounds or poor lighting conditions.

[0005] Deep learning-based methods: Use advanced target detection algorithms (such as Faster R-CNN, SSD, and YOLO series) to locate individuals in the video, and combine tracking algorithms (such as SORT and DeepSORT) to maintain consistency and continuity for multiple targets. Introduce action recognition networks specifically for video sequences (such as I3D, TSN, and SlowFast) to capture behavioral patterns that evolve over time. Able to handle more diverse behavioral patterns, with stronger generalization capabilities and higher accuracy. Despite excellent performance, it is still a challenge to maintain accurate tracking of a single individual in a high-density crowd environment, especially in the case of fast-moving targets or complex backgrounds, where false positives or missed detections are prone to occur. Summary of the invention

[0006] The purpose of the present invention is to provide a method for analyzing abnormal behavior of personnel based on deep learning, which can solve the above-mentioned problems and not only maintain accurate tracking of a single individual in a high-density crowd environment, but also is not prone to false alarms or missed detections in fast-moving targets or complex backgrounds.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows: a method for analyzing abnormal behavior of personnel based on deep learning, comprising the following steps;

[0008] S1, obtain a data set D for abnormal behavior detection of personnel, including several video samples, each video sample includes T image frames, and each image frame contains target detection annotations and abnormal behavior annotations;

[0009] S2, construct an improved YOLOv5 network, including S21~S23;

[0010] S21, constructing spatial feature enhancement module;

[0011] The spatial feature enhancement module includes two ConvModule modules ConvModule1 and ConvModule2, and also includes a bottleneck module, a FEM module, a layer normalization module and an addition layer;

[0012] Input features of spatial feature enhancement module The first feature is obtained by ConvModule1 Then it is divided into two paths, one of which is obtained through the bottleneck module to obtain the bottleneck features , and the other path is obtained through the FEM module to obtain the spatial attention feature , After the layer normalization module, The fusion feature is obtained by adding layers , and then get the output features through ConvModule2 ;

[0013] The bottleneck module includes N serially connected bottleneck layers, each bottleneck layer includes 2 serially connected ConvModule modules, and the output end of the bottleneck layer is residually connected to the output end;

[0014] The FEM module includes 3 ConvModule modules: The query matrix Q, key matrix K and value matrix V are generated by three ConvModule modules respectively, and the spatial attention features are obtained by the self-attention mechanism. ;

[0015] S22, obtaining a YOLOv5 network, including a backbone network and a neck network;

[0016] S23, a spatial feature enhancement module is set between the backbone network and the neck network to obtain an improved YOLOv5 network;

[0017] S3, use the abnormal behavior detection data set D to train the improved YOLOv5 network to obtain the target detection model;

[0018] S4, training the SlowFast network, including steps S41 to S44;

[0019] S41, for the video samples in the personnel abnormal behavior detection dataset D, input their image frames into the target detection model in sequence, and output the target detection result of each image frame;

[0020] S42, based on the DeepSORT algorithm, generates motion trajectory information for each target in the video sample;

[0021] S43, train the SlowFast network with video samples containing motion trajectory information and minimize the loss function Update the network parameters of the SlowFast network to obtain the trained SlowFast model. The loss function According to the following formula:

[0022] ,

[0023] Where N is the total number of video samples in a batch during training, and the i-th video sample is marked as , , , They are Adaptive weights, SlowFast network focus loss, temporal consistency loss, is the time consistency loss weight;

[0024] S44, to minimize Train the SlowFast network to obtain the SlowFast model;

[0025] S5, based on the target detection model, DeepSORT algorithm and SlowFast model, abnormal behavior recognition is performed on the target in the video sample to be identified.

[0026] Preferably, the abnormal behavior detection dataset D is the ShanghaiTech Campus Dataset, the Avenue Dataset, the CUHK Avenue Dataset, the Event Dataset and the UCF-Crimes dataset.

[0027] As a preference: the spatial attention feature , fusion features According to the following formula:

[0028] ,

[0029] ,

[0030] In the formula, is the softmax function, is the dimension of the key matrix K, S is the transpose operation, is the layer normalization operation.

[0031] As a preferred embodiment: S3 specifically includes, presetting the number of iterations, taking the image frame of the personnel abnormal behavior detection data set D as input, the annotation of the image frame as the predicted output, training the improved YOLOv5 network until convergence, and obtaining the target detection model.

[0032] As a preferred embodiment: in S43, According to the following formula:

[0033] ,

[0034] In the formula, is the balance factor and γ is the focusing parameter.

[0035] As a preferred embodiment: in S43, , According to the following formulas:

[0036] ,

[0037] ,

[0038] In the formula, For control The hyperparameter of the rate of change, For SlowFast network The predicted abnormal behavior category, for The real abnormal behavior category, , They are The predicted abnormal behavior category probability distribution of the tth and t+1th image frames in , To calculate the L2 distance.

[0039] As a preferred embodiment: S5 is specifically:

[0040] Obtain the video sample to be identified, and obtain the target detection result of each image frame through the target detection model;

[0041] Through the DeepSORT algorithm, motion trajectory information is generated for each target in the video sample to be identified;

[0042] The SlowFast model is used to identify abnormal behaviors and output the abnormal behavior category of the target.

[0043] Preferably, the target detection annotation includes a target category and a bounding box, and the abnormal behavior annotation includes an abnormal behavior category.

[0044] The structure of the YOLOv5 series target detection network is mainly composed of three parts: the backbone network Backbone, the neck network Neck and the head network Head. These three parts work together to achieve efficient feature extraction and target detection of the input image. In the backbone network, the ConvModule module is one of its important modules.

[0045] ConvModule module: This module is commonly used in the target detection networks of the YOLOv5 series to the YOLOv8 series. This module contains three functions: convolution, normalization, and activation. Specifically, it consists of a layer of 3×3Conv2d+BatchNorm+SiLU activation function. Among them, the Conv2d function is an important function in the deep learning library TensorFlow, which is used to implement the convolution operation of the convolutional neural network (CNN). Its input parameters include the input matrix, filter (weight), step size, and padding method, and its output is generally the result after convolution. BatchNorm (Batch-Normalization) is a commonly used deep learning technology used to accelerate the training of neural networks and improve the performance of the model. It normalizes the input in each hidden layer in the network to make the input distribution of each neuron more stable, which helps the convergence and training effect of the network. The SiLU function is an activation function in a neural network. Its full name is Sigmoid Gated Linear Unit, also known as the Swish function. It is a nonlinear activation function that can effectively perform nonlinear transformations on the input of the neural network.

[0046] About the framework of YOLOv5, DeepSORT and SlowFast: In the existing real-time video behavior detection, there is a detection method based on the framework of YOLOv5, DeepSORT and SlowFast. In this framework, YOLOv5 is used for target recognition, DeepSORT is used for target tracking, and SlowFast is used for action recognition. Specifically:

[0047] 1. Target detection: implemented by YOLOv5, the video sample is decomposed into separate image frames, each image frame is provided as input to YOLOv5, and YOLOv5 identifies and locates the target in each image frame, including the target's category probability and target box. The target described in the present invention is a person / person.

[0048] 2. Target tracking: It is implemented by the DeepSORT algorithm. The target frame information output by YOLOv5 is used to establish a feature matching model through the DeepSORT algorithm to associate the targets in different image frames, thereby obtaining the target's motion trajectory information.

[0049] 3. Behavior recognition: This is implemented by the SlowFast network, which uses the target's motion trajectory information obtained in the previous step to extract the target's motion features through the SlowFast algorithm. The extracted features are then classified by the classifier to identify the target's actions. The SlowFast network is specifically designed to handle action recognition tasks in video data and is able to capture behavior patterns that evolve over time. It receives target position information in a series of consecutive image frames to determine whether there is any abnormal behavior.

[0050] On this basis, the present invention improves YOLOv5 by adding a spatial feature enhancement module in its backbone network and neck network. The spatial feature enhancement module is divided into two paths, one is to extract local features through the bottleneck module, and the other is to capture global dependencies using the spatial attention mechanism through the FEM module. Such a design effectively combines local details with global context information, providing richer feature representation for downstream tasks.

[0051] The present invention also improves the loss function of the SlowFast network, based on the existing focus loss Based on the above, adaptive weights and temporal consistency loss are introduced to enhance the model's attention to difficult-to-classify samples and handle the problem of class imbalance. Dynamically adjust according to the difficulty of the sample The temporal consistency loss is used to consider the temporal characteristics of video data to ensure the consistency of prediction results between adjacent frames.

[0052] Compared with the prior art, the advantages of the present invention are:

[0053] (1) The present invention introduces a spatial feature enhancement module in YOLOv5, which can adaptively emphasize the areas in the image that are more important to the task while suppressing irrelevant or redundant information. This allows the model to focus more on the key parts of the target object, thereby improving the feature representation ability and the generalization performance of the model. In the spatial feature enhancement module, two paths are designed to extract local features and capture global dependencies. This design effectively combines local details with global context information, providing richer feature representation for downstream tasks.

[0054] (2) The present invention introduces an adaptive weight mechanism to dynamically adjust the loss weight of each sample, where the adaptation of the i-th video sample is According to its formula, when , The closer it is, the more accurate the prediction is. The larger the value of , the higher the weight that can be given to the video sample. This design allows the model to pay more attention to samples that are difficult to classify, thereby improving the model's ability to learn complex or ambiguous samples.

[0055] (3) The present invention introduces temporal consistency loss and uses the L2 distance between the probability distributions of predicted categories of adjacent image frames to ensure the consistency of prediction results in the video sequence. This module is particularly suitable for the temporal characteristics of video data, ensuring the consistency and stability of predictions between adjacent frames in action recognition tasks. It can effectively improve the performance of the model when processing continuous video streams and enhance the capture of temporal dependencies.

[0056] In summary, the present invention uses adaptive weights to solve the problem of category imbalance and uses temporal information to improve the accuracy of video action recognition, thereby improving the robustness and accuracy of the model for complex scenes. It also reduces the demand for computing resources through lightweight design, providing greater flexibility and efficiency for practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a schematic diagram of the spatial feature enhancement module structure;

[0058] Figure 2 Schematic diagram of the bottleneck layer structure;

[0059] Figure 3 It is a schematic diagram of the FEM module structure;

[0060] Figure 4 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described below in conjunction with embodiments.

[0062] The present invention will be further described below in conjunction with the accompanying drawings.

[0063] Example 1: See Figures 1 to 4 ,A method for analyzing abnormal behavior of personnel based on deep learning, comprises the following steps;

[0064] S1, obtain a data set D for abnormal behavior detection of personnel, including several video samples, each video sample includes T image frames, and each image frame contains target detection annotations and abnormal behavior annotations;

[0065] S2, construct an improved YOLOv5 network, including S21~S23;

[0066] S21, constructing spatial feature enhancement module;

[0067] The spatial feature enhancement module includes two ConvModule modules ConvModule1 and ConvModule2, and also includes a bottleneck module, a FEM module, a layer normalization module and an addition layer;

[0068] Input features of spatial feature enhancement module The first feature is obtained by ConvModule1 Then it is divided into two paths, one of which is obtained through the bottleneck module to obtain the bottleneck features , and the other path is obtained through the FEM module to obtain the spatial attention feature , After the layer normalization module, The fusion feature is obtained by adding layers , and then get the output features through ConvModule2 ;

[0069] The bottleneck module includes N serially connected bottleneck layers, each bottleneck layer includes 2 serially connected ConvModule modules, and the output end of the bottleneck layer is residually connected to the output end;

[0070] The FEM module includes 3 ConvModule modules: The query matrix Q, key matrix K and value matrix V are generated by three ConvModule modules respectively, and the spatial attention features are obtained by the self-attention mechanism. ;

[0071] S22, obtaining a YOLOv5 network, including a backbone network and a neck network;

[0072] S23, a spatial feature enhancement module is set between the backbone network and the neck network to obtain an improved YOLOv5 network;

[0073] S3, use the abnormal behavior detection data set D to train the improved YOLOv5 network to obtain the target detection model;

[0074] S4, training the SlowFast network, including steps S41 to S44;

[0075] S41, for the video samples in the personnel abnormal behavior detection dataset D, input their image frames into the target detection model in sequence, and output the target detection result of each image frame;

[0076] S42, based on the DeepSORT algorithm, generates motion trajectory information for each target in the video sample;

[0077] S43, train the SlowFast network with video samples containing motion trajectory information and minimize the loss function Update the network parameters of the SlowFast network to obtain the trained SlowFast model. The loss function According to the following formula:

[0078] ,

[0079] Where N is the total number of video samples in a batch during training, and the i-th video sample is marked as , , , They are Adaptive weights, SlowFast network focus loss, temporal consistency loss, is the time consistency loss weight;

[0080] S44, to minimize Train the SlowFast network to obtain the SlowFast model;

[0081] S5, based on the target detection model, DeepSORT algorithm and SlowFast model, abnormal behavior recognition is performed on the target in the video sample to be identified.

[0082] In this embodiment: the abnormal behavior detection dataset D is the ShanghaiTech Campus Dataset, Avenue Dataset, CUHK Avenue Dataset, Event Dataset and UCF-Crimes dataset.

[0083] The spatial attention feature , fusion features According to the following formula:

[0084] ,

[0085] ,

[0086] In the formula, is the softmax function, is the dimension of the key matrix K, S is the transpose operation, is the layer normalization operation.

[0087] Specifically, S3 presets the number of iterations, takes the image frame of the abnormal behavior detection dataset D as input, the annotation of the image frame as the predicted output, trains the improved YOLOv5 network until convergence, and obtains the target detection model.

[0088] In S43, According to the following formula:

[0089] ,

[0090] In the formula, is the balance factor and γ is the focusing parameter.

[0091] In S43, , According to the following formulas:

[0092] ,

[0093] ,

[0094] In the formula, For control The hyperparameter of the rate of change, For SlowFast network The predicted abnormal behavior category, for The real abnormal behavior category, , They are The predicted abnormal behavior category probability distribution of the tth and t+1th image frames in , To calculate the L2 distance.

[0095] S5 is specifically as follows: obtaining a video sample to be identified, and obtaining the target detection result of each image frame through the target detection model; generating motion trajectory information for each target in the video sample to be identified through the DeepSORT algorithm; performing abnormal behavior identification through the SlowFast model, and outputting the abnormal behavior category of the target.

[0096] The target detection annotation includes a target category and a bounding box, and the abnormal behavior annotation includes an abnormal behavior category.

[0097] In this embodiment, regarding the annotation of the data set: the public data set is selected during the training process of the present invention, and the samples are all annotated, including target detection annotation and abnormal behavior annotation. If the self-made data set is used, the following methods can be used for annotation:

[0098] (1) Object detection and annotation: For each image frame in the video stream, mark all the people or other related objects that appear, and assign category labels and bounding boxes to them. Category labels include people, vehicles, etc. This step is usually implemented using labeling tools such as LabelImg, CVAT, etc.

[0099] (2) Behavior segmentation: Identify time periods containing abnormal behaviors and extract these time periods from the video stream as independent behavior segments. Each segment should have clear start and end timestamps.

[0100] (3) For each behavior segment, specify the corresponding abnormal behavior category: The category definition should be as detailed and representative as possible, covering all possible abnormal situations. For example, in a security monitoring scenario, it can include running, falling, fighting, and leaving items behind.

[0101] Example 2: See Figures 1 to 4 In order to verify the effectiveness of the method proposed in this invention, we selected the public ShanghaiTech Campus Dataset, which was created by Shanghai Jiao Tong University and contains video clips of pedestrian behavior in a campus environment. It is particularly suitable for testing crowd abnormal behavior detection systems. In the experiment, we used the average frame-level AUC (Area Under the Curve) as the evaluation indicator, which is a widely accepted method for measuring the performance of binary classification models, especially in anomaly detection tasks. In this experiment, we compared six different model methods, including the present invention. These models cover the current mainstream abnormal behavior detection technologies. The results are shown in Table 1 below:

[0102] Table 1. Abnormal behavior detection results of different models

[0103] method FFP MemAE MTP MPN LLSH The present invention AUC 72.8% 71.2% 76.0% 73.8% 77.6% 79.3% ,

[0104] As can be seen from Table 1, the present invention achieves the highest average frame-level AUC value of 79.3%, which is significantly higher than the other five methods. This result shows that by introducing the spatial feature enhancement module and adaptive weighted loss, the present invention not only improves the robustness and accuracy of abnormal behaviors in complex scenarios, but also effectively improves the flexibility and efficiency of the system.

[0105] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for analyzing abnormal behavior of personnel based on deep learning, characterized by: The steps include: S1, obtain a data set D for abnormal behavior detection of personnel, including several video samples, each video sample includes T image frames, and each image frame contains target detection annotations and abnormal behavior annotations; S2, construct an improved YOLOv5 network, including S21~S23; S21, constructing spatial feature enhancement module; The spatial feature enhancement module includes two ConvModule modules ConvModule1 and ConvModule2, and also includes a bottleneck module, a FEM module, a layer normalization module and an addition layer; Input features of spatial feature enhancement module The first feature is obtained by ConvModule1 Then it is divided into two paths, one of which is obtained through the bottleneck module to obtain the bottleneck features , and the other path is obtained through the FEM module to obtain the spatial attention feature , After the layer normalization module, The fusion feature is obtained by adding layers , and then get the output features through ConvModule2 ; The bottleneck module includes N serially connected bottleneck layers, each bottleneck layer includes 2 serially connected ConvModule modules, and the input and output ends of the bottleneck layer are residually connected; The FEM module includes 3 ConvModule modules: The query matrix Q, key matrix K and value matrix V are generated by three ConvModule modules respectively, and the spatial attention features are obtained by the self-attention mechanism. ; S22, obtaining a YOLOv5 network, including a backbone network and a neck network; S23, a spatial feature enhancement module is set between the backbone network and the neck network to obtain an improved YOLOv5 network; S3, use the abnormal behavior detection data set D to train the improved YOLOv5 network to obtain the target detection model; S4, training the SlowFast network, including steps S41 to S44; S41, for the video samples in the personnel abnormal behavior detection dataset D, input their image frames into the target detection model in sequence, and output the target detection result of each image frame; S42, based on the DeepSORT algorithm, generates motion trajectory information for each target in the video sample; S43, train the SlowFast network with video samples containing motion trajectory information and minimize the loss function Update the network parameters of the SlowFast network to obtain the trained SlowFast model. The loss function According to the following formula: ， Where N is the total number of video samples in a batch during training, and the i-th video sample is marked as , , , They are Adaptive weights, SlowFast network focus loss, temporal consistency loss, is the time consistency loss weight; S44, to minimize Train the SlowFast network to obtain the SlowFast model; S5, based on the target detection model, DeepSORT algorithm and SlowFast model, abnormal behavior recognition is performed on the target in the video sample to be identified.

2. According to the method for analyzing abnormal behavior of personnel based on deep learning according to claim 1, it is characterized in that: The abnormal behavior detection dataset D includes ShanghaiTech Campus Dataset, Avenue Dataset, CUHK Avenue Dataset, Event Dataset and UCF-Crimes.

3. According to the method for analyzing abnormal behavior of personnel based on deep learning according to claim 1, it is characterized in that: The spatial attention feature , fusion features According to the following formula: ， ， In the formula, is the softmax function, is the dimension of the key matrix K, S is the transpose operation, is the layer normalization operation.

4. The method for analyzing abnormal behavior of personnel based on deep learning according to claim 1, characterized in that: Specifically, S3 presets the number of iterations, takes the image frame of the abnormal behavior detection dataset D as input, the annotation of the image frame as the predicted output, trains the improved YOLOv5 network until convergence, and obtains the target detection model.

5. The method for analyzing abnormal behavior of personnel based on deep learning according to claim 1, characterized in that: In S43, According to the following formula: ， In the formula, is the balance factor, γ is the focusing parameter, For SlowFast network The predicted abnormal behavior category.

6. The method for analyzing abnormal behavior of personnel based on deep learning according to claim 1, characterized in that: In S43, , According to the following formulas: ， ， In the formula, For control The hyperparameter of the rate of change, For SlowFast network The predicted abnormal behavior category, for The real abnormal behavior category, , They are The predicted abnormal behavior category probability distribution of the tth and t+1th image frames in , To calculate the L2 distance.

7. The method for analyzing abnormal behavior of personnel based on deep learning according to claim 1, characterized in that: S5 specifically includes: Obtain the video sample to be identified, and obtain the target detection result of each image frame through the target detection model; Through the DeepSORT algorithm, motion trajectory information is generated for each target in the video sample to be identified; The SlowFast model is used to identify abnormal behaviors and output the abnormal behavior category of the target.

8. The method for analyzing abnormal behavior of personnel based on deep learning according to claim 1, characterized in that: The target detection annotation includes a target category and a bounding box, and the abnormal behavior annotation includes an abnormal behavior category.

Images