Abnormal behavior detection algorithm based on deep learning

Abstract

The present invention discloses an abnormal behavior detection algorithm based on deep learning, and specifically relates to the technical field of target detection. The Yolov5 of the present invention has a lightweight model structure, which significantly improves the detection speed and is more real-time compared with other technologies; and BNFNet is a lightweight classification network designed by the present invention, which has a simple and efficient structure and can effectively extract feature information of sub-images. When used in combination with Yolov5, it not only ensures real-time performance, but also improves the network's ability to distinguish abnormal behaviors.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to an abnormal behavior detection algorithm based on deep learning. Background Art

[0002] Abnormal behavior detection technology is mainly used to identify and prevent abnormal or harmful behavior patterns, and is widely used in video surveillance, network security, financial transaction monitoring and other fields. Abnormal behavior detection technology is mainly divided into two categories: traditional machine learning methods and deep learning methods. Abnormal behavior detection based on traditional machine learning generally uses manual feature methods to extract behavioral features, which can extract features such as pedestrian appearance, movement, and time and space. Feature extraction refers to the process of extracting key information from video data to characterize behavior. Abnormal behavior detection based on deep learning methods generally uses convolutional neural networks (CNN) and recurrent convolutional neural networks (RNN) to learn the feature representation of data through multi-layer networks. Machine learning methods require a large amount of labeled data for training, the model has poor interpretability, and is sensitive to hyperparameters, while deep learning methods require a large amount of computing resources and data, and the model is more black-box and has poor interpretability.

[0003] Video communication, processing and understanding are the three core technologies of video surveillance technology. At present, the mainstream network video surveillance system consists of embedded video surveillance terminals and PC-based network video servers.

[0004] There are two main methods for detecting abnormal behavior: local target detection and bilinear detection, which include three functional modules:

[0005] 1. Separate foreground and background and detect moving targets;

[0006] 2. Extract features of the target to represent motion behavior;

[0007] 3. Use anomaly detection algorithms to identify behaviors.

[0008] Local target detection methods usually only focus on local areas in the video, and tend to ignore global context information, resulting in insufficient understanding of the overall scene. When the target is occluded by other objects, local target detection methods may not be able to accurately identify the target, and are prone to missed detection or false detection. Light changes can affect the appearance characteristics of local targets, resulting in reduced detection accuracy. When processing large-scale video data, local target detection methods may require a lot of computing resources, resulting in reduced real-time and efficiency. Bilinear detection methods need to consider the interaction between targets, involving more complex model and algorithm design, which increases the difficulty of implementation and debugging. Usually a large amount of labeled data is required for training, and high requirements are placed on data quality and quantity. The parameter settings in the bilinear detection method have a great impact on the detection effect, and require experienced professionals to tune, lacking versatility and adaptability. In complex scenarios, such as multi-target interaction, fast movement, etc., the bilinear detection method may not be able to effectively capture and identify abnormal behaviors. Summary of the invention

[0009] To this end, the present invention provides an abnormal behavior detection algorithm based on deep learning to solve the problems raised in the background technology.

[0010] In order to achieve the above object, the present invention provides the following technical solution: an abnormal behavior detection algorithm based on deep learning, comprising:

[0011] (1) Preparation of datasets;

[0012] First, different abnormal behavior materials are shot at the same angle of different scenes in the experimental environment and at different angles of the same scene; the acquired video data is extracted from interval frame images as required; and the acquired images are labeled with the categories of abnormal behaviors;

[0013] (2) Building a target detection model;

[0014] The input end, Backbone, Neck, and Prediction of the target detection model form the target detection network Yolo v5. The input end of Yolo v5 needs to preprocess the image.

[0015] Yolo (You only look once): object detection algorithm

[0016] CSPDarknet53 (Cross Stage Partial Darknet53): Cross-stage partial Darknet53 model

[0017] The specific steps of image preprocessing of the target detection model Yolo v5 include: using the adaptive anchor frame calculation and adaptive image scaling methods to scale the input image to the same size as the input required by the target detection model Yolo v5, and then normalizing, randomly cropping, color space conversion, and using Mosaic data enhancement to improve the training speed and network accuracy of the target detection model Yolo v5;

[0018] (3) Construction of classification model;

[0019] Build a lightweight classification model. Use DWConv to build a Bottleneck block with a residual structure, and use a dual-branch merging structure to build a Fire block with feature compression and expansion capabilities. Connect the Bottleneck block and the Fire block by adjusting the channel dimension to form a BNF block. Stack the BNF blocks twice to form the classification model BNFNet.

[0020] (4) Model training: The image data labeled with abnormal behaviors are separated into a training set for training the target detection model Yolo v5. During the training process, the parameters are iteratively updated through back propagation, including the parameters of the convolution layer, the activation layer, and the normalization layer. When the network training reaches a certain level and the accuracy of the validation set no longer increases, the network training stops and the target detection model Yolo v5 with the best test results on the validation set is saved.

[0021] After labeling the acquired images with abnormal behavior categories, extract the labeled abnormal behavior sub-images, classify and save all sub-images and use them for training the classification model BNFNet. When the network training reaches a certain level and the accuracy of the verification set no longer increases, the network training stops and the classification model BNFNet with the best test results on the verification set is saved.

[0022] (5) Model connection: The coordinate information of the target detection model Yolo v5 is obtained, the sub-image corresponding to the prediction box is extracted and processed, and then passed to the classification model BNFNet to further determine the category of the detected abnormal behavior and output the correct category to the reasoning process.

[0023] Preferably, Labelimg software is used to label the acquired images with categories of abnormal behaviors and to separate training sets.

[0024] Preferably, Backbone adopts the CSPDarknet53 structure and uses the Focus structure as the baseline network; Neck is a series of network layers used to combine and mix image features, improve the robustness and diversity of features, and transmit image features to the prediction layer; Prediction predicts image features, generates bounding boxes and predicts the category of the target.

[0025] Preferably, the training of the target detection model Yolo v5 adopts the SGD optimizer, and the initial learning rate is set to 0.01, and the loss function adopts CIoU Loss.

[0026] Preferably, the classification model BNFNet is trained using the Adam optimizer, the initial learning rate is set to 0.001, and the loss function uses the Cross Entropy Loss.

[0027] Preferably, the one-dimensional data output by the target detection model Yolo v5 includes, in addition to coordinate information, coordinate information, confidence and category information.

[0028] The present invention has the following advantages:

[0029] The embodiments of the present invention can monitor abnormal behaviors in the large environment of the library, especially in the reading space, including but not limited to falling, fighting and smoking. Yolo v5 has a lightweight model structure, which significantly improves the detection speed and is more real-time compared with other technologies. BNFNet is a lightweight classification network designed by the present invention. It has a simple and efficient structure and can effectively extract the feature information of sub-images. When used in combination with Yolo v5, compared with the existing technology, it not only ensures real-time performance, but also improves the network's ability to distinguish abnormal behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Provide a schematic diagram of the algorithm structure for this embodiment;

[0031] Figure 2 This is an example diagram of the algorithm processing flow and results provided in this embodiment. DETAILED DESCRIPTION

[0032] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Embodiment 1: This embodiment provides an abnormal behavior detection algorithm based on deep learning, including:

[0034] (1) Preparation of datasets;

[0035] First, different abnormal behavior materials are shot at the same angle of different scenes in the experimental environment and at different angles of the same scene; the acquired video data is extracted from interval frame images as required; and the acquired images are labeled with the categories of abnormal behaviors;

[0036] (2) Building a target detection model;

[0037] The input end, Backbone, Neck, and Prediction of the target detection model form the target detection network Yolo v5. The input end of Yolo v5 needs to preprocess the image.

[0038] The specific steps of image preprocessing of the target detection model Yolo v5 include: using the adaptive anchor frame calculation and adaptive image scaling methods to scale the input image to the same size as the input required by the target detection model Yolo v5, and then normalizing, randomly cropping, color space conversion, and using Mosaic data enhancement to improve the training speed and network accuracy of the target detection model Yolo v5;

[0039] (3) Construction of classification model;

[0040] Build a lightweight classification model. Use DWConv to build a Bottleneck block with a residual structure, and use a dual-branch merging structure to build a Fire block with feature compression and expansion capabilities. Connect the Bottleneck block and the Fire block by adjusting the channel dimension to form a BNF block. Stack the BNF blocks twice to form the classification model BNFNet.

[0041] Among them, DWConv (Depthwise Separable Convolution) refers to depthwise separable convolution.

[0042] (4) Model training: The image data labeled with abnormal behaviors are separated into a training set for training the target detection model Yolo v5. During the training process, the parameters are iteratively updated through back propagation, including the parameters of the convolution layer, the activation layer, and the normalization layer. When the network training reaches a certain level and the accuracy of the validation set no longer increases, the network training stops and the target detection model Yolo v5 with the best test results on the validation set is saved.

[0043] After labeling the acquired images with abnormal behavior categories, extract the labeled abnormal behavior sub-images, classify and save all sub-images and use them for training the classification model BNFNet. When the network training reaches a certain level and the accuracy of the verification set no longer increases, the network training stops and the classification model BNFNet with the best test results on the verification set is saved.

[0044] (5) Model connection: The coordinate information of the target detection model Yolo v5 is obtained, the sub-image corresponding to the prediction box is extracted and processed, and then passed to the classification model BNFNet to further determine the category of the detected abnormal behavior and output the correct category to the reasoning process.

[0045] Preferably, Labelimg software is used to label the acquired images with categories of abnormal behaviors and to separate training sets.

[0046] Preferably, Backbone adopts the CSPDarknet53 structure and uses the Focus structure as the baseline network; Neck is a series of network layers used to combine and mix image features, improve the robustness and diversity of features, and transmit image features to the prediction layer; Prediction predicts image features, generates bounding boxes and predicts the category of the target.

[0047] Preferably, the training of the target detection model Yolo v5 adopts the SGD optimizer, and the initial learning rate is set to 0.01, and the loss function adopts CIoU Loss.

[0048] Among them, CIoU Loss (Complete Intersection over Union Loss) refers to the complete intersection over union loss, and SGD (Stochastic Gradient Descent) refers to the stochastic gradient descent method.

[0049] Preferably, the classification model BNFNet is trained using the Adam optimizer, the initial learning rate is set to 0.001, and the loss function uses the Cross Entropy Loss.

[0050] Among them, Adam (Adaptive Moment Estimation) refers to adaptive moment estimation.

[0051] Preferably, the one-dimensional data output by the target detection model Yolo v5 includes, in addition to coordinate information, coordinate information, confidence and category information.

[0052] Example 2

[0053] Based on Example 1, this example further discloses an abnormal behavior detection algorithm based on deep learning, which is characterized by: using the above detection algorithm to realize abnormal behavior monitoring of readers in the smart reading space.

[0054] Specifically, Example 2 is described in detail through the following implementation scenario.

[0055] The method for detecting abnormal behavior of readers in a smart reading space based on deep learning provided by the present invention, for the training of the target detection model Yolo v5, firstly preprocesses the input image of the library reading space, especially performs Mosaic data enhancement, enriches the detection data set, improves the generalization of the network, and then imports the pre-trained weights trained on the coco data set. After a certain number of training iterations, the network accuracy reaches fitting, and the target detection model Yolo v5 weights are saved.

[0056] For the training of the classification model BNFNet, the input image data is first scaled to obtain a scale that meets the expected input of the network, and then the network is initialized by kaiming to accelerate the learning convergence of the network. Similarly, after a certain number of training iterations, the network accuracy reaches fitting and the model weights are saved. Finally, the two networks are connected, and the respective best model weights are imported. The abnormal behavior image of the library to be tested is input, and the preliminary judgment result is first obtained through the target detection model Yolo v5. The judgment result is further judged by the classification model BNFNet to obtain the final abnormal behavior judgment. The structural reference of the abnormal behavior detection algorithm based on deep learning provided by the present invention Figure 1 , the detailed steps are described below:

[0057] Step 1: Dataset production. The library of Nanjing University of Aeronautics and Astronautics was used as the data collection scene. Different abnormal behavior materials were shot at the same angle of different scenes in the library and at different angles of the same scene. The shooting format was MP4, the resolution was 1080p, the frame rate was 60hz, and the shooting device was iPhone13. For all the videos shot, sampling was performed every 30 frames, and the sampled images totaled 3050. Then, Labelimg software was used to annotate the data set. The abnormal behaviors shot included falling, fighting, and smoking. For the behavior of falling, it was marked as stumble, and the target area was selected. For the behavior of fighting, it was marked as fight, and the target area was selected. For the behavior of smoking, the smoke was marked as smoke, and other long objects similar to smoke were marked as no_smoke. In the end, there were 1766 images containing smoke, 390 containing no_smoke, 365 containing fight, and 529 containing stumble. The annotation process was completed by two researchers in this field to reduce the influence of human subjective factors. The above data will be used for training the Yolo v5 network. For the data of the classification network, this embodiment extracts all the marked areas and saves them as sub-images of the corresponding categories. Finally, 2278 images of the smoke class, 1648 images of the no_smoke class, 556 images of the stumble class, and 365 images of the fight class are obtained, totaling 4847 images.

[0058] Step 2: Network construction. The overall network of this embodiment includes two parts, namely the target detection model Yolov5 and the classification model BNFNet. The construction of the target detection model Yolo v5 consists of the input end, Backbone, Neck, and Prediction. The input end first stretches the image size to 608×608, and then uses Mosaic data enhancement, that is, four images are spliced ​​by random scaling, random cropping, and random arrangement to form a new image. In addition, the input end will also adaptively calculate the anchor frame. When the training starts, the corresponding starting prediction frame is generated according to the initial anchor frame. Backbone uses the Focus structure to slice the input image, that is, each image takes a value every other pixel, so as to obtain four images. These four images will not lose information, but concentrate the width and height information in the channel space and expand it to the channel dimension. In addition, Backbone also uses the CSP structure. CSP splits the feature map into two parts, one of which is convolved, and the other is spliced ​​with the convolution result of the first part in the channel dimension. The CSP structure is mainly used to solve the problem of excessive computation in training from the perspective of network structure design. In the process of network optimization, the repeated calculation of gradient information leads to excessive computation. Neck adopts the FPN+PAN structure. PAN draws on the idea of ​​PANet in the field of image segmentation and outputs feature maps of different scales after downsampling. FPN is a top-down fusion and transmission of high-level feature information through upsampling, and finally obtains the predicted feature map. There is a bottom-up feature pyramid after FPN, and two PAN structures are included in it. In the FPN structure, the top-down transmission can extract strong semantic features, while the bottom-up transmission can extract strong positioning features. Through this structure, the feature pyramid can aggregate parameters between different backbone layers and detection layers, so as to achieve better detection effect. Prediction includes Bounding box loss function and NMS non-maximum suppression, among which the Bounding box loss function uses CIoU_Loss as the loss function. The BNFNet network is mainly composed of a Bottom block and a Fire block. The Bottle block of the present invention uses DWConv inside. The detailed description is that the input feature map first passes through a 1×1 convolution layer, then a 3×3 DWConv convolution layer, and then a 1×1 convolution layer to obtain the output and then add it to the input. Each convolution layer is linked to a BatchNorm and a ReLU activation function.Fire block uses a dual-branch structure to compress and then expand the feature map to further extract feature information. The detailed description is that the input feature map first passes through a 3×3 convolution layer to reduce the channel dimension and compress the feature information into high-level semantic information, and then expands it through two branches, one of which is a 1×1 convolution layer and the other is a 3×3 DWConv convolution layer. The convolution operations of the two branches will expand the channel dimension and amplify the high-level semantic information. Finally, the outputs of the two branches are merged in the channel dimension, and each convolution layer is connected to a BatchNorm and a ReLU activation function. In addition, in order to enable this algorithm to process video stream data, the input video stream data or real-time monitoring video stream data will be transformed into frames of images as input, and the prediction box and confidence information output by the network will be returned to each frame of the image and streamed again as video output.

[0059] Step 3: Model training. In this embodiment, the PyTorch deep learning framework is used to build all networks. The programming language used is python3.6, the CPU is Intel i7-77003.6GHz, and the GPU is NVIDIA RTX 2080ti. First, the data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:1. The training set is used for network training, validation set testing, and the best weights are saved. The test set identifies the generalization effect of the network. The training of the target detection model Yolo v5 uses the official pre-trained weights, SGD is used as the optimizer for model training, the batch size is 4, the initialization learning rate is 0.01, the momentum is 0.9, and the training rounds are 300 rounds. Since the classification model BNFNet is a network proposed by the present invention, there is no pre-trained weight. Adam is used as the optimizer for model training, cross entropy is used as the loss function, the batch size is 4, the initialization learning rate is 0.001, the weight decay is 0.0001, the batch size is 8, and the training rounds are 200.

[0060] Step 4: Effect evaluation. In order to verify the performance of the model, the test is conducted on the test set. The images of the test set are processed by the target detection model Yolo v5 to obtain the location information of the determination area, and the sub-images of the area to which they belong are passed to the classification model BNFNet to obtain the category determination of abnormal behavior. Figure 2 shown.

[0061] In order to ensure the stability of the order of the library reading space and overcome the lack of real-time performance and low accuracy of abnormal behavior judgment of existing abnormal behavior detection technology, the present invention provides an abnormal behavior detection algorithm with secondary classification capability based on the target detection model Yolo v5. In actual scenarios, especially in the reading space of the library, the shooting area can be monitored in real time through a high-position camera to timely identify three abnormal behaviors: falling, fighting and smoking.

[0062] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. Abnormal behavior detection algorithm based on deep learning, characterized by: include: (1) Preparation of datasets; First, different abnormal behavior materials are shot at the same angle of different scenes in the experimental environment and at different angles of the same scene; the acquired video data is extracted from interval frame images as required; and the acquired images are labeled with the categories of abnormal behaviors; (2) Building a target detection model; The input end, Backbone, Neck, and Prediction of the target detection model form the target detection network Yolo v5. The input end of Yolo v5 needs to preprocess the image. The specific steps of image preprocessing of the target detection model Yolo v5 include: using the adaptive anchor frame calculation and adaptive image scaling methods to scale the input image to the same size as the input required by the target detection model Yolo v5, and then normalizing, randomly cropping, color space conversion, and using Mosaic data enhancement to improve the training speed and network accuracy of the target detection model Yolo v5; (3) Construction of classification model; Build a lightweight classification model. Use DWConv to build a Bottleneck block with a residual structure, and use a dual-branch merging structure to build a Fire block with feature compression and expansion capabilities. Connect the Bottleneck block and the Fire block by adjusting the channel dimension to form a BNF block. Stack the BNF blocks twice to form the classification model BNFNet. (4) Model training: The image data annotated with abnormal behaviors are separated into a training set for training the target detection model Yolo v5. During the training process, the model parameters are iteratively updated through back propagation, including the convolution layer parameters, activation layer parameters, and normalization layer parameters. When the network training reaches a certain level and the accuracy of the validation set no longer increases, the network training stops and the target detection model Yolo v5 with the best test results on the validation set is saved. After labeling the acquired images with abnormal behavior categories, extract the labeled abnormal behavior sub-images, classify and save all sub-images and use them for training the classification model BNFNet. When the network training reaches a certain level and the accuracy of the verification set no longer increases, the network training stops and the classification model BNFNet with the best test results on the verification set is saved. (5) Model connection: The coordinate information of the target detection model Yolo v5 is obtained, the sub-image corresponding to the prediction box is extracted and processed, and then passed to the classification model BNFNet to further determine the category of the detected abnormal behavior and output the correct category to the reasoning process.

2. The abnormal behavior detection algorithm based on deep learning according to claim 1, characterized in that: Labelimg software was used to label the acquired images with abnormal behavior categories and to separate them into training sets.

3. The abnormal behavior detection algorithm based on deep learning according to claim 1, characterized in that: Backbone adopts the CSPDarknet53 structure and uses the Focus structure as the baseline network; Neck is a series of network layers used to combine and mix image features, improve the robustness and diversity of features, and transmit image features to the prediction layer; Prediction predicts image features, generates bounding boxes and predicts the category of the target.

4. The abnormal behavior detection algorithm based on deep learning according to claim 1, characterized in that: The training of the target detection model Yolo v5 uses the SGD optimizer, sets the initial learning rate to 0.01, and uses CIoU Loss as the loss function.

5. The abnormal behavior detection algorithm based on deep learning according to claim 1, characterized in that: The classification model BNFNet is trained using the Adam optimizer, with the initial learning rate set to 0.001 and the loss function using Cross EntropyLoss.

6. The abnormal behavior detection algorithm based on deep learning according to claim 1, characterized in that: The one-dimensional data output by the target detection model Yolo v5 includes not only coordinate information but also confidence and category information.

Images