A system and method for detecting violent behavior in surveillance video
By adopting 3D convolutional neural network and CenterNet object detection network in the field of video understanding, combined with data annotation and image enhancement technology, the problems of difficulty in encoding dynamic motion information and high calculation cost in video understanding in the prior art are solved, and high efficiency is achieved , accurate detection of violent behavior.
Patent Information
- Application Number
- CN202210317073.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-29
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-03-29
AI Technical Summary
In the field of video understanding, brute-action detection methods based on 2DCNN and RNN are difficult to effectively encode dynamic motion information. Although 3D convolutional neural networks improve performance, their calculation costs are high and their robustness to combat changes are limited, making it difficult to achieve. Efficient video understanding.
A violent behavior detection system in surveillance video is adopted, including data extraction, data annotation, image enhancement, model training and violent behavior detection modules, and is decomposed into 2D spatial convolution and 1D temporal convolution using 3D convolution neural network (P3D). In addition, combined with the CenterNet object detection network, model training is carried out by minimizing the loss function to achieve accurate detection of violent behavior in video.
It realizes accurate identification of violent behavior in video and high-precision target detection, reduces calculation costs and the number of parameters, improves the robustness of counter-change changes, and is suitable for brute-action detection of video streams in real-time monitoring.
Smart Images

Figure BDA0003570133080000031 
Figure BDA0003570133080000033 
Figure BDA0003570133080000036
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of deep learning, and in particular relates to a system and method for detecting violent behavior in surveillance videos. Background Art
[0002] With the development of information technology, intelligent surveillance equipment has been widely used throughout the country. Violent scenes can be recorded by surveillance cameras, and detecting violent incidents in these videos is crucial to public safety. However, with a large number of videos generated every second, manual detection by staff can only be used for a limited number of videos. Therefore, from a practical application point of view, automated violence detection is a meaningful research that can be applied to many fields, such as intelligent surveillance, evaluating videos uploaded to mobile applications, and prison guard robots. Therefore, it is of far-reaching significance to study an effective method that can automatically monitor and detect violent behavior in surveillance videos.
[0003] Computer vision has been developing continuously with the improvement of computing power and the availability of large-scale datasets. Deep learning is a key technology in computer vision and has achieved remarkable milestones in many fields such as image classification and object detection, so it can be introduced to solve the problem of violence detection. Compared with methods based on hand-crafted features, deep learning methods have produced huge improvements in robustness and accuracy. However, this method has limitations in its application to video understanding, because videos are time sequences of frames, and 2DCNN cannot encode dynamic motion information. To this end, some scholars have combined video sequences with recurrent neural networks (RNN). In addition, since 3D convolutional neural networks (C3D) encode spatiotemporal information by extending 2D filters to the time axis, C3D is introduced to deal with the problem of violence detection. By using 3D filters to capture spatial and temporal information in videos, they are successfully applied to video action recognition tasks.
[0004] Recurrent neural network (RNN)-based violent behavior detection methods encode dynamic motion information to some extent, however, these methods have performance flaws as they do not perform convolutions across multiple frames in the early layers of the network. Video-based action recognition is a challenging task because traditional 2DCNN models do not encode temporal information.
[0005] For the above problems, some people have proposed to use 3D convolutional neural networks to encode temporal information. Such models have significantly improved performance, but they are computationally expensive and require a large number of parameters. In addition, the features extracted by 3D convolutional neural networks are very effective in combating various spatial changes caused by changes in lighting, posture, scale, etc., but their performance for large-scale video understanding is still limited. Summary of the invention
[0006] In view of the deficiencies of the prior art, the present invention provides a system and method for detecting violent behavior in surveillance videos.
[0007] A violent behavior detection system in surveillance videos includes: a data extraction module, a data annotation module, an image enhancement module, a model training module, a violent behavior detection module, and a monitoring log module;
[0008] The data extraction module extracts image frames from the video stream data pushed by the surveillance camera and outputs them to the data annotation module; the data annotation module annotates the image frames to see whether they contain violent behavior and the area where the violent behavior occurs, and outputs them to the image enhancement module; the image enhancement module performs sharpening, histogram equalization, and resolution adjustment processing on the annotated frame images according to their quality, and outputs them to the model training module; the model training module trains them, and obtains a violent behavior detection module after the training; the violent behavior detection module detects the newly input unannotated image enhancement pictures, and finally stores the output results in the monitoring log module;
[0009] The violent behavior detection module includes a violent behavior classification model and a violent behavior target detection model. Unlabeled images are first input into the image enhancement module for processing and then sent to the violent behavior detection module. In the violent behavior detection module, the images are first processed by the violent behavior classification model to output the classification results. Based on the classification results, the images containing violent behaviors are extracted and input into the violent behavior target detection model to output the location where the violent behaviors occur.
[0010] On the other hand, a method for detecting violent behavior in surveillance videos is implemented based on a system for detecting violent behavior in surveillance videos, and comprises the following steps:
[0011] Step 1: Obtain video data captured by surveillance cameras in indoor and outdoor places. The video data is pushed to the video extraction module in the form of a video stream. The video extraction module performs frame extraction processing on the video data and extracts a certain amount of video data images;
[0012] Step 2: Annotate the extracted video data image to determine whether it contains violent behavior and the area where the violent behavior occurs;
[0013] Step 3: Input the annotated video data image into the image enhancement module, perform image enhancement according to resolution and light factors, use histogram equalization to significantly enhance the image contrast, and appropriately adjust the resolution of the video data image as needed;
[0014] Step 4: Input the video data image processed in step 3 into the model training module to obtain a violent behavior detection module;
[0015] Step 4.1: Input the video data image labeled whether it contains violent behavior and image enhancement into the 3D convolutional neural network, and train the model to obtain a violent behavior classification model;
[0016] The 3D convolutional neural network uses the P3D convolution structure to decompose the 3D convolution into 2D spatial convolution and 1D temporal convolution;
[0017] Step 4.2: Input the video data image with the marked violent area and image enhancement into the CenterNet target detection network, and train the model to the violent behavior target detection model;
[0018] Training CenterNet requires minimizing the loss function, which consists of three parts: key point prediction loss, discrete offset prediction loss, and prediction box size loss.
[0019] Step 4.2.1: The key point prediction loss formula is as follows:
[0020]
[0021] Among them, L k is the key point prediction loss, N is the number of key points in the input image, α and β are hyperparameters, is the prediction result, y xyc is the real result; xy represents the coordinates of the image pixels, and c represents the category;
[0022] Step 4.2.2: In order to correct the predicted key points, the discrete offset prediction loss is introduced, and the formula is as follows:
[0023]
[0024] Among them, L off is the discrete offset prediction loss, R is the number of key points in the input image, is the predicted offset, is the predicted key point position, and p is the actual key point position;
[0025] Step 4.2.3: In addition to the key points, we need to predict the area where the violence occurs. We also need to predict the length and width of the border marking the area where the violence occurs, and then introduce the prediction box size loss. The formula is as follows:
[0026]
[0027] Among them, L size is the prediction box size loss, Predicted length and width, S k is the actual length and width;
[0028] Step 4.2.4: Finally, the three loss functions are integrated into a total loss function. As long as the loss function is minimized, the target detection model converges:
[0029] L det =L k +λ size L size +λ off L off
[0030] Among them, L det is the total loss function, λ size is the weight of the discrete offset prediction loss, usually set to 0.1, λ off is the weight of the loss of the predicted box size, usually set to 1;
[0031] Step 5: Input the video image data acquired in real time by the surveillance camera into the violent behavior classification model to obtain the violent behavior classification result, and obtain the score of each frame of the image sent to the model containing violent behavior;
[0032] Step 6: Based on the results of the violent behavior classification model, select the top k images with the highest scores, keep their original order in the video stream, and form a new image sequence;
[0033] Step 7: Input the top k image sequences with the highest scores into the violent behavior target detection model to obtain the violent behavior target detection results, and obtain the center point coordinates, length and width of the violent behavior area in each image input into the model, and the confidence of the area;
[0034] Step 8: Finally, the violent behavior detection results are sent to the running user terminal and stored in the log module;
[0035] Step 9: Periodically select some monitoring data for re-labeling, send them to the violent behavior detection module for testing, calculate the classification accuracy and the IOU of target detection, and obtain image frames with poor detection results; use the image frames with poor detection results to train and improve the violent behavior classification model and violent behavior target detection model;
[0036] The violent behavior classification model in surveillance videos obtained through training can classify the video images extracted from the video stream sent back by the video surveillance camera, and select the images containing violent behavior. The violent behavior target detection model in surveillance videos can further detect these images to achieve high-precision violent behavior area target detection.
[0037] Beneficial technical effects of the present invention:
[0038] The present invention provides a method for detecting violent behavior in surveillance videos. A wireless surveillance camera collects real-time images in a scene, and a deep learning classification model and a deep learning target detection model are used to train labeled images containing violent behavior and normal images, so as to accurately identify violent behavior. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 A schematic diagram of a flow chart of a method for detecting violent behavior in a surveillance video according to an embodiment of the present invention;
[0040] Figure 2 A schematic diagram of the structure of a system for detecting violent behavior in surveillance videos according to an embodiment of the present invention;
[0041] Figure 3 Schematic diagram of the structure of an algorithm model for detecting violent behavior targets in surveillance videos according to an embodiment of the present invention. DETAILED DESCRIPTION
[0042] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:
[0043] The invention provides a system and method for detecting violent behavior in surveillance videos.
[0044] A system for detecting violent behavior in surveillance videos includes: a data extraction module, a data annotation module, an image enhancement module, a model training module, a violent behavior detection module, and a monitoring log module; as shown in the attached Figure 2 As shown;
[0045] The data extraction module extracts image frames from the video stream data pushed by the surveillance camera and outputs them to the data annotation module; the data annotation module annotates the image frames to see whether they contain violent behavior and the area where the violent behavior occurs, and outputs them to the image enhancement module; the image enhancement module performs sharpening, histogram equalization, and resolution adjustment processing on the annotated frame images according to their quality, and outputs them to the model training module; the model training module trains them, and obtains a violent behavior detection module after the training; the violent behavior detection module detects the newly input unannotated but image-enhanced images, and finally stores the output results in the monitoring log module;
[0046] The violent behavior detection module includes a violent behavior classification model and a violent behavior target detection model. Unlabeled images are first input into the image enhancement module for processing and then sent to the violent behavior detection module. In the violent behavior detection module, the images are first processed by the violent behavior classification model to output the classification results. Based on the classification results, the images containing violent behaviors are extracted and input into the violent behavior target detection model to output the location where the violent behaviors occurred. The violent behavior target detection algorithm model structure is shown in the attached figure. Figure 3 As shown;
[0047] On the other hand, a method for detecting violent behavior in surveillance videos is implemented based on a system for detecting violent behavior in surveillance videos; Figure 1 As shown; including the following steps:
[0048] Step 1: Obtain video data captured by surveillance cameras in indoor and outdoor places. The video data is pushed to the video extraction module in the form of a video stream. The video extraction module performs frame extraction processing on the video data and extracts a certain amount of video data images;
[0049] The video data captured by the surveillance camera deployed at the top of the corridor in the building is uploaded to the host where the algorithm is located (it can also be said to be uploaded to the algorithm server). The RTSP protocol is used in the process of uploading the video stream to ensure the real-time nature of the data;
[0050] The video data uploaded to the host (or server) is input into the data extraction module, and some surveillance video frames between aisles are randomly selected to calculate the similarity and determine the image extraction time interval;
[0051] Step 2: Annotate the extracted video data image to determine whether it contains violent behavior and the area where the violent behavior occurs;
[0052] Marking some video image data with at least whether it contains violent behavior and the area where the violent behavior occurs, and requiring the marking to be stored in an XML file format;
[0053] Step 3: Input the annotated video data image into the image enhancement module, perform image enhancement based on factors such as resolution and light, use histogram equalization to significantly enhance the image contrast, and appropriately adjust the resolution of the video data image as needed;
[0054] The labeled image is sent to the image enhancement module for histogram equalization adjustment, and then the image resolution is set to 512*512;
[0055] Step 4: Input the video data image processed in step 3 into the model training module to obtain a violent behavior detection module;
[0056] Step 4.1: Input the video data image labeled whether it contains violent behavior and image enhancement into the 3D convolutional neural network, and train the model to obtain a violent behavior classification module;
[0057] The 3D convolutional neural network uses the P3D convolution structure to decompose the 3D convolution into 2D spatial convolution and 1D temporal convolution. This not only takes time information into consideration, but also greatly reduces the parameters of the neural network and improves the efficiency of algorithm execution.
[0058] Step 4.2: Input the video data image with the marked violent area and image enhancement into the CenterNet target detection network, and train the model to the violent behavior target detection model;
[0059] Training CenterNet requires minimizing the loss function, which consists of three parts: key point prediction loss, discrete offset prediction loss, and prediction box size loss.
[0060] Step 4.2.1: The key point prediction loss formula is as follows:
[0061]
[0062] Among them, L k is the key point prediction loss, N is the number of key points in the input image, α and β are hyperparameters, is the prediction result, y xyc is the real result; xy represents the coordinates of the image pixels, and c represents the category;
[0063] Step 4.2.2: In order to correct the predicted key points, the discrete offset prediction loss is introduced, and the formula is as follows:
[0064]
[0065] Among them, L off is the discrete offset prediction loss, R is the number of key points in the input image, is the predicted offset, is the predicted key point position, and p is the actual key point position;
[0066] Step 4.2.3: In addition to the key points, we need to predict the area where the violence occurs. We also need to predict the length and width of the border marking the area where the violence occurs, and then introduce the prediction box size loss. The formula is as follows:
[0067]
[0068] Among them, L size is the prediction box size loss, Predicted length and width, S k is the actual length and width;
[0069] Step 4.2.1: Finally, the three loss functions are integrated into a total loss function. As long as the loss function is minimized, the target detection model converges:
[0070] L det =L k +λ size L size +λ offL off
[0071] Among them, L det is the total loss function, λ size is the weight of the discrete offset prediction loss, usually set to 0.1, λ off is the weight of the loss of the predicted box size, usually set to 1;
[0072] Step 5: Input the video image data acquired in real time by the surveillance camera into the violent behavior classification model to obtain the violent behavior classification result, and obtain the score of each frame of the image sent to the model containing violent behavior;
[0073] The processed image is sent to the violence detection module. The image is first sent to the violence classification model to obtain the violence classification result, and the score of each frame of the image sent to the model containing violence is obtained. The score is a decimal between 0 and 1, indicating the probability that the image contains violence.
[0074] Step 6: Based on the results of the violent behavior classification model, select the top k images with the highest scores, keep their original order in the video stream, and form a new image sequence;
[0075] Step 7: Input the top k image sequences with the highest scores into the violent behavior target detection model to obtain the violent behavior target detection results, and obtain the area where the violent behavior occurs and the confidence of the area for each image input into the model;
[0076] The first k image sequences with the highest scores are input into the violent behavior target detection module to detect the coordinates of the center point of the violent behavior area, the length and width of the area, and the confidence of the area.
[0077] Step 8: Finally, the violent behavior detection results are sent to the running user terminal and stored in the log module;
[0078] Step 9: Periodically select some monitoring data for re-labeling, send them to the violent behavior detection module for testing, calculate the classification accuracy and the IOU of target detection, and obtain image frames with poor detection results; use the image frames with poor detection results to train and improve the violent behavior classification model and violent behavior target detection model;
[0079] The violent behavior classification model in the surveillance video obtained through training can classify the video images extracted from the video stream sent back by the video surveillance camera, and select the images containing violent behavior. The violent behavior target detection model in the surveillance video can further detect these images to achieve high-precision violent behavior area target detection; it is widely applicable to the application scenarios of violent behavior detection;
[0080] The violent behavior classification model in surveillance videos obtained through training can classify video images extracted from the video stream sent back by the video surveillance camera and select images containing violent behavior. The violent behavior target detection model in surveillance videos can further detect these images. The final average precision (AP) can reach 45.1, realizing reliable violent behavior detection, which is widely applicable to application scenarios of violent behavior detection.
Claims
1. A system for detecting violent behavior in surveillance videos, characterized in that: Specifically, it includes data extraction module, data annotation module, image enhancement module, model training module, violence detection module and monitoring log module; The data extraction module extracts image frames from the video stream data pushed by the surveillance camera and outputs them to the data annotation module; The data annotation module annotates the image frame to determine whether it contains violent behavior and the area where the violent behavior occurs, and outputs the information to the image enhancement module; The image enhancement module performs sharpening, histogram equalization, and resolution adjustment on the annotated frame images according to their quality, and outputs them to the model training module; The model training module trains it, and after the training, a violent behavior detection module is obtained; the violent behavior detection module detects the newly input unlabeled image enhancement picture, and finally stores the output result in the monitoring log module; The violent behavior detection module includes a violent behavior classification model and a violent behavior target detection model; the unlabeled image is first input into the image enhancement module for processing, and then sent to the violent behavior detection module; in the violent behavior detection module, the image is first processed by the violent behavior classification model to output the classification result, and according to the classification result, the image containing violent behavior is extracted and input into the violent behavior target detection model to output the location where the violent behavior occurs; Input the video data images that are labeled whether they contain violent behavior and undergo image enhancement into a 3D convolutional neural network, and train the model to obtain a violent behavior classification model; The video data images with the marked violent areas and enhanced images are input into the CenterNet target detection network, and the model is trained to the violent behavior target detection model; The 3D convolutional neural network uses the convolution structure of P3D, and the 3D convolution is decomposed into 2D spatial convolution and 1D temporal convolution.
2. A method for detecting violent behavior in surveillance videos is implemented based on a system for detecting violent behavior in surveillance videos as claimed in claim 1, characterized in that: The following steps are involved: Step 1: Obtain video data captured by surveillance cameras in indoor and outdoor places. The video data is pushed to the video extraction module in the form of a video stream. The video extraction module performs frame extraction processing on the video data and extracts a certain amount of video data images; Step 2: Annotate the extracted video data image to determine whether it contains violent behavior and the area where the violent behavior occurs; Step 3: Input the annotated video data image into the image enhancement module, perform image enhancement according to resolution and light factors, use histogram equalization to significantly enhance the image contrast, and appropriately adjust the resolution of the video data image as needed; Step 4: Input the video data image processed in step 3 into the model training module to obtain a violent behavior detection module; Step 5: Input the video image data acquired in real time by the surveillance camera into the violent behavior classification model to obtain the violent behavior classification result, and obtain the score of each frame of the image sent to the model containing violent behavior; Step 6: Based on the results of the violent behavior classification model, select the top k images with the highest scores, keep their original order in the video stream, and form a new image sequence; Step 7: Input the top k image sequences with the highest scores into the violent behavior target detection model to obtain the violent behavior target detection results, and obtain the coordinates of the center point of the violent behavior area, the length and width of the area, and the confidence of the area for each image input into the model; Step 8: Finally, the violent behavior detection results are sent to the running user terminal and stored in the log module; Step 9: Regularly select some monitoring data for re-labeling, send it to the violent behavior detection module for testing, calculate the classification accuracy and the IOU of target detection, and obtain image frames with poor detection results; Use image frames with poor detection results to train and improve the violence classification model and the violence target detection model; The violent behavior classification model in surveillance videos obtained through training can classify the video images extracted from the video stream sent back by the video surveillance camera, and select the images containing violent behavior. The violent behavior target detection model in surveillance videos can further detect these images to achieve high-precision violent behavior area target detection.
3. A method for detecting violent behavior in surveillance video according to claim 2, characterized in that: The step 4 is specifically as follows: Step 4.1: Input the video data image labeled whether it contains violent behavior and image enhancement into the 3D convolutional neural network, and train the model to obtain a violent behavior classification model; Step 4.2: Input the video data image with the marked violent area and image enhancement into the CenterNet target detection network, and train the model to a violent behavior target detection model.
4. A method for detecting violent behavior in surveillance video according to claim 3, characterized in that: The training of CenterNet requires minimizing the loss function, which consists of three parts: key point prediction loss, discrete offset prediction loss, and prediction box size loss.
5. The method for detecting violent behavior in surveillance video according to claim 3, characterized in that: The step 4.2 is specifically as follows: Step 4.2.1: The key point prediction loss formula is as follows: Among them, L k is the key point prediction loss, N is the number of key points in the input image, α and β are hyperparameters, is the prediction result, Y xyc is the real result; xy represents the coordinates of the image pixels, and c represents the category; Step 4.2.2: In order to correct the predicted key points, the discrete offset prediction loss is introduced, and the formula is as follows: Among them, L off is the discrete offset prediction loss, R is the number of key points in the input image, is the predicted offset, is the predicted key point position, and p is the actual key point position; Step 4.2.3: In addition to the key points, we need to predict the area where the violence occurs. We also need to predict the length and width of the border marking the area where the violence occurs, and then introduce the prediction box size loss. The formula is as follows: Among them, L size is the prediction box size loss, Predicted length and width, S k is the actual length and width; Step 4.2.4: Finally, the three loss functions are integrated into a total loss function. As long as the loss function is minimized, the target detection model converges: L det =L k +λ size L size +λ off L off Among them, L det is the total loss function, λ size is the weight of the discrete offset prediction loss, set to 0.1, λ off is the weight of the prediction box size loss, set to 1.
