Violence detection method, electronic device, and storage medium

Abstract

A method, electronic device, and storage medium for detecting violent behavior are disclosed, integrating real-time target detection and real-time posture detection. Target detection delineates regions of interest (ROIs) suggesting potential violations, followed by magnified secondary posture detection within these regions to determine the number of people and their postures. This information serves as the basis for determining the presence of violent behavior in the area. This invention addresses the problems of existing methods relying solely on target detection, which suffer from insufficient accuracy in real-world scenarios due to lack of data training and lighting conditions; and the inaccuracy in posture detection when the image size is too large and the person occupies a small portion of the frame. This invention balances real-time performance and accuracy, significantly reducing the false positive rate, and is of great significance to the security field.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and is applicable to various occasions where personnel safety needs to be monitored. It involves judging whether there is violent behavior in people in the picture, detecting fighting behavior based on deep neural networks, and detecting and mining human posture. It is a fighting behavior detection method based on target detection and posture detection. Background Technology

[0002] Violent behavior identification and recognition is currently a hot research topic. Detecting violent behavior through video surveillance allows for timely intervention to prevent injuries and fatalities. However, current violent behavior detection algorithms still have some problems. Using only general target detection results in a high false positive rate; for example, normal actions like two people talking, brushing past each other, or carrying objects can be misidentified as fighting. Relying solely on pose algorithms leads to missed detections and numerous low-confidence key points due to blurred poses during fights and a large background area in the input image causing a low signal-to-noise ratio. Furthermore, many poses may resemble fighting postures, such as two people passing an object, leading to false positives. Using complex large models for detection results in excessive computation and unreal-time performance issues. Additionally, violent behavior is a dynamic, temporal event; relying solely on pose detection for each frame is prone to false positives. Methods like optical flow, which calculate continuous human movements, are highly sensitive to light intensity, and the high speed of movement during fights makes them difficult to use and deploy with ordinary surveillance cameras.

[0003] Meanwhile, the complexity of application scenarios and the lack of datasets for those scenarios also pose challenges to ordinary single-stage detection. Summary of the Invention

[0004] The problem this invention aims to solve is that existing methods for detecting specific human behaviors are inaccurate when relying solely on target detection for violent behavior detection, and inaccurate and potentially missed detection methods when relying solely on posture detection. Furthermore, human movements are dynamic, and computational methods such as optical flow are computationally intensive and cannot meet the requirements for lightweight deployment.

[0005] The technical solution of this invention is: a method for detecting violent behavior, based on target detection and pose detection, comprising the following steps:

[0006] Step 1: Construct a fighting behavior dataset and a human posture dataset;

[0007] Step 2: Train a YOLOv7 model with spatial attention mechanism using a fighting behavior dataset to obtain a target of interest detection model for violent behavior;

[0008] Step 3: Train the YOLOv8 pose detection network using the human pose dataset to obtain the human pose model;

[0009] Step 4: Based on the usage scenario, set the fight count, fight count threshold, and fight posture feature samples, and initialize the fight count to 0;

[0010] Step 5: Read the video, input the video frames into the target of interest detection model in Step 2, and the detection output will show areas where violent behavior may occur;

[0011] Step 6: Use the human pose model obtained in Step 3 to detect human poses within the region, and obtain the number of people in the region and the pose points of each person.

[0012] Step 7: Based on the number of people in the area, if there are fewer than two people, discard the detection result and continue to detect the next frame; if there are more than two people, calculate the pixel area of ​​each person in the area.

[0013] Step 8: Calculate the total pixel area of ​​the region and the size ratio of each person in the region. If the current person's size ratio in the image is less than 0.3, then the current person is considered as background and the data of the current person is discarded. If the size ratio is greater than 0.3, then the person is considered a suspicious person. This process continues until a set of all suspicious people in the region is obtained.

[0014] Step 9: If there is more than one suspicious person in the set of suspicious persons, select the suspicious person according to the detection results of the pose detection and obtain the pose point data of the person. If there are fewer than two suspicious persons, return to Step 5 to continue detecting the next frame.

[0015] Step 10: Match the current suspicious person with the remaining suspicious persons in pairs to determine whether there has been contact between the persons;

[0016] Step 11: Compare the current suspicious person's posture with the configured fighting posture feature samples to determine whether the person is in a fighting posture.

[0017] Step 12: Combining the results from Step 10 and Step 11, if the current suspicious person is in contact with others and is in a fighting posture, it is determined to be in a fighting state. At this time, the fighting state count is incremented by 10. It is then determined whether the fighting state count is greater than the threshold. If it is, an alarm is issued and the detection proceeds to the next frame; otherwise, the detection proceeds directly to the next frame. If the current suspicious person is not in contact with others or is not in a fighting posture, it is marked as not in a fighting state, and the process returns to Step 9 to judge the next suspicious person until all suspicious persons in the current frame have been judged.

[0018] Furthermore, in step 12, the fight count is incremented by 10 when a fight exists and decremented by 2 when no fight exists. The minimum fight count is 0. The fight count threshold is set based on the presence of fights within n frames, thus enabling a comprehensive temporal judgment of fights.

[0019] The present invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program. The at least one instruction or at least one program is loaded and executed by the processor to implement the above-mentioned region of interest target detection model, human pose model, and violent behavior detection method, and to detect violent behavior in the input video.

[0020] The present invention also provides that the computer-readable storage medium stores at least one instruction or at least one program, which, when executed, implements the above-mentioned target of interest detection model, human posture model, and violent behavior detection method.

[0021] The beneficial effects of this invention are as follows: This invention combines target detection and posture detection. First, target detection is used to delineate the region of interest where there is suspected violation. Then, a magnified secondary posture detection is performed in this region to determine the number of people and their postures within the region. This serves as the basis for judging whether there is violent behavior in the region. The dual judgment improves the detection accuracy. At the same time, by setting various thresholds, the detection method can be applied to various occasions. Furthermore, by counting fights, dynamic temporal judgment of the behavior of people in the video is achieved. There is no problem of easy misjudgment in single-frame judgment, and there is no need to obtain continuous motion information of people through complex optical flow methods, so as to judge the dynamic behavior of people.

[0022] This invention leverages the speed of YOLOv7 object detection to achieve real-time target of interest (ROI) detection. However, these targets include areas that genuinely indicate fighting as well as areas that are mistakenly identified as fighting, leading to false detections. To address this, the affected areas are extracted, and pose detection results are used for state determination. This invention first uses object detection to predict ROIs that may indicate fighting, then performs pose detection on a portion of the designated area, improving the signal-to-noise ratio of the pose detection input source and making the results more accurate. This method improves upon the problem of false detection when using only object detection methods like YOLOv7 or only pose detection for fighting detection, maintaining high accuracy, a very low false positive rate, and flexible application scenarios.

[0023] The threshold of this invention can be flexibly adjusted according to actual scenarios and application requirements. For example, in darker scenes, the confidence level can be slightly reduced to reduce missed detections caused by lighting. In scenes with good lighting and clear human movement, the confidence level can be appropriately increased to raise the counting threshold and prevent false alarms caused by individual actions. The camera angle can also be taken into account to expand the range of posture standards when comparing fighting postures, thereby achieving the desired effect. Attached Figure Description

[0024] Figure 1 This is a flowchart of a fighting behavior detection method based on target detection and posture detection according to the present invention.

[0025] Figure 2 This is a flowchart illustrating the training process of the target of interest detection model and the human pose model in this invention.

[0026] Figure 3 The diagram shows the network structure of the YOLOv7 model with spatial attention mechanism introduced in this invention. (a) is the improved RFCBAM module used in YOLOv7, (b) is the improved BACKBONE backbone network, and (c) is the improved ELAN module.

[0027] Figure 4 The test was conducted in a corridor scene to implement this invention, and the output test result image is shown. Detailed Implementation

[0028] The implementation of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] like Figure 1 As shown, this invention integrates real-time target detection and real-time pose detection. It requires only a small amount of data to delineate regions of interest (ROIs) with suspected violations through target detection. Then, a secondary pose detection is performed within this region to determine the number of people and their poses, which serves as the basis for judging whether violent behavior exists in the area. The implementation involves the following steps.

[0030] First, train the target of interest detection model and the human pose model, such as Figure 2 As shown, the implementation is as follows: steps 1 to 3.

[0031] Step 1: Construct a fighting behavior dataset and a human pose dataset. Images of violent acts are collected, labeled, and augmented to create the fighting behavior dataset. The human pose dataset can be constructed using publicly available datasets or by self-labeling images of violent acts. Self-labeled data is also preferably augmented to expand the training data. Furthermore, publicly available datasets often fall short in practical applications, necessitating the creation of custom datasets. This invention collects and labels images from cameras within the violent behavior scenes to be detected, using these images as supplementary data to fine-tune the target of interest detection model and the human pose model.

[0032] Data enhancement includes:

[0033] Step 1.1: Perform scalar transformations on the images, such as cropping, scaling, translation, and jittering, to expand the dataset;

[0034] Step 1.2: The image is transformed by adding noise, adjusting contrast, brightness, and saturation to achieve frequency domain transformation and expand the dataset.

[0035] Step 2: Train a YOLOv7 model with spatial attention mechanism using a fighting behavior dataset to obtain a target of interest detection model for violent behavior. This YOLOv7 model with spatial attention mechanism integrates the RFCBAM spatial receptive field attention mechanism in the BACKBONE part, and adds a SwinTransformer module at the end of the BACKBONE part to facilitate YOLOv7's processing of super-resolution images. The ordinary Conv modules in the ELAN module of the BACKBONE are replaced with GhostConv modules, and some ordinary Conv modules in the Head part are also replaced with GhostConv modules to compensate for the additional computational cost caused by the RFCBAM module. Figure 3 As shown, the specific implementation is as follows.

[0036] Extract the BACKBONE network from YOLOv7; truncate it before the first ELAN module, and add an RFCBAM module in front of it, as shown in the image. Figure 3As shown in (a), the feature map output from the previous layer is input into two branches. The first branch passes through the GlobalAvgPool layer, Linear layer, ReLU layer, Linear layer, and Sigmoid layer in sequence. The second branch passes through the GroupConv layer, Normalize layer, ReLU layer, Adjust Shape layer, AvgPool layer, Max Pool layer, Conv layer, and Sigmoid layer in sequence. The results of the two branches are combined through a Re-weight layer and then fused together through a GhostConv layer to complete the spatial attention calculation.

[0037] Adding the SwingTransformer module to the end of the BACKBONE section helps YOLOv7 handle super-resolution images. The improved BACKBONE is as follows: Figure 3 As shown in (b). Improvements were also made to the ELAN module of BACKBONE, replacing Conv with the GhostConv module, and replacing Conv in the Head section with the GhostConv module to compensate for the additional computational cost caused by the RFCBAM module, such as... Figure 3 As shown in (c).

[0038] For solutions combining YOLOv7 with RFCBAM, please refer to the arXiv search result "RFAConv: Innovating SpatialAttention and Standard Convolutional Operation". For solutions combining YOLOv7 with SwinTransformer, please refer to the ICCV 2021 paper "Swin Transformer: Hierarchical Vision Transformer using ShiftedWindows".

[0039] How to introduce a suitable attention mechanism module (RFAconv instead of CBAM), how to maintain real-time performance after finding the region of interest and then performing pose detection (by fusing some special Conv layers), and the mentioned reduction of the detection range is actually a data preprocessing technique. In this case, we use deep learning itself to perform data preprocessing without changing the image itself.

[0040] Step 3: Train the human pose dataset into the YOLOv8 pose detection network to obtain the human pose model;

[0041] Step 4: Based on the usage scenario, initialize the fight count to 0, and set the fight count threshold and fight posture feature samples;

[0042] Step 5: Read the video and feed the video frames into the object detection network in Step 2 to obtain areas where violent behavior may occur;

[0043] Step 6: Using the human pose model obtained in Step 3, perform human pose detection in areas where violent behavior may occur, obtaining the number of people in the area and the pose points of each person. In practice, the obtained area is cropped into a new image matrix, i.e., a screenshot of the area, with matrix elements being RGB pixels. These are stored in a new memory area in pixel order for pose detection.

[0044] Step 7: Determine the number of people in the area. If there are fewer than two people, discard the detection result and continue to detect the next frame. If there are more than two people, calculate the pixel area of ​​each person in the area.

[0045] Step 8: Calculate the total pixel area of ​​the region and the size ratio of each person in the region. If the current person's size ratio in the image is less than 0.3, then the current person is considered as background and the data of the current person is discarded. If the size ratio is greater than 0.3, then the person is considered a suspicious person. This process continues until a set of all suspicious people in the region is obtained.

[0046] Step 8.1: The pixel area of ​​the person is the total number of pixels in the person frame. The pixel area of ​​each person is the length multiplied by the width W of the person frame output during human pose detection in Step 6. person ×H person The total pixel area of ​​the region is the length x width W of the region of interest obtained in Step 5. area ×H area Calculate the screen ratio according to formula (1):

[0047]

[0048] Step 8.2: Filter the indicators for each person calculated in Step 8.1. Analyze those below 0.3 as potential or false positives and delete them. Keep the rest.

[0049] Step 9: If there is more than one suspicious person, select the suspicious person in order of the pose detection results and obtain the data of that person. If there are fewer than two suspicious persons, return to Step 5 to continue detecting the next frame.

[0050] Step 10: Match the current suspicious person with the remaining suspicious persons in pairs to determine whether there is any contact between the persons.

[0051] Step 10.1: Based on the calculated pixel area S of each person p1 and S p2Intersection over Union (IOU) is calculated for each pair of regions sequentially. The IOU value is calculated by dividing the area of ​​intersection of the two regions by the area of ​​union of the two regions. The IOU value between each pair of regions is calculated using formula (2).

[0052]

[0053] Step 10.2: Determine if the IOU value is greater than 0.2. If it is greater than 0.2, then the two people are in contact; otherwise, they are not in contact.

[0054] Step 11: Compare the current suspicious person's posture with the configured fighting posture feature samples to determine whether the person is in a fighting posture.

[0055] Step 11.1: Based on the position of the suspicious person's pose points, filter according to the scene conditions, discarding points whose confidence level does not meet the confidence threshold. The confidence threshold is set according to the scene brightness and camera angle. The pose detection of this invention is consistent 2D pose detection, which is equivalent to projecting onto a plane. However, cameras generally monitor from a high position to cover more area. Considering the extreme case, a person viewed from directly above can only see the head and shoulders. In this case, pose detection based on 2D is impossible. Useless detection data is filtered out by using the confidence threshold.

[0056] Step 11.2: Compare the similarity between the fighting posture feature samples and the posture of the current target. If the similarity meets the requirements, it is determined to be a fighting state; otherwise, it is determined to be a non-fighting state. The preset fighting posture feature samples are related to the angle of the video camera. For example, for a side-view camera, the angle between the elbow and palm and the ground, the angle between the elbow and shoulder, and the position of the arm and forearm can be calculated. Figure 4 If the boy on the right is in a fighting state, and his arms are in a fist-like posture and his arms overlap with the detection frame of another person, then he is judged to be in a fighting state; otherwise, he is judged to be in a non-fighting state.

[0057] Step 12: Combining the results from Step 10 and Step 11, if the current suspicious person is in contact with others and is in a fighting posture, it is determined to be in a fighting state. At this time, the fighting state count is incremented by k. It is determined whether the fighting state count is greater than the threshold. If it is greater, an alarm is issued and the process proceeds to the next frame; otherwise, the process proceeds directly to the next frame. If the current suspicious person is not in contact with others or is not in a fighting posture, it is marked as not in a fighting state, and the process returns to Step 9 to judge the next suspicious person, until all suspicious persons in the current frame have been judged.

[0058] As a preferred method, in Step 12, the fight count is incremented by k when a fight is present, and decremented by j when no fight is present. k and j are integers, and the minimum fight count is 0. The fight count threshold is set based on the presence of fights within n frames, achieving a comprehensive temporal judgment of fight behavior. In this invention, once a fight is detected in the current video frame, the fight count is increased and the detection proceeds to the next frame. The fight count for a single video frame only increases by k to prevent a sudden increase in the count. The minimum fight count is 0 and not negative to prevent the decrement of the count when there is no fight in multiple frames from affecting the increment of the count for subsequent fights. An example illustrating the fight count setting of this invention is as follows: For instance, setting the fight count to +10 when a fight is present and -2 when no fight is present, with a fight count threshold of 70, then the presence of fights in 7 consecutive frames or intermittent fights within 12 frames will be judged as the presence of a fight and trigger an alarm. This aligns with the dynamic characteristics of actual human behavior, enabling accurate judgment regardless of whether the action is continuous or intermittent.

[0059] This invention also provides an electronic device, including a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the region of interest target detection model, human pose model, and violence detection method described in the above embodiments, thereby enabling violence detection on input video. This electronic device is intended to represent various forms of digital electronic computer devices, such as laptops, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, smartphones, wearable devices, and other similar data computing devices. The processor can be various general-purpose and / or dedicated processing components with processing and computing capabilities. Some examples of processors include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc.

[0060] This invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, causes the computer to perform a method according to an embodiment of the invention. The program code for implementing the method of the invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a standalone software package, or entirely on a remote machine or server.

Claims

1. A method for detecting violent behavior, characterized by: Based on object detection and pose detection, the following steps are included: Step 1: Construct a fighting behavior dataset and a human posture dataset; Step 2: Train a YOLOv7 model incorporating spatial attention using a brawl behavior dataset to obtain a target of interest detection model for violent behavior; the YOLOv7 model incorporating spatial attention is implemented as follows: For the BACKBONE network in YOLOv7, the first ELAN module is truncated, and an RFCBAM module is added before it. Specifically, the input is fed into two branches. The first branch passes through the GlobalAvgPool layer, Linear layer, ReLU layer, Linear layer, and Sigmoid layer in sequence. The second branch passes through the GroupConv layer, Normalize layer, ReLU layer, AdjustShape layer, AvgPool layer, Max Pool layer, Conv layer, and Sigmoid layer in sequence. The results of the two branches are combined through a Re-weight layer and then fused together through a GhostConv layer to complete the spatial attention calculation. The result of the attention calculation is used for YOLOv7 detection, and the final output is the region where violent behavior may exist. The SwinTransformer module was added to the end of BACKBONE to facilitate YOLOV7's processing of super-resolution images; the Conv module in BACKBONE was replaced with the GhostConv module, and the Conv module in the Head section was also replaced with the GhostConv module to compensate for the additional computation caused by the RFCBAM module. Step 3: Train the YOLOv8 pose detection network using the human pose dataset to obtain the human pose model; Step 4: Based on the usage scenario, set the fight count, fight count threshold, and fight posture feature samples, and initialize the fight count to 0; Step 5: Read the video, input the video frames into the target of interest detection model in Step 2, and the detection output will show areas where violent behavior may occur; Step 6: Use the human pose model obtained in Step 3 to detect human poses within the region, and obtain the number of people in the region and the pose points of each person. Step 7: Based on the number of people in the area, if there are fewer than two people, discard the detection result and continue to detect the next frame; if there are more than two people, calculate the pixel area of ​​each person in the area. Step 8: Calculate the total pixel area of ​​the region and the size ratio of each person in the region. If the current person's size ratio in the image is less than 0.3, then the current person is considered as background and the data of the current person is discarded. If the size ratio is greater than 0.3, then the person is considered a suspicious person. This process continues until a set of all suspicious people in the region is obtained. Step 9: If there is more than one suspicious person in the set of suspicious persons, select the suspicious person according to the detection results of the pose detection and obtain the pose point data of the person. If there are fewer than two suspicious persons, return to Step 5 to continue detecting the next frame. Step 10: Match the current suspicious person with the remaining suspicious persons in pairs to determine whether there has been contact between the persons; Step 11: Compare the current suspicious person's posture with the configured fighting posture feature samples to determine whether the person is in a fighting posture. Step 12: Combining the results from Step 10 and Step 11, if the current suspicious person is in contact with others and is in a fighting posture, it is determined to be in a fighting state. At this time, the fighting state count is incremented by k. It is determined whether the fighting state count is greater than the threshold. If it is greater, an alarm is issued and the detection proceeds to the next frame. Otherwise, the detection proceeds directly to the next frame. If the current suspicious person is not in contact with others or is not in a fighting posture, it is marked as not in a fighting state, and the process returns to Step 9 to judge the next suspicious person until all suspicious persons in the current frame have been judged.

2. The violent behavior detection method according to claim 1, characterized in that in Step 12, when a fighting state exists, the fighting count is incremented by k, and when no fighting state exists, the fighting count is decremented by j, where k and j are integers, the minimum fighting count is 0, and the fighting count threshold is set according to the situation of fighting behavior within n frames, so as to realize the temporal comprehensive judgment of fighting behavior.

3. The violent behavior detection method according to claim 1, characterized in that: In step 1, images of violent behavior are collected, labeled, and then augmented to form a fighting behavior dataset. The human posture dataset is constructed using public datasets or based on images of violent behavior, labeled, and augmented to form a human posture dataset.

4. The violent behavior detection method according to claim 3, characterized in that: Images from cameras in the violent behavior scene to be detected are collected, labeled, and used as supplementary data to fine-tune the target of interest detection model and the human pose model.

5. The violent behavior detection method according to claim 3, characterized in that: Data augmentation includes: 1.1) Perform sizing transformations on the images, including cropping, scaling, translation, and jittering, to expand the dataset; 1.2) The image is transformed by adding noise, adjusting contrast, brightness and saturation to achieve frequency domain transformation and expand the dataset.

6. The violent behavior detection method according to claim 1, characterized in that determining whether there is contact between the individuals in step 10 specifically involves: 10.1) Based on the pixel area of ​​each person obtained in step 6, extract each pair of people in turn and perform an IOU (Intersection over Union) calculation. The IOU value is calculated by dividing the area of ​​intersection of the two regions by the area of ​​union of the two regions. 10.2) Determine if the IOU value is greater than 0.

2. If it is greater than 0.2, then the two people are in contact; otherwise, they are not in contact.

7. The violent behavior detection method according to claim 1, characterized in that step 11, determining whether a person is in a fighting posture, specifically comprises: 11.1) For the pose point data of suspicious persons, filter according to the scene conditions and discard points whose confidence does not meet the confidence threshold. The confidence threshold is set according to the scene brightness and camera angle. 11.2) Compare the similarity between the preset fighting posture feature samples and the posture of the current target. The fighting posture feature samples are related to the angle of the video camera. If the similarity meets the requirements, it is determined to be a fighting state; otherwise, it is determined to be a non-fighting state.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the region of interest target detection model, human pose model and violent behavior detection method as described in any one of claims 1-7, and to detect violent behavior in the input video.

9. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores at least one instruction or at least one program, which, when executed, implements the region of interest target detection model, human posture model, and violence behavior detection method as described in any one of claims 1-7.

Images