Method, device, medium and product for detecting violence in surveillance video
By adaptively adjusting the threshold for determining personnel rectangles and using a violent behavior recognition model with a Transformer encoder, the problem of inaccurate identification of personnel depth differences in the monitoring system was solved, achieving more accurate violent behavior detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAMEN MILESIGHT IOT CO LTD
- Filing Date
- 2025-11-05
- Publication Date
- 2026-06-19
Smart Images
Figure CN121505512B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision, and in particular to a method, apparatus, equipment, and product for detecting violent behavior in surveillance videos. Background Technology
[0002] With the acceleration of urbanization and the increasing demand for public safety, video surveillance systems have been widely used in various public places such as airports, train stations, banks, schools, and streets, becoming an important technical means to maintain social security and prevent crime. However, the complex actions and behaviors detected in most surveillance systems still rely heavily on manual monitoring and analysis. Faced with massive and constantly generated video data, security personnel struggle to make timely analyses and judgments, and are prone to missed or false alarms due to fatigue and decreased attention, thus missing the best opportunity to intervene and prevent violent incidents from occurring or escalating. To overcome the limitations of manual monitoring, computer vision-based methods for detecting violent behavior have emerged.
[0003] Patent application number 202310758668.1 proposes a video analysis-based method for identifying fighting behavior. This method first determines whether continuous and frequent close-range human contact exists in the surveillance video through human detection and normalized distance calculation. Then, it fuses and extracts regions exhibiting interactive behavior and inputs them into an improved lightweight X3D video classification model. This model not only outputs behavior classification results but also achieves precise localization of the fighting area through category-activated heatmap branches that share fully connected layer parameters. Finally, through post-processing of the heatmap and filtering optimization of historical recognition results, it enables the identification, location, and alarm of fighting behavior on edge devices.
[0004] However, because the horizontal installation angle of surveillance cameras is usually small, when two people have significantly different actual depths but are on the same line of sight, their center points in the image will be relatively close. The recognition method in the aforementioned patent application uses normalized pixel distance as the criterion, that is, the pixel distance between the center points of the two people's detection boxes divided by the area of the smaller of the two boxes; this method has the following problem: as long as the area of the smallest bounding box and the pixel distance between their center points remain unchanged, the normalized distance remains unchanged regardless of how the area of the other person's box changes. Figure 1As shown, the center pixel distances of Person 1, Person 2, and Person 3 in the image are the same, and the smallest bounding box is Person 1, resulting in consistent normalized distance calculation results. However, in real-world scenarios, Person 1 and Person 2 are spatially close, while Person 1 is far away from Person 3, exhibiting significant depth differences. This method, relying solely on normalized distance in the image plane, fails to fully integrate depth-sensing information, making it difficult to effectively distinguish between different pairs of people located on the same line of sight but with significantly different actual depths. Furthermore, since the normalized pixel distance itself has insufficient precision, relying on this distance, the Intersection over Union (IOU), and two fixed thresholds to determine the actual distance relationship of people in 3D space will also yield inaccurate results. Summary of the Invention
[0005] To address the aforementioned problems in the prior art, embodiments of the present invention provide a method, device, medium, and product for detecting violent behavior in surveillance videos.
[0006] To achieve the above objectives, on the one hand, a method for detecting violent behavior in surveillance videos is provided, including:
[0007] S1, perform personnel target detection on the frame images in the surveillance video to obtain multiple personnel targets, where each personnel target is identified by a rectangular box;
[0008] S2, calculate the distance d and distance threshold thresh between the bounding boxes of every two person targets in a multi-person target group, and determine that the two corresponding person targets belong to a dense group of people when the distance d is less than the distance threshold thresh; where:
[0009]
[0010]
[0011]
[0012]
[0013]
[0014]
[0015] and These represent the x and y coordinates of the first rectangle corresponding to person p1, respectively. and These represent the x and y coordinates of the second rectangle corresponding to person p2, respectively; and These represent the areas of the first rectangle and the second rectangle, respectively. and These represent the diagonal lengths of the first rectangle and the second rectangle, respectively. and This represents the width and height of the first rectangle; and This represents the width and height of the second rectangle; factor represents a predetermined constant factor.
[0016] S3, based on the spatial relationship diagram of dense people in the frame image, identify the region where dense people are connected, and take the smallest bounding rectangle of the rectangle of dense people in the connected region as the dense area of people.
[0017] S4, starting from the current frame, capture one image of the densely populated area every t seconds, for a predetermined number of consecutive captures, where the value of t is preset;
[0018] S5, input the predetermined number of captured image frames into the pre-trained violence behavior recognition model, and the violence behavior recognition model outputs the determination result of whether violence exists in the current scene.
[0019] Preferably, in the method, in step S3, a breadth-first search algorithm is applied to identify connected regions based on the spatial relationship graph of dense people in the frame image.
[0020] Preferably, in the method, the violent behavior recognition model includes: a 2D convolutional neural network and a Transformer encoder; the Transformer encoder includes: a self-attention mechanism layer, a feedforward neural network layer, a linear layer and a Sigmoid layer.
[0021] Preferably, in the method, the violent behavior recognition model processes a predetermined number of input image frames and outputs a violent behavior confidence score; when the violent behavior confidence score is greater than a predetermined confidence score threshold, it is determined that violent behavior exists; otherwise, it is determined that no violent behavior exists.
[0022] Preferably, in the method, the predetermined number of frame image sequences are N RGB images with width W and height H; the processing performed by the violence behavior recognition model on the input predetermined number of frame image sequences includes:
[0023] Feature extraction is performed on each of the N RGB images through multiple 2D convolutional layers, and N first feature maps with a width of W / 32, a height of H / 32, and a number of channels of C are output.
[0024] Flatten the first feature map along the first, third, and fourth dimensions to obtain a second feature map with a height of C and a width of N×(H / 32)×(W / 32);
[0025] The second feature map is concatenated with the first classification feature vector with height C and width 1 to obtain a third feature map with height C and width L, and the third feature map is input into the Transformer encoder; where L=(N×(H / 32)×(W / 32)+1);
[0026] The Transformer encoder performs self-attention mechanism calculation on the third feature map to obtain the self-attention mechanism calculation result. The self-attention mechanism calculation result is then processed by a feedforward neural network, and the result of the feature processing is processed by a classification feature vector function to obtain a second classification feature vector. The obtained classification feature vector is then processed by a linear layer mapping and a sigmoid activation function.
[0027] Preferably, in the method, the initial value of the first classification feature vector is a random number with a mean of 0 and a variance of 0.02; the value of the first classification feature vector is updated during training.
[0028] On the other hand, an electronic device is also provided, including a memory and a processor, the memory storing at least one program, the at least one program being executed by the processor to implement the steps of the method for detecting violent behavior in surveillance video as described in any of the above.
[0029] In another aspect, a computer-readable storage medium is also provided, wherein at least one program is stored therein, the at least one program being executed by a processor to implement the steps of the method for detecting violent behavior in surveillance video as described in any of the above.
[0030] In another aspect, a computer program product is also provided, comprising a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of the method for detecting violent behavior in surveillance video as described in any of the above.
[0031] The above technical solution has the following technical effects:
[0032] The technical solution of this invention uses a judgment threshold that is adaptively adjusted based on the area information of the personnel target detection rectangle when determining dense crowds; specifically, when the area difference of the rectangles corresponding to different personnel increases, the adjustment factor... The value approaches 0, thus attenuating the judgment threshold. Since people are considered to be in a densely populated state when the distance between them is less than the threshold, this increases the difficulty of classifying them as densely populated, making false judgments less likely. For example, if two people are of different sizes within the camera's view, and they are far apart, the greater the area difference, the smaller the threshold will be, making it less likely that these two targets will be considered densely populated; conversely, if the area difference decreases, the adjustment factor... When the threshold value approaches 1, the judgment threshold is not affected by attenuation or the attenuation is small, reducing the difficulty of determining dense states. Therefore, the technical solution of this invention effectively overcomes the shortcomings of existing fixed threshold methods in depth perception, and can more realistically and accurately reflect the actual positional relationship of people in three-dimensional space, improving the reliability and scene adaptability of the judgment results. In particular, it can effectively distinguish multiple person targets located on the same line of sight but with significant differences in actual depth.
[0033] In a further embodiment, the violent behavior recognition model used in this embodiment of the invention overcomes the limitation of the perception range caused by the use of the STMM module in existing violent behavior detection models by introducing a global self-attention mechanism, and significantly improves the recognition accuracy and generalization performance of the model in complex scenarios. Attached Figure Description
[0034] Figure 1 A schematic diagram of a scene captured by a horizontally mounted surveillance camera with a small included angle;
[0035] Figure 2 This is a flowchart illustrating a method for detecting violent behavior in surveillance video according to an embodiment of the present invention.
[0036] Figure 3 This is a schematic diagram of the violent behavior recognition model used in a method for detecting violent behavior in surveillance video according to an embodiment of the present invention. Detailed Implementation
[0037] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments and to explain the operating principles of the embodiments in conjunction with the relevant descriptions in the specification. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention. Components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.
[0038] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.
[0039] Example 1:
[0040] Figure 2 This is a flowchart illustrating a method for detecting violent behavior in surveillance video according to an embodiment of the present invention. Figure 2 The method for detecting violent behavior in surveillance video in this embodiment includes the following steps:
[0041] S1, perform personnel target detection on the frame images in the surveillance video to obtain multiple personnel targets, where each personnel target is identified by a rectangular box;
[0042] S2, calculate the distance d and distance threshold thresh between the bounding boxes of every two person targets in a multi-person target group, and determine that the two corresponding person targets belong to a dense group of people when the distance d is less than the distance threshold thresh; where:
[0043]
[0044]
[0045]
[0046]
[0047]
[0048]
[0049] and These represent the x and y coordinates of the first rectangle corresponding to person p1, respectively. and These represent the x and y coordinates of the second rectangle corresponding to person p2, respectively; and These represent the areas of the first and second rectangles, respectively. and These represent the diagonal lengths of the first and second rectangles, respectively. and This represents the width and height of the first rectangle; and This represents the width and height of the second rectangle; factor represents a predetermined constant factor; preferably, factor is 1.5;
[0050] In this step, when determining whether there are densely packed people, i.e. whether there are people in a dense state, the area information of the rectangular frame of the target person is introduced to adaptively adjust the threshold for judging densely packed people. This can effectively overcome the shortcomings of the fixed threshold method used in the prior art in terms of depth perception, and avoid misjudging people with large area differences and far actual distances as densely packed people. This situation is prone to occur in images captured by side-mounted cameras, thus reflecting the actual positional relationship of people in three-dimensional space more realistically and accurately.
[0051] This step determines the positional relationship of people in the image based on the detection boxes (rectangles) of the detected personnel targets, and can then determine whether there are densely populated areas through step S3.
[0052] S3. Based on the spatial relationship diagram of dense people in the frame image, identify the connected regions of dense people, and take the smallest bounding rectangle of the connected dense people rectangles in the connected regions as the dense people region.
[0053] In one specific implementation, based on the spatial relationship diagram of dense people in the frame image, a breadth-first search algorithm is applied to identify connected regions; the connected regions are the regions connected by the rectangular boxes of dense people; the breadth-first search algorithm is a search algorithm of the prior art, and will not be described in detail here to avoid confusing the inventive point of this invention;
[0054] S4, starting from the current frame, capture an image of a densely populated area every t seconds, for a predetermined number of consecutive captures, where the value of t is preset;
[0055] S5, input the predetermined number of captured image frames into the pre-trained violence behavior recognition model, and the violence behavior recognition model outputs the determination result of whether violence exists in the current scene.
[0056] Preferably, in an embodiment of the present invention, the violence behavior recognition model includes: a 2D convolutional neural network and a Transformer encoder; the Transformer encoder includes: a self-attention mechanism layer, a feedforward neural network layer, a linear layer, and a sigmoid layer. After processing a predetermined number of input image frames, the violence behavior recognition model outputs a violence behavior confidence score; when the violence behavior confidence score is greater than a predetermined confidence score threshold, it is determined that violence behavior exists; otherwise, it is determined that no violence behavior exists.
[0057] Figure 3 This is a schematic diagram of the violent behavior recognition model structure used in a method according to an embodiment of the present invention. Figure 3 The input to the violent behavior recognition model is: It consists of N RGB video frame images with a width of W and a height of H, and is represented by... express, Representing the real number field. The violent behavior recognition model first extracts features from each frame of the N input images using multiple 2D convolutional layers, resulting in N output feature maps with a width of W / 32, a height of H / 32, and C channels. This indicates the output feature map. Flattening the map along the first, third, and fourth dimensions yields a feature map with height C and width N×(H / 32)×(W / 32), which, together with the classification feature vector of height C and width 1, forms the first classification feature vector. The features are concatenated to form a feature map with height C and width L. The first classification feature vector mentioned above is added separately, and its initial value is a random number with a mean of 0 and a variance of 0.02, which will be updated during training. The feature map L = (N × (H / 32) × (W / 32) + 1) is used as the input to the Transformer encoder. express.
[0058] Existing technologies typically use 3D convolutional models, while the inventors of this invention, considering that the violent behaviors to be detected usually involve actions, i.e., hands and feet with certain movement trajectories, use image sequences as input, and extract features from each dense region of N images through a single 2D convolutional neural network in N batches; at the same time, a Transformer encoder is used to associate these easily identifiable fighting features and output the corresponding confidence scores.
[0059] The Transformer encoder first processes the input... Perform self-attention mechanism calculations:
[0060]
[0061] Where Q, K, and V are inputs The feature matrix is mapped from three linear layers, where Q is the query matrix, K is the key matrix, and V is the value matrix.
[0062] Then, after feature processing by the feedforward neural network, the classification feature vector obtained by the classification feature vector function will be... The confidence level of the final brute-force behavior is obtained by processing through linear layer mapping and the sigmoid activation function. :
[0063]
[0064]
[0065]
[0066] in, This represents the processing performed by the feedforward neural network, where Linear represents a linear layer and F represents the feature vector for classification. Function. When If the predetermined confidence threshold is 0.5, then violent behavior is determined to exist; otherwise, no violent behavior is found.
[0067] Example 2:
[0068] The present invention also provides an electronic device including a memory and a processor, the memory storing at least one program, the at least one program being executed by the processor to implement the steps of the method for detecting violent behavior in surveillance video as described above.
[0069] The aforementioned electronic device can be a computer unit, which may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer unit may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described structure of the computer unit is merely an example and does not constitute a limitation on the computer unit. It may include more or fewer components, or combine certain components, or use different components. For example, the computer unit may also include input / output devices, network access devices, buses, etc., and this embodiment of the invention does not limit this.
[0070] Furthermore, as an executable solution, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, connecting various parts of the entire computer unit via various interfaces and lines.
[0071] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer unit by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0072] Example 3:
[0073] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the embodiments of the present invention.
[0074] If the modules / units integrated in the computer unit are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.
[0075] Example 4:
[0076] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps described above.
[0077] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.
Claims
1. A method of monitoring video for violence detection, the method comprising: include: S1, perform personnel target detection on the frame images in the surveillance video to obtain multiple personnel targets, where each personnel target is identified by a rectangular box; S2, calculate the distance d and distance threshold thresh between the bounding boxes of every two person targets in a multi-person target group, and determine that the two corresponding person targets belong to a dense group of people when the distance d is less than the distance threshold thresh; where: and These represent the x and y coordinates of the first rectangle corresponding to person p1, respectively. and These represent the x and y coordinates of the second rectangle corresponding to person p2, respectively; and These represent the areas of the first rectangle and the second rectangle, respectively. and These represent the diagonal lengths of the first rectangle and the second rectangle, respectively. and This represents the width and height of the first rectangle; and This represents the width and height of the second rectangle; factor represents a predetermined constant factor. S3, based on the spatial relationship diagram of dense people in the frame image, identify the region where dense people are connected, and take the smallest bounding rectangle of the rectangle of dense people in the connected region as the dense area of people. S4, starting from the current frame, capture one image of the densely populated area every t seconds, for a predetermined number of consecutive captures, where the value of t is preset; S5, input the predetermined number of captured image frames into the pre-trained violence behavior recognition model, and the violence behavior recognition model outputs the determination result of whether violence exists in the current scene.
2. The method according to claim 1, characterized in that, In step S3, based on the spatial relationship diagram of dense people in the frame image, a breadth-first search algorithm is applied to identify connected regions.
3. The method according to claim 1, characterized in that, The violent behavior recognition model includes a 2D convolutional neural network and a Transformer encoder; the Transformer encoder includes a self-attention mechanism layer, a feedforward neural network layer, a linear layer, and a Sigmoid layer.
4. The method according to claim 3, characterized in that, The violent behavior recognition model processes a predetermined number of image frames as input and outputs a violent behavior confidence score. When the violent behavior confidence score is greater than a predetermined confidence score threshold, it is determined that violent behavior exists. Otherwise, it is determined that no violence occurred.
5. The method according to claim 2, characterized in that, The predetermined number of frame image sequences consists of N RGB images with a width of W and a height of H; the processing performed by the violence behavior recognition model on the input predetermined number of frame image sequences includes: Feature extraction is performed on each of the N RGB images through multiple 2D convolutional layers, and N first feature maps with a width of W / 32, a height of H / 32, and a number of channels of C are output. Flatten the first feature map along the first, third, and fourth dimensions to obtain a second feature map with a height of C and a width of N×(H / 32)×(W / 32); The second feature map is concatenated with the first classification feature vector with height C and width 1 to obtain a third feature map with height C and width L, and the third feature map is input into the Transformer encoder; where L=(N×(H / 32)×(W / 32)+1); The Transformer encoder performs self-attention mechanism calculation on the third feature map to obtain the self-attention mechanism calculation result. The self-attention mechanism calculation result is then processed by a feedforward neural network, and the result of the feature processing is processed by a classification feature vector function to obtain a second classification feature vector. The obtained classification feature vector is then processed by a linear layer mapping and a sigmoid activation function.
6. The method according to claim 5, characterized in that, The initial value of the first classification feature vector is a random number with a mean of 0 and a variance of 0.02; the value of the first classification feature vector is updated during training.
7. An electronic device, characterized in that, The device includes a memory and a processor, the memory storing at least one program, which is executed by the processor to implement the steps of the method for detecting violent behavior in surveillance video as described in any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that, The storage medium stores at least one program, which is executed by a processor to implement the steps of the method for detecting violent behavior in surveillance video as described in any one of claims 1 to 6.
9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for detecting violent behavior in surveillance video as described in any one of claims 1 to 6.