A method for detecting and tracking violence based on audio and video

Abstract

The application discloses a violence detection and tracking method based on audio and video, and belongs to the technical field of violence detection and tracking. The application solves the problems of low detection accuracy, slow response speed, high dependence on hardware and the need for manual intervention in the prior art. The application combines the violence recognition results of video data and audio data, eliminates the misrecognition of the current violence recognition result based on historical violence recognition results, improves the accuracy of violence recognition, controls the movement direction of the camera based on historical information, and improves the detection range of a single camera. The YOLO model is used to recognize the position of violent personnel, and the identity features of the violent personnel are extracted. The violence detection and tracking method has the characteristics of high detection accuracy, fast response speed, low dependence on hardware, and no need for manual intervention. The method can be applied to violence detection and tracking.

Description

Technical Field

[0001] This invention belongs to the field of violence detection and tracking technology, specifically relating to a method for detecting and tracking violent behavior based on audio and video. Background Technology

[0002] With the rapid development of technology and the times, advancements in camera technology and artificial intelligence (AI) have provided new solutions for social security and public safety. Traditional security monitoring systems often suffer from problems such as high latency, low accuracy, and reliance on human intervention when violent incidents occur.

[0003] Traditional surveillance systems often only provide video footage after the fact, lacking real-time early warning and intervention capabilities, posing a significant challenge to the prevention and control of violent incidents. Despite numerous efforts to strengthen public security management across various regions, relying solely on human patrols and traditional surveillance equipment is often insufficient to promptly detect and prevent these sudden acts of violence. Camera-based body capture violence detection and alarm systems have emerged to address this issue, aiming to enhance the real-time detection, tracking, and response capabilities to violent incidents through efficient technological means, and also possessing the ability to identify perpetrators.

[0004] In summary, existing methods for detecting violent incidents still suffer from low detection accuracy, slow response speed, high dependence on hardware, and the need for human intervention. Therefore, how to utilize advanced technologies to improve the accuracy of violent incident detection, the speed of response to violent incidents, reduce dependence on hardware, and eliminate the need for human intervention has become an important issue in current social security management. Summary of the Invention

[0005] The purpose of this invention is to address the problems of low detection accuracy, slow response speed, high hardware dependence, and the need for manual intervention in existing violent incident detection methods, and to propose a violent behavior detection and tracking method based on audio and video.

[0006] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a method for detecting and tracking violent behavior based on audio and video, the method specifically includes the following steps:

[0007] Step 1: Perform brute-force identification on the current frame of the surveillance video and the corresponding audio separately, and then perform weighted fusion on the obtained brute-force identification results. Use the weighted fusion result as the probability that the current frame of the surveillance video is identified as brute-force.

[0008] Step 2: Compare the probability of the current frame of the surveillance video being identified as violent with the violence identification threshold, and obtain the violence identification result of the current frame of the surveillance video based on the comparison result;

[0009] Step 3: Perform temporal filtering based on the brute-force identification results of historical surveillance videos and the brute-force identification results of the current frame of surveillance video, and determine whether a brute-force response is required based on the temporal filtering results;

[0010] If a violent response is required, extract the information of the violent individuals in the current frame of the surveillance video before proceeding to step four.

[0011] If a brute-force response is not required, proceed directly to step four;

[0012] Step 4: Move the camera to the next position to continue capturing the next frame of surveillance video, and then return to Step 1 to execute the captured next frame of surveillance video.

[0013] The beneficial effects of this invention are:

[0014] This invention uses a 3D-CNN model for feature extraction from video data, and an SVM model for violence detection. For audio data, it uses Mel-ray features and a 2D-CNN model for violence detection. By combining the violence detection results from both video and audio data, and then using historical violence detection results to eliminate false positives in the current violence detection, the accuracy of violence detection can be improved. Furthermore, by controlling the camera to move in the direction of potential violence based on historical information, the detection range of a single camera is increased, ensuring response and tracking of violence. The violence perpetrator information extraction part uses a YOLO model to identify the location of the violence perpetrator and extract their identity features. This invention's violence detection and tracking method can detect violence in real time for rapid response. It features high detection accuracy, fast response speed, low hardware dependence, and requires no human intervention throughout the process. Therefore, it can improve the real-time detection, tracking, and response capabilities for violent incidents, thereby enhancing public safety.

[0015] Experiments show that the violent behavior detection method of the present invention can achieve an accuracy of about 97% on both the UBI dataset and the Hockey dataset. Attached Figure Description

[0016] Figure 1 This is a flowchart of a method for detecting and tracking violent behavior based on audio and video according to the present invention;

[0017] Figure 2 This is a flowchart of the brute-force identification section;

[0018] Figure 3 This is a flowchart for surveillance video detection;

[0019] Figure 4 This is a schematic diagram of the 3D-ResNet structure;

[0020] Figure 5This is a schematic diagram of the 3D-InceptionNet structure;

[0021] Figure 6 It is a prediction probability diagram for time-lapse prediction;

[0022] In the figure, the horizontal axis represents the probability of brute force given by the model, and the vertical axis represents the number of samples.

[0023] Figure 7 It is a full-sample prediction probability map;

[0024] Figure 8 This is a flowchart for brute-force detection of audio.

[0025] Figure 9 This is a schematic diagram of a Markov decision chain;

[0026] Figure 10 This is a schematic diagram of the camera movement trajectory obtained by the Bayesian decision method;

[0027] In the diagram, the horizontal axis represents the camera's position along the x-axis, and the vertical axis represents the camera's position along the y-axis.

[0028] Figure 11 This is a schematic diagram of the camera movement trajectory obtained by reinforcement learning.

[0029] Figure 12 This is a flowchart for extracting information on violent individuals;

[0030] Figure 13 This is a speed comparison chart of YOLOv3 with other methods;

[0031] In the figure, the horizontal axis represents the model inference time, and the vertical axis represents the mAP score on the COCO dataset.

[0032] Figure 14 This is a diagram of the YOLOv3 network architecture;

[0033] Figure 15 It is a schematic diagram of a bounding box with dimensional priors and location predictions;

[0034] Figure 16 This is a structural diagram of the CNN-REID network. Detailed Implementation

[0035] Specific implementation method one: Combining Figure 1 This embodiment describes a method for detecting and tracking violent behavior based on audio and video. The method specifically includes the following steps:

[0036] Step 1: Perform brute force identification on the current frame of the surveillance video and the corresponding audio separately, and then perform weighted fusion on the obtained brute force identification results (that is, perform weighted fusion on the probability that the video is identified as brute force and the probability that the audio is identified as brute force, and the obtained weighted fusion result is the probability that brute force has occurred). Use the weighted fusion result as the probability that the current frame of the surveillance video is identified as brute force.

[0037] Step 2: Compare the probability of the current frame of the surveillance video being identified as violent with the violence identification threshold, and obtain the violence identification result of the current frame of the surveillance video based on the comparison result;

[0038] Step 3: Perform temporal filtering based on the brute-force identification results of historical surveillance videos and the brute-force identification results of the current frame of surveillance video, and determine whether a brute-force response is required based on the temporal filtering results;

[0039] If a violent response is required, extract the information of the violent individuals in the current frame of the surveillance video before proceeding to step four.

[0040] If a brute-force response is not required, proceed directly to step four;

[0041] Step 4: Move the camera to the next position to continue capturing the next frame of surveillance video, and then return to Step 1 to execute the captured next frame of surveillance video.

[0042] Modern surveillance systems integrate camera and radio functions, capturing both live video and audio. Traditional methods often struggle to distinguish between normal interactions and genuine acts of violence, leading to numerous misjudgments. The inclusion of audio data significantly improves accuracy, greatly reducing the probability of false positives. Furthermore, during detection, temporal filtering eliminates false positives based on historical data, and reinforcement learning and online learning enable self-movement control of the camera to expand the detection range.

[0043] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that the brute-force identification of the current frame of monitored video and the corresponding audio is performed separately as follows:

[0044] The 3D-CNN network is used to identify violence in the current frame of the surveillance video. The 3D-CNN network is used to extract features from the current frame of the surveillance video. The feature extraction results are then passed through a support vector machine or a fully connected neural network. The output of the support vector machine or the fully connected neural network is probability-calibrated, and the probability calibration result is used as the initial probability that the current frame of the surveillance video is identified as violence.

[0045] The method for brute-force identification of the audio corresponding to the current frame of the surveillance video uses Mel features and a 2D-CNN network. First, Mel features are extracted from the audio corresponding to the current frame of the surveillance video. Then, the extracted Mel features are passed through a 2D-CNN network. The output of the 2D-CNN network is used as the probability that the audio corresponding to the current frame of the surveillance video is identified as brute-force.

[0046] The other steps and parameters are the same as in Specific Implementation Method 1.

[0047] The 3D-CNN network of this invention is pre-trained on the Kinetics-700 pose recognition dataset and exhibits strong feature extraction capabilities. Support Vector Machines (SVMs) are used for feature-based classification to obtain brute-force detection results. Since the SVM's objective function is to maximize the minimum projection distance of different data categories to the classification hyperplane, it exhibits high robustness and can still effectively classify data even with a limited brute-force detection dataset.

[0048] This implementation uses a 3D-CNN network to extract features from video data simultaneously in both temporal and spatial dimensions, and then performs classification. Compared to traditional pose recognition methods based on optical flow estimation, it has advantages such as high accuracy and low computational cost; compared to the CNN-RNN architecture that extracts spatial features first and then temporal features, it has advantages in both accuracy and real-time performance; and compared to methods based on Transformers variants, it has significant advantages in both computational cost and real-time performance.

[0049] In the brute-force detection of audio data, the audio data is first preprocessed. For excessively long audio data, only the first half is extracted, which reduces computational load without losing too much important information. For excessively short audio data, zero-padding is used to make it of uniform length. This preprocessing method not only simplifies the complexity of data processing but also ensures the consistency of input data.

[0050] This invention uses Mel features to convert preprocessed audio data to the frequency domain. In violence detection scenarios, extracting Mel frequency cepstral coefficients (MFCCs) can help identify violent behavioral features such as screams and fighting sounds in the audio data, providing important information for subsequent classification tasks.

[0051] By combining the feature extraction capabilities of MFCC with the powerful classification capabilities of 2D convolutional neural networks, it is possible to efficiently and accurately identify violent behavior features in audio data, achieving high accuracy and real-time performance in audio violence detection, and providing important technical support for maintaining public safety.

[0052] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method Two in that the probability calibration of the output of the support vector machine or fully connected neural network is specifically as follows:

[0053]

[0054] Where y = 1 indicates that the current frame of the surveillance video is identified as violent, p(y = 1|z) is the initial probability of the current frame of the surveillance video being identified as violent after processing the current frame of the surveillance video, e is the base of the natural logarithm, and z is the output of the support vector machine or fully connected neural network.

[0055] The other steps and parameters are the same as in Specific Implementation Method Two.

[0056] The initial probability of the current frame of the surveillance video being identified as violent is then fused with the result obtained from audio processing. The fused result is the identification result of step one.

[0057] Specific Implementation Method Four: This implementation method differs from Specific Implementation Method Three in that, in step two, the probability of the current frame of the surveillance video being identified as violent is compared with a violence identification threshold, and the violence identification result of the current frame of the surveillance video is obtained based on the comparison result; the specific process is as follows:

[0058] Step Two: Define the occurrence of violence as x = 1 and the occurrence of non-violence as x = 0. Let C be the loss incurred when a non-violent video is identified as a violent video. 01 The loss caused by identifying a violent video as a non-violent video is denoted as C. 10 The loss for correctly identifying a violent video as a violent video is denoted as C. 11 The loss for correctly identifying a nonviolent video as a nonviolent video is denoted as C. 00 Then the expected value E[C] of the loss C is:

[0059]

[0060] Where P(x=0) is the probability that the actual situation is non-violent, P(y=1|x=0) is the probability that the model predicts the value as violent when the actual situation is non-violent (for videos where the actual situation is non-violent, the fusion result after processing the audio and video is P(y=1|x=0), that is, the model's prediction value here is the fusion result after processing the audio and video), P(y=0|x=0) is the probability that the model predicts the value as non-violent when the actual situation is non-violent, P(x=1) is the probability that the actual situation is violent, P(y=1|x=1) is the probability that the model predicts the value as violent when the actual situation is violent, P(y=0|x=1) is the probability that the model predicts the value as non-violent when the actual situation is violent, P(y|x=0) is the probability that the model predicts the output as y when the actual situation is non-violent, P(y|x=1) is the probability that the model predicts the output as y when the actual situation is violent, and y=0 or 1;

[0061] Step 22: Calculate the brute-force discrimination threshold y based on the expected loss value. T (That is, calculated based on the stationary point of the expected loss):

[0062]

[0063] Steps 2 and 3: Compare the probability of the current frame of the surveillance video being identified as violent with the violence identification threshold;

[0064] If the probability of the current frame of the surveillance video being identified as violent is greater than or equal to the violence identification threshold, then the violence identification result is that the current frame of the surveillance video contains violence.

[0065] If the probability of the current frame of the surveillance video being identified as violent is less than the violence identification threshold, then the violence identification result is that the current frame of the surveillance video does not contain violence.

[0066] The other steps and parameters are the same as in Specific Implementation Method 3.

[0067] Specific Implementation Method Five: This implementation method differs from Specific Implementation Method Four in that, in step three, temporal filtering is performed based on the brute-force identification results of historical surveillance videos and the brute-force identification results of the current frame surveillance video. The result of the temporal filtering is then used to determine whether a brute-force response is necessary. The specific process is as follows:

[0068] The number of frames in the historical surveillance video is set to 3. That is, the 3 frames of surveillance video before the current frame are selected to perform time-series filtering on the violence identification result of the current frame. The automaton state composed of the violence identification results of the selected 3 historical surveillance videos is denoted as abc, where the values ​​of a, b, and c are all 0 or 1. 1 represents that the corresponding surveillance video is identified as having violence, and 0 represents that the corresponding surveillance video is identified as not having violence.

[0069] If the result of brute-force identification of the current frame of the monitored video is denoted as d, then the automaton state will be converted to bcd;

[0070] When the state bcd is 000, 010, or 100, a brute-force response is not required.

[0071] If the state bcd is not 000, 010, or 100, a brute-force response is required.

[0072] The other steps and parameters are the same as in Specific Implementation Method Four.

[0073] This implementation method effectively eliminates false identification through time-series filtering, reduces the false judgment rate, and improves the accuracy and stability of the detection method.

[0074] Specific Implementation Method Six: This implementation method differs from Specific Implementation Method Four in that, in step three, temporal filtering is performed based on the brute-force identification results of historical surveillance videos and the brute-force identification results of the current frame of surveillance video. The result of the temporal filtering is then used to determine whether a brute-force response is necessary. The specific process is as follows:

[0075] Step 3. Calculate p(y) t |z t =i):

[0076]

[0077] Where p(y t ) is the surveillance video at time t that is predicted as y. t The probability, y t =0 or 1, y t =1 indicates that the surveillance video at time t is predicted to be violent, y t =0 indicates that the surveillance video at time t is predicted to be non-violent; p(y t |z t =i) is the prediction result y when the actual situation of the surveillance video at time t is i. t The probability of p(z) t =i|y t y is the prediction result of the surveillance video at time t. t When z is the probability that the actual situation is i; t It is the actual situation of the surveillance video at time t;

[0078] Define the recursive function α at time t. t (i):

[0079] α t (i)=p(y 1：t , z t =i)=∑ j p(y 1：t-1 , zt-1 =j)p(z t =i|z t-1 =j)p(y t |z t =i)

[0080] =p(y t |z t =i)Σ j α t-1 (p(z t =i|z t-1 =j)

[0081] Where p(y 1:t-1 ,z t-1 =j) represents the actual situation of the surveillance video at time t-1, where j is the actual situation and y is the predicted result of the surveillance video from time 1 to time t-1. 1:t-1 The probability of p(z) t =i|z t-1 =j) is the probability that the actual situation of the surveillance video at time t-1 is j, and the actual situation of the surveillance video at time t is i; α t-1 (j) is the recursive function at time t-1; where y 1:t-1 Each parameter in the value ranges from 0 to 1.

[0082]

[0083] Where p(y1: t , z t =i) represents the actual situation of the surveillance video at time t as i, and the predicted result of the surveillance video from time 1 to time t as y1: t Probability, p(z) t =i|y1: t The prediction result of the surveillance video from time 1 to time t is y1: t When the actual situation is i, the probability is i.

[0084] Step 3.2: Compare p(y) t =1|z t =i) and the magnitude of the brute-force discrimination threshold:

[0085] If p(y t =1|z t =i) If the threshold for brute-force discrimination is greater than or equal to the threshold, then a brute-force response is required; otherwise, no brute-force response is required.

[0086] The other steps and parameters are the same as in Specific Implementation Method Four.

[0087] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Method Five or Six in that the moving camera continues to capture the next frame of monitoring video at the next location. The calculation method for the target location is as follows:

[0088] After calculating the intermediate variable w, the posterior probability p(x|μ0,∑0,μ) of the target position x of the camera movement is calculated based on the intermediate variable w. x , ∑ x ,y):

[0089] p(x|μ0,∑0,μ x , ∑ x ,y)∝p(μ0,∑0,μ x , ∑ x ,y|x)=p(μ0,∑0|x)p(μ x , ∑ x |x)p(y|x)∝p(μ0,∑0,x)p(μ x , ∑ x ,x)p(y,x)∝p(x|μ0,∑0)p(x|μ x , ∑ x )p(y)

[0090]

[0091]

[0092]

[0093] Where μ0 is the most likely angle for the camera to capture, μ x This represents the position of the camera when capturing the current frame of the surveillance video. ∝ indicates that the left side of the formula is proportional to the right side of the formula. p(μ0, Σ0, μ) x , ∑ x Let y|x) represent the angle that the camera is most likely to capture when the target position of the camera is x, where μ0 is the angle, ∑0 is the prior distribution covariance matrix of the camera position, and μ is the current position of the camera. x The camera movement probability covariance matrix is ​​∑ x The probability that a surveillance video taken at the target location is identified as y, where y = 0 or 1, and p(μ0, ∑0|x) represents the probability that the angle captured by the camera is μ0 when the target location of the camera is x. The probability that the prior distribution covariance matrix of the camera position is ∑0 is also given by p(μ0, ∑0|x). x , ∑ x |x) represents the current position of the camera when the target position is x and the current position of the camera is μ. x The camera movement probability covariance matrix is ​​∑ xLet p(y|x) represent the probability that the surveillance video captured by the camera at target position x is identified as y, and p(μ0,∑0,x) represent the probability that the camera's most likely shooting angle is μ0, the prior distribution covariance matrix of the camera position is ∑0, and the target position is x. x ,∑ x (x) represents the current position of the camera as μ. x The camera movement probability covariance matrix is ​​∑ x The probability that the target location is x, p(y,x) represents the joint probability that the surveillance video captured by the camera at target location x is identified as y, and p(x|μ0,∑0) represents the probability that the next target location of the camera is x when the most likely shooting angle of the camera is μ0 and the prior distribution covariance matrix of the camera position is ∑0. x ,∑ x ) indicates that the current position of the camera is μ x The camera movement probability covariance matrix is ​​∑ x When the next target location is x, p(y) represents the probability that the surveillance video captured by the camera is identified as y, and ∑0 represents the prior distribution covariance matrix of the camera location. x Let represent the covariance matrix of the camera movement probability, α represent the hyperparameter of the probability distribution of the brute force value, and (x-μ0)′ be the transpose of x-μ0. x )′ is x-μ x The transpose of , w′ is the transpose of w, the superscript -1 represents the inverse of the matrix, and N represents the normal distribution;

[0094] Then, based on the posterior probability p(x|μ0,∑0,μ) x ,∑ x ,y) to obtain the target position x of the camera movement.

[0095] The other steps and parameters are the same as in specific implementation methods five or six.

[0096] This implementation can track violence by controlling the movement of the camera, ensuring continuous monitoring of violent behavior and increasing the detection range of a single camera.

[0097] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Method Five or Six in that the moving camera continues to capture the next frame of monitoring video at the next location. The calculation method for the target location is as follows:

[0098] Let the position of the camera when capturing the current frame of the surveillance video be x. Maximize the expected value of brute force y by determining the strategy p(a) for moving 'a':

[0099]

[0100] Where q(a|x) is the probability that the camera moves to position a when the current position is x, under any policy, and E q(a|x) [y=1] represents the expectation that the surveillance video captured by the camera after moving a to the new target position from the current position x will be identified as violent, and p(a|x) is the probability that the camera moves a when the current position is x under the target policy (i.e. the policy corresponding to the maximum expectation).

[0101] The target position of the camera movement is determined by using the movement 'a' corresponding to the highest probability p(a|x) and the current position 'x'.

[0102] The other steps and parameters are the same as in specific implementation methods five or six.

[0103] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Method Seven in that the intermediate variable w is calculated as follows:

[0104]

[0105] Where X is the position of the camera when the current frame of the surveillance video is captured, Y is the probability that the current frame of the surveillance video is identified as violent, ‖·‖ is the L2 norm, X′ is the transpose of X, γ represents the regularization coefficient, I is the identity matrix, and the superscript -1 represents the inverse of the matrix.

[0106] The other steps and parameters are the same as in Specific Implementation Method Seven.

[0107] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Method One in that the specific process of extracting information about violent individuals from the current frame of the surveillance video is as follows:

[0108] Step 1: Use the YOLOv3 model to identify the location of the person in the current frame of the surveillance video;

[0109] Step 2: Use a CNN-REID network to extract information about violent individuals from the image of the area where the person's image is located;

[0110] The CNN-REID network includes, in sequence, a convolutional layer, a batch normalization layer, a ReLU activation function layer, a first ResBlock module, a second ResBlock module, a third ResBlock module, a fourth ResBlock module, a fifth ResBlock module, an adaptive max pooling layer, a Flatten layer, a Linear layer, and a Sigmoid activation function layer.

[0111] The working process of the CNN-REID network is as follows:

[0112] The image of the region where the person is located is used as the input to the CNN-REID network. Within the CNN-REID network, the input image is used as the input to the convolutional layer.

[0113] Use the output of the convolutional layer as the input of the BN layer;

[0114] Use the output of the BN layer as the input of the ReLU activation function layer;

[0115] The output of the ReLU activation function layer is used as the input of the first ResBlock module;

[0116] Use the output of the first ResBlock module as the input of the second ResBlock module;

[0117] Use the output of the second ResBlock module as the input of the third ResBlock module;

[0118] Use the output of the third ResBlock module as the input of the fourth ResBlock module;

[0119] Use the output of the fourth ResBlock module as the input of the fifth ResBlock module;

[0120] The output of the fifth ResBlock module is used as the input to the adaptive max pooling layer;

[0121] Use the output of the adaptive max pooling layer as the input of the Flatten layer;

[0122] Use the output of the Flatten layer as the input of the Linear layer;

[0123] Use the output of the Linear layer as the input of the Sigmoid activation function layer;

[0124] The output of the Sigmoid activation function layer is used as the extracted information about the violent perpetrators;

[0125] The first ResBlock module includes a first branch, a second branch, and a norm unit. The first branch includes a first res unit, a second res unit, and a convolutional layer, and the second branch includes a convolutional layer.

[0126] Within the first ResBlock module, the input of the first ResBlock module is first used as the input of the two branches, and then the outputs of the two branches are connected. The connection result is passed through the norm unit, and the output of the norm unit is used as the output of the first ResBlock module.

[0127] The first res unit contains, in sequence, a convolutional layer and a ReLU activation function layer;

[0128] The norm unit contains a ReLU activation function layer and a BN layer in sequence.

[0129] The other steps and parameters are the same as in Specific Implementation Method 1.

[0130] In this embodiment, the structures and operating processes of the second, third, fourth, and fifth ResBlock modules are the same as those of the first ResBlock module. The structure and operating process of the second res unit are also the same as those of the first res unit.

[0131] Example

[0132] This embodiment proposes a method for detecting and tracking violent behavior based on audio and video, the method specifically including the following steps:

[0133] Step 1: Perform brute-force recognition on the current frame of the monitored video and the corresponding audio separately, and then fuse the brute-force recognition results to obtain the initial brute-force recognition result for the current frame.

[0134] Among them, a 3D-CNN network is used for brute-force identification of the current frame of the surveillance video, such as... Figure 3 As shown, a 3D-CNN network is used to extract features from the current frame of the surveillance video. The feature extraction results are then passed through a support vector machine or a fully connected neural network (a fully connected neural network consisting of at least two fully connected layers connected sequentially). In other words, a fully connected layer used for classification in the original 3D-CNN network is replaced by a fully connected neural network or a support vector machine.

[0135] Comparative experiments were conducted between the 3D-CNN network used in this invention and traditional optical flow-based pose estimation and CNN-RNN architectures, confirming that the 3D-CNN architecture used in this invention has advantages in both accuracy and real-time performance.

[0136] The experimental dataset was obtained by filtering and merging approximately 6000 videos from five existing datasets. Specifically, after removing videos that were too long or otherwise problematic, 4800 videos were selected as the training set, and the remaining 801 videos were used as the test set. A summary of the merged datasets is shown in Table 1.

[0137] Table 1 Violent Video Detection Dataset

[0138]

[0139]

[0140] Table 1 lists the references for the dataset Real Life Violence Situations: “M. Soliman, M. Kamal, M. Nashed, Y. Mostafa, B. Chawky, D. Khattab, “Violence Recognition from Videos using Deep Learning Techniques”, Proc. 9th International Conference on Intelligent Computing and Information Systems (ICICIS'19), Cairo, pp. 79-84, 2019.”; the reference for the dataset UBI-fight: “Degardin B, Proenca H. Human Activity Analysis: Iterative Weak / Self-Supervised Learning Frameworks for Detecting Abnormal Events [C] / / 2020 IEEE International Joint Conference on Biometrics (IJCB). IEEE, 2020. DOI: 10.1109 / IJCB48548.2020.9304905.”; and the reference for the dataset surf: “ GA HKEkenel, “Vision-based Fight Detection from Surveillance Cameras”, IEEE / EURASIP 9th International Conference on Image Processing Theory, Tools and Applications, Istanbul, Turkey, November 2019.; Violence detection dataset reference: “Bianculli M, Falcionelli N, Sernani P, et al. A dataset for automatic violence detection in videos[J]. Data in brief, 2020, 33:106587.”; RWF-2000 dataset reference: “Cheng M, Cai K, Li M. RWF-2000: An Open Large Scale Video Database for Violence Detection[J]. 2019.”.

[0141] The video was read and resized using OpenCV. Every 2-4 frames, an image was selected from the video and added to the keyframe sequence. For excessively long videos, only the first half was extracted. Pixels were rendered using RGB channels and normalized to the [-1, 1] range.

[0142] 3D-ResNet and 3D-InceptionNet, both based on the 3D-CNN architecture, were tested and pre-trained on the Kinetics dataset (Carreira J, Noland E, Hillier C, et al. A short note on the kinetics-700humanaction dataset[J]. arXiv preprint arXiv:1907.06987,2019.). This dataset contains 650,000 videos with 700 different human poses, providing rich scene representation and thus enabling the models to have strong generalization capabilities.

[0143] 3D-ResNet (R3D) is a 3D extension of 2D-ResNet. The only difference is that the convolutional layers are replaced with 3D convolutions. The structure of each residual module is as follows: Figure 4 As shown, 3D-InceptionNet (I3D) is a 3D extension of InceptionNet, with the following structure: Figure 5 As shown.

[0144] This invention provides R3D-34, R3D-50, and I3D paired with fully connected neural networks or SVMs, trained and tested with various specifications including a maximum width of 128 or 256, a maximum number of frames of 32, 48, or 64, and an inter-frame interval of 0.08, 0.125, or 0.17 seconds, all demonstrating good performance. The performance on the test set is shown in Table 2.

[0145] Table 2 Model Performance

[0146]

[0147]

[0148] Probabilistic calibration of the Support Vector Machine (SVM) was performed using logistic regression with only one parameter. The final logistic regression result was 1.02 for almost all parameters. Therefore, when the SVM output is z, the probability y that the current frame of the monitored video is a violent video is:

[0149]

[0150] Where y = 1 indicates that the current frame of the surveillance video is identified as violent, p(y = 1|z) is the initial probability of the current frame of the surveillance video being identified as violent after processing the current frame of the surveillance video, e is the base of the natural logarithm, and z is the output of the support vector machine or fully connected neural network.

[0151] The relationship between the output probability of the calibrated SVM and the sample probability on the statistical test set is as follows: Figure 6 and Figure 7 As shown, the prediction probabilities are polarized, indicating that the SVM's judgment is not ambiguous, and the prediction probability approaches 0.5 when the prediction is wrong, indicating that the prediction probability has strong reference value.

[0152] To verify the superiority of the 3D-CNN architecture of this invention, traditional schemes based on HOF features (Kumar RN, Chandrakala D S. DETECTING AGGRESSIVE HUMAN BEHAVIOR IN PUBLIC ENVIRONMENTS[J].[2024-03-18].) and CNN-RNN architecture were tested. The HOF scheme uses HOG to extract image feature points, uses local least squares to obtain the optical flow at the feature points (i.e., the movement speed of the feature points), and calculates the histogram of the optical flow distribution to obtain HOF features, which are then classified using SVM. The CNN-RNN architecture uses ResNet-34 to extract features for each frame, and then uses GRU or Attention for classification. The test results are shown in Table 3:

[0153] Table 3

[0154]

[0155] As shown in Table 3, the HOF scheme is inferior to CNN-3D in both accuracy and latency.

[0156] The process of brute-force detection for audio is as follows: Figure 8 As shown, the training samples selected as the 2D-CNN model for audio violence detection come from an audio dataset, supplemented by audio tracks extracted from violent videos. However, the audio tracks in the violent video dataset mostly come from monitoring rooms or have no audio at all; only some videos in the Real Life Violence Situations dataset contain fighting audio. Therefore, this invention uses moviepy to extract the fighting audio from this dataset, as well as the audio from non-violent videos in all violent video detection datasets, and uses non-violent audio from the Audio-based Violence Detection Dataset as supplementary material.

[0157] The acquired dataset is saved in WAV format, and the dataset specifications are shown in Table 4:

[0158] Table 4 Audio Dataset Specifications

[0159]

[0160] When processing audio data, Mel features are used to process the audio to the frequency domain. For excessively long audio segments, only the first half is taken; for excessively short audio segments, zeros are used for padding. Then, 2D convolutions are used for classification. Since audio is relatively simple and the dataset is small, the number of convolutional layers is kept to a minimum, the stride is maximized, and dropout is added to prevent overfitting. The 2D-CNN network model structure is shown in Table 5.

[0161] Table 5 Violent Audio Detection Model

[0162]

[0163] The final test performance is as follows:

[0164] Table 6 Performance of Violent Audio Detection

[0165]

[0166] Step 2: Calculate the brute-force discrimination threshold

[0167] Define the occurrence of violence as x = 1 and the occurrence of non-violence as x = 0. Let C be the loss for identifying non-violent video as violent video. 01 The loss from identifying violent videos as non-violent videos is denoted as C. 10The loss for correctly identifying a violent video as a violent video is denoted as C. 11 The loss for correctly identifying a nonviolent video as a nonviolent video is denoted as C. 00 Then the expected value of the loss C is:

[0168]

[0169] Where P(x=0) is the probability that the actual situation is non-violent, P(y=1|x=0) is the probability that the model predicts the value as violent when the actual situation is non-violent, P(y=0|x=0) is the probability that the model predicts the value as non-violent when the actual situation is non-violent, P(x=1) is the probability that the actual situation is violent, P(y=1|x=1) is the probability that the model predicts the value as violent when the actual situation is violent, P(y=0|x=1) is the probability that the model predicts the value as non-violent when the actual situation is violent, P(y|x=0) is the probability that the model output is y when the actual situation is non-violent, and P(y|x=1) is the probability that the model output is y when the actual situation is violent.

[0170] Calculate the brute-force discrimination threshold y based on the expected loss value. T (That is, calculated based on the stationary point of the expected loss):

[0171]

[0172] In this invention, C is set 10 -c 00 =C 01 -C 11 =1, P(x=0)=0.5, at this time the brute-force discrimination threshold is 0.5.

[0173] The probability of the current frame of the surveillance video being identified as violent is compared with a violence identification threshold. Based on the comparison result, the violence identification result of the current frame of the surveillance video is obtained, that is:

[0174] If the probability of the current frame of the surveillance video being identified as violent is greater than or equal to the violence identification threshold, then the violence identification result is that the current frame of the surveillance video contains violence.

[0175] If the probability of the current frame of the surveillance video being identified as violent is less than the violence identification threshold, then the violence identification result is that the current frame of the surveillance video does not contain violence.

[0176] Step 3: Since violent behavior is continuous, if violent behavior occurs, the model should continuously detect violent behavior. Similarly, if violent behavior does not occur, the model should continuously detect non-violent behavior. Therefore, referring to historical model recognition results can, to some extent, avoid model misidentification. This invention uses automata or hidden Markov chains for temporal filtering.

[0177] 1. Automatic machine

[0178] Automata can recognize some non-contextual language and can deterministically determine the reliability of the current recognition based on a limited set of historical results. Taking a history length of 3 as an example, the automaton's states are binary numbers 000 to 111. In state abc, if the model's recognition result is d, the automaton's state will transition to bcd, and its state transition function is:

[0179] δ abc (d)=bcd

[0180] When the status is 110, 101, 011, the model continuously detects violent events, but intermittently, so it is a level 2 warning.

[0181] When the status is 111, the model continuously detects violence, which is a level 3 warning.

[0182] When the status is 001, a violent event is detected, but the possibility of false identification cannot be ruled out, therefore it is a Level 1 warning;

[0183] If the status is 000, 010, or 100 and is continuously identified as non-violent, it means that the actual situation is non-violent. If violence is identified, it is likely a false alarm and therefore no warning is issued.

[0184] If an alert is issued, a forceful response is required; if no alert is issued, a forceful response is not required.

[0185] 2. Hidden Markov Chain

[0186] When considering practical applications Figure 9 Given a Markov chain of events, where the hidden state z follows a Bernoulli distribution indicating whether a fight occurs, and the observable state y represents the model output value, then the observation probability matrix at time t is p(y...). t |z t =i), representing the probability that the model output is y when the actual value is i. This probability can be estimated using a Gaussian mixture distribution, or by assuming y t To achieve a uniform distribution, the following approximation can be made:

[0187]

[0188] Where p(y t ) is at time t, i.e., when the current frame of the monitored video is predicted as y. t The probability, p(z) t =i|y t ) is at time t, i.e., when the current frame of the monitored video is predicted as y. t The probability that the actual value is i, p(y) t |z t=i) is the prediction of y when the actual situation of the current frame of the monitored video is i at time t. t The probability of;

[0189] At this point, the observation probability matrix is ​​the probability output by the model. To obtain the posterior probability of the hidden state, a recursive function α is defined. t (i):

[0190] α t (i)=p(y 1:t , z t =i)=∑ j p(y 1：t-1 ,z t-1 =j)p(z t =i|z t-1 =j)p(y t |z t =i)

[0191] =p(y t |z t =i)Σ j α t-1 (j)p(z t =i|z t-1 =j)

[0192] Where p(y 1:t-1 ,z t-1 =j) is the actual value at time t-1 is j, and the predicted value from time 1 to time t-1 is y. 1:t-1 Probability, p(z) t =i|z t-1 =j) is the probability that the actual situation at time t-1 is j and the actual situation at time t is i;

[0193] This function can be solved using dynamic programming. Using the maximum posterior of the hidden state at time t as the model's discrimination result, the posterior probability is:

[0194]

[0195] Where p(y1: t , z t =i) is the actual value at time t is i, and the predicted value from time 1 to time t is y1: t Probability; p(z) t =i|y1: t The prediction result of the surveillance video from time 1 to time t is y1: t When the actual situation is i, the probability is i.

[0196] Compare p(y) t =1|z t =i) and the magnitude of the brute-force discrimination threshold:

[0197] If p(y t =1|z t =i) If the threshold for brute-force discrimination is greater than or equal to the threshold, then a brute-force response is required; otherwise, no brute-force response is required.

[0198] Both the initial brute-force prior probability and the state transition probability can be user-defined. If undefined, a brute-force prior of 0.5 and a transition probability matrix maintaining the current state of 0.7 are used. Examples are shown in Table 7.

[0199] Table 7. Recognition results after HMM processing

[0200] Probability after processing 0.65 0.40 0.32 0.50 0.45 0.34 0.61 0.89 0.36 0.41

[0201] As can be seen, the method of the present invention effectively eliminates the possibility of false identification.

[0202] Then, based on the time-series filtering results, it is determined whether a violent response is needed. If a violent response is needed, the violent person information in the current frame of the surveillance video is extracted, and the camera is moved to the next position. If a violent response is not needed, the violent person information in the current frame of the surveillance video does not need to be extracted, and the camera position is moved directly.

[0203] The target location that the camera needs to move to is calculated based on Bayesian decision-making or reinforcement learning schemes.

[0204] 1. Bayesian decision making

[0205] Define the distribution of the brute force value y captured by the camera at the new position under a reasonable movement 'a', and then calculate the posterior of 'a' with respect to 'y'.

[0206] First, we define the prior distribution of 'a'. Since the camera has a primary responsible angle μ0, for example, a street camera should primarily capture the street surface when there is no obvious violent event, we define the prior distribution of the target position x of the camera movement as follows:

[0207] p(x|μ0,Σ0)=N(x|μ0,∑0)

[0208] Where ∑0 is the covariance matrix of the prior distribution.

[0209] The camera's movement distance in a single operation should be finite; therefore, given the current camera position μ... x At that time, the expected position x of the camera target at the next moment should remain unchanged, therefore its distribution in this sense is defined as:

[0210] p(x|μ x , ∑ x )=N(x|μ0,∑0)

[0211] Given a reasonable camera movement scheme, the captured brute-force value y should be as large as possible, and its distribution is defined as follows:

[0212]

[0213] in, α is the normalization coefficient, and α is the hyperparameter.

[0214] Calculate w using either method (1) or method (2) as follows:

[0215] Method (1)

[0216] Using linear ridge regression to fit the relationship between x and y, although x and y are not linearly related, the camera's movement distance is limited in the short term, and x and y exhibit a linear relationship locally.

[0217]

[0218] Where X and Y are the camera positions and captured brute force values ​​at historical moments, respectively, and α is the regularization coefficient, representing the confidence level of w prior.

[0219] Method (2)

[0220] When using linear ridge regression, all information outside the memory is lost, which may be wasteful. Therefore, Kalman filtering can also be used to update w. Kalman filtering uses a global loss function to solve for w at time t:

[0221]

[0222] The final iterative expression is:

[0223] H t =δH t-1 +X t′ X t

[0224] w t =w t-1 +(H t ) -1 X t′ Y t

[0225] This method can preserve historical information relatively well.

[0226] Assuming all knowledge is unknown, the camera position is completely random, and the position cannot be accurately determined based on brute force, then, given the main position, the current position, and the brute force value for each position, and assuming these three are independent, the posterior probability of the camera's next position is:

[0227] p(x|μ0,∑0,μ x , ∑x ,y)∝p(μ0,∑0,μ x , ∑ x ,y|x)=p(μ0,Σ0|x)p(μ x , Σ x |x)p(y|x)∝p(μ0,∑0,x)p(μ x , ∑ x ,x)p(y,x)

[0228]

[0229]

[0230]

[0231] In this case, the distribution is Gaussian when the summation is positive, so sampling is sufficient.

[0232] Based on the posterior probability p(x|μ0,∑0,μ x , ∑ x The target position of the camera movement can be obtained by using the Bayesian decision method. When the brute force occurs in the upper left corner, the camera movement trajectory obtained is as follows: Figure 10 As shown, the camera will quickly probe towards the upper left corner and converge to a certain point.

[0233] 2. Strengthen learning

[0234] In the policy gradient scheme, given the current input position x, the policy p(a) for moving a maximizes the expected value of the brute force y:

[0235]

[0236] When solving using the gradient descent method, its derivative with respect to q(a|x) is:

[0237]

[0238]

[0239] Therefore, given a dataset A, Y, X following a distribution q(a|x), iterating using the policy gradient method is equivalent to finding the stationary point of the following objective function:

[0240] L(q)=E x～X，y～Y，a～A [log(q(a|x))(k+y)]

[0241] Where k is a constant;

[0242] Let q(a|x) be a Gaussian distribution whose expectation is a linear combination of positions x expanded in high dimension:

[0243]

[0244] Where w′ is the transpose of the model parameters, ∑ is the covariance matrix, and Φ(x) is any nonlinear mapping at position x;

[0245] Therefore, the original problem becomes a weighted least squares problem:

[0246] E x～X，y～Y，a～A [log(q(a|x))(k+y)]|∝E x～X，y～Y，a～A [(aw′Φ(x))′(α-w′Φ(x))(k+y)]

[0247] Introducing the Gaussian prior of w, its maximum a posteriori estimate is obtained as follows:

[0248] w′Φ(x)=Φ(x)′X′(n w Λ -1 +XX′) -1 A

[0249]

[0250] Where, n w Let w be the prior probability degrees of freedom. The value of k does not affect the expected convergence of q(a|x), but it can be set to the mean of y to reduce the convergence variance. Use the kernel trick to avoid explicitly constructing Φ:

[0251] K(||x1-x2|| 2 )=Φ(x1)′Φ(x2)

[0252] Meanwhile, to prevent cameras from leaving the main area for extended periods, the system is set to move the cameras back to the main area if the violence level is consistently low within a certain timeframe.

[0253] The camera movement trajectory was obtained by testing in the same environment as the Bayesian decision method, as shown below. Figure 11 As shown, the camera also achieved focusing. Therefore, reinforcement learning methods can also capture brute-force areas (top left corner in the image).

[0254] After the camera moves to the target location, it continues to capture the next frame of the monitoring video, and then returns to step one for the captured next frame of the monitoring video;

[0255] The process for extracting information on violent individuals is as follows: Figure 12As shown, it mainly includes two parts: collecting the location of people in the surveillance video based on YOLOv3 and extracting features from the collected people. The people feature extraction adopts the pedestrian re-identification (CNN-REID) model trained on the PA-100k dataset to identify many features of people. Features such as age, gender, and clothing are extracted from the people detected in the first part.

[0256] 1. Human Image Extraction

[0257] YOLO (You Only Look Once) is a popular object detection and image segmentation model developed by Joseph Redmon and Ali Farhadi of the University of Washington. Released in 2015, YOLO quickly gained popularity due to its high speed and accuracy. The YOLOv3 used in this invention is the third version in the YOLO (You Only Look Once) algorithm family.

[0258] YOLO's basic principle is to view object detection as a regression problem involving spatially separated bounding boxes and associated class probabilities. In each evaluation, the neural network can directly predict bounding boxes and classification probabilities from the complete image, such as determining that the bounded portion belongs to a person. Because YOLO's detection pipeline is a single network, end-to-end optimization of detection performance is possible. YOLO's biggest advantage over other detection methods, such as DPM and R-CNN, is its high detection efficiency. Figure 13 As shown, the accuracy of extracting specific individuals from surveillance images is also higher. The initial version of YOLO's basic model could process images in real time at a high speed of 45 frames per second, while the v3 version used in this invention is an update from the initial version of YOLO. The structure of the v3 version is as follows... Figure 14 As shown, when the image size is 320×320, YOLOv3 runs at 28.2mAP in 22 milliseconds, which is as accurate as SSD, but three times faster.

[0259] YOLO's core idea is to divide an image into W×W grids, with each grid responsible for objects whose center point falls within that grid. The network will then process each bounding box (t... x ,t y ,t w ,t h Predict 4 coordinates (Wang Xinpeng. Research on Garbage Detection Algorithm Based on YOLOv3 [D]. Chongqing: Chongqing University of Posts and Telecommunications, 2022.). Record the offset of the cell from the top left corner of the image as (c x ,c y The bounding box has a width and height of p. w p hThe prediction corresponds to:

[0260] b x =σ(t) x )+x

[0261] b y =σ(t) y )+y

[0262]

[0263]

[0264] Figure 15 The specific meaning of each quantity is intuitively explained. The network uses the sigmoid method to constrain the predicted value t. x ,t y Within the mathematical scope, and effectively accelerated network convergence, by adding (c x ,c y The offset yields the center coordinates, for t w ,t h The size data of the prediction block is processed to obtain the final prediction result.

[0265] To minimize the loss of information in the graphs during processing, YOLOv3 uses feature maps of three different scales. In designing the model, the deeper residual network Darknet-53 was chosen as the backbone feature extraction network. Darknet-53 abandons common pooling and fully connected operations, and its convolutional layers use 1×1 and 3×3 kernels of different sizes to extract features. After convolution calculation, normalization and LeakyReLU are added (Redmon, Joseph and Ali Farhadi. “YOLOv3: An Incremental Improvement.” ArXiv abs / 1804.02767(2018):n.pag.).

[0266] The YOLOv3 network structure can be roughly divided into three parts: Backbone, FPN, and YOLO Head. The Backbone is primarily composed of Darknet-53. The FPN is a feature pyramid model that first performs dimensionality reduction and feature extraction from the image through convolution from bottom to top, and then fuses different semantic information from multiple levels of feature maps from top to bottom through upsampling and feature fusion. This way, it possesses both high-level semantic information and low-level contour information. Finally, the YOLO Head is a decoder that outputs three-dimensional values ​​w and h, which are denoted as b, representing the width and height of the predicted bounding box. w ,b hEach point corresponds to a grid in the original image, and the channel corresponds to the predicted value of that anchor.

[0267] 2. Identity Feature Extraction

[0268] A person re-identification model was used to extract the identity of violent individuals. The model was trained on the PA-100K dataset using a small model consisting of multiple convolutional layers, BatchNorm layers, LeakyReLU activation function, and adaptive max-pooling layers. The resulting model can read image data and output feature extraction results including, but not limited to, gender, age, clothing, and hairstyle, which is helpful for locating violent individuals. The final model achieved a test accuracy of 86%.

[0269] The CNN-REID network model first performs a convolution, then normalizes the result, and applies the LeakyReLU activation function. It then uses five consecutive ResBlock layers, processes the result using an adaptive max-pooling layer, transforms the result into a one-dimensional array using Flatten, and applies a linear transformation. Finally, it uses a Sigmoid transformation to calculate the feature extraction results, including but not limited to gender, age, clothing, and hairstyle. We trained this model on the PA-100K dataset, and the test accuracy reached over 86%.

[0270] Overall, the violent behavior detection method of the present invention has the following characteristics:

[0271] 1) Violent behavior detection based on dual audio and video signals

[0272] This invention uses a weighted geometric average of video and audio recognition results as the final recognition result. For video recognition, it abandons the traditional method of extracting features separately for space and time, and the use of large-scale models. Instead, it employs a 3D CNN network with smaller parameters that simultaneously extracts spatiotemporal features. Logistic regression is used for probability calibration to obtain a probabilistic interpretation of the output. For audio, Mel features are used to extract features from the input audio, which are then classified using a small 2D CNN model.

[0273] The method of this invention achieves a brute-force detection framework with a latency of less than 1 second on the CPU, while ensuring a video recognition accuracy of over 90%.

[0274] 2) Intelligent tracking and optimization of audio and video signals

[0275] In traditional violence detection methods, older cameras are mostly stationary, so the movement of the viewing angle is not considered during detection. With the advancement of technology, cameras have been largely replaced by movable ones. A reasonable camera movement strategy can significantly increase the detection range of a single camera. Therefore, this invention proposes two sets of autonomous camera movement strategies to control the movement of the cameras, effectively expanding the detection range of a single camera and improving the traceability of violent behavior.

[0276] 3) Misidentification elimination based on prior knowledge

[0277] Even machine learning-based data-driven violence detection models, which integrate video and audio information, can still misidentify violence. However, considering that violent incidents are often not isolated events but rather continuous behaviors over a relatively long period, historical identification results can be used to determine the reliability of the latest identification, thereby eliminating false identifications.

[0278] This invention proposes two methods for eliminating false positives. The automaton method requires constructing an automaton based on historically detected violent behaviors, building a state transition function based on rules, and assigning a violence index of 1 to 3 to each state. The higher the violence index, the higher the probability of a violent event occurring. The hidden Markov chain method fully utilizes the hidden state z of the hidden Markov chain. By modeling the real situation as the hidden state z and the identification result as the observed state, the posterior probability of the current hidden state is calculated to determine the probability of violence occurring under the current observation result, thereby correcting false positives and obtaining better judgment results.

[0279] 4) Intelligent control based on reinforcement learning

[0280] This invention utilizes reinforcement learning technology to achieve monitoring and steering control, thereby enabling the tracking and observation of violent behavior. The goal of reinforcement learning is to allow an agent to gradually learn the optimal behavioral strategy through continuous trial and error, enabling it to make optimal decisions when facing similar environments in the future. This invention employs a policy-based reinforcement learning method, using a monitoring device with a rotating viewing angle as the agent. Initially, the device is allowed to rotate randomly. The detected violence value is then used as the reward function, and the policy gradient method is used to obtain the optimal strategy that maximizes the expected future reward, thus achieving camera movement control.

[0281] After a period of training, the surveillance system will "learn" to chase scenes containing violence to maximize its built-in expectations. Visually, this means the surveillance will always be focused on scenes of violence, and it will be able to capture the scene well even if the people in the violent conflict move.

[0282] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for detecting and tracking violent behavior based on audio and video, characterized in that, The method specifically includes the following steps: Step 1: Perform brute-force identification on the current frame of the surveillance video and the corresponding audio separately, and then perform weighted fusion on the obtained brute-force identification results. Use the weighted fusion result as the probability that the current frame of the surveillance video is identified as brute-force. Step 2: Compare the probability of the current frame of the surveillance video being identified as violent with the violence identification threshold, and obtain the violence identification result of the current frame of the surveillance video based on the comparison result; The specific process of step two is as follows: Step Two: Define the occurrence of violence as... Non-violent occurrence The loss caused by non-violent videos being identified as violent videos is denoted as The loss of identifying violent videos as non-violent videos is denoted as The loss caused by correctly identifying a violent video as a violent video is recorded as The loss caused by correctly identifying nonviolent videos as nonviolent videos is denoted as Then the loss Expected value for: in, It represents the probability that the actual situation is non-violent. It represents the probability that the model predicts a violent value when the actual situation is non-violent. It represents the probability that the model predicts a non-violent outcome when the actual situation is non-violent. It represents the probability that the actual situation is violent. It represents the probability that the model predicts a violent value when the actual situation is violent. It represents the probability that the model predicts a non-violent outcome when the actual situation is violent. When the actual situation is non-violent, the model's predicted output is: The probability, When the actual situation is brute force, the model's predicted output is: The probability, ; Step 22: Calculate the brute-force discrimination threshold based on the expected loss value. : Steps 2 and 3: Compare the probability of the current frame of the surveillance video being identified as violent with the violence identification threshold; If the probability of the current frame of the surveillance video being identified as violent is greater than or equal to the violence identification threshold, then the violence identification result is that the current frame of the surveillance video contains violence. If the probability of the current frame of the surveillance video being identified as violent is less than the violence identification threshold, then the violence identification result is that the current frame of the surveillance video does not contain violence. Step 3: Perform temporal filtering based on the brute-force identification results of historical surveillance videos and the brute-force identification results of the current frame of surveillance video, and determine whether a brute-force response is required based on the temporal filtering results; If a violent response is required, extract the information of the violent individuals in the current frame of the surveillance video before proceeding to step four. If a brute-force response is not required, proceed directly to step four; Step 4: Move the camera to the next position to continue capturing the next frame of surveillance video, and then return to Step 1 to execute the captured next frame of surveillance video.

2. The method for detecting and tracking violent behavior based on audio and video as described in claim 1, characterized in that, The brute-force identification of the current frame of monitored video and the corresponding audio is as follows: The 3D-CNN network is used to identify violence in the current frame of the surveillance video. The 3D-CNN network is used to extract features from the current frame of the surveillance video. The feature extraction results are then passed through a support vector machine or a fully connected neural network. The output of the support vector machine or the fully connected neural network is probability-calibrated, and the probability calibration result is used as the initial probability that the current frame of the surveillance video is identified as violence. The brute-force identification of the audio corresponding to the current frame of the surveillance video uses Mel features and a 2D-CNN network. First, Mel features are extracted from the audio corresponding to the current frame of the surveillance video. Then, the extracted Mel features are processed through a 2D-CNN network. The output of the 2D-CNN network is used as the probability that the audio corresponding to the current frame of the monitored video is identified as violent.

3. The method for detecting and tracking violent behavior based on audio and video according to claim 2, characterized in that, The specific steps for performing probability calibration on the output of the support vector machine or fully connected neural network are as follows: in, This indicates that the current frame of the surveillance video has been identified as violent. This is the initial probability that the current frame of the surveillance video is identified as violent, obtained by processing the current frame of the surveillance video. It is the base of the natural logarithm. It is the output of a support vector machine or a fully connected neural network.

4. The method for detecting and tracking violent behavior based on audio and video according to claim 3, characterized in that, In step three, temporal filtering is performed based on the brute-force identification results of historical surveillance videos and the brute-force identification results of the current frame of surveillance video. The result of the temporal filtering is then used to determine whether a brute-force response is necessary. The specific process is as follows: The number of frames in the historical surveillance video is set to 3. That is, the 3 frames of surveillance video before the current frame are selected to perform time-series filtering on the violence identification result of the current frame. The automaton state composed of the violence identification results of the selected 3 historical surveillance videos is denoted as abc, where the values ​​of a, b, and c are all 0 or 1. 1 represents that the corresponding surveillance video is identified as having violence, and 0 represents that the corresponding surveillance video is identified as not having violence. If the result of brute-force identification of the current frame of the monitored video is denoted as d, then the automaton state will be converted to bcd; When the state bcd is 000, 010, or 100, a brute-force response is not required. If the state bcd is not 000, 010, or 100, a brute-force response is required.

5. The method for detecting and tracking violent behavior based on audio and video according to claim 3, characterized in that, In step three, temporal filtering is performed based on the violent identification results of historical surveillance videos and the violent identification results of the current frame surveillance video, and it is determined whether a violent response is needed based on the temporal filtering results. The specific process is as follows: Step 31: Calculation : in, The surveillance video at time t is predicted as The probability, =0 or 1, =1 indicates that the surveillance video at time t is predicted to be violent. =0 indicates that the surveillance video at time t is predicted to be non-violent; When the actual situation of the surveillance video at time t is i, the prediction result is: The probability of; The prediction result of the surveillance video at time t is When, the probability that the actual situation is i; It is the actual situation of the surveillance video at time t; Define the recursive function at time t. : in, Let j be the actual situation of the surveillance video at time t-1, and let j be the predicted result of the surveillance video from time 1 to time t-1. The probability of; It is the probability that the actual situation of the surveillance video at time t-1 is j and the actual situation of the surveillance video at time t is i. It is a recursive function at time t-1; in, Let i be the actual situation of the surveillance video at time t, and let i be the predicted result of the surveillance video from time 1 to time t. probability, The prediction result for the surveillance video from time 1 to time t is: When, the probability that the actual situation is i; Step 3.2 Comparison The size of the threshold for distinguishing violence: like If the threshold is greater than or equal to the brute-force discrimination threshold, a brute-force response is required; otherwise, no brute-force response is required.

6. A method for detecting and tracking violent behavior based on audio and video according to claim 4 or 5, characterized in that, The moving camera continues to capture the next frame of monitoring video at the next location. The method for calculating the target location is as follows: Calculate intermediate variables Then, based on intermediate variables Calculate the target position of the camera movement posterior probability : in, It's the angle of the camera shot. It refers to the position of the camera when capturing the current frame of the surveillance video. This means that the left side of the formula is proportional to the right side of the formula. This indicates that when the target position of the camera is x, the angle at which the camera captures the image is... The prior distribution covariance matrix of the camera location is: The camera's current location is The camera movement probability covariance matrix is The probability that a surveillance video taken at the target location is identified as y, where y = 0 or 1. This indicates that when the target position of the camera is x, the angle at which the camera captures the image is... The prior distribution covariance matrix of the camera location is: The probability, This indicates that when the target position of the camera is x, the current position of the camera is... The camera movement probability covariance matrix is The probability, This represents the probability that the surveillance video captured at the target location x will be identified as y. This indicates the angle at which the camera is shooting. The prior distribution covariance matrix of the camera location is: The probability that the target position is x. Indicates the current location of the camera. The camera movement probability covariance matrix is The probability that the target position is x. This represents the joint probability that the target location x of the camera and the surveillance video captured at that target location are identified as y. This indicates the angle at which the camera is shooting. The prior distribution covariance matrix of the camera location is: At that time, what is the probability that the next target position of the camera is x? Indicates the current location of the camera. The camera movement probability covariance matrix is When the next target position is x, what is the probability? This represents the probability that the surveillance video captured by the camera is identified as y. The prior distribution covariance matrix representing the camera location. This represents the covariance matrix of the camera movement probability. The hyperparameter representing the probability distribution of brute force values. yes transpose, yes transpose, yes The transpose of the matrix, where the superscript -1 represents the inverse of the matrix. Represents a normal distribution; Then based on the posterior probability Obtain the target position of the camera movement .

7. A method for detecting and tracking violent behavior based on audio and video according to claim 4 or 5, characterized in that, The moving camera continues to capture the next frame of monitoring video at the next location. The method for calculating the target location is as follows: Let the position of the camera when capturing the current frame of surveillance video be x, and determine the strategy for moving 'a'. To maximize the expected value of y: in, Let x be the probability that the camera moves to position a when the current position is x, under any given strategy. This represents the expectation that the surveillance video captured by the camera after moving 'a' from its current position x to the new target position will be identified as violent. Under the target strategy, the probability that the camera moves to position a when the current position is x; Utilize the highest probability The corresponding movement 'a' and the current position 'x' determine the target position of the camera movement.

8. The method for detecting and tracking violent behavior based on audio and video according to claim 6, characterized in that, The intermediate variable The calculation method is as follows: Where X is the position of the camera when the current frame of the surveillance video is captured, and Y is the probability that the current frame of the surveillance video is identified as violent. It is the L2 norm. It is the transpose of X. Represents the coefficient of the regularization term. It is an identity matrix, and the superscript -1 represents the inverse of the matrix.

9. The method for detecting and tracking violent behavior based on audio and video according to claim 1, characterized in that, The specific process for extracting information about violent individuals from the current frame of the surveillance video is as follows: Step 1: Use the YOLOv3 model to identify the location of the person in the current frame of the surveillance video; Step 2: Use a CNN-REID network to extract information about violent individuals from the image of the area where the person's image is located; The CNN-REID network includes, in sequence, a convolutional layer, a batch normalization layer, a ReLU activation function layer, a first ResBlock module, a second ResBlock module, a third ResBlock module, a fourth ResBlock module, a fifth ResBlock module, an adaptive max pooling layer, a Flatten layer, a Linear layer, and a Sigmoid activation function layer. The working process of the CNN-REID network is as follows: The image of the region where the person is located is used as the input to the CNN-REID network. Within the CNN-REID network, the input image is used as the input to the convolutional layer. Use the output of the convolutional layer as the input of the BN layer; Use the output of the BN layer as the input of the ReLU activation function layer; The output of the ReLU activation function layer is used as the input of the first ResBlock module; Use the output of the first ResBlock module as the input of the second ResBlock module; Use the output of the second ResBlock module as the input of the third ResBlock module; Use the output of the third ResBlock module as the input of the fourth ResBlock module; Use the output of the fourth ResBlock module as the input of the fifth ResBlock module; The output of the fifth ResBlock module is used as the input to the adaptive max pooling layer; Use the output of the adaptive max pooling layer as the input of the Flatten layer; Use the output of the Flatten layer as the input of the Linear layer; Use the output of the Linear layer as the input of the Sigmoid activation function layer; The output of the Sigmoid activation function layer is used as the extracted information about the violent perpetrators; The first ResBlock module includes a first branch, a second branch, and a norm unit. The first branch includes a first res unit, a second res unit, and a convolutional layer, and the second branch includes a convolutional layer. Within the first ResBlock module, the input of the first ResBlock module is first used as the input of the two branches, and then the outputs of the two branches are connected. The connection result is passed through the norm unit, and the output of the norm unit is used as the output of the first ResBlock module. The first res unit contains, in sequence, a convolutional layer and a ReLU activation function layer; The norm unit contains a ReLU activation function layer and a BN layer in sequence.

Images