Violent behavior detection and tracking method based on audio and video

Abstract

The invention discloses a violent behavior detection and tracking method based on audios and videos, and belongs to the technical field of violent behavior detection and tracking. The method solves the problems that an existing violent event detection method is low in detection precision, low in response speed and high in degree of dependence on hardware and needs manual intervention. The violence recognition results of the video data and the audio data are integrated, the current violence recognition result is subjected to error recognition elimination based on the historical violence recognition result, the violence recognition precision can be improved, the camera is controlled to move in the direction where violence possibly occurs based on the historical information, and the detection range of a single camera is widened. A YOLO model is adopted to identify the position of a violent person, and then identity feature extraction is carried out on the violent person. The violent behavior detection and tracking method has the advantages of being high in detection precision, high in response speed, low in dependence on hardware and free of manual intervention in the whole process. The method can be applied to violent behavior detection and tracking.

Description

Technical Field

[0001] The present invention belongs to the technical field of violence detection and tracking, and in particular relates to a method for detecting and tracking violent behavior based on audio and video. Background Art

[0002] With the rapid development of science and technology, the advancement of camera technology and artificial intelligence (AI) has provided new solutions for social order and public safety. Traditional security monitoring systems often have problems such as high latency, low accuracy, and reliance on manual intervention when violent incidents occur.

[0003] Traditional monitoring systems can only provide video data after the fact, and lack the ability to warn and intervene in real time, which makes the prevention and control of violent incidents face huge challenges. Although various places have made many efforts to strengthen public security management, it is often difficult to detect and stop these sudden violent incidents in time by relying solely on human patrols and traditional monitoring equipment. The camera-based body capture violence detection and alarm system came into being in this context, aiming to improve the real-time detection, tracking and response capabilities of violent incidents through efficient technical means, and has the ability to extract the identities of violent personnel.

[0004] In summary, the existing violent incident detection methods still have problems such as low detection accuracy, slow response speed, high dependence on hardware and the need for human intervention. Therefore, how to use advanced technical means to improve the accuracy of violent incident detection, the response speed to violent incidents, reduce dependence on hardware and get rid of human intervention has become an important issue in current social security management. Summary of the invention

[0005] The purpose of the present invention is to solve the problems of low detection accuracy, slow response speed, high dependence on hardware and need for manual intervention in existing violent incident detection methods, and to propose a violent behavior detection and tracking method based on audio and video.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a method for detecting and tracking violent behavior based on audio and video, the method specifically comprising the following steps:

[0007] Step 1: After respectively performing violence recognition on the current frame surveillance video and the audio corresponding to the current frame surveillance video, weighted fusion is performed on the obtained violence recognition results, and the weighted fusion result is used as the probability that the current frame surveillance video is recognized as violence;

[0008] Step 2: comparing the probability that the current frame of surveillance video is identified as violence with the violence identification threshold, and obtaining a violence identification result for the current frame of surveillance video according to the comparison result;

[0009] Step 3: Perform time series filtering based on the violence recognition results of the historical surveillance video and the violence recognition results of the current frame surveillance video, and determine whether a violent response is required based on the time series filtering results;

[0010] If a violent response is required, extract the violent person information in the current frame surveillance video and then execute step 4;

[0011] If no violent response is required, proceed directly to step 4;

[0012] Step 4: Move the camera to the next position to continue collecting the next frame of surveillance video, and then return to step 1 for the collected next frame of surveillance video.

[0013] The beneficial effects of the present invention are:

[0014] The present invention uses a 3D-CNN model to extract features from video data, uses SVM to detect violence from video data, uses Mel features and a 2D-CNN model to detect violence from audio data, and by integrating the violence recognition results of video data and audio data, and then eliminating misidentification of the current violence recognition results based on historical violence recognition results, the accuracy of violence recognition can be improved, and the camera is controlled to move in the direction where violence may occur based on historical information, thereby increasing the detection range of a single camera and ensuring response and tracking of violence. The violent personnel information extraction part uses a YOLO model to identify the location of violent personnel and extract identity features of violent personnel. The violent behavior detection and tracking method of the present invention can detect violence in real time for rapid response, and has the characteristics of high detection accuracy, fast response speed, low dependence on hardware, and no need for manual intervention throughout the process. Therefore, it can enhance the real-time detection, tracking and response capabilities of violent incidents to improve public safety.

[0015] Experiments show that the accuracy of the violent behavior detection method of the present invention on the ubi dataset and the hockey dataset can reach about 97%. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a method for detecting and tracking violent behavior based on audio and video of the present invention;

[0017] Figure 2 This is the flowchart of the violence identification part;

[0018] Figure 3 It is a flow chart of surveillance video detection;

[0019] Figure 4 This is a schematic diagram of the structure of 3D-ResNet;

[0020] Figure 5This is a schematic diagram of the structure of 3D-InceptionNet;

[0021] Figure 6 It is the prediction probability map when the prediction is wrong;

[0022] In the figure, the horizontal axis is the probability of violence given by the model, and the vertical axis is the number of samples;

[0023] Figure 7 is the predicted probability plot for the entire sample;

[0024] Figure 8 It is a flowchart for violence detection on audio;

[0025] Fig. 9 is a schematic diagram of a Markov decision chain;

[0026] Fig.10 It is a schematic diagram of the camera movement trajectory obtained by the Bayesian decision method;

[0027] In the figure, the horizontal axis represents the position of the camera in the x-axis direction, and the vertical axis represents the position of the camera in the y-axis direction;

[0028] Fig.11 It is a schematic diagram of the camera movement trajectory obtained by the reinforcement learning method;

[0029] Fig.12 It is a flowchart of violent personnel information extraction;

[0030] Fig.13 This is a speed comparison chart of YOLOv3 and other methods;

[0031] In the figure, the horizontal axis is the model inference time, and the vertical axis is the mAP score on the COCO dataset;

[0032] Fig.14 This is the network structure diagram of YOLOv3;

[0033] Fig.15 is a diagram of a bounding box with dimension priors and position predictions;

[0034] Fig.16 This is the structural diagram of the CNN-REID network. DETAILED DESCRIPTION

[0035] Specific implementation method 1: Combination Figure 1 The present embodiment describes a method for detecting and tracking violent behavior based on audio and video, and the method specifically comprises the following steps:

[0036] Step 1: After respectively performing violence identification on the current frame surveillance video and the audio corresponding to the current frame surveillance video, weighted fusion is performed on the obtained violence identification results (i.e., weighted fusion is performed on the probability of the video being identified as violence and the probability of the audio being identified as violence, and the obtained weighted fusion result is the probability of violence occurring at present), and the weighted fusion result is used as the probability of the current frame surveillance video being identified as violence;

[0037] Step 2: comparing the probability that the current frame of surveillance video is identified as violence with the violence identification threshold, and obtaining a violence identification result for the current frame of surveillance video according to the comparison result;

[0038] Step 3: Perform time series filtering based on the violence recognition results of the historical surveillance video and the violence recognition results of the current frame surveillance video, and determine whether a violent response is required based on the time series filtering results;

[0039] If a violent response is required, extract the violent person information in the current frame surveillance video and then execute step 4;

[0040] If no violent response is required, proceed directly to step 4;

[0041] Step 4: Move the camera to the next position to continue collecting the next frame of surveillance video, and then return to step 1 for the collected next frame of surveillance video.

[0042] New surveillance has integrated the functions of cameras and radios, which can not only capture real-time images, but also the corresponding sounds. Traditional methods often have difficulty distinguishing normal interactive behaviors from real violent behaviors, resulting in many misjudgments. Now, with the addition of sound data, the detection results are more accurate, greatly reducing the probability of misidentification. Moreover, during the detection process, time series filtering eliminates misidentification based on historical recognition results, and uses reinforcement learning and online learning to achieve self-movement control of the camera to increase the detection range.

[0043] Specific implementation method 2: This implementation method is different from the specific implementation method 1 in that the current frame monitoring video and the audio corresponding to the current frame monitoring video are respectively subjected to violence recognition, specifically:

[0044] The 3D-CNN network is used to identify violence in the current frame of surveillance video. The 3D-CNN network is used to extract features from the current frame of surveillance video. The feature extraction results are then passed through a support vector machine or a fully connected neural network. The output of the support vector machine or the fully connected neural network is probability calibrated and the probability calibration result is used as the initial probability of the current frame of surveillance video being identified as violence.

[0045] Mel features and 2D-CNN network are used to identify violence in the audio corresponding to the current frame of surveillance video. First, Mel features are extracted from the audio corresponding to the current frame of surveillance video, and then the extracted Mel features are passed through the 2D-CNN network. The output of the 2D-CNN network is used as the probability that the audio corresponding to the current frame of surveillance video is identified as violence.

[0046] The other steps and parameters are the same as those in the first embodiment.

[0047] The 3D-CNN network of the present invention is pre-trained on the Kinetics-700 gesture recognition data set and has a strong feature extraction capability. A support vector machine is used to classify based on the features to obtain a violence detection result. The support vector machine takes maximizing the minimum projection distance of different categories of data to the classification hyperplane as the objective function, so it has high robustness and can still effectively classify when the size of the violence detection data set is limited.

[0048] This implementation uses a 3D-CNN network to extract features from video data in both time and space dimensions and classify them. Compared with the traditional posture recognition method based on optical flow estimation, it has the advantages of high accuracy and low computational complexity; compared with the method based on the CNN-RNN architecture that extracts spatial features first and then temporal features, it has advantages in accuracy and real-time performance; compared with the method based on Transformers variants, it has great advantages in computational complexity and real-time performance.

[0049] In the process of detecting violence on audio data, the audio data is first preprocessed. For audio data that is too long, only the first half is extracted. This can reduce the amount of calculation without losing too much important information. For audio data that is too short, the zero-padding method is used to make it reach a uniform length. This preprocessing method not only simplifies the complexity of data processing, but also ensures the consistency of input data.

[0050] The present invention uses Mel features to convert the preprocessed audio data into the frequency domain. In the violence detection scenario, extracting Mel frequency cepstral coefficients (MFCC) can help identify the characteristics of violent behaviors such as screaming and fighting sounds in the audio data, providing important information for subsequent classification tasks.

[0051] By combining the feature extraction capabilities of MFCC and the powerful classification capabilities of 2D convolutional neural networks, it is possible to efficiently and accurately identify the characteristics of violent behavior in audio data, achieve high accuracy and real-time performance in audio violence detection, and provide important technical support for maintaining public safety.

[0052] Specific implementation method three: This implementation method is different from the specific implementation method two in that the output of the support vector machine or the fully connected neural network is subjected to probability calibration, specifically:

[0053]

[0054] Among them, y=1 indicates that the current frame surveillance video is identified as violent, p(y=1|z) is the initial probability of the current frame surveillance video being identified as violent obtained by processing the current frame surveillance video, e is the base of the natural logarithm, and z is the output of the support vector machine or the fully connected neural network.

[0055] The other steps and parameters are the same as those in the second specific implementation manner.

[0056] Then, the initial probability of the current frame surveillance video being identified as violent is fused with the result obtained by processing the audio, and the fusion result is the recognition result of step one.

[0057] Specific implementation method 4: This implementation method is different from specific implementation method 3 in that, in step 2, the probability of the current frame surveillance video being identified as violence is compared with the violence identification threshold, and the violence identification result of the current frame surveillance video is obtained according to the comparison result; the specific process is:

[0058] Step 21: Define x=1 for violent video and x=0 for non-violent video. The loss of non-violent video being identified as violent video is recorded as C. 01 , the loss of identifying a violent video as a non-violent video is recorded as C 10 , the loss of a violent video being correctly identified as a violent video is recorded as C 11 , the loss of correctly identifying a non-violent video as a non-violent video is recorded as C 00 , then the expected value E[C] of the loss C is:

[0059]

[0060] Among them, P(x=0) is the probability that the actual situation is non-violent, P(y=1|x=0) is the probability that the model predicts violence when the actual situation is non-violent (for videos where the actual situation is non-violent, the fusion result after processing the audio and video is P(y=1|x=0), that is, the prediction value of the model here is the fusion result after processing the audio and video), P(y=0|x=0) is the probability that the model predicts non-violent when the actual situation is non-violent, P(x=1) is the probability that the actual situation is violent, P(y=1|x=1) is the probability that the model predicts violence when the actual situation is violent, P(y=0|x=1) is the probability that the model predicts non-violent when the actual situation is violent, P(y|x=0) is the probability that the model predicts y when the actual situation is non-violent, P(y|x=1) is the probability that the model predicts y when the actual situation is violent, y=0 or 1;

[0061] Step 2: Calculate the brute force discrimination threshold y based on the expected value of loss T (i.e. calculated based on the stationary point of the expected value of loss):

[0062]

[0063] Step 23: comparing the probability of the current frame surveillance video being identified as violence with the violence identification threshold;

[0064] If the probability that the current frame of surveillance video is identified as violence is greater than or equal to the violence identification threshold, the violence identification result is that the current frame of surveillance video contains violence;

[0065] If the probability that the current frame surveillance video is identified as violence is less than the violence identification threshold, the violence identification result is that there is no violence in the current frame surveillance video.

[0066] The other steps and parameters are the same as those in the third specific implementation manner.

[0067] Specific implementation mode 5: This implementation mode is different from specific implementation mode 4 in that, in step 3, time series filtering is performed based on the violence recognition results of the historical monitoring video and the violence recognition results of the current frame monitoring video, and whether a violent response is required is determined based on the time series filtering results; the specific process is:

[0068] The number of frames of historical surveillance video is set to 3, that is, the 3 surveillance video frames before the current surveillance video frame are selected to perform time-series filtering on the violence recognition result of the current surveillance video frame, and the automaton state composed of the violence recognition results of the selected 3 historical surveillance video frames is recorded as abc, where the values ​​of a, b, and c are all 0 or 1, 1 represents that the corresponding surveillance video is recognized as having violence, and 0 represents that the corresponding surveillance video is recognized as not having violence;

[0069] The violence recognition result of the current frame surveillance video is recorded as d, and the automaton state will be converted to bcd;

[0070] When the status bcd is 000, 010, or 100, no violent response is required;

[0071] When the status bcd is not 000, 010, or 100, a brute force response is required.

[0072] The other steps and parameters are the same as those in the fourth embodiment.

[0073] This implementation can effectively eliminate the phenomenon of misidentification through time series filtering, reduce the misjudgment rate, and improve the accuracy and stability of the detection method.

[0074] Specific implementation method six: This implementation method is different from specific implementation method four in that, in step three, time series filtering is performed based on the violence recognition results of the historical monitoring video and the violence recognition results of the current frame monitoring video, and whether a violent response is required is determined based on the time series filtering results; the specific process is:

[0075] Step 3. Calculate p(y t |z t =i):

[0076]

[0077] Among them, p(y t ) is the surveillance video at time t predicted as y t The probability of y t =0 or 1, y t =1 means the surveillance video at time t is predicted to be violent, y t = 0 means that the surveillance video at time t is predicted to be non-violent; p(y t |z t =i) is the actual situation of the surveillance video at time t when i is the predicted result y t The probability of t =i|y t ) is the prediction result of the surveillance video at time t, which is y t When , the probability that the true situation is i; z t is the actual situation of the surveillance video at time t;

[0078] Define the recursive function α at time t t (i)

[0079] α t (i) = p(y 1：t , z t =i)=∑ j p(y 1：t-1 , zt-1 =j)p(z t =i|z t-1 =j)p(y t |z t =i)

[0080] =p(y t |z t =i)Σ j α t-1 (p(z t =i|z t-1 =j)

[0081] Among them, p(y 1:t-1 ,z t-1 =j) is the actual situation of the surveillance video at time t-1 is j, and the prediction result of the surveillance video from time 1 to time t-1 is y 1:t-1 The probability of t =i|z t-1 =j) is the probability that the real situation of the surveillance video at time t-1 is j and the real situation of the surveillance video at time t is i; α t-1 (j) is the recursive function at time t-1; where y 1:t-1 The value of each parameter in is also 0 or 1;

[0082]

[0083] Among them, p(y1: t , z t =i) is the actual situation of the surveillance video at time t is i, and the prediction result of the surveillance video from time 1 to time t is y1: t Probability, p(z t =i|y1: t ) is the prediction result of the surveillance video from time 1 to time t, which is y1: t When , the probability that the true situation is i;

[0084] Step 32: Compare p(y t =1|z t =i) and the size of the brute force identification threshold:

[0085] If p(y t =1|z t =i) is greater than or equal to the violence identification threshold, a violent response is required, otherwise no violent response is required.

[0086] The other steps and parameters are the same as those in the fourth embodiment.

[0087] Specific implementation method 7: This implementation method is different from specific implementation methods 5 or 6 in that the mobile camera moves to the next position to continue collecting the next frame of monitoring video, and the calculation method of the target position moved to is:

[0088] After calculating the intermediate variable w, the posterior probability p(x|μ0,∑0,μ x ,∑ x , y):

[0089] p(x|μ0,∑0,μ x ,∑ x , y)∝p(μ0,∑0,μ x ,∑ x , y|x)=p(μ0,∑0|x)p(μ x ,∑ x |x)p(y|x)∝p(μ0,∑0,x)p(μ x ,∑ x ,x)p(y,x)∝p(x|μ0,∑0)p(x|μ x ,∑ x )p(y)

[0090]

[0091]

[0092]

[0093] Among them, μ0 is the most likely angle for the camera to capture, μ x is the position of the camera when the current frame of surveillance video is captured. ∝ indicates that the left side of the formula is proportional to the right side of the formula. p(μ0, Σ0, μ x ,∑ x , y|x) means that when the camera target position is x, the most likely shooting angle of the camera is μ0, the prior distribution covariance matrix of the camera position is ∑0, and the current position of the camera is μ x , the camera movement probability covariance matrix is ​​∑ x , the probability that the surveillance video shot at the target position is identified as y, y = 0 or 1, p(μ0, ∑0|x) represents the probability that when the camera target position is x, the most likely shooting angle of the camera is μ0 and the prior distribution covariance matrix of the camera position is ∑0, p(μ x ,∑ x |x) means that when the camera target position is x, the camera current position is μ x , the camera movement probability covariance matrix is ​​∑ xp(y|x) represents the probability that the surveillance video shot at the target position is identified as y when the camera target position is x. p(μ0,∑0,x) represents the probability that the most likely shooting angle of the camera is μ0, the prior distribution covariance matrix of the camera position is ∑0, and the target position is x. p(μ x ,∑ x ,x) indicates that the current position of the camera is μ x , the camera movement probability covariance matrix is ​​∑ x , the probability that the target position is x, p(y,x) represents the joint probability that the camera target position x and the surveillance video shot at the target position are identified as y, p(x|μ0,∑0) represents the probability that the next target position of the camera is x when the most likely shooting angle of the camera is μ0 and the prior distribution covariance matrix of the camera position is ∑0, p(x|μ x ,∑ x ) indicates that the current position of the camera is μ x , the camera movement probability covariance matrix is ​​∑ x , the probability that the next target position is x, p(y) represents the probability that the surveillance video captured by the camera is identified as y, ∑0 represents the prior distribution covariance matrix of the camera position, ∑ x represents the camera movement probability covariance matrix, α represents the hyperparameter of the probability of violence value distribution, (x-μ0)′ is the transpose of x-μ0, (x-μ x )′ is x-μ x is the transpose of w, w′ is the transpose of w, the superscript -1 represents the inverse of the matrix, and N represents the normal distribution;

[0094] Then according to the posterior probability p(x|μ0,∑0,μ x ,∑ x ,y) to get the target position x where the camera moves.

[0095] The other steps and parameters are the same as those in the fifth or sixth embodiment.

[0096] This implementation can track violence by controlling the movement of the camera to ensure continuous monitoring of violent behavior and increase the detection range of a single camera.

[0097] Specific implementation eight: This implementation differs from specific implementation five or six in that the mobile camera moves to the next position to continue collecting the next frame of surveillance video, and the calculation method for the target position moved to is:

[0098] Let the position of the camera when shooting the current frame of surveillance video be x, and maximize the expectation of violence y by determining the strategy p(a) of moving a:

[0099]

[0100] Among them, q(a|x) is the probability that the camera moves to a when the current position is x under any strategy, E q(a|x) [y=1] represents the expectation that the surveillance video captured by the camera from the current position x and after the camera moves a to the new target position will be identified as violence. p(a|x) is the probability that the camera moves a when the current position is x under the target strategy (i.e., the strategy corresponding to the maximum expectation).

[0101] The target position of the camera movement is determined using the movement a corresponding to the maximum probability p(a|x) and the current position x.

[0102] The other steps and parameters are the same as those in the fifth or sixth embodiment.

[0103] Specific implementation method 9: This implementation method is different from specific implementation method 7 in that the calculation method of the intermediate variable w is:

[0104]

[0105] Where X is the position of the camera when the current frame of surveillance video is taken, Y is the probability that the current frame of surveillance video is identified as violent, ‖·‖ is the L2 norm, X′ is the transpose of X, γ represents the regularization term coefficient, I is the identity matrix, and the superscript -1 represents the inverse of the matrix.

[0106] The other steps and parameters are the same as those in the seventh embodiment.

[0107] Specific implementation method 10: This implementation method is different from the specific implementation method 1 in that the specific process of extracting the violent person information in the current frame surveillance video is as follows:

[0108] Step 1: Use the YOLOv3 model to identify the position of the portrait in the current frame surveillance video;

[0109] Step 2: Use the CNN-REID network to extract information about violent individuals from the image in the area where the portrait is located;

[0110] The CNN-REID network includes a convolution layer, a BN layer, a Relu activation function layer, a first ResBlock module, a second ResBlock module, a third ResBlock module, a fourth ResBlock module, a fifth ResBlock module, an adaptive maximum pooling layer, a Flatten layer, a Linear layer and a Sigmoid activation function layer in sequence;

[0111] The working process of CNN-REID network is:

[0112] The image of the area where the portrait is located is used as the input of the CNN-REID network. In the CNN-REID network, the input image is used as the input of the convolutional layer;

[0113] Use the output of the convolutional layer as the input of the BN layer;

[0114] Use the output of the BN layer as the input of the Relu activation function layer;

[0115] The output of the Relu activation function layer is used as the input of the first ResBlock module;

[0116] Use the output of the first ResBlock module as the input of the second ResBlock module;

[0117] The output of the second ResBlock module is used as the input of the third ResBlock module;

[0118] The output of the third ResBlock module is used as the input of the fourth ResBlock module;

[0119] The output of the fourth ResBlock module is used as the input of the fifth ResBlock module;

[0120] The output of the fifth ResBlock module is used as the input of the adaptive maximum pooling layer;

[0121] Use the output of the adaptive max pooling layer as the input of the Flatten layer;

[0122] Use the output of the Flatten layer as the input of the Linear layer;

[0123] Use the output of the Linear layer as the input of the Sigmoid activation function layer;

[0124] The output of the Sigmoid activation function layer is used as the extracted violent personnel information;

[0125] The first ResBlock module includes a first branch, a second branch and a norm unit, wherein the first branch includes a first res unit, a second res unit and a convolutional layer, and the second branch includes a convolutional layer;

[0126] In the first ResBlock module, firstly, the input of the first ResBlock module is used as the input of the two branches, then the outputs of the two branches are connected, the connection result passes through the norm unit, and the output of the norm unit is used as the output of the first ResBlock module;

[0127] The first res unit includes a convolutional layer and a Relu activation function layer in sequence;

[0128] The norm unit includes the Relu activation function layer and the BN layer in sequence.

[0129] The other steps and parameters are the same as those in the first embodiment.

[0130] In this embodiment, the structures and working processes of the second ResBlock module, the third ResBlock module, the fourth ResBlock module and the fifth ResBlock module are the same as those of the first ResBlock module. The structure and working process of the second res unit are the same as those of the first res unit.

[0131] Example

[0132] This embodiment proposes a method for detecting and tracking violent behavior based on audio and video, and the method specifically includes the following steps:

[0133] Step 1: After respectively performing violence recognition on the current frame surveillance video and the audio corresponding to the current frame surveillance video, the obtained violence recognition results are fused to obtain an initial violence recognition result of the current frame;

[0134] Among them, the 3D-CNN network is used to identify violence in the current frame surveillance video, such as Figure 3 As shown, the 3D-CNN network is used to extract features of the current frame surveillance video, and then the feature extraction results are passed through a support vector machine or a fully connected neural network (a fully connected neural network composed of at least two layers of fully connected layers connected in sequence), that is, a fully connected layer used for classification in the original 3D-CNN network is replaced by a fully connected neural network or a support vector machine.

[0135] The 3D-CNN network used in the present invention is compared with the traditional optical flow-based posture estimation and CNN-RNN architecture, which confirms that the 3D-CNN architecture used in the present invention has advantages in both accuracy and real-time performance.

[0136] The experimental dataset was obtained by screening and merging a total of about 6,000 videos from 5 existing datasets. That is, after removing special videos such as those that are too long, 4,800 videos were selected as the training set, and the remaining 801 videos were used as the test set. The merged dataset is summarized in Table 1:

[0137] Table 1. Violent video detection dataset

[0138]

[0139]

[0140] In Table 1, the dataset Real Life Violence Situations references "M.Soliman, M.Kamal, M.Nashed, Y.Mostafa, B.Chawky, D.Khattab, "Violence Recognition from Videos usingDeep Learning Techniques", Proc.9th International Conference on IntelligentComputing and Information Systems(ICICIS'19), Cairo, pp.79-84, 2019."; the dataset UBI-fight references "DegardinB, Proenca H.Human Activity Analysis:Iterative Weak / Self-Supervised Learning Frameworks for Detecting Abnormal Events[C] / / 2020IEEE International Joint Conference on Biometrics(IJCB).IEEE, 2020.DOI:10.1109 / IJCB48548.2020.9304905."; the dataset surf references " GA HKEkenel, "Vision-based Fight Detection from Surveillance Cameras", IEEE / EURASIP 9th International Conference on Image Processing Theory, Tools and Applications, Istanbul, Turkey, November 2019."; Dataset Violence detection dataset reference "Bianculli M, Falcionelli N, Sernani P, et al. A dataset for automatic violence detection in videos[J]. Data in brief, 2020, 33: 106587."; Dataset RWF-2000 reference "Cheng M, Cai K, Li M. RWF-2000: An Open Large Scale Video Database for Violence Detection[J]. 2019.".

[0141] Use opencv to read the video and resize it. Select an image from the video every 2-4 frames and add it to the key frame sequence. For long videos, only the first half is captured. Pixels use RGB channels and are normalized to the [-1,1] range.

[0142] The 3D-ResNet and 3D-InceptionNet under the 3D-CNN architecture were tested. Both were pre-trained on the Kinetics (Carreira J, Noland E, Hillier C, et al. A short note on the kinetics-700 human action dataset [J]. arXiv preprint arXiv: 1907.06987, 2019.) dataset. The dataset contains 650,000 videos of 700 human poses with rich scenes, which can make the model have strong generalization ability.

[0143] 3D-ResNet (R3D) is a 3D extension of 2D-ResNet. It has no difference except that the convolution layer is modified to 3D convolution. The structure of each residual module is as follows Figure 4 As shown, 3D-InceptionNet (I3D) is a 3D extension of InceptionNet, with a structure like Figure 5 shown.

[0144] The present invention provides R3D-34, R3D-50, I3D with fully connected neural network or SVM, and trains and tests them with multiple specifications of maximum width of 128 or 256, maximum number of frames of 32, 48 or 64, and inter-frame interval of 0.08, 0.125 or 0.17s, all of which show good performance. The performance on the test set is shown in Table 2:

[0145] Table 2 Model performance

[0146]

[0147]

[0148] The probability of the support vector machine (SVM) is calibrated using logistic regression with only one parameter. The logistic regression solution is 1.02 under almost all parameters. Therefore, when the SVM output is z, the probability y that the current frame surveillance video is a violent video is:

[0149]

[0150] Among them, y=1 indicates that the current frame surveillance video is identified as violent, p(y=1|z) is the initial probability of the current frame surveillance video being identified as violent obtained by processing the current frame surveillance video, e is the base of the natural logarithm, and z is the output of the support vector machine or the fully connected neural network.

[0151] The relationship between the output probability of the SVM and the sample after calibration under the statistical test set is as follows: Figure 6 and Figure 7 As shown in the figure, the prediction probability is polarized, indicating that the judgment of SVM is not ambiguous, and the prediction probability when the prediction is wrong is close to 0.5, indicating that the prediction probability has a strong reference value.

[0152] To verify the superiority of the 3D-CNN architecture of the present invention, the traditional scheme based on HOF features (Kumar RN, Chandrakala DS. DETECTING AGGRESSIVE HUMAN BEHAVIOR IN PUBLIC ENVIRONMENTS [J]. [2024-03-18].) and CNN-RNN architecture was tested. The HOF scheme uses HOG to extract image feature points, uses local least squares to obtain the optical flow at the feature points, that is, the moving speed of the feature points, and obtains the HOF features by statistically analyzing the histogram of the optical flow distribution, and then uses SVM for classification; the CNN-RNN architecture uses ResNet-34 to extract the features of each frame, and then uses GRU or Attention for classification. The test results are shown in Table 3:

[0153] Table 3

[0154]

[0155] As shown in Table 3, the accuracy and latency of the HOF scheme are inferior to those of CNN-3D.

[0156] The process of violence detection on audio is as follows Figure 8 As shown, the training samples selected as the audio violence detection 2D-CNN model come from the audio dataset, and the audio tracks are extracted from the violent videos for supplementation. However, the audio tracks in the violent video dataset mostly come from the monitoring room or have no audio, and only some videos in the Real Life Violence Situations dataset contain fighting audio. Therefore, the present invention uses moviepy to extract the fighting audio in the dataset and the audio in the non-violent videos in all the violent video detection datasets, and uses the non-violent audio in the Audio-based Violence Detection Dataset as a supplement.

[0157] The acquired data set is saved in wav format, and the data set specifications are shown in Table 4:

[0158] Table 4 Audio dataset specifications

[0159]

[0160] When processing audio data, Mel features are used to process audio to the frequency domain. For audio that is too long, only the first half is taken. For audio that is too short, 0 is used to fill it. Then 2D convolution is used for classification. Since the audio is relatively simple and the data set is small, the number of convolution layers is as small as possible, the step size is as large as possible, and dropout is added to avoid overfitting. The 2D-CNN network model structure is shown in Table 5:

[0161] Table 5 Violent audio detection model

[0162]

[0163] The final test performance is as follows:

[0164] Table 6 Violent audio detection performance

[0165]

[0166] Step 2: Calculate the brute force identification threshold

[0167] Define violence as x=1, non-violence as x=0, and the loss of identifying a non-violent video as a violent video is denoted by C 01 , the loss of identifying violent videos as non-violent videos is recorded as C 10, the loss of a violent video being correctly identified as a violent video is recorded as C 11 , the loss of correctly identifying a non-violent video as a non-violent video is recorded as C 00 , then the expected value of the loss C is:

[0168]

[0169] Among them, P(x=0) is the probability that the actual situation is non-violent, P(y=1|x=0) is the probability that the model predicts violence when the actual situation is non-violent, P(y=0|x=0) is the probability that the model predicts non-violent when the actual situation is non-violent, P(x=1) is the probability that the actual situation is violent, P(y=1|x=1) is the probability that the model predicts violence when the actual situation is violent, P(y=0|x=1) is the probability that the model predicts non-violent when the actual situation is violent, P(y|x=0) is the probability that the model output is y when the actual situation is non-violent, and P(y|x=1) is the probability that the model output is y when the actual situation is violent;

[0170] Calculate the violent discrimination threshold y based on the loss expectation value T (i.e. calculated based on the stationary point of the expected value of loss):

[0171]

[0172] In the present invention, C is set 10 -c 00 =C 01 -C 11 =1, P(x=0)=0.5, and the violence identification threshold is 0.5.

[0173] The probability of the current frame surveillance video being identified as violence is compared with the violence identification threshold, and the violence identification result of the current frame surveillance video is obtained according to the comparison result, that is:

[0174] If the probability that the current frame of surveillance video is identified as violence is greater than or equal to the violence identification threshold, the violence identification result is that the current frame of surveillance video contains violence;

[0175] If the probability that the current frame surveillance video is identified as violence is less than the violence identification threshold, the violence identification result is that there is no violence in the current frame surveillance video.

[0176] Step 3: Since violent behavior is continuous, if violent behavior occurs, the model should continuously detect violent behavior. Similarly, if violent behavior does not occur, the model should continuously detect non-violent behavior. Therefore, referring to the historical model recognition results can avoid model misrecognition to a certain extent. The present invention uses an automaton or hidden Markov chain method for time series filtering.

[0177] 1. Automatic machine

[0178] The automaton can recognize some non-context-related languages, and it can determine whether the current recognition is reliable with certainty through limited historical results. Taking the history length of 3 as an example, the state of the automaton is binary numbers 000 to 111. When it is in state abc and receives the model recognition result d, the automaton state will be converted to bcd, and its state transition function is:

[0179] δ abc (d) = bcd

[0180] When the status is 110, 101, and 011, the model detects violent incidents continuously, but intermittently, so it is a level 2 warning;

[0181] When the status is 111, the model continuously detects violence, which is a level 3 warning;

[0182] When the status is 001, a violent incident is detected, but the possibility of misidentification cannot be ruled out, so it is a level 1 warning;

[0183] When the status is 000, 010, and 100, it is continuously identified as non-violence, which means that it is actually non-violence. If violence is identified, it is most likely a misidentification, so no warning is issued.

[0184] If an early warning occurs, a violent response is required; if no early warning occurs, a violent response is not required.

[0185] 2. Hidden Markov Chain

[0186] Considering the practical application Fig. 9 The hidden state z is the Bernoulli distribution of whether a fight occurs, and the observable state y is the model output value. Then the observation probability matrix at time t is p(y t |z t =i), which represents the probability that the model output is y when the actual situation is i. This probability can be estimated using a Gaussian mixture distribution, or we can assume that y t is uniformly distributed, and the following approximation can be made:

[0187]

[0188] Among them, p(y t ) is the time t, that is, the current frame surveillance video is predicted to be y t The probability, p(z t =i|y t ) is the time t, that is, the current frame surveillance video is predicted to be y t The probability that the true situation is i, p(y t |z t=i) is time t, that is, when the actual situation of the current frame surveillance video is i, the prediction is y t probability;

[0189] At this time, the observation probability matrix is ​​the probability of the model output. In order to obtain the posterior probability of the hidden state, define the recursive function α t (i)

[0190] α t (i) = p(y 1:t , z t =i)=∑ j p(y 1：t-1 ,z t-1 =j)p(z t =i|z t-1 =j)p(y t |z t =i)

[0191] =p(y t |z t =i)Σ j α t-1 (j)p(z t =i|z t-1 =j)

[0192] Among them, p(y 1:t-1 ,z t-1 =j) means the actual situation at time t-1 is j, and the predicted value from time 1 to time t-1 is y 1:t-1 Probability, p(z t =i|z t-1 =j) is the probability that the true situation at time t-1 is j and the true situation at time t is i;

[0193] This function can be solved by dynamic programming. Using the maximum posterior of the hidden state at time t as the model identification result, the posterior probability is:

[0194]

[0195] Among them, p(y1: t , z t =i) means the actual situation at time t is i, and the predicted value from 1 to time t is y1: t Probability; p(z t =i|y1: t ) is the prediction result of the surveillance video from time 1 to time t, which is y1: t When , the probability that the true situation is i;

[0196] Compare p(y t =1|z t =i) and the size of the brute force identification threshold:

[0197] If p(y t =1|z t =i) is greater than or equal to the violence identification threshold, a violent response is required, otherwise no violent response is required.

[0198] The initial brute force prior probability and state transition probability can be customized by the user. When not defined, a brute force prior of 0.5 and a transition probability matrix of 0.7 are used to maintain the current state. An example is shown in Table 7:

[0199] Table 7 Recognition status after HMM processing

[0200] Identification probability 0.7 0.4 0.4 0.6 0.5 0.4 0.7 0.9 0.3 0.5 Probability after treatment 0.65 0.40 0.32 0.50 0.45 0.34 0.61 0.89 0.36 0.41

[0201] It can be seen that the method of the present invention effectively removes the suspected misidentification situations.

[0202] Then, based on the results of the time series filtering, it is determined whether a violent response is needed. If a violent response is needed, the information of the violent person in the current frame surveillance video is extracted, and the camera is moved to the next position. If a violent response is not needed, there is no need to extract the information of the violent person in the current frame surveillance video, that is, the camera position is directly moved.

[0203] The target position to which the camera needs to move is calculated based on a Bayesian decision or reinforcement learning scheme.

[0204] 1. Bayesian Decision Making

[0205] Define the distribution of the violence value y captured by the camera at the new position under a reasonable movement a, and then calculate the posterior of a with respect to y.

[0206] First, we define the prior distribution of a. Since the camera has a main angle μ0, for example, street cameras should mainly shoot the street when there is no obvious violence, so the prior distribution of the target position x of the camera movement is defined as:

[0207] p(x|μ0,Σ0)=N(x|μ0,Σ0)

[0208] Among them, ∑0 is the covariance matrix of the prior distribution.

[0209] The camera is required to move a limited distance at a time, so when the current camera position μ is known x When , the expectation of the camera target position x at the next moment should remain unchanged, so its distribution in this sense is defined as:

[0210] p(x|μ x ,∑ x )=N(x|μ0,∑0)

[0211] When the camera movement scheme is reasonable, the captured violence value y should be as large as possible, and its distribution is defined as:

[0212]

[0213] in, is the normalization coefficient and α is a hyperparameter.

[0214] Calculate w using the following method (1) or method (2):

[0215] Method (1)

[0216] Use linear ridge regression to fit the relationship between x and y. Although x and y are not linear, the camera moves a limited distance in the short term, and x and y are linear in the local area:

[0217]

[0218] Where X and Y are the camera position and the captured violence value at the historical moment, respectively, and α is the regularization coefficient, which represents the prior confidence in w.

[0219] Method (2)

[0220] When using linear ridge regression, all information outside the memory bank is lost, which may be wasteful. Therefore, Kalman filtering can also be used to update w. Kalman filtering uses a global loss function to solve w at time t:

[0221]

[0222] The final iterative solution is:

[0223] H t =δH t-1 +X t′ X t

[0224] w t =w t-1 +(H t ) -1 X t′ Y t

[0225] This method can better preserve historical information.

[0226] Assuming that all knowledge is unknown, the camera position is completely random, and the position cannot be accurately determined based on the violence value. When the main position, current position, and violence value of each position are known, assuming that these three are independent of each other, the posterior probability of the next position of the camera is:

[0227] p(x|μ0,∑0,μ x ,∑x , y)∝p(μ0,∑0,μ x ,∑ x , y|x)=p(μ0,Σ0|x)p(μ x ,Σ x |x)p(y|x)∝p(μ0,∑0,x)p(μ x ,∑ x ,x)p(y,x)

[0228]

[0229]

[0230]

[0231] Among them, when ∑ is positive, the distribution is Gaussian distribution and sampling is sufficient.

[0232] According to the posterior probability p(x|μ0,∑0,μ x ,∑ x , y) can obtain the target position of the camera movement. When the violence occurs in the upper left corner, the camera movement trajectory obtained by the Bayesian decision method is as follows Fig.10 As shown, it can be seen that the camera will quickly detect towards the upper left corner and converge to a certain point.

[0233] 2. Reinforcement Learning

[0234] In the policy gradient scheme, when the current input position is x, the strategy p(a) of moving a maximizes the expectation of the violence value y:

[0235]

[0236] When solving by gradient descent method, its derivative with respect to q(a|x) is:

[0237]

[0238]

[0239] Therefore, given a data set A, Y, X that obeys the distribution of q(a|x), using the policy gradient method to iterate is equivalent to finding the stationary point of the following objective function:

[0240] L(q)=E x～X，y～Y，a～A [log(q(a|x))(k+y)]

[0241] Where k is a constant;

[0242] Let q(a|x) be the Gaussian distribution with high-dimensional expansion at position x and linear combination as the desired value:

[0243]

[0244] Where w′ is the transpose of the model parameters, ∑ is the covariance matrix, and Φ(x) is any nonlinear mapping of position x;

[0245] Therefore, the original problem becomes a weighted least squares problem:

[0246] E x～X，y～Y，a～A [log(q(a|x))(k+y)]|∝E x～X，y～Y，a～A [(aw′Φ(x))′(α-w′Φ(x))(k+y)]

[0247] Introducing the Gaussian prior of w, the maximum a posteriori estimate is obtained as:

[0248] w′Φ(x)=Φ(x)′X′(n w Λ -1 +XX′) -1 A

[0249]

[0250] Among them, n w is the w prior probability degrees of freedom, the value of k does not affect the convergence expectation of q(a|x), but can be set to the mean of y to reduce the convergence variance. Use the kernel trick to avoid explicitly constructing Φ:

[0251] K(||x1-x2|| 2 )=Φ(x1)′Φ(x2)

[0252] At the same time, in order to prevent the camera from leaving the main area for a long time, the camera is set to move to the main area if it detects low violence values ​​within a certain period of time.

[0253] The same environment as the Bayesian decision method was used for testing, and the camera movement trajectory obtained was as follows Fig.11 As shown in the figure, it can be seen that the camera also achieves focus. Therefore, the reinforcement learning method can also capture the violent area (the upper left corner in the figure).

[0254] After the camera moves to the target position, it continues to collect the next frame of surveillance video, and then returns to step 1 for the collected next frame of surveillance video;

[0255] The process of extracting information about violent individuals is as follows: Fig.12As shown in the figure, it mainly includes two parts: collecting the position of the portrait in the surveillance video based on YOLOv3 and extracting the features of the collected portrait. The portrait feature extraction adopts the pedestrian re-identification (CNN-REID) model trained on the PA-100k dataset for identifying many human features. The features of the portrait detected in the first part are extracted, such as age, gender, clothing, etc.

[0256] 1. Portrait Extraction

[0257] YOLO (You Only Look Once) is a popular object detection and image segmentation model developed by Joseph Redmon and Ali Farhadi of the University of Washington. YOLO was launched in 2015 and quickly gained popularity due to its high speed and high accuracy. YOLOv3 used in the present invention is the third version in the YOLO (You Only Look Once) algorithm series.

[0258] The basic principle of YOLO is to treat object detection as a regression problem of spatially separated bounding boxes and associated class probabilities. In each evaluation, the neural network can directly predict bounding boxes and classification probabilities from the complete image, such as determining that the framed part belongs to a person. Since YOLO's detection pipeline is a single network, it can be directly optimized end-to-end on the detection performance. Compared with other detection methods, such as DPM and R-CNN, the biggest advantage of YOLO is its high detection efficiency, such as Fig.13 As shown in the figure, the accuracy of extracting specific people from surveillance images is also higher. The basic model of the first version of YOLO can process images in real time at a high speed of 45 frames per second, and the v3 version used in the present invention is updated from the first version of YOLO. The structure of the v3 version is as follows: Fig.14 As shown, when the image size is 320×320, YOLOv3 runs in 22 milliseconds at 28.2mAP and is as accurate as SSD, but three times faster.

[0259] The idea of ​​YOLO is to divide an image into W×W grids, each of which is responsible for the target whose center point falls in the grid. The network will generate a bounding box for each (t x ,t y ,t w ,t h ) predicts 4 coordinates (Wang Xinpeng. Research on garbage detection algorithm based on YOLOv3 [D]. Chongqing: Chongqing University of Posts and Telecommunications, 2022.). The offset between the cell and the upper left corner of the image is recorded as (c x ,c y ), the width and height of the bounding box are p w , p h, then the prediction corresponds to:

[0260] b x =σ(t x )+x

[0261] b y =σ(t y )+y

[0262]

[0263]

[0264] Fig.15 The specific meaning of each quantity is intuitively explained. The network uses the sigmoid method to limit the predicted value t x ,t y The mathematical range of the network is effectively accelerated. x ,c y ) offset to get the center coordinates, for t w ,t h The size data of the prediction block is obtained by processing, and finally the prediction result is obtained.

[0265] In order to reduce the damage to the information in the image during processing, YOLOv3 uses three feature maps of different scales. When designing the model, the residual network Darknet-53 with deeper layers is selected as the backbone feature extraction network. The Darknet-53 network abandons the common pooling and full connection operations. Its convolution layer is designed with two different sizes of convolution kernels, 1×1 and 3×3, to extract features. Normalization and LeakyReLU are added after the convolution calculation (Redmon, Joseph and Ali Farhadi. "YOLOv3: An Incremental Improvement." ArXiv abs / 1804.02767(2018):n.pag.).

[0266] The YOLOv3 network structure can be roughly divided into three parts: Backbone, FPN and Yolo Head. The main body of the Backbone part is Darknet-53, and FPN is a feature pyramid model. From bottom to top, the image is reduced in dimension and features are extracted through convolution, and then the different semantic information of multi-level feature maps is fused from top to bottom through upsampling and feature fusion. In this way, both high-level semantic information and low-level contour information are obtained. Finally, the Yolo Head part is a decoder, and the output is three-dimensional w, h. Since the meaning of the expression is the width and height of the prediction box, it is recorded as b w ,b hEach point corresponds to the grid divided by the original image, and the channel corresponds to the predicted value of the anchor.

[0267] 2. Identity feature extraction

[0268] The person re-identification model is used to extract the identity of violent persons. The person re-identification model is trained from the PA-100K dataset using a small model consisting of multiple convolutional layers, BatchNorm layers, LeakyReLU activation functions, and adaptive maximum pooling layers. The resulting person re-identification model can read image data and output feature extraction results including but not limited to gender, age, clothing, hairstyle, and many other features, which is helpful for locating violent persons. The final model test accuracy can reach 86%.

[0269] The CNN-REID network model first performs a convolution and then normalizes the convolution result, and then uses the activation function LeakyReLU. Then the ResBlock module is used 5 times in succession, and the result is processed by the adaptive maximum pooling layer, and the result is converted into a one-dimensional array by Flatten, and a linear transformation is used. Finally, the result is calculated by Sigmoid transformation to output the feature extraction results including but not limited to gender, age, clothing, hairstyle and many other features. We use the PA-100K dataset and put it into this model for training, and the model test accuracy can reach more than 86%.

[0270] In general, the violent behavior detection method of the present invention has the following characteristics:

[0271] 1) Violence detection based on audio and video dual signals

[0272] The present invention uses the weighted geometric mean of the video recognition results and the audio recognition results as the final recognition result. In terms of picture recognition, the traditional method of extracting features separately in space and time and the large-scale model are abandoned, and a 3D CNN network with a small parameter scale and simultaneous extraction of spatiotemporal features is used for feature extraction. In order to obtain a probabilistic explanation of the output, logistic regression is used for probability calibration. In terms of audio, Mel features are used to extract features from the input audio, and then classified by a small 2D CNN model.

[0273] The method of the invention realizes a violence detection framework with a CPU delay of less than 1 second under the premise that the video recognition accuracy reaches more than 90%.

[0274] 2) Intelligent tracking and optimization of audio and video signals

[0275] In traditional violence detection methods, old cameras are mostly fixed in place, so the movement of the viewing angle is not considered during detection. With the advancement of the times, cameras have been replaced with movable cameras on a large scale. A reasonable camera movement strategy can greatly increase the detection range of a single camera, so the present invention proposes two sets of camera autonomous movement strategies to control the movement of the camera, effectively expand the detection range of a single camera, and improve the traceability of violent behavior.

[0276] 3) Elimination of misidentification based on prior knowledge

[0277] The violence detection model driven by machine learning data still has the possibility of misidentification even if it integrates information from both video and audio dimensions. However, considering that violent incidents are often not accidental at a certain moment, but continuous behavior over a long period of time, the reliability of the latest recognition results can be judged by historical recognition results, thereby eliminating misidentification.

[0278] The present invention proposes two methods for eliminating false recognition. The automaton method requires that an automaton be constructed based on violent behaviors detected in history, a state transfer function be constructed based on rules, and a violence index of 1 to 3 be confirmed for each state. The higher the violence index, the higher the probability of a violent incident. The hidden Markov chain method makes full use of the hidden state z of the hidden Markov chain, models the actual situation as the hidden state z, models the recognition result as the observation state, and determines the probability of whether violence occurs under the current observation result by calculating the posterior probability of the current hidden state, thereby correcting the false judgment and obtaining a better judgment effect.

[0279] 4) Intelligent control based on reinforcement learning

[0280] The present invention uses reinforcement learning technology to realize monitoring steering control, thereby tracking and observing violent behavior. The goal of reinforcement learning is to allow the intelligent agent to gradually learn the optimal behavior strategy in the process of continuous trial and error, so that it can make the best decision when facing similar environments in the future. In the present invention, a policy-based reinforcement learning method is used, and a monitoring device that can rotate the viewing angle is used as an intelligent agent. It is allowed to rotate randomly at the beginning, and then the detected violence value is used as a reward function. Then, the policy gradient method is used to obtain the optimal strategy that maximizes the expected future return to realize camera movement control.

[0281] After a period of training, the surveillance camera will "learn" to chase the scenes containing violence to maximize the built-in expectation. From a visual perspective, the surveillance camera will always focus on the scenes where violence occurs, and can capture the scenes well even if the characters in the violent conflict are displaced.

[0282] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for detecting and tracking violent behavior based on audio and video, characterized in that: The method specifically comprises the following steps: Step 1: After respectively performing violence recognition on the current frame surveillance video and the audio corresponding to the current frame surveillance video, weighted fusion is performed on the obtained violence recognition results, and the weighted fusion result is used as the probability that the current frame surveillance video is recognized as violence; Step 2: comparing the probability that the current frame of surveillance video is identified as violence with the violence identification threshold, and obtaining a violence identification result for the current frame of surveillance video according to the comparison result; Step 3: Perform time series filtering based on the violence recognition results of the historical surveillance video and the violence recognition results of the current frame surveillance video, and determine whether a violent response is required based on the time series filtering results; If a violent response is required, extract the violent person information in the current frame surveillance video and then execute step 4; If no violent response is required, proceed directly to step 4; Step 4: Move the camera to the next position to continue collecting the next frame of surveillance video, and then return to step 1 for the collected next frame of surveillance video.

2. The method for detecting and tracking violent behavior based on audio and video according to claim 1, characterized in that: The violence recognition is performed on the current frame surveillance video and the audio corresponding to the current frame surveillance video respectively, specifically: The 3D-CNN network is used to identify violence in the current frame of surveillance video. The 3D-CNN network is used to extract features from the current frame of surveillance video. The feature extraction results are then passed through a support vector machine or a fully connected neural network. The output of the support vector machine or the fully connected neural network is probability calibrated and the probability calibration result is used as the initial probability of the current frame of surveillance video being identified as violence. Mel features and 2D-CNN network are used to identify the audio corresponding to the current frame of surveillance video. First, Mel features are extracted from the audio corresponding to the current frame of surveillance video, and then the extracted Mel features are passed through the 2D-CNN network. The output of the 2D-CNN network is used as the probability that the audio corresponding to the current frame of surveillance video is identified as violent.

3. The method for detecting and tracking violent behavior based on audio and video according to claim 2, characterized in that: The probability calibration of the output of the support vector machine or the fully connected neural network is specifically as follows: Among them, y=1 indicates that the current frame surveillance video is identified as violent, p(y=1|z) is the initial probability of the current frame surveillance video being identified as violent obtained by processing the current frame surveillance video, e is the base of the natural logarithm, and z is the output of the support vector machine or the fully connected neural network.

4. The method for detecting and tracking violent behavior based on audio and video according to claim 3, characterized in that: In the step 2, the probability of the current frame surveillance video being identified as violence is compared with the violence identification threshold, and the violence identification result of the current frame surveillance video is obtained according to the comparison result; the specific process is: Step 21: Define x=1 for violent video and x=0 for non-violent video. The loss of non-violent video being identified as violent video is recorded as C. 01 , the loss of identifying a violent video as a non-violent video is recorded as C 10 , the loss of a violent video being correctly identified as a violent video is recorded as C 11 , the loss of correctly identifying a non-violent video as a non-violent video is recorded as C 00 , then the expected value E[C] of the loss C is: Where P(x=0) is the probability that the true situation is non-violent, P(y=1|x=0) is the probability that the model predicts violence when the true situation is non-violent, P(y=0|x=0) is the probability that the model predicts non-violent when the true situation is non-violent, P(x=1) is the probability that the true situation is violent, P(y=1|x=1) is the probability that the model predicts violence when the true situation is violent, P(y=0|x=1) is the probability that the model predicts non-violent when the true situation is violent, P(y|x=0) is the probability that the model predicts y when the true situation is non-violent, P(y|x=1) is the probability that the model predicts y when the true situation is violent, y=0 or 1; Step 2: Calculate the brute force discrimination threshold y based on the expected value of loss T : Step 23: comparing the probability of the current frame surveillance video being identified as violence with the violence identification threshold; If the probability that the current frame of surveillance video is identified as violence is greater than or equal to the violence identification threshold, the violence identification result is that the current frame of surveillance video contains violence; If the probability that the current frame surveillance video is identified as violence is less than the violence identification threshold, the violence identification result is that there is no violence in the current frame surveillance video.

5. The method for detecting and tracking violent behavior based on audio and video according to claim 4, characterized in that: In the step three, time series filtering is performed based on the violence recognition results of the historical monitoring video and the violence recognition results of the current frame monitoring video, and whether a violent response is required is determined based on the time series filtering results; the specific process is: The number of frames of historical surveillance video is set to 3, that is, the 3 surveillance video frames before the current surveillance video frame are selected to perform time-series filtering on the violence recognition result of the current surveillance video frame, and the automaton state composed of the violence recognition results of the selected 3 historical surveillance video frames is recorded as abc, where the values ​​of a, b, and c are all 0 or 1, 1 represents that the corresponding surveillance video is recognized as having violence, and 0 represents that the corresponding surveillance video is recognized as not having violence; The violence recognition result of the current frame surveillance video is recorded as d, and the automaton state will be converted to bcd; When the status bcd is 000, 010, or 100, no violent response is required; When the status bcd is not 000, 010, or 100, a brute force response is required.

6. The method for detecting and tracking violent behavior based on audio and video according to claim 4, characterized in that: In the step three, time series filtering is performed based on the violence recognition results of the historical monitoring video and the violence recognition results of the current frame monitoring video, and whether a violent response is required is determined based on the time series filtering results; the specific process is: Step 31. Calculate p(y t |z t =i): Among them, p(y t ) is the surveillance video at time t predicted as y t The probability of y t =0 or 1, y t =1 means the surveillance video at time t is predicted to be violent, y t = 0 means that the surveillance video at time t is predicted to be non-violent; p(y t |z t =i) is the actual situation of the surveillance video at time t when i is the predicted result y t The probability of t =i|y t ) is the prediction result of the surveillance video at time t, which is y t When , the probability that the true situation is i; z t is the actual situation of the surveillance video at time t; Define the recursive function α at time t t (i) Among them, p(y 1:t-1 ,z t-1 =j) is the actual situation of the surveillance video at time t-1 is j, and the prediction result of the surveillance video from time 1 to time t-1 is y 1:t-1 The probability of t =i|z t-1 =j) is the probability that the real situation of the surveillance video at time t-1 is j and the real situation of the surveillance video at time t is i; α t-1 (j) is the recursive function at time t-1; Among them, p(y 1:t ,z t =i) is the actual situation of the surveillance video at time t is i, and the prediction result of the surveillance video from time 1 to time t is y 1:t Probability, p(z t =i|y 1:t ) is the prediction result of the surveillance video from time 1 to time t, which is y 1:t When , the probability that the true situation is i; Step 32: Compare p(y t =1|z t =i) and the size of the brute force identification threshold: If p(y t =1|z t =i) is greater than or equal to the violence identification threshold, a violent response is required, otherwise no violent response is required.

7. A method for detecting and tracking violent behavior based on audio and video according to claim 5 or 6, characterized in that: The mobile camera moves to the next position to continue collecting the next frame of surveillance video. The calculation method for the target position moved to is: After calculating the intermediate variable w, the posterior probability p(x|μ0,∑0,μ x ,∑ x ,y): p(x|μ0,∑0,μ x ,∑ x ,y)∝p(μ0,∑0,μ x ,∑ x ,y|x)=p(μ0,∑0|x)p(μ x ,∑ x |x)p(y|x) ∝p(μ0,∑0,x)p(μ x ,∑ x ,x)p(y,x)∝p(x|μ0,∑0)p(x|μ x ,∑ x )p(y) Among them, μ0 is the angle of the camera, μ x is the position of the camera when the current frame of surveillance video is captured. ∝ indicates that the left side of the formula is proportional to the right side of the formula. p(μ0,∑0,μ x ,∑ x ,y|x) means that when the camera target position is x, the camera shooting angle is μ0, the prior distribution covariance matrix of the camera position is ∑0, and the current position of the camera is μ x , the camera movement probability covariance matrix is ​​∑ x , the probability that the surveillance video shot at the target position is identified as y, y = 0 or 1, p(μ0,∑0|x) represents the probability that the camera shooting angle is μ0 and the prior distribution covariance matrix of the camera position is ∑0 when the camera target position is x, p(μ x ,∑ x |x) means that when the camera target position is x, the camera current position is μ x , the camera movement probability covariance matrix is ​​∑ x p(y|x) represents the probability that the surveillance video shot at the target position is identified as y when the camera target position is x. p(μ0,∑0,x) represents the probability that the camera shooting angle is μ0, the prior distribution covariance matrix of the camera position is ∑0, and the target position is x. p(μ x ,∑ x ,x) indicates that the current position of the camera is μ x , the camera movement probability covariance matrix is ​​∑ x , the probability that the target position is x, p(y,x) represents the joint probability that the camera target position x and the surveillance video shot at the target position are identified as y, p(x|μ0,∑0) represents the probability that the next target position of the camera is x when the camera shooting angle is μ0 and the prior distribution covariance matrix of the camera position is ∑0, p(x|μ x ,∑ x ) indicates that the current position of the camera is μ x , the camera movement probability covariance matrix is ​​∑ x , the probability that the next target position is x, p(y) represents the probability that the surveillance video captured by the camera is identified as y, ∑0 represents the prior distribution covariance matrix of the camera position, ∑ x represents the camera movement probability covariance matrix, α represents the hyperparameter of the probability of violence value distribution, (x-μ0)′ is the transpose of x-μ0, (x-μ x )′ is x-μ x is the transpose of w, w′ is the transpose of w, the superscript -1 represents the inverse of the matrix, and N represents the normal distribution; Then according to the posterior probability p(x|μ0,∑0,μ x ,∑ x ,y) to get the target position x where the camera moves.

8. The method for detecting and tracking violent behavior based on audio and video according to claim 5 or 6, characterized in that: The mobile camera moves to the next position to continue collecting the next frame of surveillance video. The calculation method for the target position moved to is: Let the position of the camera when shooting the current frame of surveillance video be x, and maximize the expectation of violence y by determining the strategy p(a) of moving a: Among them, q(a|x) is the probability that the camera moves to a when the current position is x under any strategy, E q(a|x) [y=1] represents the expectation that the surveillance video captured by the camera from the current position x and after moving a to the new target position will be identified as violence. p(a|x) is the probability that the camera moves a when the current position is x under the target strategy. The target position of the camera movement is determined using the movement a corresponding to the maximum probability p(a|x) and the current position x.

9. The method for detecting and tracking violent behavior based on audio and video according to claim 7, characterized in that: The calculation method of the intermediate variable w is: Where X is the position of the camera when the current frame of surveillance video is taken, Y is the probability that the current frame of surveillance video is identified as violent, ‖·‖ is the L2 norm, X′ is the transpose of X, γ represents the regularization term coefficient, I is the identity matrix, and the superscript -1 represents the inverse of the matrix.

10. The method for detecting and tracking violent behavior based on audio and video according to claim 1, characterized in that: The specific process of extracting the violent personnel information in the current frame surveillance video is as follows: Step 1: Use the YOLOv3 model to identify the position of the portrait in the current frame surveillance video; Step 2: Use the CNN-REID network to extract information about violent individuals from the image in the area where the portrait is located; The CNN-REID network includes a convolution layer, a BN layer, a Relu activation function layer, a first ResBlock module, a second ResBlock module, a third ResBlock module, a fourth ResBlock module, a fifth ResBlock module, an adaptive maximum pooling layer, a Flatten layer, a Linear layer and a Sigmoid activation function layer in sequence; The working process of CNN-REID network is: The image of the area where the portrait is located is used as the input of the CNN-REID network. In the CNN-REID network, the input image is used as the input of the convolutional layer; Use the output of the convolutional layer as the input of the BN layer; Use the output of the BN layer as the input of the Relu activation function layer; The output of the Relu activation function layer is used as the input of the first ResBlock module; Use the output of the first ResBlock module as the input of the second ResBlock module; The output of the second ResBlock module is used as the input of the third ResBlock module; The output of the third ResBlock module is used as the input of the fourth ResBlock module; The output of the fourth ResBlock module is used as the input of the fifth ResBlock module; The output of the fifth ResBlock module is used as the input of the adaptive maximum pooling layer; Use the output of the adaptive max pooling layer as the input of the Flatten layer; Use the output of the Flatten layer as the input of the Linear layer; Use the output of the Linear layer as the input of the Sigmoid activation function layer; The output of the Sigmoid activation function layer is used as the extracted violent personnel information; The first ResBlock module includes a first branch, a second branch and a norm unit, wherein the first branch includes a first res unit, a second res unit and a convolutional layer, and the second branch includes a convolutional layer; In the first ResBlock module, firstly, the input of the first ResBlock module is used as the input of the two branches, then the outputs of the two branches are connected, the connection result passes through the norm unit, and the output of the norm unit is used as the output of the first ResBlock module; The first res unit includes a convolutional layer and a Relu activation function layer in sequence; The norm unit includes the Relu activation function layer and the BN layer in sequence.

Images