A weakly supervised video anomaly detection method, system, device and medium

Abstract

The application discloses a weakly supervised video anomaly detection method, system, device and medium, and the weakly supervised video anomaly detection method comprises the following steps: acquiring a video frame feature sequence to be detected; based on the acquired video frame feature sequence to be detected, an anomaly detection model pre-trained is used for video anomaly detection, and a video anomaly detection result is obtained. In the technical scheme provided by the application, the multi-instance learning loss function based on the abnormal proportion and the multi-stage self-training framework guided by the abnormal proportion can accurately detect the existing anomalies (including simple abnormal frames and difficult abnormal frames) in the video.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a weakly supervised video anomaly detection method, system, device and medium. BACKGROUND

[0002] Video anomaly detection aims to find and locate abnormal behaviors or dangerous behaviors (for example, behaviors threatening public safety) in surveillance videos. In the training set of weakly supervised video anomaly detection, there are normal videos and abnormal videos, as well as their corresponding video-level class labels. The time of the occurrence of abnormal events is unknown. Illustratively, the video anomaly detection task judges whether any frame in a video is an abnormal frame, and then detects the abnormal event in the video.

[0003] At the present stage, the existing weakly supervised video anomaly detection method still has the following problems and defects: anomaly detection needs to detect both simple anomalies and difficult anomalies in abnormal events, but the existing method usually only considers the learning of simple anomalies and can only detect simple anomalies (illustratively, abnormal frames that are obviously different from normal frames); the learning of difficult anomalies is ignored, and difficult anomalies (illustratively, abnormal frames that are similar to normal frames and often occur at the beginning of an abnormal event) cannot be detected; further illustratively, for a car accident event, the moment of vehicle collision is a simple anomaly, and the abnormal speed or trajectory of the vehicle is a difficult anomaly. SUMMARY

[0004] The present application aims to provide a weakly supervised video anomaly detection method, system, device and medium to solve one or more of the above technical problems. The weakly supervised video anomaly detection method provided by the present application can accurately detect anomalies (including simple anomalies and difficult anomalies) in a video.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] The first aspect of the present application provides a weakly supervised video anomaly detection method, comprising the following steps:

[0007] obtaining a video frame feature sequence to be detected;

[0008] based on the obtained video frame feature sequence to be detected, performing video anomaly detection using a pre-trained anomaly detection model to obtain a video anomaly detection result;

[0009] The training steps of the pre-trained anomaly detection model include:

[0010] (1) obtaining an abnormality proportion value λ small , an abnormality proportion value λ largeand a training sample set; wherein each training sample in the training sample set comprises T segment features of a normal video and T segment features of an abnormal video;

[0011] (2) taking the abnormal proportion value λ small , the training sample set as input, training the first anomaly detection network to reach a preset convergence condition, and obtaining the trained first anomaly detection network; wherein the loss function used is a multi-instance learning loss function based on an abnormal proportion;

[0012] (3) based on the trained first anomaly detection network, obtaining a high-confidence sample index and a pseudo label thereof; taking the abnormal proportion value λ large , the high-confidence sample index and the pseudo label thereof, and the training sample set as input, training the second anomaly detection network to reach a preset convergence condition, and obtaining the trained second anomaly detection network; wherein the loss function used is a combined loss of a multi-instance learning loss based on an abnormal proportion and a classification loss;

[0013] (4) when training the third anomaly detection network to the Kth anomaly detection network, for the kth anomaly detection network, based on the trained (k-1) th anomaly detection network, obtaining an adaptive abnormal proportion value λ of an abnormal video in each training sample, a high-confidence sample index and a pseudo label thereof, taking the adaptive abnormal proportion value λ, the high-confidence sample index and the pseudo label thereof, and the training sample set as input, training the kth anomaly detection network to reach a preset convergence condition, and obtaining the trained kth anomaly detection network; wherein the loss function used is a combined loss of a multi-instance learning loss based on an abnormal proportion and a classification loss; and the finally trained Kth anomaly detection network is the pre-trained anomaly detection model.

[0014] Further improvements of the present application are that the network architectures of the first anomaly detection network to the Kth anomaly detection network are the same, and each comprises:

[0015] a time sequence convolution layer for inputting a video frame feature sequence to perform feature extraction and outputting extracted intermediate features;

[0016] a standard multi-head self-attention layer for inputting the intermediate features to perform feature fusion and outputting fused features;

[0017] a classification layer for inputting the fused features to perform prediction and outputting a predicted video anomaly detection result.

[0018] Further improvements of the present application are that step (2) specifically comprises the following steps:

[0019] inputting the training sample set into the first anomaly detection network to obtain predicted anomaly scores of normal video clips and predicted anomaly scores of abnormal video clips; calculating an anomaly proportion-based multi-instance learning loss based on the anomaly proportion value λ small and the predicted anomaly scores, and optimizing network parameters to obtain the trained first anomaly detection network;

[0020] The expression of the anomaly proportion-based multi-instance learning loss function is as follows.

[0021]

[0022] In the formula, r + and r - are predicted anomaly proportions of normal and abnormal videos respectively; α is a weight of normal and abnormal video constraints; λ small is a given anomaly proportion value; is a predicted anomaly score of the tth video clip of an abnormal video, is a predicted anomaly score of the tth video clip of a normal video, and T is the length of a video clip in a video.

[0023] The further improvement of the application is that step (3) specifically comprises the following steps:

[0024] Based on the trained first anomaly detection network, high-confidence sample indexes and pseudo labels thereof are obtained; the training sample set is input into a second anomaly detection network to obtain predicted anomaly scores of normal video clips and predicted anomaly scores of abnormal video clips; an anomaly proportion-based multi-instance learning loss is calculated based on the anomaly proportion value λ large and the predicted anomaly scores; a classification loss is calculated based on the predicted anomaly scores and the obtained high-confidence sample indexes and pseudo labels thereof; a combination of the anomaly proportion-based multi-instance learning loss and the classification loss is taken as a total loss function, network parameters are optimized, and a trained second anomaly detection network is obtained;

[0025] The expression of the total loss function is as follows.

[0026]

[0027]

[0028]

[0029] In the formula, s is a binary cross-entropy loss; β is a hyperparameter for balancing the contributions of the AR-MIL loss and the classification loss; s i is a predicted anomaly score of the ith video clip; y iis the pseudo label of the i-th video segment; N is the total number of pseudo label samples; is the abnormal proportion based multi-instance loss; r + and r - are the predicted abnormal proportions of normal and abnormal videos respectively; alpha is the weight of normal and abnormal video constraints; lambda large is the given abnormal proportion value; is the predicted abnormal score of the t-th video segment of the abnormal video, is the predicted abnormal score of the t-th video segment of the normal video, and T is the length of the video segment in a video.

[0030] Further improvement of the application is that step (4) specifically comprises the following steps:

[0031] For the k-th anomaly detection network, based on the trained (k-1)-th anomaly detection network, the adaptive abnormal proportion value lambda of the abnormal video in each training sample, the high confidence sample index and its pseudo label are obtained; the training sample set is input into the k-th anomaly detection network to obtain the predicted abnormal score of the normal video segment and the abnormal score of the abnormal video segment; the abnormal proportion based multi-instance learning loss is calculated based on the adaptive abnormal proportion value lambda and the predicted abnormal score; the classification loss is calculated based on the predicted abnormal score and the obtained high confidence sample index and its pseudo label; the combination of the abnormal proportion based multi-instance learning loss and the classification loss is taken as the total loss function, the network parameters are optimized, and the trained k-th anomaly detection network is obtained;

[0032] wherein the expression of the total loss function is,

[0033]

[0034]

[0035]

[0036] wherein, is a binary cross-entropy loss; beta is a hyperparameter for balancing the contribution of AR-MIL loss and classification loss; s i is the predicted abnormal score of the i-th video segment; y i is the pseudo label of the i-th video segment; N is the total number of pseudo label samples; is the abnormal proportion based multi-instance loss; r + and r - are the predicted abnormal proportions of normal and abnormal videos respectively; alpha is the weight of normal and abnormal video constraints; lambda is the given abnormal proportion value; is the predicted abnormal score of the t-th video segment of the abnormal video, is the predicted anomaly score of the t-th video segment of the normal video, and T is the length of the video segment in a video.

[0037] The further improvement of the present application is that the step of obtaining the high-confidence sample index and the pseudo label thereof specifically comprises:

[0038] inputting the abnormal video feature sequence into the trained anomaly detection network to obtain an anomaly score of the video segment, sorting the anomaly scores in descending order, marking the video segment with an anomaly score in the upper quartile as abnormal, and marking the video segment with an anomaly score in the lower quartile as normal; wherein, in addition to the samples in the abnormal video, the pseudo label samples also include samples in the normal video.

[0039] The further improvement of the present application is that the step of obtaining the adaptive anomaly proportion value λ of the abnormal video in each training sample specifically comprises:

[0040] inputting the abnormal video feature sequence into the trained anomaly detection network to obtain an anomaly detection result, setting the adaptive anomaly proportion value λ as the predicted anomaly proportion at the threshold value μ, and the expression is,

[0041]

[0042] In the formula, is the predicted anomaly score of the t-th video segment of the abnormal video; μ is a threshold value; and T is the length of the video segment in a video. is an indicator function, and when outputs 1 when greater than μ, and otherwise outputs 0.

[0043] The second aspect of the present application provides a weakly supervised video anomaly detection system, comprising:

[0044] a data acquisition module configured to acquire a video frame feature sequence to be detected;

[0045] a result acquisition module configured to perform video anomaly detection based on the acquired video frame feature sequence to be detected, using a pre-trained anomaly detection model, and obtain a video anomaly detection result;

[0046] The training steps of the pre-trained anomaly detection model comprise:

[0047] (1) obtaining an anomaly proportion value λ small , an anomaly proportion value λ large , and a training sample set; wherein each training sample in the training sample set comprises T segment features of a normal video and T segment features of an abnormal video.

[0048] (2) setting the anomaly proportion value λsmall , the first anomaly detection network is trained with the training sample set as input until a preset convergence condition is reached, and a trained first anomaly detection network is obtained;The loss function used is a multi-instance learning loss function based on abnormal proportion;

[0049] (3) Based on the trained first anomaly detection network, high confidence sample index and its pseudo label are obtained;With abnormal proportion value λ large , high confidence sample index and its pseudo label, and training sample set as input, the second anomaly detection network is trained until a preset convergence condition is reached, and a trained second anomaly detection network is obtained;The loss function used is a combination loss of multi-instance learning loss and classification loss based on abnormal proportion;

[0050] (4) when training the third anomaly detection network to the Kth anomaly detection network, for the kth anomaly detection network, the adaptive abnormal proportion value λ of the abnormal video in each training sample is obtained based on the trained k-1th anomaly detection network, the high confidence sample index and its pseudo label, the adaptive abnormal proportion value λ, the high confidence sample index and its pseudo label, and the training sample set are input to the kth anomaly detection network, and the kth anomaly detection network is trained until a preset convergence condition is reached, and a trained kth anomaly detection network is obtained;The loss function used is a combination loss of multi-instance learning loss and classification loss based on abnormal proportion;The final trained Kth anomaly detection network is the pre-trained anomaly detection model.

[0051] The third aspect of the present application provides an electronic device, comprising:

[0052] at least one processor;And

[0053] the memory connected in communication with the at least one processor;Wherein

[0054] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the weakly supervised video anomaly detection method according to any one of the first aspect of the present application.

[0055] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the weakly supervised video anomaly detection method according to any one of the first aspect of the present application.

[0056] Compared with the prior art, the present application has the following beneficial effects:

[0057] In the technical solution provided by the application, the multi-instance learning loss function based on the abnormal proportion and the multi-stage self-training framework guided by the abnormal proportion can accurately detect the abnormalities (including simple abnormal frames and difficult abnormal frames) in the video. Specifically, the loss function based on the abnormal proportion can ensure that more difficult abnormal samples are mined when the abnormal proportion in the video is less than a given value. The multi-stage self-training framework guided by the abnormal proportion is the core of the application and is divided into three stages. The self-starting stage enables the trained model to learn the characteristics of simple abnormal samples and detect simple abnormal samples. The difficult abnormal sample mining stage enables the trained model to mine difficult abnormal samples. The adaptive self-training stage enables the trained model to have more accurate judgment of abnormalities. The three stages are progressive and gradually detect the abnormalities in the video, including simple abnormalities and difficult abnormal samples. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following briefly introduces the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0059] Figure 1 is a flowchart of a weakly supervised video anomaly detection method provided by an embodiment of the present application;

[0060] Figure 2 is a flowchart of model training in an embodiment of the present application;

[0061] Figure 3 is a structure diagram of a multi-stage self-training framework (ARMS) guided by the abnormal proportion in an embodiment of the present application; wherein, Figure 3 (a) in the multi-stage self-training process guided by the abnormal proportion, Figure 3 (b) is a multi-instance learning (AR-MIL) loss function diagram based on the abnormal proportion, Figure 3 (c) is a pseudo-label generation process diagram, Figure 3 (d) is a self-adaptive abnormal proportion generation process diagram;

[0062] Figure 4 is a partial result diagram on the UCF-Crime and ShanghaiTech data sets in an embodiment of the present application; wherein, Figure 4 (a) is a prediction result diagram of RoadAccidents004, Figure 4 (b) is a prediction result diagram of Explosion010,Figure 4 Fig. 2 is a schematic diagram of a prediction result of Fighting018 in the embodiment of the present application;

[0063] Figure 5 Fig. 1 is a schematic diagram of a weakly supervised video anomaly detection system provided by the embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0065] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0066] The present application will be described in further detail below with reference to the drawings:

[0067] Referring to Figure 1 The weakly supervised video anomaly detection method disclosed in the embodiment of the present application comprises the following steps:

[0068] Step 1: Obtain a video frame feature sequence to be detected.

[0069] Step 2: Based on the video frame feature sequence to be detected obtained in step 1, use a pre-trained anomaly detection model to perform video anomaly detection, and obtain a video anomaly detection result.

[0070] In the technical solution provided by the embodiment of the present application, the multi-instance learning loss function based on the abnormal proportion and the multi-stage self-training framework guided by the abnormal proportion can accurately detect the anomalies (including simple anomaly frames and difficult anomaly frames) existing in the video.

[0071] In the embodiment of the present application, the training and obtaining step of the pre-trained anomaly detection model comprises:

[0072] obtain a training sample set and an abnormality proportion value λ small obtain a training sample set and an abnormality proportion value λ large ; each training sample comprises T segment features of a normal video and T segment features of an abnormal video

[0073] obtain a training sample set and an abnormality proportion value λ small , the training sample set and the abnormality proportion value λ

[0074] The first anomaly detection network comprises:

[0075] a time sequence convolution layer configured to input a video frame feature sequence to perform feature extraction and output extracted intermediate features;

[0076] a standard multi-head self-attention layer configured to input the intermediate features to perform feature fusion and output fused features;

[0077] a classification layer configured to input the fused features to perform prediction and output a predicted video anomaly detection result;

[0078] The loss function used in the training of the first anomaly detection network is an abnormality proportion-based multi-instance learning (AR-MIL) loss function.

[0079] obtain high-confidence sample indexes and pseudo labels thereof based on the trained first anomaly detection network; obtain a training sample set and an abnormality proportion value λ large , the high-confidence sample indexes and the pseudo labels thereof, and the training sample set as inputs to train a second anomaly detection network until a preset convergence condition is reached to obtain a trained second anomaly detection network.

[0080] The second anomaly detection network comprises:

[0081] a time sequence convolution layer configured to input a video frame feature sequence to perform feature extraction and output extracted intermediate features;

[0082] a standard multi-head self-attention layer configured to input the intermediate features to perform feature fusion and output fused features;

[0083] a classification layer configured to input the fused features to perform prediction and output a predicted video anomaly detection result;

[0084] The loss function used in the training of the second anomaly detection network is a combined loss of an abnormality proportion-based multi-instance learning (AR-MIL) loss and a classification loss.

[0085] ​In the training of the third to the Kth anomaly detection network, for the kth anomaly detection network, based on the trained (k-1)th anomaly detection network, the adaptive anomaly proportion value λ, the high-confidence sample index and the pseudo label of the abnormal video in each training sample are obtained, and the trained (k-1)th anomaly detection network, the adaptive anomaly proportion value λ, the high-confidence sample index and the pseudo label, and the training sample set are taken as inputs to train the kth anomaly detection network until a preset convergence condition is reached, and the trained kth anomaly detection network is obtained; wherein the loss function used is a combined loss of the multi-instance learning loss based on the anomaly proportion and the classification loss.

[0086] The final trained Kth anomaly detection network is the pre-trained anomaly detection model;

[0087] The final anomaly detection model comprises:

[0088] A time sequence convolution layer is configured to input a video frame feature sequence for feature extraction, and output extracted intermediate features;

[0089] A standard multi-head self-attention layer is configured to input the intermediate features for feature fusion, and output fused features;

[0090] A classification layer is configured to input the fused features for prediction, and output a predicted video anomaly detection result.

[0091] Referring to Figure 3 In the anomaly detection model, the AR-MIL loss forces at least a proportion λ of abnormal samples in the abnormal video, and no abnormal samples in the normal video, and the expression is:

[0092]

[0093]

[0094] wherein, r + and r - are the predicted anomaly proportions of the normal and abnormal videos, λ is the set minimum anomaly proportion of the abnormal video, and α is the weight of the normal and abnormal video constraints; is the predicted anomaly score of the tth video segment of the abnormal video, is the predicted anomaly score of the tth video segment of the normal video, and T is the length of the video segment in a video. In explanation, the first term of the AR-MIL loss controls that the anomaly proportion of the normal video should be as small as possible, because there is no abnormal frame in the normal video, and the second term controls that the proportion of the abnormal frame in the abnormal video should not be less than λ; the AR-MIL loss can realize the mining of the specified proportion of abnormal samples, and further has the ability to mine difficult abnormal samples.

[0095] Referring toFigure 3 In the embodiment of the present application, the method for obtaining high-confidence sample indexes and their pseudo labels comprises the following steps:

[0096] The abnormal video feature sequence is input into the trained abnormality detection network to obtain abnormality scores of video clips in the video The abnormality scores are sorted in descending order, and the video clips with abnormality scores in the upper quartile are marked as abnormal, and the video clips with abnormality scores in the lower quartile are marked as normal. In addition, in addition to the samples in the abnormal video, the pseudo label samples also include samples in the normal video, and the pseudo labels of all pseudo label samples are marked as Where N is the total number of pseudo label samples.

[0097] Please refer to Figure 3 In the embodiment of the present application, the method for obtaining the adaptive abnormality proportion λ of each training abnormal video comprises the following steps:

[0098] The abnormal video feature sequence is input into the trained abnormality detection network to obtain abnormality detection results The adaptive abnormality proportion λ is set as the predicted abnormality proportion under the threshold μ:

[0099]

[0100] Where, is the predicted abnormality score of the tth video clip of the abnormal video; μ is a threshold; T is the length of a video clip in a video; is an indicator function, which outputs 1 when and outputs 0 otherwise.

[0101] In the embodiment of the present application, the step of obtaining the classification loss function comprises:

[0102] The training sample set is input into the abnormality detection network to obtain the predicted abnormality scores of normal video clips and the abnormality scores of abnormal video clips The training sample set is input into the trained abnormality detection network to obtain high-confidence sample indexes and their pseudo labels According to the video sample index and the predicted abnormality score and The predicted abnormality score of the high-confidence clip is found The classification loss is calculated:

[0103]

[0104] In the embodiment of the application, the combination of the abnormal proportion-based multi-instance learning (AR-MIL) loss and the classification loss is as follows:

[0105]

[0106] wherein, is a binary cross-entropy loss, s i is the predicted abnormal score of the i-th video segment, y i is the pseudo label of the i-th segment, β is a hyperparameter for balancing the contributions of the AR-MIL loss and the classification loss, and N is the total number of pseudo label samples.

[0107] For details, please refer to Figure 2 and Figure 3 , in order to mine difficult abnormal samples while detecting simple abnormal samples, the embodiment of the application specifically discloses a weakly supervised video anomaly detection method, which comprises the following steps:

[0108] Step 1, collecting a training sample set; wherein each training sample comprises T segment features of a normal video and T segment features of an abnormal video;

[0109] Step 2, in the self-starting phase, according to the training sample set provided in step 1 and a given abnormal proportion value λ small , a trained first anomaly detection network is obtained; specifically, it can specifically comprise:

[0110] 2.1) constructing an anomaly detection network, inputting the training sample set obtained in step 1 into the anomaly detection network to obtain the predicted abnormal scores of the normal video segments and the predicted abnormal scores of the abnormal video segments;

[0111] 2.2) giving an abnormal proportion value λ small , calculating an abnormal proportion-based multi-instance learning (AR-MIL) loss based on λ small , the predicted abnormal scores obtained in step 2.1), and optimizing the network parameters to obtain the trained first anomaly detection network;

[0112] Step 3, difficult abnormal sample mining phase, using the trained first anomaly detection network obtained in step 2 to obtain high-confidence sample indexes and their pseudo labels; given an abnormal proportion value λ large , the high-confidence sample indexes and their pseudo labels and the training sample set provided in step 1 are used together to train a second anomaly detection network to obtain a trained second anomaly detection network capable of mining difficult abnormal samples; specifically, it can specifically comprise:

[0113] 3.1) using the trained first anomaly detection network in step 2 to obtain high-confidence sample indexes and their pseudo labels;

[0114] 3.2) Constructing the second anomaly detection network, inputting the training sample set obtained in step 1 into the second anomaly detection network to obtain the predicted anomaly scores of normal video clips and the predicted anomaly scores of abnormal video clips;

[0115] 3.3) Given the anomaly proportion value λ large , based on λ large , the predicted anomaly scores obtained in step 3.2), calculate the AR-MIL loss, based on the predicted anomaly scores obtained in step 3.2) and the high-confidence sample index and its pseudo label obtained in step 3.1), calculate the classification loss, the combination of the AR-MIL loss and the classification loss as the total loss function, optimize the network parameters, and obtain the trained second anomaly detection network capable of detecting difficult anomaly samples;

[0116] Step 4, adaptive self-training phase, using the second anomaly detection network trained in step 3 to obtain the adaptive anomaly proportion λ, and obtaining the high-confidence sample index and its pseudo label. The anomaly proportion value λ, the pseudo label sample index and its pseudo label, and the training sample set provided in step 1 are used together to train the third anomaly detection network, and the trained third anomaly detection network is obtained, and the current iteration round k is set to k+1; specific examples can specifically include:

[0117] 4.1) Using the anomaly detection network trained in step 3 to obtain the high-confidence sample index and its pseudo label;

[0118] 4.2) Using the anomaly detection network trained in step 3 to obtain the updated anomaly proportion λ;

[0119] 4.3) Constructing the third anomaly detection network, inputting the training sample set obtained in step 1 into the anomaly detection network to obtain the predicted anomaly scores of normal video clips and the predicted anomaly scores of abnormal video clips;

[0120] 4.4) Based on the anomaly proportion λ obtained in step 4.2, the anomaly scores obtained in step 4.3) to calculate the AR-MIL loss, based on the anomaly scores obtained in step 4.3) and the high-confidence sample index and its pseudo label obtained in step 4.1) to calculate the classification loss, the combination of the AR-MIL loss and the classification loss as the total loss function, optimize the network parameters, and obtain the trained anomaly detection network. Set the current iteration round k=k+1;

[0121] Step 5, adaptive self-training phase, repeatedly using the trained third anomaly detection network to generate high-confidence sample index and its pseudo label and anomaly proportion value λ, iteratively training a new anomaly detection network until k>K, and obtaining the trained third anomaly detection network as the final anomaly detection model.

[0122] In the embodiment of the present invention, the abnormality ratio λ small , the training sample set is taken as input, and the trained first anomaly detection model realizes the learning of detecting abnormal samples; based on the first anomaly detection model trained in the previous stage, the anomaly proportion λ large , training sample set, and the second anomaly detection model trained in the subsequent difficult anomaly sample mining stage realizes the mining of difficult anomaly samples; based on the second anomaly detection model and training sample set trained in the previous stage, the subsequent adaptive self-training stage realizes setting different anomaly proportions for different abnormal videos, and the trained third anomaly detection model realizes the detection of abnormal samples, including simple anomaly samples and difficult anomaly samples.

[0123] The core of a weakly supervised video anomaly detection method in an embodiment of the present invention is a multi-stage self-training framework guided by anomaly proportion. The steps are as follows:

[0124] Step 1: Acquire a training sample set, where each training sample includes: T segment features of a normal video and T segment features of an abnormal video.

[0125] 1.1) Collect and obtain the training sample set, process the video into video segments, and extract their corresponding features to obtain T segment features of a video in D represents the feature dimension.

[0126] In the embodiment of the present invention, a video is first divided into frames, and then the video frames are divided into segments with 16 frames as a segment. Next, a pre-trained feature extraction network is used to extract the features of the video segments, and finally T segments are selected from them to form the feature of the video. in D represents the feature dimension. In the embodiment of the present invention, the features of multiple segments of a normal video are represented as The features of multiple clips of an abnormal video are represented as

[0127] Step 2, the self-starting phase, based on the training sample set provided in step 1 and the given abnormality ratio value λ small , get the trained first anomaly detection network, and set the current iteration round k = 1:

[0128] 2.1) Construct a first anomaly detection network, input the training sample set obtained in step 1 into the anomaly detection network, and obtain predicted anomaly scores of normal video segments and abnormal video segments; exemplary methods may include:

[0129] 2.1.1) constructing a first anomaly detection network; the anomaly detection network is composed of a time convolution layer, a standard multi-head self-attention layer, and a classification layer;

[0130] 2.1.2) predicting anomaly scores of video clips; wherein the training sample set obtained in step 1 is input into the anomaly detection network constructed in step 2.1.1) to obtain predicted anomaly scores of normal video clips and anomaly scores of abnormal video clips

[0131] 2.2) Given an abnormality proportion value λ small , based on λ small , the predicted anomaly scores obtained in step 2.1), calculate the anomaly proportion-based multi-instance learning (AR-MIL) loss, and optimize the network parameters to obtain the trained anomaly detection network, set the current iteration round k = 1.

[0132] 2.2.1) Obtain the loss function value. Based on the anomaly scores obtained in step 2.1) and the given abnormality proportion λ small , calculate the anomaly proportion-based multi-instance learning (AR-MIL) loss function of the embodiment of the present application.

[0133] 2.2.2) Optimize network parameters. Based on the loss function obtained in step 2.2.1), optimize the network parameters to obtain the trained first anomaly detection network.

[0134] Step 3, difficult anomaly sample mining phase, using the trained first anomaly detection network obtained in step 2 to obtain high confidence sample index and its pseudo label, given the abnormality proportion value λ large , the high confidence sample index and its pseudo label and the training sample set provided in step 1 are used together to train a new anomaly detection network to obtain a trained second anomaly detection network capable of mining difficult anomaly samples. Set the current iteration round k = 2:

[0135] 3.1) Use the trained anomaly detection network in step 2 to obtain high confidence sample index and its pseudo label.

[0136] The pseudo label generation method proposed in the embodiment of the present application is as follows: input the features of multiple clips of an abnormal video obtained in step 1 into the trained anomaly detection network to obtain the predicted In the multiple clips of the abnormal video, the labels of the samples with anomaly scores in the upper quartile are marked as abnormal, and the samples with anomaly scores in the lower quartile are marked as normal, so that the high confidence pseudo label samples in the abnormal video are obtained. In addition, in addition to the samples in the abnormal video, the pseudo label samples also include samples in the normal video, and the pseudo labels of all pseudo label samples are marked as where N is the total number of pseudo-label samples.

[0137] 3.2) Constructing the second anomaly detection network, inputting the training sample set obtained in step 1 into the anomaly detection network to obtain the predicted anomaly scores of normal video clips and the predicted anomaly scores of abnormal video clips;

[0138] Constructing the second anomaly detection network, wherein the network structure is the same as that in step 2.1.1), and inputting the video clips obtained in step 1 into the network to obtain the predicted anomaly scores of normal video clips and the predicted anomaly scores of abnormal video clips

[0139] 3.3) Given the anomaly proportion value λ large , calculate the AR-MIL loss based on λ large , the predicted anomaly scores obtained in step 3.2), calculate the classification loss based on the predicted anomaly scores obtained in step 3.2) and the high-confidence sample index and its pseudo-label obtained in step 3.1), and combine the AR-MIL loss and the classification loss as the total loss function, optimize the network parameters, and obtain the trained second anomaly detection network capable of detecting difficult anomaly samples. Set the current iteration round k = 2.

[0140] 3.3.1) Based on the anomaly scores obtained in step 3.2) and the given anomaly proportion λ large , calculate the anomaly proportion-based multiple instance learning (AR-MIL) loss of the embodiment of the present application.

[0141] 3.3.2) Based on the anomaly scores obtained in step 3.2) and the high-confidence sample index and its pseudo-label obtained in step 3.1 find the predicted anomaly scores of high-confidence clips Next, calculate the classification loss:

[0142]

[0143] 3.3.3) Combine the AR-MIL loss and the classification loss as the total loss function, and the formula is as follows:

[0144]

[0145] where, is a binary cross-entropy loss, s i is the predicted anomaly score of a video clip, y i is the pseudo-label of the clip, β is a hyperparameter for balancing the contributions of the AR-MIL loss and the classification loss, and N is the total number of pseudo-label samples.

[0146] Based on the obtained loss function, the network parameters are optimized to obtain a trained abnormality detection network capable of detecting difficult abnormal samples.

[0147] Step 4, adaptive self-training phase, obtaining adaptive abnormality proportion λ using the second abnormality detection network trained in step 3, and obtaining high-confidence sample index and its pseudo label. The abnormality proportion value λ, the high-confidence sample index and its pseudo label, and the training sample set provided in step 1 are used to train a new third abnormality detection network, and a trained third abnormality detection network is obtained, and the current iteration round k is set to k+1:

[0148] 4.1) Obtain high-confidence sample index and its pseudo label using the second abnormality detection network trained in step 3.

[0149] Here, the high-confidence sample index and its pseudo label are obtained in the same way as step 3.1) Where N is the total number of pseudo label samples, and the model used to obtain the prediction result is replaced by the abnormality detection network trained in step 3.

[0150] 4.2) Obtain updated abnormality proportion λ using the second abnormality detection network trained in step 3.

[0151] The updating method of the abnormality proportion λ proposed in the embodiment of the application is as follows: input the features of multiple segments of an abnormal video obtained in step 1 into the abnormality detection network trained in step 3, obtain the predicted abnormality score of the abnormal segment Set the updated λ as the predicted abnormality proportion under the threshold μ:

[0152]

[0153] Wherein, is the predicted abnormality score of the tth video segment of the abnormal video; μ is a threshold; T is the length of the video segment in a video; is an indicator function, when Output 1 when greater than μ, otherwise output 0. For example, the abnormality proportion of the last iteration round is directly used as the threshold μ here.

[0154] 4.3) Construct a new third abnormality detection network, input the training sample set obtained in step 1 into the abnormality detection network, and obtain the predicted abnormality score of the normal video segment and the predicted abnormality score of the abnormal video segment.

[0155] Construct a new abnormality detection network, wherein the network structure is the same as that in step 2.1.1), and input the video segment obtained in step 1 into the network to obtain the predicted abnormality score of the normal video segment and the predicted abnormality score of the abnormal video segment

[0156] 4.4) Calculate the AR-MIL loss based on the abnormal proportion λ obtained in step 4.2 and the abnormal score obtained in step 4.3, calculate the classification loss based on the abnormal score obtained in step 4.3 and the high-confidence sample index and its pseudo label obtained in step 4.1, the combination of the AR-MIL loss and the classification loss as the total loss function, optimize the network parameters, and obtain the trained anomaly detection network. Set the current iteration round k = k + 1.

[0157] 4.4.1) Calculate the abnormal proportion-based multi-instance learning (AR-MIL) loss of the embodiment of the application based on the abnormal score obtained in step 4.3 and the abnormal proportion λ obtained in step 4.2.

[0158] 4.4.2) Calculate the classification loss based on the abnormal score obtained in step 4.3 and the high-confidence sample index and its pseudo label obtained in step 4.1

[0159] 4.4.3) The combination of the AR-MIL loss and the classification loss as the total loss function, wherein the expression of the loss function is the same as in step 3.3.3. Based on the obtained loss function, optimize the network parameters, and obtain the trained anomaly detection network.

[0160] Step 5, adaptive self-training phase, repeatedly use the trained anomaly detection network to generate high-confidence sample index and its pseudo label and abnormal proportion value λ, iteratively train a new anomaly detection network until k > K, obtain the final anomaly detection network, and its corresponding anomaly detection result.

[0161] ​In the embodiment of the present application, in view of the problem that the existing method cannot detect difficult abnormal samples, a multi-instance learning loss function based on abnormal proportion and a new weakly supervised video anomaly detection method are specifically proposed. The multi-instance learning loss function based on abnormal proportion (AR-MIL) forces at least a certain proportion of abnormal frames in an abnormal video, and no abnormal frames in a normal video to guide the learning of difficult abnormal samples. The core of the new weakly supervised video anomaly detection method is a multi-stage self-training framework guided by abnormal proportion (ARMS), which includes three stages: self-starting stage, difficult abnormal sample mining stage, and adaptive self-training stage. The three stages respectively realize the learning of simple abnormal samples, difficult abnormal sample mining, and setting different abnormal proportions for different abnormal videos. The three stages are progressive, and gradually enable simple abnormal and difficult abnormal samples to be detected at the same time. The weakly supervised video anomaly detection is realized. In the multi-instance learning loss function based on abnormal proportion of the embodiment of the present application, the abnormal scores of the predicted normal video segments and the abnormal video segments, and the video labels and abnormal proportion values λ are obtained; the multi-instance learning loss function based on abnormal proportion (AR-MIL) is calculated by using the provided predicted abnormal scores and the given abnormal proportion λ. The multi-instance learning loss function based on abnormal proportion of the present application forces at least a certain proportion of abnormal samples in an abnormal video, and no abnormal samples in a normal video, which can realize the mining of abnormal samples with a specified proportion, and further has the ability to mine difficult abnormal samples.

[0162] In summary, the embodiment of the present application specifically proposes a new weakly supervised video anomaly detection method in view of the problem that the existing method cannot detect difficult abnormal samples. The core of the embodiment of the present application is a multi-stage self-training (ARMS) framework guided by abnormal proportion, which includes three stages: self-starting stage, difficult abnormal sample mining stage, and adaptive self-training stage. The three stages respectively realize the learning of simple abnormal samples, difficult abnormal sample mining, and setting different abnormal proportions for different abnormal videos. The three stages are progressive, and gradually enable simple abnormal and difficult abnormal samples to be detected at the same time, realizing weakly supervised video anomaly detection. The weakly supervised video anomaly detection method of the embodiment of the present application is compared with the existing method through qualitative and quantitative experimental analysis, and the effectiveness of the method is verified on two public data sets UCF-Crime and ShanghaiTech.

[0163] Table 1 is the quantitative experimental results of the weakly supervised video anomaly detection method of the embodiment of the present application on UCF-Crime and ShanghaiTech. It can be seen that the method of the embodiment of the present application is optimal in performance on the two data sets, reaching 85.79% AUC on UCF-Crime and 97.48% AUC on ShanghaiTech. Compared with the baseline method, the method of the embodiment of the present application has a very obvious improvement. Therefore, from the quantitative results, the weakly supervised video anomaly detection method of the embodiment of the present application can detect accurate abnormal samples, including difficult abnormal samples and simple abnormal samples.

[0164] Table 1. Experimental results of the method on UCF-Crime and ShanghaiTech data sets

[0165]

[0166] Please refer to Figure 4 , Figure 4 is the qualitative experimental results of the weakly supervised video anomaly detection method of the embodiment of the present application, that is, the anomaly detection results on the UCF-Crime test set. It can be seen that the method of the embodiment of the present application can accurately detect normal and abnormal. Figure 4 In (a) of, the method of the embodiment of the present application accurately detects the difficult abnormality (abnormal driving speed) and the simple abnormality (vehicle collision) in the traffic accident event, but the baseline method cannot detect the abnormality in the video. Figure 4 In (b) of, the method of the embodiment of the present application completely predicts all abnormal frames of the explosion event, while the abnormal frames predicted by the baseline method are incomplete. Figure 4 In (c) of, the method of the embodiment of the present application predicts all abnormal frames of the fighting event, but the baseline method still has incomplete prediction. Therefore, from the qualitative results, the weakly supervised video anomaly detection method of the embodiment of the present application can detect accurate abnormal samples, especially difficult abnormal samples.

[0167] The following is the device embodiment of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the device embodiment, please refer to the method embodiment of the present application.

[0168] Please refer to Figure 5 , the weakly supervised video anomaly detection system provided by the embodiment of the present application comprises:

[0169] The data acquisition module is configured to acquire a video frame feature sequence to be detected.

[0170] The result acquisition module is configured to perform video anomaly detection by using a pre-trained anomaly detection model based on the acquired video frame feature sequence to be detected, and obtain a video anomaly detection result.

[0171] The training step of the pre-trained anomaly detection model comprises:

[0172] (1) obtaining an anomaly proportion value λ small , an anomaly proportion value λ large , and a training sample set; wherein each training sample in the training sample set comprises T segment features of a normal video and T segment features of an abnormal video;

[0173] (2) training a first anomaly detection network by taking the anomaly proportion value λ small , the training sample set as input, until a preset convergence condition is reached, to obtain a trained first anomaly detection network; wherein a loss function used is a multi-instance learning loss function based on the anomaly proportion;

[0174] (3) based on the trained first anomaly detection network, obtaining high-confidence sample indexes and pseudo labels thereof; taking the anomaly proportion value λ large , the high-confidence sample indexes and the pseudo labels thereof, and the training sample set as input, training a second anomaly detection network until a preset convergence condition is reached, to obtain a trained second anomaly detection network; wherein a loss function used is a combined loss of a multi-instance learning loss based on the anomaly proportion and a classification loss;

[0175] (4) when training the third anomaly detection network to the Kth anomaly detection network, for the kth anomaly detection network, obtaining an adaptive anomaly proportion value λ , high-confidence sample indexes and pseudo labels thereof of an abnormal video in each training sample based on the trained (k-1)th anomaly detection network, taking the adaptive anomaly proportion value λ , the high-confidence sample indexes and the pseudo labels thereof, and the training sample set as input, training the kth anomaly detection network until a preset convergence condition is reached, to obtain a trained kth anomaly detection network; wherein a loss function used is a combined loss of a multi-instance learning loss based on the anomaly proportion and a classification loss; and the finally trained Kth anomaly detection network is the pre-trained anomaly detection model.

[0176] In another embodiment of the present application, a computer device is provided, which comprises a processor and a memory, the memory is configured to store a computer program, the computer program comprises program instructions, and the processor is configured to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the computer storage medium to implement a corresponding method process or a corresponding function; the processor in the embodiments of the present application can be used for the operation of the weakly supervised video anomaly detection method.

[0177] In another embodiment of the present application, the present application further provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in the computer device, and is configured to store programs and data. It can be understood that the computer readable storage medium herein can include the built-in storage medium in the computer device, and of course can also include the expansion storage medium supported by the computer device. The computer readable storage medium provides a storage space, and the storage space stores the operating system of the terminal. Moreover, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and the instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory. One or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the weakly supervised video anomaly detection method in the above embodiments.

[0178] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0179] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0180] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0182] Finally, it should be noted that the above-mentioned embodiments are merely intended for describing and illustrating, not limiting the technical solutions of the present application. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. A weakly supervised video anomaly detection method, characterized in that: The following steps are involved: Obtaining a video frame feature sequence to be detected; Based on the acquired video frame feature sequence to be detected, using a pre-trained anomaly detection model to perform video anomaly detection to obtain a video anomaly detection result; The training steps of the pre-trained anomaly detection model include: (1) Obtain the abnormal proportion value λ small , abnormal proportion value λ large And a training sample set; wherein each training sample in the training sample set includes T segment features of a normal video and T segment features of an abnormal video; (2) Taking the abnormal proportion value λ small , the training sample set is used as input, the first anomaly detection network is trained to achieve a preset convergence condition, and a trained first anomaly detection network is obtained; wherein the loss function used is a multi-instance learning loss function based on the anomaly ratio; (3) Based on the trained first anomaly detection network, obtain the high confidence sample index and its pseudo label; take the anomaly ratio value λ large , high-confidence sample indexes and their pseudo labels, and training sample sets are used as input to train the second anomaly detection network until a preset convergence condition is reached, thereby obtaining a trained second anomaly detection network; wherein the loss function used is a combination of multi-instance learning loss based on anomaly proportion and classification loss; (4) When training the third anomaly detection network to the Kth anomaly detection network, for the kth anomaly detection network, the adaptive anomaly ratio value λ, the high confidence sample index and its pseudo-label of the abnormal video in each training sample are obtained based on the trained k-1th anomaly detection network, and the kth anomaly detection network is trained with the adaptive anomaly ratio value λ, the high confidence sample index and its pseudo-label, and the training sample set as input to achieve the preset convergence condition and obtain the trained kth anomaly detection network; wherein the loss function adopted is a combination loss of multi-instance learning loss based on anomaly ratio and classification loss; and the finally trained Kth anomaly detection network is the pre-trained anomaly detection model.

2. A weakly supervised video anomaly detection method according to claim 1, characterized in that The network architectures of the first anomaly detection network to the Kth anomaly detection network are the same, and all include: Temporal convolution layer, which is used to input the video frame feature sequence for feature extraction and output the extracted intermediate features; A standard multi-head self-attention layer is used to input the intermediate features for feature fusion and output the fused features; The classification layer is used to input the fused features for prediction and output the predicted video anomaly detection results.

3. A weakly supervised video anomaly detection method according to claim 1, characterized in that: Step (2) specifically includes the following steps: The training sample set is input into the first anomaly detection network to obtain the predicted anomaly scores of normal video segments and abnormal video segments; based on the anomaly proportion value λ small and the predicted anomaly score, calculate the multi-instance learning loss based on the anomaly ratio, and optimize the network parameters to obtain the trained first anomaly detection network; Among them, the multi-instance learning loss function based on the abnormal proportion The expression is, Where r + and r - are the abnormal proportions of predicted normal and abnormal videos respectively; α is the weight of normal and abnormal video constraints; is the predicted anomaly score of the tth video clip of the anomaly video, is the predicted anomaly score of the t-th video segment of a normal video, and T is the length of the video segment in a video.

4. A weakly supervised video anomaly detection method according to claim 1, characterized in that Step (3) specifically includes the following steps: Based on the trained first anomaly detection network, high confidence sample indexes and their pseudo labels are obtained; the training sample set is input into the second anomaly detection network to obtain the predicted anomaly scores of normal video segments and abnormal video segments; based on the anomaly proportion value λ large The multi-instance learning loss based on anomaly proportion is calculated based on the predicted anomaly score and the obtained high-confidence sample index and its pseudo-label. The combination of the multi-instance learning loss based on anomaly proportion and the classification loss is used as the total loss function to optimize the network parameters and obtain a trained second anomaly detection network. Among them, the expression of the total loss function is: Where, is a binary cross entropy loss; β is a hyperparameter used to balance the contribution of AR-MIL loss and classification loss; s i is the predicted anomaly score of the i-th video clip; y i is the pseudo label of the i-th video clip; N is the total number of pseudo label samples; is the multi-instance loss based on the anomaly ratio; r + and r - are the abnormal proportions of predicted normal and abnormal videos respectively; α is the weight of normal and abnormal video constraints; is the predicted anomaly score of the tth video clip of the anomaly video, is the predicted anomaly score of the t-th video segment of a normal video, and T is the length of the video segment in a video.

5. The weakly supervised video anomaly detection method according to claim 1, characterized in that: Step (4) specifically includes the following steps: For the kth anomaly detection network, based on the trained k-1th anomaly detection network, the adaptive anomaly ratio value λ, the high-confidence sample index and its pseudo-label of the abnormal video in each training sample are obtained; the training sample set is input into the kth anomaly detection network to obtain the predicted anomaly score of the normal video segment and the anomaly score of the abnormal video segment; the multi-instance learning loss based on the anomaly ratio is calculated based on the adaptive anomaly ratio value λ and the predicted anomaly score; the classification loss is calculated based on the predicted anomaly score and the obtained high-confidence sample index and its pseudo-label; the combination of the multi-instance learning loss based on the anomaly ratio and the classification loss is used as the total loss function to optimize the network parameters and obtain the trained kth anomaly detection network; Among them, the expression of the total loss function is: Where, is a binary cross entropy loss; β is a hyperparameter used to balance the contribution of AR-MIL loss and classification loss; s i is the predicted anomaly score of the i-th video clip; y i is the pseudo label of the i-th video clip; N is the total number of pseudo label samples; is the multi-instance loss based on the anomaly ratio; r + and r - are the abnormal proportions of predicted normal and abnormal videos respectively; α is the weight of normal and abnormal video constraints; is the predicted anomaly score of the tth video clip of the anomaly video, is the predicted anomaly score of the t-th video segment of a normal video, and T is the length of the video segment in a video.

6. A weakly supervised video anomaly detection method according to claim 1, characterized in that: The steps to obtain high-confidence sample indexes and their pseudo labels include: The abnormal video feature sequence is input into the trained anomaly detection network to obtain the anomaly score of the video clip. The anomaly score is sorted in descending order. Video clips with anomaly scores in the upper quartile are marked as abnormal, and video clips with anomaly scores in the lower quartile are marked as normal. In addition to samples in abnormal videos, pseudo-label samples also include samples in normal videos.

7. The weakly supervised video anomaly detection method according to claim 1, characterized in that: The steps of obtaining the adaptive abnormality ratio value λ of abnormal videos in each training sample specifically include: The abnormal video feature sequence is input into the trained anomaly detection network to obtain the anomaly detection result. The adaptive anomaly ratio value λ is set to the predicted anomaly ratio under the threshold μ, which is expressed as follows: Where, is the predicted anomaly score of the tth video clip of the anomaly video; μ is a threshold; T is the length of the video clip in a video; is an indicator function, when If it is greater than μ, output 1, otherwise output 0.

8. A weakly supervised video anomaly detection system, characterized in that: include: A data acquisition module is used to obtain a feature sequence of a video frame to be detected; A result acquisition module is used to perform video anomaly detection based on the acquired video frame feature sequence to be detected using a pre-trained anomaly detection model to obtain a video anomaly detection result; The training steps of the pre-trained anomaly detection model include: (1) Obtain the abnormal proportion value λ small , abnormal proportion value λ large And a training sample set; wherein each training sample in the training sample set includes T segment features of a normal video and T segment features of an abnormal video; (2) Taking the abnormal proportion value λ small , the training sample set is used as input, the first anomaly detection network is trained to achieve a preset convergence condition, and a trained first anomaly detection network is obtained; wherein the loss function used is a multi-instance learning loss function based on the anomaly ratio; (3) Based on the trained first anomaly detection network, obtain the high confidence sample index and its pseudo label; take the anomaly ratio value λ large , high-confidence sample indexes and their pseudo labels, and training sample sets are used as input to train the second anomaly detection network until a preset convergence condition is reached, thereby obtaining a trained second anomaly detection network; wherein the loss function used is a combination of multi-instance learning loss based on anomaly proportion and classification loss; (4) When training the third anomaly detection network to the Kth anomaly detection network, for the kth anomaly detection network, the adaptive anomaly ratio value λ, the high confidence sample index and its pseudo-label of the abnormal video in each training sample are obtained based on the trained k-1th anomaly detection network, and the kth anomaly detection network is trained with the adaptive anomaly ratio value λ, the high confidence sample index and its pseudo-label, and the training sample set as input to achieve the preset convergence condition and obtain the trained kth anomaly detection network; wherein the loss function adopted is a combination loss of multi-instance learning loss based on anomaly ratio and classification loss; and the finally trained Kth anomaly detection network is the pre-trained anomaly detection model.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the weakly supervised video anomaly detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the weakly supervised video anomaly detection method according to any one of claims 1 to 7 is implemented.

Images