Weak supervision video anomaly detection method based on course learning and contrast learning strategies

Abstract

The invention belongs to the field of artificial intelligence and video anomaly detection, and discloses a weak supervision video anomaly detection method based on curriculum learning and contrast learning strategies. A course learning strategy is designed by simulating a human learning mode, an anomaly detection model is trained through a contrast learning method based on a normal prototype, and the method comprises the following steps: preprocessing a video and extracting an optical flow image; video appearance and motion features are extracted and fused; inputting into a pre-classifier to obtain a pseudo label; judging the anomaly detection difficulty according to the false label, and dividing a simple set and a difficult set; initializing a model by using a simple set; and performing two-stage training by using the difficult set, and updating model parameters through normal prototype contrast learning. According to the invention, the discrimination capability of a weak supervision video anomaly detection model can be effectively improved, the problems of model deviation, fuzzy positive and abnormal boundaries and insufficient model discrimination capability under the influence of noise in the prior art are solved to a great extent, and the accuracy of video anomaly detection is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and video anomaly detection, and relates to a weakly supervised video anomaly detection method based on curriculum learning and contrast learning strategies. Background Art

[0002] Video anomaly detection refers to identifying behaviors or activities that are contrary to normal patterns from video data. With the advancement of communication technology, video is expected to further become the mainstream information dissemination medium in society, and the analysis of massive surveillance videos has gradually become an important research topic in the field of social security. As one of the core technologies of surveillance video analysis, video anomaly detection plays a vital role in maintaining public safety. Through video anomaly detection technology, not only can the probability of false detection and missed detection caused by manual identification be reduced, but also a large amount of manpower, material resources and financial resources can be consumed.

[0003] Video anomaly detection has several difficulties due to the particularity of the problem. Abnormal events have special essential characteristics such as openness, sparsity, suddenness, and fuzziness, which are not easy to detect. Specifically, openness refers to the diversity of abnormal events, and it is impossible to list all types one by one for classification and judgment. Sparsity means that the probability of abnormal events is significantly lower than that of normal events, and there is a significant imbalance between normal and abnormal data. Suddenness means that abnormal events occur suddenly and are difficult to predict. Fuzziness refers to the lack of clear definition standards between normal and abnormal samples. When dealing with video anomaly detection problems, it is necessary to model complex and high-dimensional video data, which is very challenging to deal with. In addition, when dealing with video anomaly detection tasks, it is also necessary to model high-dimensional video data. Video data contains rich image sequences, spatiotemporal information, etc. How to effectively extract and utilize this information and build a model that can accurately capture abnormal features is very challenging.

[0004] In the field of video anomaly detection, if supervised methods are used to annotate video data frame by frame, it will greatly consume manpower and material resources. Therefore, video anomaly detection is gradually developing in the direction of unsupervised and weakly supervised methods. Among them, weakly supervised methods have become the mainstream video anomaly detection methods due to their low annotation cost and high accuracy. Specifically, weakly supervised video anomaly detection means that only video-level labels are available in the training video, and there are no frame-level annotations, but the model needs to detect frame-level or segment-level anomalies. Among the current weakly supervised anomaly detection methods, the main one is based on multi-instance learning methods. Sultani et al. first proposed a deep multi-instance learning ranking detection paradigm, which regards the video features extracted by the pre-trained model as a package, and the segments in the video as examples in the package. Through the package-level labels, the ranking loss is used to make the examples with the highest abnormality in the positive package (abnormal video) and the examples with the highest abnormality in the negative package (normal video) as far away as possible. Here, the example with the highest abnormality in the positive bag can be regarded as the most likely abnormal segment, while the example with the highest abnormality in the negative bag can be regarded as a normal difficult sample (Sultani W, Chen C, Shah M. Real-World Anomaly Detection in Surveillance Videos[C] / / 2018IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2018: 6479-6488). On this basis, in order to better measure the degree of anomaly, Tian et al. introduced robust temporal feature magnitude learning, which trains the feature magnitude learning function to effectively identify positive examples, thereby significantly improving the robustness of the multi-instance learning method to negative examples of abnormal videos. At the same time, it also uses dilated convolution and self-attention mechanisms to capture long-distance and short-distance temporal dependencies (Tian Y, Pang G, Chen Y, et al. Weakly-supervised video anomaly detection with robust temporal feature magnitude learning [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 4975-4986). In addition, some methods specifically modify the objective function in the task framework to further constrain the search space so that there is a large gap between the lowest and highest scores of the input instances in the positive bag, while the corresponding gap in the negative bag is smaller. Or the video anomaly detection problem is regarded as an anomaly score regression task, and the difference between abnormal examples and normal examples is increased through dynamic multi-instance learning loss, and the difference in normal examples is reduced through center loss.

[0005] However, existing methods generally have model bias problems, which is closely related to the guidance of noise on the model. There must be a lot of noise under weak supervision conditions, and the current model chooses to focus only on the most representative features to avoid noise interference. However, since the model cannot clearly distinguish which data belongs to noise, this strategy may cause it to ignore some key information when it is executed, or mistakenly learn noise information as effective features, causing model bias and insufficient discrimination ability under the influence of noise. On the other hand, in the process of anomaly detection, the definition of abnormal events often lacks clear and definite criteria, resulting in unclear positive and abnormal boundaries, insignificant distinction between positive and abnormal events, and low accuracy and reliability of anomaly detection. For example, in some complex scenarios, some normal samples may be misjudged as abnormal because they have local features similar to abnormal samples, while some slightly abnormal samples may be missed because their overall features are close to normal samples. Therefore, how to better distinguish noise from effective features, avoid the model from being guided by noise and clarify the positive and abnormal boundaries is a problem we are concerned about.

[0006] Based on this, the present invention proposes a weakly supervised video anomaly detection method based on curriculum learning and contrastive learning strategies, formulates a selective pseudo-label curriculum learning strategy, alleviates the influence of sample noise under weak supervision conditions, avoids model bias, guides it to a better parameter space, and improves the quality of training; studies a contrastive learning method based on normal prototype, closely maps similar instances in the latent space according to the normal prototype knowledge, clarifies the decision boundary, and thus improves the accuracy of positive anomaly detection. Summary of the invention

[0007] In order to solve the above problems, the purpose of the present invention is to provide a weakly supervised video anomaly detection method based on curriculum learning and contrastive learning strategies. The method designs a training framework based on curriculum learning strategy and normal prototype contrastive learning, uses curriculum learning strategy to reduce model bias and improve the model's robustness to noise, and uses contrastive learning method based on normal prototype to expand the distance between normal and abnormal features, clarify the decision boundary, and improve the performance of video anomaly detection.

[0008] A weakly supervised video anomaly detection method based on curriculum learning and contrastive learning strategy includes the following steps:

[0009] Step 1: Divide the video into a training set and a test set, and split each video into RGB video frames according to its frame rate;

[0010] Step 2: Use the TV-LI algorithm to extract the corresponding optical flow image from the RGB video frame;

[0011] Step 3: Use the pre-trained I3D network (Inflated 3D ConvNet) as the feature extraction network to extract the appearance features and motion features of the videos in the training set;

[0012] The training video V in the training set includes abnormal videos and normal videos. Under weak supervision, the abnormal video is V a , the video level label is Y a =1, including abnormal and normal clips; normal video is V n , the video level label is Y n = 0, only normal segments are included; training video V = V a ∪V n , ∪ represents the union; according to the multi-instance learning method, each video in the training set is regarded as a different video package, and the clips in the video package are regarded as examples;

[0013] Each video packet is divided into Z non-overlapping video segments, and the i-th video segment is denoted as v i , using the I3D network pre-trained on the public dataset Kinetics-400 as the feature extraction network, the RGB image v of the input video clip i RGB To I3D network pre-trained on RGB images I3D RGB In the example, the optical flow image v of the input video clip is i Optical-Flow To the I3D network pre-trained on optical flow images I3D Optical-Flow The extracted RGB feature of the i-th video clip is The optical flow feature is Concatenate the two network output features to get the fusion feature x i , the calculation formula is as follows:

[0014]

[0015] in, Represents a splicing operation;

[0016] Normal video V n The feature representation of is a set of Z normal segment features Abnormal Video V a The feature representation of is a set of Z abnormal fragment features

[0017] Step 4: Use the pre-classifier to estimate the video anomaly score and generate segment-level pseudo labels for the video;

[0018] The pre-classifier is a 3-layer fully connected neural network. The first fully connected layer contains 512 units, the second fully connected layer contains 32 units, and the final output layer contains 1 unit. 60% dropout regularization is used between the fully connected layers. ReLU activation function and Sigmoid activation function are added after the first and last layers respectively. Adagrad optimizer with an initial learning rate of 0.001 is used. The pre-classifier is trained using binary cross entropy loss and deep ranking loss.

[0019] The binary cross entropy loss function is calculated as:

[0020]

[0021] Among them, y represents the video level label, *∈{n,a} represents normal and abnormal, represents the scores of normal and abnormal video clips after detection by the pre-classifier, and E[.] represents the mathematical expectation;

[0022] The maximum score of the abnormal clip in the abnormal video is restricted to be greater than the maximum score of the normal clip in the normal video, and the temporal smoothing between the abnormal scores is achieved by minimizing the score difference between adjacent video clips; the deep ranking loss L including the temporal smoothing calculation is rank The calculation formula is:

[0023]

[0024] Among them, max means taking the maximum value of a video score, x l Represents the characteristics of the current moment, x l+1 Indicates the characteristics of the next moment; (*) + It means taking the maximum value between the expression * and 0, ε is a hyperparameter representing the boundary, and α is a hyperparameter used to balance the sorting loss and the time smoothing part;

[0025] The total training loss L F (X) The calculation formula is:

[0026] L F (X) = BCE F (X)+λ1L rank (X) (4)

[0027] Among them, λ1 is a hyperparameter used to balance the binary cross entropy loss and the depth ranking loss;

[0028] The scores of each video after the pre-classifier, i.e., the segment-level pseudo-labels The normal video pseudo label set is The set of abnormal video pseudo labels is

[0029] Step 5: Define the difficulty judgment rule, make difficulty judgment based on the pseudo-label value, and divide the training set into a simple set S and a difficult set H;

[0030] Use the pseudo-labels obtained in step 4 to define the difficulty determination rules and perform difficulty determination;

[0031] Select t samples with anomaly scores closest to 1 from the abnormal video as the confident positive sample feature set S that is likely to be abnormal a ; Select t samples S with abnormal scores closest to 0 from normal videos n , as the confident negative sample feature set; the intersection of the confident positive sample feature set and the confident negative sample feature set is taken as the simple set S, and the remaining samples have more noise or are not significantly distinguished, the detection results are prone to deviation, and the recognition is difficult, so they are collected as the difficult set H; the difficult set H contains abnormal fragments H a and normal fragment H n ; The calculation formula for dividing the set according to the difficulty determination rule is as follows:

[0032]

[0033] S=S a ∪S n (6)

[0034] H a =X a -S a ,H n =X n -S n (7)

[0035] H=H a ∪H n (8)

[0036] in, represents the index corresponding to the t values ​​that make the expression * the smallest, i represents the i-th video clip, and - represents the relative complement;

[0037] Step 6: Construct a normal prototype vector set P, take the difficult set constructed in step 5 to initialize it, and calculate the average value of the normal data features in the difficult set as the normal prototype vector;

[0038] Establish a normal prototype vector set P = {p1, p2, ..., p k} to represent the representative features of normal samples, k is the number of normal prototype vectors; it will contain m n The normal sample difficulty set of samples is evenly divided into k subsets, each subset is used to initialize a normal prototype vector and continuously updated during training; the jth normal prototype vector is represented by pj ∈R d , d is the characteristic dimension, and the calculation formula is as follows:

[0039]

[0040] in, represents the sample in the i-th normal sample difficulty set, Indicates rounding up;

[0041] Step 7: Execute the selective pseudo-label course learning strategy to imitate the human learning process from easy to difficult knowledge. First, use the simple set constructed in step 5 to initialize the discriminant model and update the model parameters.

[0042] The discriminant model is set as a 3-layer fully connected neural network consistent with the pre-classifier. The discriminant model is trained using binary cross entropy loss and deep ranking loss. At this time, the input data of the discriminant model is a simple set; the total loss L F (S) is expressed as:

[0043] L F (S)=BCE F (S)+λ1L rank (S) (10)

[0044] Among them, λ1 is a hyperparameter used to balance the binary cross entropy loss and the depth ranking loss;

[0045] Step 8: Use the difficulty set constructed in step 5 to perform the second stage training on the discriminant model; in the second stage training, adopt the normal prototype contrast learning strategy, use the normal prototype vector constructed in step 6 to calculate the similarity between the normal feature and the normal prototype, use the contrast loss to update the discriminant model, and dynamically update the normal prototype during training;

[0046] The discriminant model is trained in the second stage using the difficult set, and a binary cross entropy loss function is constructed for key fragments with high scores. The binary cross entropy loss function BCE of the second stage training R (H) The calculation formula is as follows:

[0047]

[0048] Among them, E(.) represents the mathematical expectation, Indicates the maximum value of the scores of normal and abnormal videos after being judged by the model;

[0049] Contrasting learning is performed on the errors caused by non-critical segments. The normal prototype vector set established in step 6 is used to construct the normal prototype contrast loss L S-P(H) consists of two parts. The first part pushes normal samples closer to the normal prototype, and the second part pushes abnormal samples away from the normal prototype. The calculation formula is as follows:

[0050]

[0051] Among them, b n is a vector of all 1s, indicating the target similarity between the normal sample and the closest normal prototype; H n represents the difficult set of normal samples, H a represents the difficult set of abnormal samples, T represents transposition, P k is a set of normal prototype vectors, m n 、m a They represent the number of samples in the normal sample difficulty set and the number of samples in the abnormal sample difficulty set respectively, and ||.||2 means taking the l2 norm;

[0052] The total training loss at this stage is expressed as:

[0053] L R (H)=BCE R (H)+λ1'L rank (H)+λ2L S-P (H) (13)

[0054] Among them, λ1' and λ2 are hyperparameters used to balance the binary cross entropy loss function, ranking loss and normal prototype contrast loss of the two-stage training;

[0055] Step 9: After obtaining features of the test set images using steps 2 and 3, use the best discriminant model trained in step 8 and the learned normal prototype to calculate the anomaly score, thus obtaining the video anomaly detection result;

[0056] In the inference phase, the normality score is obtained by the maximum similarity between the test sample and the learned normal prototype, as follows:

[0057]

[0058] in, represents the i-th test sample, and Respectively represent the discriminant model parameters and normal prototype vector learned during training;

[0059] The total anomaly score of the i-th segment of the video The discriminant model score is added to the negative number of the normality score. The formula is expressed as:

[0060]

[0061] represents the score of the best discriminant model trained by the test sample input, Represents the normality score; the larger the anomaly score, the higher the degree of abnormality of the video clip. By judging the value of the anomaly score, the weakly supervised video anomaly detection result is finally obtained.

[0062] Compared with the prior art, the present invention has the following beneficial effects: the present invention reduces the noise guidance that cannot be ignored under weak supervision conditions by imitating the course learning strategy of human learning, measures the difficulty of data samples for model discrimination according to the noise level of pseudo labels and the confidence level of abnormal fragments, realizes the denoising process and guides the model to converge. In addition, the present invention designs a contrast learning method based on the normal prototype, uses a normal sample with stability to model the contrast relationship between the sample and the normal prototype, thereby adding constraints to the fuzzy sample, clarifying the positive and abnormal boundaries, and effectively improving the accuracy of video anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A model diagram of a weakly supervised video anomaly detection method based on curriculum learning and contrastive learning strategies of the present invention;

[0064] Figure 2 is a flow chart of training steps in a specific embodiment of the present invention;

[0065] Figure 3 The figure is a flow chart of the test steps of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. The present invention includes but is not limited to the following embodiments.

[0067] like Figure 1 As shown, the present invention provides a weakly supervised video anomaly detection method based on curriculum learning and contrastive learning strategies. Figure 2 As shown, the specific implementation process is as follows:

[0068] 1. Video preprocessing and extraction of optical flow images

[0069] The videos are divided into training set and test set, and each video is divided into RGB video frames according to its frame rate. The training video V in the training set contains abnormal videos and normal videos. Under weak supervision, the abnormal video is V a , the video level label is Y a =1, including abnormal and normal clips; normal video is V n , the video level label is Y n= 0, only normal segments are included. Referring to the method of multi-instance learning, each video in the training set is regarded as a different video package, and the segments in the package are regarded as examples. The size of each video frame is resized to 240×320 pixels, and the frame rate is fixed to 30fps. The TV-LI algorithm is used to extract the optical flow map corresponding to the continuous RGB image.

[0070] 2. Video feature extraction

[0071] The interval between video frames is short, and calculating features frame by frame is prone to redundancy. Each video packet is divided into Z non-overlapping segments, and the i-th segment is recorded as v i , each clip contains 16 consecutive frames. The I3D network pre-trained on the public dataset Kinetics-400 is used as the feature extraction network, and the RGB image v of the video clip is input. i RGB To I3D network pre-trained on RGB images I3D RGB In the example, the optical flow image v of the input video clip is i Optical-Flow To the I3D network pre-trained on optical flow images I3D Optical-Flow The extracted RGB feature of the i-th video clip is The optical flow feature is Concatenate the two network output features to get the fusion feature x i , the calculation formula is as follows:

[0072]

[0073] in, Represents a concatenation operation.

[0074] Normal video V n The feature representation of is a set of Z normal segment features Abnormal Video V a The feature representation of is a set of Z abnormal fragment features

[0075] 3. Determination of the difficulty of anomaly detection

[0076] Use the pre-classifier to estimate the anomaly score of the training set videos and obtain pseudo labels. The pre-classifier is a 3-layer fully connected neural network. The first fully connected layer contains 512 units, the second fully connected layer contains 32 units, and the last output layer contains 1 unit. 60% dropout regularization is used between the fully connected layers, and ReLU and Sigmoid activation functions are added after the first and last layers respectively. The Adagrad optimizer with an initial learning rate of 0.001 is used. The network is trained using binary cross entropy loss and deep ranking loss.

[0077] The binary cross entropy loss function is calculated as:

[0078]

[0079] Among them, y represents the video level label, *∈{n,a} represents normal and abnormal, represents the scores of normal and abnormal video clips after detection by the pre-classifier, and E[.] represents the mathematical expectation.

[0080] The maximum score of the abnormal segment in the abnormal video is restricted to be greater than the maximum score of the normal segment in the normal video, and the temporal smoothing between the abnormal scores is achieved by minimizing the score difference between adjacent video segments. rank The calculation formula is:

[0081]

[0082] Among them, max means taking the maximum value of a video score, x l Represents the characteristics of the current moment, x l+1 Indicates the characteristics of the next moment. (*) + It represents the maximum value between the expression * and 0, ε is a hyperparameter representing the boundary, and α is a hyperparameter used to balance the sorting loss and the time smoothing part.

[0083] The total training loss L F (X) The calculation formula is:

[0084]

[0085] Among them, λ1 is a hyperparameter used to balance the binary cross entropy loss and the depth ranking loss.

[0086] The scores of each video after the pre-classifier, i.e., the segment-level pseudo-labels The normal video pseudo label set is The set of abnormal video pseudo labels is

[0087] The difficulty judgment rule is defined using pseudo labels for difficulty judgment. The classification confidence corresponding to significant normal or abnormal is higher, the boundary is clear, and it is easier to distinguish. T samples with abnormal scores closest to 1 are selected from the abnormal video as the confident positive sample feature set S that is likely to be abnormal a . Select t samples S with anomaly scores closest to 0 from the normal video n, as the confident negative sample feature set. The intersection of the confident positive sample feature set and the confident negative sample feature set is taken as a simple set S. The remaining samples have more noise or are not significantly distinguished. The detection results are prone to deviation and the recognition is difficult. They are collected as a difficult set H. The difficult set H contains abnormal fragments H a and normal fragment H n The calculation formula for dividing the set according to the difficulty determination rule is as follows:

[0088]

[0089] S=S a ∪S n (twenty one)

[0090] H a =X a -S a ,H n =X n -S n (twenty two)

[0091] H=H a ∪H n (twenty three)

[0092] in, It represents the index corresponding to the t values ​​that make the expression * the smallest, i represents the ith video clip, and - represents the relative complement.

[0093] 4. Initialize the normal prototype vector

[0094] Construct a normal prototype vector set P, initialize it with the difficult set, and calculate the average value of the normal data features in the difficult set as the initial normal prototype vector. Establish a normal prototype vector set P = {p1, p2, ..., p k} to represent the representative characteristics of normal samples, k is the number of normal prototypes. It will contain m n The normal sample difficulty set of samples is evenly divided into k subsets, each of which is used to initialize a normal prototype and is continuously updated during training. The jth normal prototype is denoted as p j ∈R d , d is the characteristic dimension, and the calculation formula is as follows:

[0095]

[0096] in represents the sample in the i-th normal sample difficulty set, Indicates rounding up.

[0097] 5. Simple set initialization discriminant model

[0098] The discriminant model is set as a 3-layer fully connected neural network consistent with the pre-classifier, and the discriminant model is trained using binary cross entropy loss and deep ranking loss. At this time, the input data of the discriminant model is a simple set. The total loss function L F (S) is expressed as:

[0099] L F (S)=BCE F (S)+λ1L rank (S) (25)

[0100] Among them, λ1 is a hyperparameter used to balance the binary cross entropy loss and the depth ranking loss.

[0101] 6. Use the difficult set for two-stage training

[0102] The discriminant model is trained in the second stage using the difficulty set constructed in step 5. In the second stage of training, a normal prototype contrast learning strategy is adopted, and the normal prototype vector constructed in step 6 is used to calculate the similarity between the normal feature and the normal prototype. The discriminant model is updated using contrast loss, and the normal prototype is dynamically updated during training.

[0103] The discriminant model is trained in the second stage using the difficult set, and a binary cross entropy loss function is constructed for key fragments with high scores. The binary cross entropy loss function BCE of the second stage training R (H) The calculation formula is as follows:

[0104]

[0105] Among them, E(.) represents the mathematical expectation, Indicates the maximum value of the scores of normal and abnormal videos after being judged by the model.

[0106] Contrasting learning is performed on the errors caused by non-critical segments. The normal prototype vector set established in step 6 is used to construct the normal prototype contrast loss L S-P (H) consists of two parts. The first part pushes normal samples closer to the normal prototype, and the second part pushes abnormal samples away from the normal prototype. The calculation formula is as follows:

[0107]

[0108] Among them, b n is a vector of all 1s, indicating the target similarity between the normal sample and the closest normal prototype. n represents the difficult set of normal samples, H a represents the difficult set of abnormal samples, T represents transposition, P k is a set of normal prototype vectors, m n 、m aThey represent the number of samples in the normal sample difficulty set and the number of samples in the abnormal sample difficulty set respectively, and ||.||2 means taking the l2 norm.

[0109] The total training loss at this stage is expressed as:

[0110] L R (H)=BCE R (H)+λ1'L rank (H)+λ2L S-P (H) (28)

[0111] Among them, λ1' and λ2 are hyperparameters used to balance the binary cross entropy loss function, ranking loss and normal prototype contrast loss of the two-stage training.

[0112] 7. Test video anomaly detection

[0113] like Figure 3 As shown in the figure, the video to be tested in the test set is preprocessed and the optical flow image is extracted, which is then input into the I3D network for feature extraction. Then the test sample is input into the best discriminant model obtained through training and the maximum similarity is calculated with the updated normal prototype to obtain the normality score. The calculation formula is as follows:

[0114]

[0115] in, represents the i-th test sample, and They represent the discriminant model parameters and normal prototype learned during training respectively.

[0116] The total anomaly score of the i-th segment of the video The discriminant model score is added to the negative number of the normality score. The formula is expressed as:

[0117]

[0118] represents the score of the best discriminant model trained by the test sample input, Represents the normality score. The larger the anomaly score, the higher the abnormality of the video clip. The weakly supervised video anomaly detection result is obtained by judging the value of the anomaly score.

[0119] In summary, the present invention discloses a weakly supervised video anomaly detection method based on curriculum learning and contrastive learning strategies. The present invention designs a curriculum learning strategy by imitating the learning method of humans, and trains an anomaly detection model by contrastive learning method based on normal prototype, thereby avoiding the model deviation problem caused by noise under weak supervision to a great extent, clearly distinguishing the positive and abnormal boundaries, and effectively improving the accuracy of video anomaly detection.

[0120] Firstly, a curriculum learning strategy that imitates human learning is used to formulate a video anomaly detection difficulty judgment standard that comprehensively considers the amount of noise and the degree of abnormality in the video, and pseudo labels are used to classify video samples. Secondly, samples of different difficulty levels are gradually added to the training process, and different loss functions are used for staged training to achieve denoising and guide the convergence direction of the model. Finally, a normal prototype is constructed and dynamically updated, and normal prototype contrast learning is used to distinguish the positive and abnormal boundaries of fuzzy samples, effectively improving the accuracy of video anomaly detection.

Claims

1. A weakly supervised video anomaly detection method based on curriculum learning and contrastive learning strategy, characterized in that: The following steps are involved: Step 1: Divide the video into training set and test set, and split each video into RGB video frames according to its frame rate; Step 2: Use the TV-LI algorithm to extract the corresponding optical flow image from the RGB video frame; Step 3: Use the pre-trained I3D network as the feature extraction network to extract the appearance features and motion features of the videos in the training set; The training video V in the training set includes abnormal videos and normal videos. Under weak supervision, the abnormal video is V a , the video level label is Y a =1, including abnormal and normal clips; normal video is V n , the video level label is Y n = 0, only normal segments are included; training video V = V a ∪V n , ∪ represents the union; according to the multi-instance learning method, each video in the training set is regarded as a different video package, and the clips in the video package are regarded as examples; Each video packet is divided into Z non-overlapping video segments, and the i-th video segment is denoted as v i , using the I3D network pre-trained on the public dataset Kinetics-400 as the feature extraction network, the RGB image v of the input video clip i RGB To I3D network pre-trained on RGB images I3D RGB In the example, the optical flow image v of the input video clip is i Optical-Flow To the I3D network pre-trained on optical flow images I3D Optical-Flow The extracted RGB feature of the i-th video clip is The optical flow feature is Concatenate the two network output features to get the fusion feature x i , the calculation formula is as follows: in, Represents a splicing operation; Normal video V n The feature representation of is a set of Z normal segment features Abnormal Video V a The feature representation of is a set of Z abnormal fragment features Step 4: Use the pre-classifier to estimate the video anomaly score and generate segment-level pseudo labels for the video; The pre-classifier is a 3-layer fully connected neural network. The first fully connected layer contains 512 units, the second fully connected layer contains 32 units, and the final output layer contains 1 unit. 60% dropout regularization is used between the fully connected layers. ReLU activation function and Sigmoid activation function are added after the first and last layers respectively. Adagrad optimizer with an initial learning rate of 0.001 is used. The pre-classifier is trained using binary cross entropy loss and deep ranking loss. The binary cross entropy loss function is calculated as: Among them, y represents the video level label, *∈{n,a} represents normal and abnormal, represents the scores of normal and abnormal video clips after detection by the pre-classifier, and E[.] represents the mathematical expectation; The maximum score of the abnormal clip in the abnormal video is restricted to be greater than the maximum score of the normal clip in the normal video, and the temporal smoothing between the abnormal scores is achieved by minimizing the score difference between adjacent video clips; the deep ranking loss L including the temporal smoothing calculation is rank The calculation formula is: Among them, max means taking the maximum value of a video score, x l Represents the characteristics of the current moment, x l+1 Indicates the characteristics of the next moment; (*) + It means taking the maximum value between the expression * and 0, ε is a hyperparameter representing the boundary, and α is a hyperparameter used to balance the sorting loss and the time smoothing part; The total training loss L F (X) The calculation formula is: L F (X)=BCE F (X)+λ1L rank (X) (4) Among them, λ1 is a hyperparameter used to balance the binary cross entropy loss and the depth ranking loss; The scores of each video after the pre-classifier, i.e., the segment-level pseudo-labels The normal video pseudo label set is The set of abnormal video pseudo labels is Step 5: Define the difficulty judgment rule, make difficulty judgment based on the pseudo-label value, and divide the training set into a simple set S and a difficult set H; Use the pseudo-labels obtained in step 4 to define the difficulty determination rules and perform difficulty determination; Select t samples with anomaly scores closest to 1 from the abnormal video as the confident positive sample feature set S that is likely to be abnormal a ; Select t samples S with abnormal scores closest to 0 from normal videos n , as the confident negative sample feature set; the intersection of the confident positive sample feature set and the confident negative sample feature set is taken as the simple set S, and the remaining samples have more noise or are not significantly distinguished, the detection results are prone to deviation, and the recognition is difficult, so they are collected as the difficult set H; the difficult set H contains abnormal fragments H a and normal fragment H n ; The calculation formula for dividing the set according to the difficulty determination rule is as follows: S=S a ∪S n (6) H a =X a -S a ,H n =X n -S n (7) H=H a ∪H n (8) in, represents the index corresponding to the t values ​​that make the expression * the smallest, i represents the i-th video clip, and - represents the relative complement; Step 6: Construct a normal prototype vector set P, take the difficult set constructed in step 5 to initialize it, and calculate the average value of the normal data features in the difficult set as the normal prototype vector; Establish a normal prototype vector set P = {p1, p2, ..., p k } to represent the representative features of normal samples, k is the number of normal prototype vectors; it will contain m n The normal sample difficulty set of samples is evenly divided into k subsets, each subset is used to initialize a normal prototype vector and continuously updated during training; the jth normal prototype vector is represented by p j ∈R d , d is the characteristic dimension, and the calculation formula is as follows: in, represents the sample in the i-th normal sample difficulty set, Indicates rounding up; Step 7: Execute the selective pseudo-label course learning strategy to imitate the human learning process from easy to difficult knowledge. First, use the simple set constructed in step 5 to initialize the discriminant model and update the model parameters. The discriminant model is set as a 3-layer fully connected neural network consistent with the pre-classifier. The discriminant model is trained using binary cross entropy loss and deep ranking loss. At this time, the input data of the discriminant model is a simple set; the total loss L F (S) is expressed as: L F (S)=BCE F (S)+λ1L rank (S) (10) Among them, λ1 is a hyperparameter used to balance the binary cross entropy loss and the depth ranking loss; Step 8: Use the difficulty set constructed in step 5 to perform the second stage training on the discriminant model; in the second stage training, adopt the normal prototype contrast learning strategy, use the normal prototype vector constructed in step 6 to calculate the similarity between the normal feature and the normal prototype, use the contrast loss to update the discriminant model, and dynamically update the normal prototype during training; The discriminant model is trained in the second stage using the difficult set, and a binary cross entropy loss function is constructed for key segments with high scores. The binary cross entropy loss function BCE of the second stage training R (H) The calculation formula is as follows: Among them, E(.) represents the mathematical expectation, Indicates the maximum value of the scores of normal and abnormal videos after being judged by the model; Contrasting learning is performed on the errors caused by non-critical segments. The normal prototype vector set established in step 6 is used to construct the normal prototype contrast loss L S-P (H) consists of two parts. The first part pushes normal samples closer to the normal prototype, and the second part pushes abnormal samples away from the normal prototype. The calculation formula is as follows: Among them, b n is a vector of all 1s, indicating the target similarity between the normal sample and the closest normal prototype; H n represents the difficult set of normal samples, H a represents the difficult set of abnormal samples, T represents transposition, P k is a set of normal prototype vectors, m n 、m a They represent the number of samples in the normal sample difficulty set and the number of samples in the abnormal sample difficulty set respectively, and ||.||2 means taking the l2 norm; The total training loss at this stage is expressed as: L R (H)=BCE R (H)+λ1'L rank (H)+λ2L S-P (H) (13) Among them, λ1' and λ2 are hyperparameters used to balance the binary cross entropy loss function, ranking loss and normal prototype contrast loss of the two-stage training; Step 9: After obtaining features of the test set images using steps 2 and 3, use the best discriminant model trained in step 8 and the learned normal prototype to calculate the anomaly score, thus obtaining the video anomaly detection result; In the inference phase, the normality score is obtained by the maximum similarity between the test sample and the learned normal prototype, as follows: in, represents the i-th test sample, and Respectively represent the discriminant model parameters and normal prototype vector learned during training; The total anomaly score of the i-th segment of the video The formula is to add the negative of the discriminant model score and the normality score: represents the score of the best discriminant model trained by the test sample input, Represents the normality score; the larger the anomaly score, the higher the degree of abnormality of the video clip. By judging the value of the anomaly score, the weakly supervised video anomaly detection result is finally obtained.

Images