Anomaly detection method based on visual language bidirectional encoding transformer and ranking loss function

Abstract

An anomaly detection method based on a visual language bidirectional encoder and a ranking loss function is firstly represented as a video package, wherein a video package containing at least one abnormal instance is a positive package, and a video package without any abnormal instance is a negative package; then both types of packages are divided into the same and fixed number of time periods, each time period is an instance in the video package, and then both types of packages are input into a visual language bidirectional encoder ViLBERT module based on a common attention and self-attention mechanism, thereby obtaining ViLBERT features; then the ViLBERT features are input into a fully connected neural network, and the neural network is trained based on a ranking loss function; finally, an anomaly detection model is constructed that effectively distinguishes abnormal events from normal events, and each instance can obtain its own anomaly score, thereby realizing video anomaly detection. This method solves the problem of lack of a large number of frame-level labels in anomaly detection, and ensures the effective distinction between abnormal events and normal events.

Description

Technical Field

[0001] The present invention belongs to the technical field of video anomaly detection, and in particular relates to an anomaly detection method based on a visual language bidirectional encoding transformer and a sorting loss function. Background Art

[0002] Anomaly detection is not only an important and active research topic in the field of computer vision, but also a very challenging task, because a variety of real-life situations may occur in an open and complex environment, so it is almost impossible to collect data on various abnormal events. In addition, due to the wide variety of abnormal events and their low frequency of occurrence, if abnormal events are collected and marked in massive video data sets, it will not only require a lot of manpower but also be very time-consuming. In recent years, some clustering methods have been used to solve the anomaly detection problem, such as Gaussian mixture models and k-means clustering algorithms. However, these methods are all unsupervised anomaly detection methods. Due to the wide variety of abnormal events, it is difficult to distinguish them through a simple classifier, so they usually encounter poor performance problems in practical applications.

[0003] In recent years, in order to further improve the performance of the model, anomaly detection methods using weak supervision have attracted more and more attention from researchers. In weak supervision methods, anomaly detection is usually expressed as a multi-instance learning (MIL) task, which has achieved certain results in practical applications, but its recognition of abnormal events is largely affected by the normal events that account for the majority, resulting in low accuracy. In order to solve this problem, Tian et al. proposed a weakly supervised anomaly detection method by learning the robust temporal feature magnitude in the MIL method (Y. Tian, ​​G. Pang, Y. Chen, R. Singh, JW Verjans, G. Carneiro, Weakly supervised video anomaly detection with robust temporal feature magnitude learning. In ICCV, 2021). Degardin et al. proposed an iterative weak / self-supervised learning method for anomaly detection (B. Degardin, H. Proena, Iterative weak / self-supervised classifification framework for abnormal events detection. In PRL, 2021). Zhong et al. designed a graph convolutional network to remove noisy labels, thereby applying supervised action classifiers to weakly supervised anomaly detection (J. Zhong, N. Li, W. Kong, S. Liu, TH Li, G. Li, Graph convolutional label noise cleaner: Train a plug-and-play action classifier for anomaly detection. In CVPR, 2019). However, none of the existing methods explicitly use attention mechanisms to detect abnormal events. Summary of the invention

[0004] To solve the above problems, the present invention provides an anomaly detection method based on a visual language bidirectional encoder transformer and a ranking loss function. The method inputs positive class packages containing abnormal instances and negative class packages containing only normal instances into a ViLBERT module based on a common attention and self-attention mechanism, extracts the ViLBERT features of the video instance, and then inputs the features into a neural network adjusted based on a new ranking loss function for training, and finally constructs a complete anomaly detection model, and each video instance can obtain its own anomaly score.

[0005] An anomaly detection method based on a visual language bidirectional encoding transformer and a sorting loss function specifically comprises the following steps:

[0006] S1, set the training set and the test set according to the video data, regard each video as a video packet, and define the video packet containing at least one abnormal instance as a positive packet B in the training set. p , the video packet that does not contain any abnormal instance is defined as a negative packet B n ;

[0007] S2, divide the positive and negative packets obtained in S1 into the same number of time segments L, each time segment represents an instance in the video packet;

[0008] S3, inputs both the positive and negative packets obtained in S2 into the ViLBERT module based on the attention mechanism, thereby extracting the ViLBERT features of all video instances;

[0009] S4, input the instance ViLBERT features obtained in S3 into a three-layer fully connected neural network, and perform iterative training based on the ranking loss function. After training, an anomaly detection model that distinguishes abnormal events from normal events is obtained, and the test set is input. Each video instance in it obtains its own anomaly score to obtain the abnormal detection result.

[0010] Furthermore, the video packet B in S1 p and B n The number of instances contained in is equal.

[0011] Furthermore, the positive packet video instances in S2 are represented by p1, p2, ..., p l ,...p L Indicates that the negative bag video instances are represented by n1, n2, ..., n l ,...n L express.

[0012] Furthermore, in S3, the attention mechanism structure of the ViLBERT module is mainly constructed by a common attention layer and a self-attention layer. The image instances in the positive bag and the negative bag are transformed to obtain visual vectors, and the visual vectors are respectively compared with W Q , W K , W VThe three matrices are multiplied to obtain three vectors Q, K, V corresponding to each positive bag instance, and three vectors Q*, K*, V* corresponding to each positive bag instance. Then the three vectors are input into the ViLBERT module, where K, V in the positive bag are exchanged with K*, V* in the negative bag to build a common attention. Then Q, K*, V* are passed through the self-attention layer, normalization layer, forward propagation layer, and normalization layer of the ViLBERT module in turn to finally obtain an output result; the same operation is performed on Q*, K, V as described above and an output result is also obtained. Finally, the two output results are spliced ​​and integrated to obtain the ViLBERT feature.

[0013] Furthermore, in S4, the ranking loss function is obtained by averaging the instance anomaly scores in the video package based on the means method, performing relaxation operations and hinge loss definitions, while considering the smoothness of time and the sparsity of abnormal events.

[0014] Furthermore, in S4, the ranking loss function is defined as follows:

[0015] L(W)=l(B p ,B n )+β3||W|| F

[0016] Where W is the weight, β3 is the balance parameter, ||W|| F is the Frobenius norm of W;

[0017]

[0018] in, and It represents the average value of the predicted anomaly scores calculated by traversing all video instances in each video package through the mean method. represents the temporal smoothness constraint, represents the sparsity constraint, λ is the relaxation parameter, and β1 and β2 are both balance parameters.

[0019] The beneficial effects of the present invention are:

[0020] (1) The present invention uses video-level labels instead of frame-level labels, which effectively solves the problem of lack of a large number of frame-level labels in anomaly detection. This approach has great advantages in labeling massive videos.

[0021] (2) The present invention introduces a ViLBERT module based on the common attention and self-attention mechanism. The ViLBERT features extracted by this module can effectively represent the features of the video instance.

[0022] (3) The present invention constructs a new ranking loss function, which is obtained by averaging the abnormality scores of instances in the video package, while taking into account the smoothness of time and the sparsity of abnormal events. The loss function constructed in this way can make the information obtained by the model more complete, and effectively improve the accuracy of the model in distinguishing abnormal events from normal events. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 4 is a flow chart of an anomaly detection method in an embodiment of the present invention.

[0024] Figure 2 4 is a structural diagram of the ViLBERT module in an embodiment of the present invention.

[0025] Figure 3 4 is a network model structure diagram of the anomaly detection method in an embodiment of the present invention.

[0026] Figure 4 This is a comparison chart of ROC curves of the method in the embodiment of the present invention and other anomaly detection methods on the UCF Crime dataset. DETAILED DESCRIPTION

[0027] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0028] The present invention is an anomaly detection method based on a combination of a visual language bidirectional encoding transformer and a new sorting loss function. The anomaly detection method specifically comprises the following steps:

[0029] S1, each video is regarded as a video packet, and the video packet containing at least one abnormal instance is regarded as a positive packet B. p , the video packets that do not contain any abnormal instances are regarded as negative packets B n , Video Package B p and B n The number of video instances contained in them is equal.

[0030] S2, divide the two types of video packets obtained in S1 into a fixed number of time periods, the number of time periods is L, each time period represents a video instance in the video packet. The video instances in the positive packet are represented by p1, p2, ..., p l ,...p L Indicates that the negative bag video instances are represented by n1, n2, ..., n l ,...n L express.

[0031] S3, the two types of packages obtained in S2 are input into the ViLBERT module based on the joint attention and self-attention mechanism, and the ViLBERT features of all instances in the video package are extracted from the module. The image instances in the positive and negative packages are transformed to obtain visual vectors, and the visual vectors are compared with W Q , W K , W V The three matrices are multiplied to obtain three vectors Q, K, V corresponding to each positive bag instance, and three vectors Q*, K*, V* corresponding to each positive bag instance. Then the three vectors are input into the ViLBERT module, where K, V in the positive bag are exchanged with K*, V* in the negative bag to build a common attention. Then Q, K*, V* are passed through the self-attention layer, normalization layer, forward propagation layer, and normalization layer of the ViLBERT module in turn to finally obtain an output result; the same operation is performed on Q*, K, V as described above and an output result is also obtained. Finally, the two output results are spliced ​​and integrated to obtain the ViLBERT feature.

[0032] S4, the instance ViLBERT features obtained in S3 are input into a three-layer fully connected neural network, which is iteratively trained based on the proposed new ranking loss function, and finally builds an anomaly detection model that can effectively distinguish abnormal events from normal events. Each video instance can obtain its own anomaly score.

[0033] The specific construction process of the new ranking loss function in this step is as follows:

[0034] S4-1, for a given abnormal video instance x a and normal video instance x r The simplest loss function is as follows:

[0035] s(x a )>s(x r )

[0036] Among them, s(x a ) and s(x r ) represent the video instance x a and x r Predicted anomaly score. However, since the labels at the video instance level are unknown, it is impossible to train the neural network using the above formula.

[0037] S4-2, in order to solve the problem in S4-1 and preserve the video instance information more completely, the objective function for multi-instance sorting proposed by this method is as follows:

[0038]

[0039] Among them, the mean method traverses all video instances in each video package and performs the ranking loss function by calculating the average of the predicted anomaly scores of all video instances in each video package. This makes the difference between the anomaly scores of abnormal instances and normal instances larger, which is further conducive to distinguishing the two.

[0040] S4-3, in order to make the gap between the normal instance anomaly scores smaller than the gap between the abnormal instance and the normal instance anomaly scores, the objective function is further adjusted as follows:

[0041]

[0042] in, The value of is greater than zero, thus ensuring that the formula in S4-2 still holds.

[0043] S4-4, in order to prevent overfitting, the formula in S4-3 is relaxed, and the adjusted objective function is as follows:

[0044]

[0045] Among them, λ is the relaxation parameter.

[0046] S4-5, the ranking loss function is defined by the hinge loss formula as follows:

[0047]

[0048] S4-6, considering the temporal smoothness and sparsity of abnormal instances, the loss function is further adjusted as follows:

[0049]

[0050] in, represents the temporal smoothness constraint, represents the sparsity constraint, β1 and β2 are both balance parameters.

[0051] S4-7, the final constructed complete new ranking loss function is defined as follows:

[0052] L(W)=l(B p ,B n )+β3||W|| F

[0053] Where W is the weight of the model, β3 is the balance parameter, ||W|| F is the Frobenius norm of W.

[0054] The present invention belongs to a weakly supervised learning method, and proposes an anomaly detection method that combines a visual language bidirectional encoder transformer with a new ranking loss function. For the first time, the attention mechanism is explicitly introduced into video anomaly detection, thereby effectively extracting the ViLBERT features of the video instance. At the same time, the present invention proposes a new ranking loss function to train the neural network, which takes into account both the integrity of the video instance information and the smoothness of time and the sparsity of abnormal events. The anomaly detection method based on the combination of a visual language bidirectional encoder transformer and a new ranking loss function proposed in the present invention ensures the effective distinction between abnormal events and normal events.

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0056] A. Experimental conditions

[0057] 1. Experimental Database

[0058] Training and testing are performed on UCF Crime and a self-designed dataset (NUT). During the anomaly detection process, the dataset is divided into a training class (if necessary, it is further divided into a training class and a validation class) and a test set. Different data sets have different divisions. The training class plus the validation class is a known class, and the test class is an unknown class. There is no intersection between the two. See Table 1 for details.

[0059] Table 1 Detailed introduction of the dataset

[0060]

[0061] 2. Experimental parameter settings

[0062] The fixed parameter settings of the model are shown in Table 2 below:

[0063] Table 2 Model fixed parameters

[0064] L λ <![CDATA[β1]]> <![CDATA[β2]]> <![CDATA[β3]]> 32 <![CDATA[8×10 5 ]]> <![CDATA[8×10 5 ]]> <![CDATA[8×10 5 ]]> <![CDATA[1×10 2 ]]>

[0065] B. Experimental results evaluation criteria

[0066] This model is designed for anomaly detection. The test categories in anomaly detection include unknown categories and known categories. AUC is used here to measure the detection effect. The higher the AUC value, the better the model effect. The calculation formula of AUC is as follows:

[0067]

[0068] Among them, M is the number of positive package samples, N is the number of negative package samples, and the numerator is the total number of combinations where the probability of the predicted result being a positive package is greater than the probability of the predicted result being a negative package.

[0069] C. Comparative test plan

[0070] This embodiment is compared with other cutting-edge anomaly detection methods on two data sets.

[0071] Table 3 Performance comparison of anomaly detection methods:

[0072] Methods UCF Crime NUT Binary classifier 50.0 - Conv-AE 50.6 - sparse 65.5 - MIL 75.4 82.1 MILR 76.7 - AMIL 78.5 86.0

[0073] AMIL is the method proposed in this paper. The comparison results with the current cutting-edge anomaly detection methods in Table 3 show that the anomaly detection effect of this method exceeds that of other compared methods. MIL is the basic model of the method proposed in this paper. After adding the attention mechanism, the effect of our model is improved by more than 3%, which objectively demonstrates the effectiveness of the method proposed in this paper.

[0074] Figure 3 The ROC curve of this paper's method is compared with other anomaly detection methods on the UCF Crime dataset. The curve shows that this paper's method is more effective than existing methods. In particular, when the false positive rate is around 0.3, this paper's method obtains a higher true positive rate than other methods, which illustrates the effectiveness of the model.

[0075] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. An anomaly detection method based on a visual language bidirectional encoder transformer and a ranking loss function, characterized in that: The anomaly detection method specifically comprises the following steps: S1, set the training set and the test set according to the video data, regard each video as a video packet, and define the video packet containing at least one abnormal instance as a positive packet B in the training set. p , the video packet that does not contain any abnormal instance is defined as a negative packet B n ; S2, divide the positive and negative packets obtained in S1 into the same number of time segments L, each time segment represents an instance in the video packet; S3, inputs both the positive and negative packets obtained in S2 into the ViLBERT module based on the attention mechanism, thereby extracting the ViLBERT features of all video instances; S4, input the instance ViLBERT features obtained in S3 into a three-layer fully connected neural network, and perform iterative training based on the ranking loss function. After training, an anomaly detection model that distinguishes abnormal events from normal events is obtained, and the test set is input. Each video instance in the model obtains its own anomaly score, and an abnormal detection result is obtained; The ranking loss function is obtained by averaging the instance anomaly scores in the video bag based on the means method, performing relaxation operations and hinge loss definition, while considering the smoothness of time and the sparsity of abnormal events; The ranking loss function is defined as follows: L(W)=l(B p ,B n )+β3||W|| F Where W is the weight, β3 is the balance parameter, ||W|| F is the Frobenius norm of W; in, and It represents the average value of the predicted anomaly scores calculated by traversing all video instances in each video package through the mean method. represents the temporal smoothness constraint, represents the sparsity constraint, λ is the relaxation parameter, and β1 and β2 are both balance parameters.

2. The anomaly detection method based on a visual language bidirectional encoder transformer and a ranking loss function according to claim 1, characterized in that: The video packet B in S1 p and B n The number of instances contained in is equal.

3. The anomaly detection method based on a visual language bidirectional encoder transformer and a ranking loss function according to claim 1, characterized in that: The positive packet video instances in S2 are represented by p1, p2, ..., p l ,...p L Indicates that the negative bag video instances are represented by n1, n2, ..., n l ,...n L express.

4. The anomaly detection method based on a visual language bidirectional encoder transformer and a ranking loss function according to claim 1, characterized in that: In S3, the attention mechanism structure of the ViLBERT module is mainly constructed by a common attention layer and a self-attention layer; the image instances in the positive bag and the negative bag are transformed to obtain visual vectors, and the visual vectors are respectively compared with W Q , W K , W V The three matrices are multiplied to obtain three vectors Q, K, V corresponding to each positive package instance, and three vectors Q*, K*, V* corresponding to each positive package instance; then the three vectors are input into the ViLBERT module, where K, V in the positive package are exchanged with K*, V* in the negative package to construct a common attention, and then Q, K*, V* are passed through the self-attention layer, normalization layer, forward propagation layer, and normalization layer of the ViLBERT module in turn to finally obtain an output result; the same operation is performed on Q*, K, V as described above and an output result is also obtained. Finally, the two output results are spliced ​​and integrated to obtain the ViLBERT feature.

Images