Video anomaly detection method based on contrast clustering learning network

Abstract

The invention belongs to the technical field of intelligent video processing, and particularly relates to a video anomaly detection method based on a contrast clustering learning network. The method comprises the following steps: step 1, preprocessing: positioning and classifying bounding boxes of all objects in a frame by using an object detector, and predicting an optical flow of each frame by using a motion estimator; step 2, comparative clustering learning: executing comparative clustering learning by using space-time representations of the two views; and step 3, abnormal score estimation: normal or abnormal scoring is carried out on the sample by using density estimation, wherein low estimation density represents abnormity. Clustering information is used for comparative learning, and meanwhile, different attributes of a normal mode are mined to learn fine-grained features. According to the method, clustering-level hidden information of different views can be effectively utilized. According to the method, a clustering scattering contrast loss function is provided to refine clustering information, and it is verified that the refined clustering information is significantly helpful for contrast learning of a video anomaly detection framework.

Description

Technical Field

[0001] The invention belongs to the technical field of intelligent video processing, and in particular relates to a video anomaly detection method based on a contrastive clustering learning network. Background Art

[0002] Video anomaly detection is a computer vision task that aims to automatically identify abnormal or uncommon events in videos, usually in the context of unsupervised or semi-supervised learning. It has important practical applications in areas such as video surveillance, intelligent transportation, and public safety. Traditional video anomaly detection methods usually rely on a large amount of labeled data and predefined abnormal patterns, which poses several significant challenges in practical applications. First, obtaining a large amount of labeled data is time-consuming, laborious, and impractical in many cases. Second, the types and forms of abnormal events are diverse, making them difficult to define and cover comprehensively. Finally, existing methods often perform poorly when dealing with unseen abnormal events and have limited generalization capabilities.

[0003] Existing video anomaly detection neural networks usually capture pixel-level spatiotemporal features, where features are usually from the entire frame, located on a grid of image blocks, or concentrated in preprocessed areas. For example, Liu et al. adopted spatial restrictions on intensity and gradient and introduced temporal constraints in video frame prediction; Le et al. used time shift to model temporal information and encode spatiotemporal information in a unified way; Zhao et al. proposed to use three-dimensional convolution to preserve temporal information in convolution and use two decoders to enhance motion characteristics in decoding. These methods fully consider spatiotemporal information and achieve good results, but they process the entire frame image, and the obtained spatiotemporal features often contain a large amount of non-target information that is weakly related to the normal mode. Georgescu et al. proposed a video anomaly detection framework that only focuses on object detection, including three branches and seven tasks, which further improves the performance of video anomaly detection. Although this object-level multi-task learning can better capture the fine-grained spatiotemporal features related to the normal mode, it will repeatedly learn some features, resulting in resource waste, further affecting the execution speed of the video anomaly detection task.

[0004] Reconstruction-based methods basically use autoencoders to reconstruct normal samples in the training phase, and then use the trained autoencoders to judge anomalies through reconstruction errors in the test phase. For example, Hasan et al. used convolutional autoencoders to learn features and capture patterns. As a pioneering work, this method is a good attempt to use convolutional autoencoders for abnormal event detection, but it also has some shortcomings, such as the inability of two-dimensional convolution to obtain time domain information. Ravanbakhsh et al. used the U-Net network structure to achieve cross-modal reconstruction of video frames and optical flow maps, and then used cross-modal reconstruction errors to judge anomalies. CAE can only capture spatial structural information but not time domain information. Gong et al. developed an improved autoencoder called memory-enhanced autoencoder. The core idea is that the designed normal memory will record the prototype of the normal mode. Given an abnormal input, the most relevant normal mode in the memory is retrieved for reconstruction, resulting in an output that is significantly different from the abnormal input. Zhao et al. proposed using three-dimensional convolution to preserve time information in convolution and using two decoders to enhance motion characteristics in decoding. Given the wide adaptability and strong generalization ability of neural networks, this does not mean that abnormal events will have higher reconstruction errors during the reconstruction process.

[0005] Future frame prediction aims to use the previous t frames to model the t+1th frame. Liu et al. proposed a video frame prediction network to predict future video frames based on previous video frames. This is the first work to use the reconstruction error of the predicted future frames to detect abnormal events. But sometimes the autoencoder generalizes so well that it can also reconstruct abnormal samples well, resulting in missed detection of anomalies. In order to alleviate this shortcoming of the autoencoder-based method, the following works have made different improvements. Liu et al. combined the reconstruction task with the prediction task and designed a composite framework based on previous studies. First, the optical flow field is reconstructed, and then the video frame is fused with the optical flow. The mixed frame is used as the input for future frame prediction, which increases the prediction error of abnormal events and better detects anomalies; Chen et al. proposed a bidirectional prediction framework with a simple structure, but forward and backward prediction in the event dimension can only detect anomalies after the abnormal behavior occurs, and cannot provide real-time warning. Le et al. modeled temporal information using time shift, encoded spatiotemporal information in a unified way, and provided a self-contained encoder based on residual attention, which introduced channel attention to more effectively utilize the cascaded channel dependencies within the decoder to predict future frames.

[0006] The video anomaly detection method based on clustering discrimination fits the normal sample space and clusters the normal samples, identifying the samples far away from the normal cluster center as abnormal. Qiu et al. designed a dual-scale feature clustering structure for the encoder, and compressed the channel and spatial structure features of the image to represent the features, obtained good encoding features, and expanded the boundary between normal and abnormal data. Georgescu et al. proposed a video anomaly detection framework that only focuses on object detection, including three branches and seven tasks, which further improved the performance of video anomaly detection. Operate at the object level. Then they determine whether each object is abnormal. This is a simpler challenge because objects have less diversity than the entire video frame. The results of the object-level method greatly exceed those of the frame-level method. Most of the existing object-level anomaly detection algorithms are based on multi-task learning. Multi-task learning can indeed fully learn the different attributes of normal patterns, so as to detect abnormal events well. However, in multi-task learning, the repeated learning of some features will lead to a waste of computing and storage resources, and reduce the speed of anomaly detection. Based on the object-level auxiliary task, the present invention uses clustering to perform comparative learning between different views, quickly and comprehensively learns the different attributes of normal patterns, and the detection speed is far faster than multi-task learning.

[0007] To address these problems, contrastive clustering learning provides an effective solution. Contrastive clustering learning does not require a large amount of labeled data. It automatically captures normal patterns by learning the internal structure and similarity of the data, thereby identifying abnormal events that deviate from the normal pattern. Summary of the invention

[0008] The present invention solves the problem that existing video anomaly detection methods are usually based on the pre-processed image area of ​​the target detector, and the existing methods rely heavily on multi-task learning, which will lead to repeated learning of certain features and waste of computing resources. The present invention proposes a video anomaly detection method based on contrastive clustering learning network.

[0009] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0010] A video anomaly detection method based on contrastive clustering learning network includes the following steps:

[0011] Step 1: Preprocessing: Use an object detector to locate and classify the bounding boxes of all objects in the frame, and apply a motion estimator to predict the optical flow of each frame.

[0012] Step 2: Contrastive clustering learning: Use the spatiotemporal representations of the two views to perform contrastive clustering learning.

[0013] Step 3: Anomaly score estimation: Use density estimation to score the samples as normal or abnormal, where low estimated density indicates anomalies.

[0014] Further as a preferred technical solution of the present invention, the step 1 specifically includes the following steps:

[0015] Use the object detector YOLOv3 to detect all objects in each frame. Object detection is performed for each frame I t Generates a set of N object mappings {x1,x2,...,x N}; Using optical flow as a preliminary stage to infer object motion is done in each pair of two consecutive frames I t-1 and I t Calculated between; the optical flow model extracts each frame I t The optical flow graph is given by Representation; The optical flow model is a twin network, which consists of two branches of feature extractors f(·|Θ) and f′(·|Θ′), without sharing parameters, where Θ and Θ′ are the parameters in the two extractors, respectively, and a predictor layer g(·|Φ) is added after the second branch, where Φ represents the parameters in the predictor layer; the feature extractors f(·|Θ) and f′(·|Θ′) are implemented by ResNet-50 for feature learning; the output features of the first feature extractor and the second feature extractor are represented as z i and And the output of the predictor layer in the second branch is represented as where i∈[1,N].

[0016] Further as a preferred technical solution of the present invention, the step 2 specifically comprises the following steps:

[0017] At the beginning, the first extractor branch f(·|Θ) is pre-trained and used to generate K clusters, which are denoted as C = {c1,c2,...,c K The clustering results are used to form pseudo labels to train the contrastive learning module;

[0018] To force f(·|Θ) to be derived from another view f′(·|Θ′) and exploit the clustering structure, three types of contrastive learning are adopted: object-level contrastive learning, inter-view cluster-level contrastive learning, and cluster-scattering contrastive learning;

[0019] Object-level contrastive learning: To make the representation z of two spatiotemporal views i and Learning from each other at the object level, introducing the feature output z in the first branch i and the predicted output of the second branch The negative cosine similarity of , with object-level contrast loss defined as follows

[0020]

[0021] Can effectively promote feature representation Learn the normal pattern properties in other views and enhance attention to certain normal pattern properties in own views;

[0022] Inter-view cluster-level contrastive learning: vector z i Input into the pre-trained clustering module and calculate the matching cluster center c k , and is defined as the center vector of the matched k-th cluster; the inter-view cluster-level contrast loss is defined as follows:

[0023]

[0024] Utilizing inter-view cluster-level contrast loss Update z i , and then update the cluster center; similar, Further improve the feature representation z i and To learn from each other the properties of normal modes;

[0025] Cluster scatter contrastive learning: For the predefined K clusters, a contrastive loss is constructed for these K clusters, because for one cluster, the remaining K-1 clusters must be negative examples; encourage feature learning and feature consistency between the two views, thereby maximizing the inter-class distance;

[0026] From the embedding space {u1,u2,...,u K} obtains K feature representations from the first view f(·|Θ) and from the embedding space The proposed clustering scattering contrast loss The definition is as follows:

[0027]

[0028] Among them, the feature representation u k and u k ' is calculated as follows:

[0029]

[0030] where p(k|·)∈{0,1} is the cluster assignment posterior probability; in formula (3) It can be roughly divided into formula (4): cluster alignment and cluster uniformity;

[0031] Furthermore, by replacing u k and u k' to rewrite Align the items as follows:

[0032]

[0033]

[0034] in represents the weight of each feature pair; Formula (10) shows that The alignment term in is expressed as a reweighted contrastive loss;

[0035] From formula (10): when z i and belong to the same cluster, that is, Then there is ω i,j >0.2; when z i and When belonging to different clusters, there is ω i,j =0; therefore, The alignments in contain only pairs of samples that belong to the same cluster; therefore, The comparison term is a generalized case of instance reweighted contrastive loss considering pseudo labels; observe that The consistency term in is to maximize the distance between instances in different clusters (j≠k); the alignment part further forces the two views to learn fine-grained spatiotemporal properties from each other, and the uniformity part further improves the difference between different kinds of spatiotemporal representations;

[0036] During the training phase, the total loss of training two network branches simultaneously is:

[0037]

[0038] When using the cross-view cluster-level contrastive loss in formula (2) During back propagation, the stop gradient operation is applied to the second network branch f′(·|Θ′); the parameter Θ′ in the second network branch is only used and Update; Additionally, the k-means algorithm is performed on the features extracted by the extractor branch f(·|Θ) instead of f′(·|Θ′) because f(·|Θ) performs more stably and produces more consistent clustering.

[0039] Further as a preferred technical solution of the present invention, the step 3 specifically comprises the following steps:

[0040] After training, the spatiotemporal representation z is used i and Test; in each inference segment c={I1,I2,...,I t} is fed frame by frame to f(·|Θ),f′(·|Θ′); use density estimation to score samples as normal or abnormal, where low estimated density indicates abnormality; compute spatiotemporal representation The L2 distance between the training cluster center and the training cluster center is expressed as s by taking their average value i

[0041] Given a test framework I t , whose score is simply the maximum score of all objects:

[0042]

[0043] The anomaly score of each frame in the segment is represented as s(c)={s(I1),s(I2),...,s(I T ); Since the abnormal events are expected to be prolonged, the result is smoothed by applying a time-domain one-dimensional Gaussian filter on s(c).

[0044] The video anomaly detection method based on contrastive clustering learning network described in the present invention adopts the above technical solution and has the following technical effects compared with the prior art:

[0045] (1) This paper proposes an effective video anomaly detection framework that uses clustering information for contrastive learning and simultaneously mines different attributes of normal patterns to learn fine-grained features.

[0046] (2) This paper proposes a new cluster-level instance loss function to perform inter-view contrastive learning, which can effectively utilize the cluster-level hidden information of different views.

[0047] (3) The present invention proposes a cluster scatter contrast loss function to refine the clustering information, and verifies that the refined clustering information is significantly helpful for the contrastive learning video anomaly detection framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of a video anomaly detection method based on a contrastive clustering learning network according to the present invention;

[0049] Figure 2 It is a schematic diagram of the comparative clustering learning network framework of the present invention;

[0050] FIG3( a ) is a schematic diagram of a test video #02 from CUHKAvenue of the present invention;

[0051] FIG3( b ) is a schematic diagram of the test video #03_0059 from Shanghai Tech of the present invention;

[0052] FIG4( a ) is a schematic diagram of the scoring of the present invention in CUHKAvenue;

[0053] FIG4( b ) is a schematic diagram of the scoring of the present invention at Shanghai Tech;

[0054] FIG. 5( a ) is a comparison of the unbalanced ratio of Shanghai Tech's CCL and CCLw / o L in the present invention. cs ;

[0055] FIG5( b ) is a graph showing the comparison of Shanghai Tech's CCL and CCLw / o L in cluster statistics of the present invention. cs ;

[0056] FIG5(c) is a comparison of the CCL and CCLw / oL of Shanghai Tech in the Micro-averaged AUC of the present invention. cs ;

[0057] Figure 6 The present invention is under two different training strategies: CCL w / o L cs (Left) and CCLw / L cs (Right), Schematic diagram of data visualization of learned features and clusters through t-SNE;

[0058] FIG. 7( a ) is a schematic diagram showing the effect of the number of predefined clusters on the CUHKAvenue dataset of the present invention;

[0059] FIG7( b ) is a schematic diagram showing the effect of the number of predefined clusters of the present invention on the Shanghai Tech dataset. DETAILED DESCRIPTION

[0060] The present invention is further explained below in detail with reference to the accompanying drawings so that those skilled in the art can more deeply understand the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention.

[0061] A video anomaly detection method based on contrastive clustering learning network includes the following steps:

[0062] Contrastive Cluster Learning (CCL) method for video anomaly detection. The method of the present invention comprises three stages: preprocessing, contrastive learning, and anomaly score estimation. In the preprocessing stage of the present invention, an off-the-shelf object detector is also used to locate and classify the bounding boxes of all objects in the frame. An off-the-shelf motion estimator is applied to predict the optical flow of each frame. Then, the present invention performs contrastive cluster learning using the spatiotemporal representation of the two views. Finally, the present invention calculates the anomaly score for each test frame. The basic framework of the network is as follows Figure 1 shown.

[0063] Pre-processing: Anomalous objects in video clips usually exhibit unusual motion or activity. Therefore, the present invention relies on a spatiotemporal representation associated with objects and motion.

[0064] Object detection. The method of the present invention models the framework by representing each object separately. Object-based representation is more effective than global frame-level representation. The present invention first detects all objects in each frame using an off-the-shelf object detector YOLOv3. Formally, the object detection of the present invention is t Generates a set of N object mappings {x1,x2,...,x N}.

[0065] Optical flow. The method of the present invention uses optical flow as a preliminary stage for inferring the motion of an object. It is based on the optical flow of each pair of two consecutive frames. t-1 and I t The present invention uses the existing optical flow model to extract each frame I t The optical flow map is composed of express.

[0066] Feature extraction. In general, the model of the present invention is a twin network, which consists of two branches of feature extractors f(·|Θ) and f′(·|Θ′), without sharing parameters, where Θ and Θ′ are the parameters in the two extractors, respectively, and a predictor layer g(·|Φ) is added after the second branch, where Φ represents the parameters in the predictor layer. The feature extractors f(·|Θ) and f′(·|Θ′) are implemented by ResNet-50 for feature learning. For simplicity, the output features of the first feature extractor and the second feature extractor are represented as z i and And the output of the predictor layer in the second branch is represented as where i∈[1,N].

[0067] Contrasting cluster learning: At the beginning, the present invention pre-trains the first extractor branch f(·|Θ) and uses f(·|Θ) to generate K clusters, which are denoted as C = {c1, c2, ..., c K The clustering results are used to form pseudo labels to train the contrastive learning module.

[0068] Although the powerful deep features from f(·|Θ) are very expressive, bundling together many different object spatiotemporal properties. To force f(·|Θ) to be learned from another view f′(·|Θ′) and exploit the clustering structure, the present invention adopts three types of contrastive learning: a) object-level contrastive learning, b) inter-view cluster-level contrastive learning, and c) cluster-scattering contrastive learning, as Figure 2 As shown:

[0069] Target-level contrastive learning. In order to force the representation z of two spatiotemporal views i and Learning from each other at the object level, the present invention introduces the feature output z in the first branch i and the predicted output of the second branch The negative cosine similarity of , with object-level contrast loss defined as follows

[0070]

[0071] L ob Can effectively promote feature representation Learn the normal pattern properties in other views and enhance the attention to certain normal pattern properties in own views.

[0072] Cluster-level contrastive learning between views. In order to further explore the hidden information in different views by using the clustering structure, the present invention proposes cluster-level contrastive learning between views. Specifically, the present invention transforms the vector z i Input into the pre-trained clustering module and calculate the matching cluster center c k , and is defined as the center vector of the matched k-th cluster. The inter-view cluster-level contrast loss is defined as follows:

[0073]

[0074] Utilizing inter-view cluster-level contrast loss Update z i , and then update the cluster center. similar, The feature representation z can be further improved i and To learn the properties of normal mode from each other.

[0075] Cluster scatter contrast learning. A good clustering should have good cluster separation, which can strengthen the distinction between normal attributes and abnormal attributes, or between different attributes, and can also better narrow the gap between the same attributes. For the predefined K clusters, the present invention constructs a contrast loss for these K clusters, because for one cluster, the remaining K-1 clusters must be negative examples. Therefore, the present invention proposes cluster scatter contrast learning, which encourages feature learning and feature consistency between two views, thereby maximizing the inter-class distance.

[0076] Specifically, suppose that the present invention embeds the space {u1,u2,...,u K} obtains K feature representations from the first view f(·|Θ) and from the embedding space Another K features in another view f′(·|Θ′) represent the clustered scattering contrast loss proposed in the present invention. The definition is as follows:

[0077]

[0078] Among them, the feature representation u k and u k ' is calculated as follows:

[0079]

[0080] where p(k|·)∈{0,1} is the cluster assignment posterior probability. In formula (3) It can be roughly divided into formula (4): cluster alignment and cluster uniformity. On the one hand, cluster alignment is to learn different spatiotemporal attributes from two views, which can stabilize the update of clusters and narrow the gap between the same normal attributes. On the other hand, cluster uniformity is to make clusters evenly distributed on a unit hypersphere, which can maximize the distance between clusters and improve the distinction between normal attributes and abnormal attributes in the test phase.

[0081] In addition, the present invention can be replaced by k and u k ' to rewrite Align the items as follows:

[0082]

[0083]

[0084] in represents the weight of each feature pair. Formula (10) shows that The alignment term in can be expressed as a reweighted contrastive loss.

[0085] From formula (10), we can know that: 1) When z i and When they belong to the same cluster, Then there is ω i,j >0.2.2) When z i and When belonging to different clusters, the present invention has ω i,j = 0. Therefore, The comparison items in contain only sample pairs that belong to the same cluster. Therefore, The comparison term of is a generalized case of instance reweighted contrastive loss considering pseudo labels. Similarly, it can be observed that The consistency term in is to maximize the distance between instances in different clusters (j≠k). Overall, the alignment part can further force the two views to learn fine-grained spatiotemporal properties from each other, and the uniformity part can further improve the difference between different kinds of spatiotemporal representations.

[0086] During the training phase, the present invention simultaneously trains the total loss of two network branches:

[0087]

[0088] Note that when using the cross-view cluster-level contrastive loss in Eq. (2) When performing back propagation, the present invention adopts a stop gradient operation for the second network branch f′(·|Θ′). Therefore, the parameter Θ′ in the second network branch only uses and Update. In addition, the present invention performs the k-means algorithm on the features extracted by the extractor branch f(·|Θ) instead of f′(·|Θ′), because f(·|Θ) performs more stably and produces more consistent clustering.

[0089] The proposed contrastive clustering learning network training process is given in Algorithm 1.

[0090]

[0091]

[0092] Anomaly score estimation: After training, the present invention uses the spatiotemporal representation z i and In each inference segment c={I1,I2,...,I t} is fed to f(·|Θ),f′(·|Θ′) frame by frame. The present invention uses density estimation to score samples as normal or abnormal, where low estimated density indicates abnormality. The present invention calculates the spatiotemporal representation The L2 distance between the training cluster center and the training cluster center is expressed as s by taking their average value i .

[0093] Given a test framework I t , its score is simply the maximum score of all objects.

[0094]

[0095] The present invention expresses the abnormal score of each frame in the segment as s(c)={s(I1),s(I2),...,s(I T ). Since the present invention expects that the abnormal event will be prolonged, the present invention smoothes the result by applying a time-domain one-dimensional Gaussian filter on s(c).

[0096] In specific implementation, the present invention provides a video anomaly detection method based on contrastive clustering learning network, comprising the following steps:

[0097] (1) Dataset Introduction

[0098] We evaluate our approach on three benchmark datasets: UCSD Ped2, CUHK Avenue, and Shanghai Tech, and compare the performance with the state-of-the-art.

[0099] UCSD Ped2. 16 short video sequences are used as training sets, and 12 short video sequences are used as test sets. Each sequence contains more than 100 frames with a resolution of 360×240.

[0100] CUHK Avenue. There are 16 training clips and 21 test clips in CUHK Avenue. The video was shot on the campus avenue of the Chinese University of Hong Kong, with a total of 30,652 frames (15,328 frames for training and 15,324 frames for testing), with a resolution of 640×360.

[0101] Shanghai Tech. Shanghai Tech is one of the most challenging datasets for video anomaly detection. It contains 13 scenes with complex lighting conditions and camera angles, 130 abnormal events and more than 270,000 training frames with a resolution of 480×856.

[0102] (2) Implementation details

[0103] In the object detection stage, we use the YOLOv 3 detector to obtain frame-level segmentation maps, which are pre-trained on the MS COCO dataset. Specifically for UCSD Ped 2, CIHK Avenue, and Shanghai Tech, we set the confidence threshold to 0.5, 0.8, and 0.8, respectively. We use the pre-trained Mask R-CNN to obtain frame-level segmentation maps, and pre-train LiteFlow 3 to obtain frame-level optical flow maps. For the size of the optical flow / object map, we adjust it to 64×64 in all three datasets. We use ResNet 50 to pre-train two network branches on ImageNet to obtain the feature representation z i ∈R D and Where D = 2048. The prediction layer is a D×D fully connected layer.

[0104] (3) Model performance comparison

[0105] Evaluation Metrics. Our study follows the popular evaluation metric in the video anomaly detection literature, measuring the frame-level Area Under the Receiver Operation Characteristic (AUC) relative to the ground-truth annotation by varying the threshold of the anomaly score. We employ two types of AUC: (i) Micro-averaged AUC, which is calculated by concatenating frames from all videos and then calculating the score. (ii) Macro-averaged AUC, which is calculated by averaging the frame-level AUC for each video.

[0106] To demonstrate the effectiveness of the proposed contrast clustering learning network, we compare our method with the state-of-the-art techniques in recent years [1, 33-41] in Table 1. We report the micro and macro average AUC (where available) for the three most commonly used public datasets: UCSD Ped 2, CUHK Avenue, and Shanghai Tech.

[0107] Table 1: AUC of different methods on UCSD Ped2, CUHKAvenue and Shanghai Tech datasets. Bold numbers indicate the best results.

[0108]

[0109] UCSD Ped 2 Results. UCSD Ped 2 is a long-standing video anomaly detection dataset and has therefore been reported by many previous papers. Most methods achieved over 94% micro average AUC on UCSD Ped 2, indicating that it is the simplest of the three public datasets. While our method is comparable to the current state-of-the-art method (HF2-VAD) in terms of performance, it also provides a higher detection speed (48FPS for our comparison clustering learning network and 13FPS for HF2-VAD). The near-perfect results of our method on Ped 2 indicate that it has actually solved the problem.

[0110] CUHK Avenue Results. CUHK Avenue is a dataset of different complexity from UCSD Ped2. Nevertheless, the method of the present invention applied to this dataset achieved a new state-of-the-art AUC (micro-averagedAUC) of 92.9%. In addition, the performance of the method of the present invention reached 95.0% macro-averaged AUC. Shanghai Tech Results. Although the previous state-of-the-art method (Georgescu et al.) exceeded the method of the present invention by 0.5% in terms of macro-averaged AUC. The method of the present invention outperformed all previous methods on the most difficult dataset,

[0111] Shanghai Tech results. In micro-averaged AUC. Therefore, our method achieves a micro-averaged AUC of 83.1%, while the highest performance achieved by the previous method (Georgescu et al.) is 82.7%, exceeding the current state-of-the-art by 0.4%.

[0112] In summary, our approach achieves state-of-the-art performance on three of the most commonly used public benchmarks.

[0113] Qualitative Analysis: In order to qualitatively analyze and compare the video anomaly detection performance of the cluster learning network, some anomaly detection examples at CUHK Avenue and Shanghai Tech are shown in Figure 3(a) and Figure 3(b).

[0114] It can be observed that the proposed method has excellent performance in detecting abnormal events. When an anomaly appears (disappears), the corresponding regularity score decreases (increases) rapidly and remains at a relatively stable level until the anomaly disappears (appears). Specifically, there are five abnormal events in the two selected test videos, and the method of the present invention provides peak anomaly scores for all five events, reaching frame-level AUCs of 99.67% and 99.87% on the two test videos, respectively. However, the method of Georgescu et al. exhibits low anomaly scores for many abnormal frames and therefore has a much higher false negative rate. Therefore, the frame-level AUCs obtained by Georgescu et al. on the two test videos are only 93.24% and 95.27%, respectively.

[0115] Ablation Experiments: To better understand the factors that affect the performance of our method, we performed ablation studies on the CUHK Avenue and Shanghai Tech datasets. We report the anomaly detection performance of all loss combinations in Table 2.

[0116] Table 2. Ablation study results for CUHK Avenue and Shanghai Tech. Anomaly detection performance is reported as micro-averaged and macro-averaged AUC. Bold numbers indicate the best results.

[0117]

[0118]

[0119] The research results of the present invention show that cluster scattering loss CUHK Avenue and Shanghai Tech provide the highest frame-level AUC, with micro-averaged AUCs of 87.2% and 78.8% respectively. At Shanghai Tech, the proposed cluster scattering loss It is close to the state-of-the-art compared with all previous video anomaly detection methods. The present invention believes that this is because it aggregates samples of the same type, separates samples of different types, promotes deep representation to contain more anomaly-related attributes, and effectively improves the discrimination of different samples.

[0120] and The combination resulted in a micro-averaged AUC of 78.4% for Shanghai Tech. and All are key to our model, and they promote deep feature representation to learn normal pattern properties in different views from object level and cluster level respectively. However, we observe that no combination of losses can achieve complete improvement when used alone. This is because all three losses have the function of promoting different views to learn different properties related to anomalies from each other. As for Avenue, our clustering-based representation performs well when combined with all losses, leading to state-of-the-art results of 92.9% micro-averaged AUC and 95.0% macro-averaged AUC.

[0121] In order to better demonstrate the effectiveness of different losses, the present invention calculates the average anomaly scores of all normal and abnormal frames in the test set corresponding to each sub-ablation study, and obtains the corresponding gap scores, such as Figure 4(a)-4(b) As shown. It can be seen that the proposed arbitrary loss can make abnormal frames get higher anomaly scores, while normal frames get lower anomaly scores. This shows that using any of the proposed losses can better distinguish normal and abnormal frames. In general, the present invention observes that using three losses is the key to achieving state-of-the-art results.

[0122] Further analysis of the proposed scattering loss: The proposed contrastive clustering learning network can encourage the uniformity of cluster representation through cluster-level scattering loss For the clustering task with 16 cluster centers, we perform spherical k-means on the learned representations. To improve the effectiveness of the proposed method, the present invention conducts the following experiments on the uniformity of representation, cluster collapse and cluster performance: Figure 5(a)-Figure 5(c) As shown. Average score Average score;

[0123] First, the present invention visualizes cluster imbalance to measure potential breakdown in the clustering process. More specifically, the present invention calculates the cluster imbalance ratio between the cluster with the fewest samples and the cluster with the most samples. Where N k is the number of samples in the kth cluster. The higher the value, the more balanced the clusters. In addition, the present invention also shows the cluster statistics of the model at the 100th epoch or the ranked number of samples in each cluster. Figure 5(a) and Figure 5(b) show the results of cluster imbalance during training and the cluster statistics of the last epoch. Figure 5(a) shows The k-means clustering procedure with produces more balanced clusters with a higher cluster imbalance rate. In contrast, The CCL clusters are highly unbalanced. Finally, Figure 5(c) shows CCL and does not contain The clustering performance of the CCLs is compared by measuring the micro-average AUC value. Compared with CCL, The CCL w / produced a higher and more stable Micro-averaged AUC. Consistent with the above analysis, the present invention concluded that CCL w / Produces more evenly represented and well-clustered samples, leading to better discrimination between normal and anomalous samples.

[0124] Data Visualization: In order to intuitively understand the performance of the proposed CCL, the author conducted a set of data visualization experiments at Shanghai Tech to visualize the clustering results of the learned features when using different training strategies: a) CCL w / and b) CCL w / o. The experimental results are shown in Figure 6As shown. The present invention can be observed that the cluster-level scatter loss helps the model distinguish those similar images while maintaining cluster compactness, and also separates normal objects and abnormal objects with AUC cluster count imbalance ratio from each other. This confirms the effectiveness of the cluster-level scatter loss proposed by the present invention. Further, the present invention analyzes how the alignment part and the uniformity part affect the AUC performance of CCL, as shown in Table 3. The present invention can see that both the alignment part and the uniformity part are essential for improving the performance of CCL, which once again proves the effectiveness of the proposed cluster-level scatter loss.

[0125] Table 3: Further study on the proposed cluster-level scattering loss. Anomaly detection performance is reported in the form of Micro-averaged and Macro-averaged AUC. Bold numbers indicate the best results.

[0126]

[0127] Further study of different cluster numbers: To further study the effect of K value in k-means, this paper reports the results of different K values ​​for CUHK Avenue and Shanghai Tech, e.g. Figure 7(a) , 7(b) As shown. For the over-clustered case (the number of clusters is greater than 40), the trends of the two datasets are similar. In other cases, if the representations are well aligned in the same semantic clusters, then over-clustering will try to destroy the structure of the clusters and push semantically similar examples away, which will definitely hurt the clustering performance. For the under-clustered case (the number of clusters is less than 28), the clustering performance of both datasets and both methods is significantly hurt.

[0128] This paper proposes a simple and efficient contrastive clustering learning method that can improve the accuracy and speed of video anomaly detection. In each frame, the present invention uses object-level and motion-level spatiotemporal representations to represent each object, and then performs density-based anomaly scoring. The contrastive clustering learning network between these different spatiotemporal views enables the present invention to achieve state-of-the-art results on Shanghai Tech, the most complex video anomaly dataset.

[0129] The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. A video anomaly detection method based on contrastive clustering learning network, characterized in that: The following steps are involved: Step 1: Preprocessing: Use an object detector to locate and classify the bounding boxes of all objects in the frame, and apply a motion estimator to predict the optical flow of each frame. Step 2: Contrastive clustering learning: Use the spatiotemporal representations of the two views to perform contrastive clustering learning. Step 3: Anomaly score estimation: Use density estimation to score the samples as normal or abnormal, where low estimated density indicates anomalies.

2. The video anomaly detection method based on contrastive clustering learning network according to claim 1 is characterized in that: The step 1 specifically comprises the following steps: Use the object detector YOLOv3 to detect all objects in each frame. Object detection is performed for each frame I t Generates a set of N object mappings {x1,x2,...,x N }; Using optical flow as a preliminary stage to infer object motion is done in each pair of two consecutive frames I t-1 and I t Calculated between; the optical flow model extracts each frame I t The optical flow graph is given by Representation; The optical flow model is a twin network, which consists of two branches of feature extractors f(·|Θ) and f′(·|Θ′), without sharing parameters, where Θ and Θ′ are the parameters in the two extractors, respectively, and a predictor layer g(·|Φ) is added after the second branch, where Φ represents the parameters in the predictor layer; the feature extractors f(·|Θ) and f′(·|Θ′) are implemented by ResNet-50 for feature learning; the output features of the first feature extractor and the second feature extractor are represented as z i and And the output of the predictor layer in the second branch is represented as where i∈[1,N].

3. The video anomaly detection method based on contrastive clustering learning network according to claim 2 is characterized in that: The step 2 specifically includes the following steps: At the beginning, the first extractor branch f(·|Θ) is pre-trained and used to generate K clusters, which are denoted as C = {c1,c2,...,c K The clustering results are used to form pseudo labels to train the contrastive learning module; To force f(·|Θ) to be derived from another view f′(·|Θ′) and exploit the clustering structure, three types of contrastive learning are adopted: object-level contrastive learning, inter-view cluster-level contrastive learning, and cluster-scattering contrastive learning; Object-level contrastive learning: To make the representation z of two spatiotemporal views i and Learning from each other at the object level, introducing the feature output z in the first branch i and the predicted output of the second branch The negative cosine similarity of , with the object-level contrast loss L defined as follows ob L ob Can effectively promote feature representation Learn the normal pattern properties in other views and enhance attention to certain normal pattern properties in own views; Inter-view cluster-level contrastive learning: vector z i Input into the pre-trained clustering module and calculate the matching cluster center c k , and is defined as the center vector of the matched k-th cluster; the inter-view cluster-level contrast loss is defined as follows: Using inter-view cluster-level contrast loss L iv Update z i , and then update the cluster center; and L ob Similar, L iv Further improve the feature representation z i and To learn from each other the properties of normal modes; Cluster scatter contrastive learning: For the predefined K clusters, a contrastive loss is constructed for these K clusters, because for one cluster, the remaining K-1 clusters must be negative examples; encourage feature learning and feature consistency between the two views, thereby maximizing the inter-class distance; From the embedding space {u1,u2,...,u K } obtains K feature representations from the first view f(·|Θ) and from the embedding space The proposed clustered scatter contrast loss L cs , defined as follows: Among them, the feature representation u k and u k ' is calculated as follows: where p(k|·)∈{0,1} is the cluster assignment posterior probability; L in formula (3) cs It can be roughly divided into formula (4): cluster alignment and cluster uniformity; Furthermore, by replacing u k and u k ′ to rewrite L cs Align the items as follows: in represents the weight of each feature pair; Formula (10) shows that L cs The alignment term in is expressed as a reweighted contrastive loss; From formula (10): when z i and belong to the same cluster, that is, Then there is ω i,j >0.2; when z i and When belonging to different clusters, there is ω i,j =0; therefore, L cs The alignments in L contain only pairs of samples that belong to the same cluster; therefore, L cs The comparison term is a generalized case of instance reweighted contrast loss considering pseudo labels; observe that L cs The consistency term in is to maximize the distance between instances in different clusters (j≠k); the alignment part further forces the two views to learn fine-grained spatiotemporal properties from each other, and the uniformity part further improves the difference between different kinds of spatiotemporal representations; During the training phase, the total loss of training two network branches simultaneously is: L total =L ob +L iv +L cs (11) When using the cross-view cluster-level contrast loss L in formula (2) iv During back propagation, the stop gradient operation is applied to the second network branch f′(·|Θ′); the parameter Θ′ in the second network branch only uses L ob and L cs Update; Additionally, the k-means algorithm is performed on the features extracted by the extractor branch f(·|Θ) instead of f′(·|Θ′) because f(·|Θ) performs more stably and produces more consistent clustering.

4. The video anomaly detection method based on contrastive clustering learning network according to claim 3 is characterized in that: The step 3 specifically includes the following steps: After training, the spatiotemporal representation z is used i and Test; in each inference segment c={I1,I2,...,I t } is fed frame by frame to f(·|Θ),f′(·|Θ′); use density estimation to score samples as normal or abnormal, where low estimated density indicates abnormality; compute spatiotemporal representation The L2 distance between the training cluster center and the training cluster center is expressed as s by taking their average value i Given a test framework I t , whose score is simply the maximum score of all objects: The anomaly score of each frame in the segment is represented as s(c)={s(I1),s(I2),...,s(I T ); Since the abnormal events are expected to be prolonged, the result is smoothed by applying a time-domain one-dimensional Gaussian filter on s(c).

Images