A video anomaly detection method based on a contrast clustering learning network

Abstract

The present application belongs to the technical field of intelligent video processing, and particularly relates to a video anomaly detection method based on a contrastive clustering learning network. The present application comprises the following steps: step 1, preprocessing: using an object detector to locate and classify the bounding boxes of all objects in the frame, and applying a motion estimator to predict the optical flow of each frame; step 2, contrastive clustering learning: using the spatiotemporal representation of two views to perform contrastive clustering learning; step 3, anomaly score estimation: using density estimation to score samples as normal or abnormal, wherein a low estimated density indicates abnormality. The present application uses clustering information for contrastive learning, and simultaneously mines different attributes of normal patterns to learn fine-grained features. The present application can effectively utilize the clustering-level hidden information of different views. The present application proposes a clustering scattering contrastive loss function to refine the clustering information, and verifies that the refined clustering information significantly helps the contrastive learning video anomaly detection framework.

Description

Technical Field

[0001] This invention belongs to the field of intelligent video processing technology, specifically relating to a video anomaly detection method based on a contrastive clustering learning network. Background Technology

[0002] Video anomaly detection is a computer vision task designed to automatically identify unusual or uncommon events in videos, typically performed in an unsupervised or semi-supervised learning context. It has significant practical applications in video surveillance, intelligent transportation, and public safety. Traditional video anomaly detection methods usually rely on large amounts of labeled data and predefined anomaly patterns, which presents several significant challenges in practical applications. First, acquiring large amounts of labeled data is both time-consuming and labor-intensive, and impractical in many cases. Second, anomaly events are diverse in type and form, making comprehensive definition and coverage difficult. Finally, existing methods often perform poorly when handling unseen anomalies, exhibiting limited generalization ability.

[0003] Existing video anomaly detection neural networks typically capture pixel-level spatiotemporal features, which are usually derived from the entire frame, located on a grid of image patches, or concentrated in preprocessing regions. For example, Liu et al. employed spatial constraints on intensity and gradients and introduced temporal constraints in video frame prediction; Le et al. used time shifts to model temporal information and encode spatiotemporal information in a unified manner; Zhao et al. proposed using 3D convolution to preserve temporal information in convolution and using two decoders to enhance motion characteristics in decoding. These methods fully consider spatiotemporal information and achieve good results, but they process the entire frame image, and the resulting spatiotemporal features often contain a large amount of non-target information that is weakly correlated with normal patterns. Georgescu et al. proposed a video anomaly detection framework that focuses only on object detection, including three branches and seven tasks, further improving the performance of video anomaly detection. While this object-level multi-task learning can better capture fine-grained spatiotemporal features related to normal patterns, it also involves repeatedly learning certain features, resulting in wasted resources and further affecting the execution speed of video anomaly detection tasks.

[0004] Reconstruction-based methods generally use autoencoders to reconstruct normal samples during the training phase, and then use the trained autoencoder to identify anomalies based on the reconstruction error during the testing phase. For example, Hasan et al. used convolutional autoencoders to learn features and capture patterns. As a pioneering work, this method is a good attempt to use convolutional autoencoders for anomaly detection, but it also has some drawbacks, such as the inability of two-dimensional convolutions to capture temporal information. Ravanbakhsh et al. used a U-Net network structure to achieve cross-modal reconstruction of video frames and optical flow maps, and then used the cross-modal reconstruction error to identify anomalies. CAE can only capture spatial structural information and not temporal information. Gong et al. developed an improved autoencoder called a memory-enhanced autoencoder. The core idea is that the designed normal memory stores the prototype of the normal pattern. Given anomaly input, the most relevant normal pattern is retrieved from the memory and reconstructed, thus producing an output significantly different from the anomaly input. Zhao et al. proposed using three-dimensional convolution to store temporal information in the convolution phase and using two decoders to enhance motion characteristics in the decoding phase. Given the wide adaptability and strong generalization ability of neural networks, this does not necessarily mean that anomalies will have higher reconstruction errors during the reconstruction process.

[0005] Future frame prediction aims to model the (t+1)th frame using the previous t frames. Liu et al. proposed a video frame prediction network to predict future video frames based on previous video frames. This was the first work to use the reconstruction error of the predicted future frame to detect anomalous events. However, sometimes the autoencoder generalizes very well, so much so that it can also reconstruct anomalous samples well, leading to missed anomalies. To mitigate this drawback of autoencoder-based methods, several works have made different improvements. Liu et al. combined previous research, designing a composite framework that integrates reconstruction and prediction tasks. First, the optical flow field is reconstructed, then the video frame and optical flow are fused, and the hybrid frame is used as input for future frame prediction, increasing the prediction error of anomalous events and thus improving anomaly detection. Chen et al. proposed a simple bidirectional prediction framework, but forward and backward prediction in the event dimension can only detect anomalies after the abnormal behavior occurs, and cannot provide real-time early warning. Le et al. used time shift to model temporal information, encoding spatiotemporal information in a unified way, and provided a self-contained encoder based on residual attention. By introducing channel attention, it can more effectively utilize the cascaded channel dependencies within the decoder to predict future frames.

[0006] Cluster-based video anomaly detection methods identify anomalies by fitting a normal sample space and clustering normal samples, identifying samples far from the normal cluster centers as anomalies. Qiu et al. designed a dual-scale feature clustering structure for the encoder, simultaneously compressing the channel and spatial structure features of the image to represent features, obtaining good encoded features and expanding the boundary between normal and anomalous data. Georgescu et al. proposed a video anomaly detection framework that focuses solely on object detection, including three branches and seven tasks, further improving the performance of video anomaly detection. They operate at the object level, then determine whether each object is anomalous. This is a simpler challenge because objects have less diversity than the entire video frame. The results of object-level methods significantly outperform those of framework-level methods. Most existing object-level anomaly detection algorithms are based on multi-task learning. Multi-task learning can indeed fully learn the different attributes of normal patterns, thus detecting anomalous events well. However, in multi-task learning, the repeated learning of some features leads to a waste of computational and storage resources, reducing the speed of anomaly detection. This invention, based on object-level auxiliary tasks, utilizes clustering for comparative learning between different views, quickly and comprehensively learning the different attributes of normal patterns, achieving a detection speed far exceeding that of multi-task learning.

[0007] To address these issues, contrastive clustering offers an effective solution. Contrastive clustering does not require large amounts of labeled data. It automatically captures normal patterns by learning the internal structure and similarities of the data, thereby identifying anomalous events that deviate from these patterns. Summary of the Invention

[0008] This invention addresses the problem that existing video anomaly detection methods typically rely on preprocessed image regions of the target detector. These methods heavily depend on multi-task learning, leading to repetitive learning of certain features and wasted computational resources. This invention proposes a video anomaly detection method based on a contrastive clustering learning network.

[0009] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0010] A video anomaly detection method based on a contrastive clustering learning network includes the following steps:

[0011] Step 1, Preprocessing: Use an object detector to locate and classify the bounding boxes of all objects in the frame, and apply a motion estimator to predict the optical flow of each frame;

[0012] Step 2, Contrastive Clustering Learning: Perform contrastive clustering learning using spatiotemporal representations of the two views;

[0013] Step 3, Anomaly Score Estimation: Use density estimation to score the samples as normal or abnormal, where a low estimated density indicates anomaly.

[0014] Furthermore, as a preferred embodiment of the present invention, step 1 specifically includes the following steps:

[0015] All objects in each frame are detected using the YOLOv3 object detector; object detection is performed per frame. Generate a set containing object mapping Using optical flow as the initial stage for inferring object motion occurs between every pair of consecutive frames. and Calculated between frames; optical flow model extracts each frame. Optical flow; optical flow diagram is composed of The optical flow model is represented as a Siamese network, consisting of a feature extractor. and It consists of two branches that do not share parameters, where and These are the parameters from the two extractors, and a predictor layer is added after the second branch. ,in Represents the parameters in the predictor layer; feature extractor and Implemented using ResNet-50 for feature learning; the output features of the first and second feature extractors are respectively represented as... and And the output of the predictor layer in the second branch is expressed as follows: ,in .

[0016] Furthermore, as a preferred embodiment of the present invention, step 2 specifically includes the following steps:

[0017] At the beginning, the first extractor branch is pre-trained. and use To generate There are clusters, denoted as: The value of ; the clustering results are used to form pseudo-labels to train the contrastive learning module;

[0018] If forced From another view It also utilizes clustering structures and employs three types of contrastive learning: target-level contrastive learning, view-to-view cluster-level contrastive learning, and cluster scattering contrastive learning.

[0019] Target-level contrastive learning: to enable the representation of two spatiotemporal views and Mutual learning at the object level incorporates feature outputs from the first branch. Predicted output of the second branch The negative cosine similarity is defined by the object-level contrast loss as follows.

[0020] (1)

[0021] It can effectively promote feature representation Learn about the normal mode properties in other views to enhance your awareness of certain normal mode properties in your own view;

[0022] Inter-view clustering-level contrastive learning: transforming vectors Input is fed into a pre-trained clustering module and the matching cluster centers are calculated. and will Defined as the first matching The center vectors of each cluster; the cluster-level contrast loss between views is defined as follows:

[0023] (2)

[0024] Using inter-view cluster-level contrast loss renew Some attributes are used to update the cluster centers; and with similar, Further improve feature representation and To learn the attributes of normal mode from each other;

[0025] Clustered scattering contrastive learning: for a predefined Clusters, for these Each cluster constructs a contrastive loss because for one cluster, the rest... Each cluster is definitely a negative example; it encourages feature learning and feature consistency between two views, thereby maximizing inter-class distance;

[0026] From embedded space The first view Obtain Each feature representation, and from the embedding space Another view China The proposed clustered scattering contrast loss is represented by several features. The definition is as follows:

[0027] (3)

[0028] (4)

[0029] Among them, feature representation and The calculation is as follows:

[0030] (5)

[0031] (6)

[0032] in It is the posterior probability of cluster assignment; in formula (3) It can be approximately divided into formula (4): cluster alignment and cluster uniformity;

[0033] In addition, by replacing and To rewrite Alignment items, as shown below:

[0034] (7)

[0035] (8)

[0036] (9)

[0037] (10)

[0038] in , , This represents the weight of each feature pair; Formula (10) shows that... The alignment term in the equation is represented as a reweighted contrast loss;

[0039] From formula (10): when and When they belong to the same cluster, that is Then there is ;when and When they belong to different clusters, there are ;therefore, The comparison terms in the table only include sample pairs belonging to the same cluster; therefore, The comparison term is a generalized case that considers instance reweighted contrast loss for pseudo-labels; it was observed that... The consistency term in the code maximizes the distance between instances in different clusters. The alignment component further forces the two views to learn fine-grained spatiotemporal properties from each other, and the uniformity component further improves the differences between different kinds of spatiotemporal representations.

[0040] During the training phase, the total loss for training both network branches simultaneously is:

[0041] (11)

[0042] When using the cross-view clustering level contrast loss in formula (2) During backpropagation, the second network branch A gradient-stopping operation was used; the parameters in the second network branch Only use and Update; additionally, for the extractor branch The extracted features are processed using the k-means algorithm instead of ,because The execution is more stable, resulting in more consistent clustering.

[0043] Furthermore, as a preferred embodiment of the present invention, step 3 specifically includes the following steps:

[0044] After training, spatiotemporal representation is used. and Perform tests; in each inference fragment Feeded frame by frame , ; Use density estimation to score samples as normal or abnormal, where low estimated density indicates abnormality; compute spatiotemporal representation. The L2 distance between the training cluster centers and the training cluster centers is expressed as their average value.

[0045] Given a test framework Its score is simply the highest score among all objects:

[0046] 1. (12)

[0047] The anomaly score for each frame in the segment is represented as: Because the abnormal event is expected to be prolonged, therefore... A one-dimensional Gaussian filter in the time domain is applied to smooth the results.

[0048] The video anomaly detection method based on a contrastive clustering learning network described in this invention has the following technical advantages compared with existing technologies:

[0049] (1) This invention proposes an effective video anomaly detection framework that utilizes clustering information for comparative learning and mines different attributes of normal patterns to learn fine-grained features.

[0050] (2) This invention proposes a new cluster-level instance loss function to perform inter-view comparative learning, which can effectively utilize the cluster-level hidden information of different views.

[0051] (3) This invention proposes a clustering scattering contrast loss function to refine the clustering information and verifies that the refined clustering information is of great help to the contrast learning video anomaly detection framework. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the video anomaly detection method based on contrastive clustering learning network of the present invention;

[0053] Figure 2 This is a schematic diagram of the contrastive clustering learning network framework of the present invention;

[0054] Figure 3(a) is a schematic diagram of the test video #02 from CUHK Avenue of the present invention;

[0055] Figure 3(b) is a schematic diagram of the test video #03_0059 from Shanghai Tech of the present invention;

[0056] Figure 4(a) is a schematic diagram of the scoring of the present invention at CUHK Avenue;

[0057] Figure 4(b) is a schematic diagram of the scoring of this invention at Shanghai Tech.

[0058] Figure 5(a) shows the comparison of CCL and CCL w / o of Shanghai Tech in terms of imbalance rate in this invention. ;

[0059] Figure 5(b) shows the comparison of CCL and CCL w / o of Shanghai Tech in cluster statistics using the present invention. ;

[0060] Figure 5(c) shows a comparison of Shanghai Tech's CCL and CCLw / o in terms of Micro-averaged AUC. ;

[0061] Figure 6 The present invention relates to two different training strategies: CCL w / o (Left) and CCL w / (Right) A diagram illustrating data visualization of learned features and clusters using t-SNE;

[0062] Figure 7(a) is a schematic diagram illustrating the impact of the predefined number of clusters in this invention on the CUHK Avenue dataset;

[0063] Figure 7(b) is a schematic diagram illustrating the impact of the predefined number of clusters in this invention on the Shanghai Tech dataset. Detailed Implementation

[0064] The present invention will be further explained in detail below with reference to the accompanying drawings, so that those skilled in the art can better understand and implement the present invention. However, the following examples are only used to explain the present invention and are not intended to limit the present invention.

[0065] A video anomaly detection method based on a contrastive clustering learning network includes the following steps:

[0066] A contrastive clustering learning (CCL) method for video anomaly detection. The method comprises three stages: preprocessing, contrastive learning, and anomaly score estimation. In the preprocessing stage, an off-the-shelf object detector is used to locate and classify the bounding boxes of all objects within a frame. An off-the-shelf motion estimator is applied to predict optical flow for each frame. Then, the method performs contrastive clustering learning using spatiotemporal representations of two views. Finally, the method calculates anomaly scores for each test frame. The basic network framework is as follows: Figure 1 As shown.

[0067] Preprocessing: Anomalous objects in video clips often exhibit unusual motion or activity. Therefore, this invention relies on spatiotemporal representations associated with objects and motion.

[0068] Object detection. The method of this invention models the frame by representing each object individually. Object-based representation is more efficient than global frame-level representation. This invention first uses the readily available object detector YOLOv3 to detect all objects in each frame. Formally, the object detection of this invention generates a set of objects for each frame. object mapping .

[0069] Optical flow. The method of this invention uses optical flow as a preliminary stage for inferring the motion of an object. It is performed on every pair of two consecutive frames. and The calculations are performed between frames. This invention uses an existing optical flow model to extract each frame. Optical flow. Optical flow diagrams are derived from... express.

[0070] Feature extraction. In general, the model of this invention is a Siamese network, consisting of a feature extractor. and It consists of two branches that do not share parameters, where and These are the parameters from the two extractors, and a predictor layer is added after the second branch. ,in This represents the parameters in the predictor layer. Feature extractor. and Implemented using ResNet-50 for feature learning. For simplicity, the output features of the first and second feature extractors are represented as follows: and And the output of the predictor layer in the second branch is expressed as follows: ,in .

[0071] Contrastive clustering learning: At the beginning, the invention pre-trains the first extractor branch. and use To generate There are clusters, denoted as: The value of . The clustering results are used to form pseudo-labels to train the contrastive learning module.

[0072] Although powerful depth features from It is very expressive, binding together the spatiotemporal properties of many different objects. (If forced...) From another view Utilizing clustering structures, this invention employs three types of contrastive learning: a) target-level contrastive learning, b) view-to-view cluster-level contrastive learning, and c) cluster scattering contrastive learning, such as... Figure 2 As shown:

[0073] Target-level contrastive learning. To force the representations of the two spatiotemporal views... and By learning from each other at the object level, this invention introduces the feature output from the first branch. Predicted output of the second branch The negative cosine similarity is defined by the object-level contrast loss as follows.

[0074] (1)

[0075] It can effectively promote feature representation Learn about normal mode properties in other views to enhance your awareness of certain normal mode properties in your own view.

[0076] Inter-view cluster-level comparative learning. To further explore hidden information in different views using clustering structures, this invention proposes inter-view cluster-level comparative learning. Specifically, this invention will use vectors... Input is fed into a pre-trained clustering module and the matching cluster centers are calculated. and will Defined as the first matching The center vectors of each cluster. The cluster-level contrast loss between views is defined as follows:

[0077] (2)

[0078] Using inter-view cluster-level contrast loss renew The cluster centers are updated by analyzing some of the attributes. similar, Feature representation can be further improved. and In order to learn the attributes of normal mode from each other.

[0079] Clustering scattering contrastive learning. A good cluster should have good cluster separation, which can strengthen the distinction between normal and abnormal attributes, or between different attributes, and can also better reduce the gap between the same attributes. For a predefined... Clusters, the present invention is for these Each cluster constructs a contrastive loss because for one cluster, the rest... Each cluster is necessarily a negative example. Therefore, this invention proposes clustered discrete contrastive learning, which encourages feature learning and feature consistency between two views, thereby maximizing inter-class distance.

[0080] Specifically, assuming the present invention originates from the embedded space The first view Obtain Each feature representation, and from the embedding space Another view China The feature representation of the clustered scattering contrast loss proposed in this invention is... The definition is as follows:

[0081] (3)

[0082] (4)

[0083] Among them, feature representation and The calculation is as follows:

[0084] (5)

[0085] (6)

[0086] in It is the posterior probability of cluster assignment. In formula (3) It can be approximately divided into formula (4): cluster alignment and cluster uniformity. On the one hand, cluster alignment learns different spatiotemporal attributes from two views, which can stabilize the update of clusters and reduce the gap between the same normal attributes. On the other hand, cluster uniformity promotes the uniform distribution of clusters on a unit hypersphere, which can maximize the distance between clusters and improve the distinguishability of normal and abnormal attributes during the testing phase.

[0087] Furthermore, the present invention can be replaced and To rewrite Alignment items, as shown below:

[0088] (7)

[0089] (8)

[0090] (9)

[0091] (10)

[0092] in , , This represents the weight of each feature pair. Formula (10) shows that... The alignment term in the equation can be represented as a reweighted contrast loss.

[0093] From formula (10), we can see that: 1) When and When they belong to the same cluster, that is Then there is 2) When and When belonging to different clusters, the present invention has .therefore, The comparison terms in the table only include sample pairs belonging to the same cluster. Therefore, The comparison term is a generalized case that considers instance reweighted contrast loss for pseudo-labels. Similarly, it can be observed that... The consistency term in the code maximizes the distance between instances in different clusters. Overall, the alignment component can further force the two views to learn fine-grained spatiotemporal properties from each other, and the uniformity component can further improve the differences between different kinds of spatiotemporal representations.

[0094] During the training phase, this invention simultaneously trains the total loss of two network branches:

[0095] (11)

[0096] Note that when using the cross-view clustering level contrast loss in formula (2) During backpropagation, this invention applies to the second network branch. A gradient stopping operation was used. Therefore, the parameters in the second network branch... Only use and Update. Furthermore, this invention addresses the extractor branch. The extracted features are processed using the k-means algorithm instead of ,because More stable execution, producing more consistent clusters

[0097] The training process for the proposed contrastive clustering learning network is given in Algorithm 1.

[0098]

[0099] Anomaly scoring estimation: After training, this invention uses spatiotemporal representation. and Conduct tests. In each inference fragment... Feeded frame by frame , This invention uses density estimation to score samples as normal or abnormal, where a low estimated density indicates anomaly. This invention calculates the spatiotemporal representation. The L2 distance between the training cluster centers and the training cluster centers is expressed as their average value. .

[0100] Given a test framework Its score is simply the highest score among all objects.

[0101] (12)

[0102] This invention represents the anomaly score of each frame in the segment as follows: Since this invention anticipates that abnormal events will be prolonged, this invention utilizes... A one-dimensional Gaussian filter in the time domain is applied to smooth the results.

[0103] In specific implementation, this invention provides a video anomaly detection method based on a contrastive clustering learning network, comprising the following steps:

[0104] (1) Introduction to the dataset

[0105] This invention evaluates its method on three benchmark datasets: UCSD Ped2, CUHK Avenue, and Shanghai Tech, and compares its performance with state-of-the-art techniques.

[0106] UCSD Ped2. A training set of 16 short video sequences and a test set of 12 short video sequences are used. Each sequence contains over 100 frames with a resolution of 360×240.

[0107] CUHK Avenue. CUHK Avenue contains 16 training videos and 21 test videos. The videos were filmed on the university campus avenue, totaling 30,652 frames (15,328 frames for training and 15,324 frames for testing), with a resolution of 640×360.

[0108] Shanghai Tech. Shanghai Tech is one of the most challenging datasets for video anomaly detection. It contains 13 scenes with complex lighting conditions and camera angles, 130 anomalous events, and more than 270,000 training frames at a resolution of 480×856.

[0109] (2) Implementation details

[0110] In the object detection stage, this invention uses a YOLOv3 detector to obtain frame-level segmentation maps, which are pre-trained on the MS COCO dataset. Specifically, for UCSD Ped 2, CIHK Avenue, and Shanghai Tech, the confidence thresholds are set to 0.5, 0.8, and 0.8, respectively. This invention uses a pre-trained Mask R-CNN to obtain frame-level segmentation maps and a pre-trained LiteFlow 3 to obtain frame-level optical flow maps. The size of the optical flow / object map is adjusted to 64 × 64 in all three datasets. This invention uses ResNet 50 to pre-train two network branches on ImageNet to obtain feature representations. and ,in The prediction layer is... Fully connected layer.

[0111] (3) Model performance comparison

[0112] Evaluation Metrics. This invention follows popular evaluation metrics in video anomaly detection literature, measuring the frame-level Area Under the Receiver Operation Characteristic (AUC) relative to ground-truth annotation by varying the anomaly score threshold. Two types of AUC are employed: (i) Micro-averaged AUC, calculated by concatenating frames from all videos and then calculating the score; and (ii) Macro-averaged AUC, calculated by averaging the frame-level AUC for each video.

[0113] To demonstrate the effectiveness of the proposed contrastive clustering learning network, this invention compares its method with recent state-of-the-art techniques in Table 1 [1,33-41]. This invention reports the micro and macro average AUC (if available) for three of the most commonly used publicly available datasets: UCSD Ped 2, CUHK Avenue, and Shanghai Tech.

[0114] Table 1: AUC of different methods on the UCSD Ped2, CUHK Avenue, and Shanghai Tech datasets. Bold numbers indicate the best results.

[0115]

[0116] Results on UCSD Ped 2. UCSD Ped 2 is a long-standing video anomaly detection dataset and has therefore been reported in many previous papers. Most methods achieve a micro average AUC of over 94% on UCSD Ped 2, indicating that it is the simplest among the three public datasets. While the method of this invention is comparable in performance to the current state-of-the-art method (HF2-VAD), it also offers significantly faster detection speeds (48 FPS for the contrastive clustering learning network of this invention, compared to 13 FPS for HF2-VAD). The near-perfect results of the method of this invention on Ped 2 demonstrate that it has effectively solved the problem.

[0117] Results for CUHK Avenue. CUHK Avenue is a dataset with a different complexity than UCSD Ped2. Nevertheless, the method of this invention applied to this dataset achieves a new state-of-the-art AUC (micro-averaged AUC) of 92.9%. Furthermore, the method of this invention achieves a macro-averaged AUC of 95.0%. Shanghai Tech Results. Although previous state-of-the-art methods (Georgescu et al.) outperformed the method of this invention by 0.5% in terms of macro-averaged AUC, the method of this invention outperforms all previous methods on the most difficult datasets.

[0118] Results from Shanghai Tech. In micro-averaged AUC. Therefore, the method of this invention achieves 83.1% micro-averaged AUC, while the highest performance achieved by the previous method (Georgescu et al.) is 82.7%, exceeding the current state-of-the-art level by 0.4%.

[0119] In summary, the method of this invention achieves state-of-the-art performance in three of the most commonly used public benchmarks.

[0120] Qualitative Analysis: To qualitatively analyze and compare the video anomaly detection performance of clustering learning networks, some anomaly detection examples at CUHK Avenue and Shanghai Tech are shown in Figures 3(a) and 3(b).

[0121] This invention demonstrates that the proposed method exhibits superior performance in detecting anomalous events. When an anomaly occurs (disappears), the corresponding regularity score rapidly decreases (increases) and remains at a relatively stable level until the anomaly disappears (appears). Specifically, in the two selected test videos, there were five anomalous events, and the method of this invention provided peak anomaly scores for all five events, achieving frame-level AUCs of 99.67% and 99.87% on the two test videos, respectively. However, the method of Georgescu et al. exhibited low anomaly scores for many anomalous frames, resulting in a much higher false negative rate. Therefore, the frame-level AUCs obtained by Georgescu et al. on the two test videos were only 93.24% and 95.27%, respectively.

[0122] Ablation experiments: To better understand the factors affecting the performance of the method of this invention, an ablation study was conducted on the CUHK Avenue and Shanghai Tech datasets. The anomaly detection performance of all loss combinations is reported in Table 2.

[0123] Table 2. Ablation study results from CUHK Avenue and Shanghai Tech. Anomaly detection performance is reported in terms of micro-averaged and macro-averaged AUC. Bold numbers indicate best results.

[0124]

[0125] The research results of this invention show that cluster scattering loss The highest frame-level AUCs were achieved at CUHK Avenue and Shanghai Tech, with micro-averaged AUCs of 87.2% and 78.8%, respectively. At Shanghai Tech, the proposed cluster scattering loss... This invention is among the most advanced in the country compared to all previous video anomaly detection methods. This is attributed to its ability to aggregate similar samples, separate dissimilar samples, and promote deeper representations that incorporate more anomaly-related attributes, effectively improving the discriminative power of different samples.

[0126] and The combination of these factors resulted in Shanghai Tech achieving a micro-averaged AUC of 78.4%. and These are all key components of the model in this invention, respectively facilitating the learning of normal pattern attributes from different views at the object and cluster levels in deep feature representations. However, this invention observes that no single combination of losses can achieve a complete improvement when used alone. This is because these three losses all have the function of promoting the mutual learning of different attributes related to anomalies among different views. As for Avenue, the clustering-based representation of this invention performs well when combined with all losses, resulting in state-of-the-art results of 92.9% micro-averaged AUC and 95.0% macro-averaged AUC.

[0127] To better demonstrate the effectiveness of different losses, this invention calculates the average anomalous score of all normal and anomalous frames in the test set corresponding to each sub-ablation study, and obtains the corresponding gap score, as shown in Figures 4(a)-4(b). It can be seen that the proposed arbitrary loss results in anomalous frames receiving higher anomalous scores, while normal frames receive lower anomalous scores. This indicates that using the proposed arbitrary loss can better distinguish between normal and anomalous frames. In summary, this invention observes that using three losses is key to achieving state-of-the-art results.

[0128] Further analysis of the proposed scattering loss: The contrastive clustering learning network of this invention can encourage the uniformity representation of clusters through cluster-level scattering loss. For a clustering task with 16 cluster centers, this invention performs spherical k-means on the learned representation. To demonstrate the cluster-level scattering loss... To assess the effectiveness of this invention, the following experiments were conducted on the uniformity of representation, cluster collapse, and cluster performance, as shown in Figures 5(a)-5(c). Average score;

[0129] First, this invention visualizes cluster imbalance to measure potential collapse during the clustering process. More specifically, this invention calculates the cluster imbalance ratio between the cluster with the fewest samples and the cluster with the most samples. ,in It is the first The number of samples in each cluster. A higher value indicates a more balanced cluster. Furthermore, this invention also shows the clustering statistics of the model at the 100th epoch, or the number of samples ranked in each cluster. Figures 5(a) and 5(b) show the results of cluster imbalance during training and the cluster statistics for the last epoch. Figure 5(a) shows the CCL w / The k-means clustering process produces more balanced clusters with higher imbalance rates. Conversely, without k-means clustering... The CCL clusters are highly unbalanced. Finally, Figure 5(c) shows the CCL clusters containing... CCL and without The clustering performance of CCLs was compared using micro-average AUC values. This invention shows that, compared to CCLs without... Compared to CCL, containing The CCL produced a higher and more stable Micro-averaged AUC. Consistent with the above analysis, this invention concludes that CCL w / This produces more uniform representations and well-clustered samples, thus better distinguishing between normal and abnormal samples.

[0130] Data Visualization: To intuitively understand the performance of the CCL proposed in this invention, a set of data visualization experiments were conducted at Shanghai Tech to visualize the clustering results of learned features when using different training strategies: a) CCL w / and b) CCL w / o. The experimental results are as follows... Figure 6 As shown in Table 3, this invention demonstrates that cluster-level scattering loss helps the model distinguish similar images while maintaining cluster tightness, and also separates normal objects from anomalous objects with imbalanced AUC cluster counts. This confirms the effectiveness of the proposed cluster-level scattering loss. Furthermore, this invention analyzes how the alignment and homogeneity portions affect the AUC performance of the CCL, as shown in Table 3. This invention shows that both the alignment and homogeneity portions are essential for improving CCL performance, further demonstrating the effectiveness of the proposed cluster-level scattering loss.

[0131] Table 3: Further investigation of the proposed cluster-level scattering loss. Anomaly detection performance is reported in the form of Micro-averaged and Macro-averaged AUC. Bold numbers indicate best results.

[0132]

[0133] Further research on different clustering numbers: To further study k-means... The present invention reports the impact of different values ​​on CUHK Avenue and Shanghai Tech. The results are shown in Figures 7(a) and 7(b). For the case of over-clustering (number of clusters greater than 40), the trends for both datasets are similar. In other cases, if the representations are well aligned within the same semantic clusters, over-clustering will attempt to disrupt the cluster structure and push away semantically similar examples, which will certainly degrade clustering performance. For the case of under-clustering (number of clusters less than 28), the clustering performance of both datasets and both methods is significantly impaired.

[0134] This invention proposes a simple yet efficient contrastive clustering learning method to improve the accuracy and speed of video anomaly detection. In each frame, the invention uses object-level and motion-level spatiotemporal representations to represent each object, followed by density-based anomaly scoring. These contrastive clustering learning networks across different spatiotemporal views enable the invention to achieve state-of-the-art results on the most complex video anomaly dataset at Shanghai Tech.

[0135] The specific implementation schemes described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific implementation schemes of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A video anomaly detection method based on a contrastive clustering learning network, characterized in that, Includes the following steps: Step 1, Preprocessing: Use an object detector to locate and classify the bounding boxes of all objects in the frame, and apply a motion estimator to predict the optical flow of each frame; Step 2, Contrastive Clustering Learning: Perform contrastive clustering learning using spatiotemporal representations of the two views; Step 2 specifically includes the following steps: At the beginning, the first extractor branch is pre-trained. and use To generate There are clusters, denoted as: The value of ; the clustering results are used to form pseudo-labels to train the contrastive learning module; If forced From another view It also utilizes clustering structures and employs three types of contrastive learning: target-level contrastive learning, view-to-view cluster-level contrastive learning, and cluster scattering contrastive learning. Target-level contrastive learning: to enable the representation of two spatiotemporal views and Mutual learning at the object level incorporates feature outputs from the first branch. Predicted output of the second branch The negative cosine similarity is defined by the object-level contrast loss as follows. ; (1) It can effectively promote feature representation Learn about the normal mode properties in other views to enhance your awareness of certain normal mode properties in your own view; Inter-view clustering-level contrastive learning: transforming vectors Input is fed into a pre-trained clustering module and the matching cluster centers are calculated. and will Defined as the first matching The center vectors of each cluster; the cluster-level contrast loss between views is defined as follows: (2) Using inter-view cluster-level contrast loss renew Some attributes are used to update the cluster centers; and with similar, Further improve feature representation and To learn the attributes of normal mode from each other; Clustered scattering contrastive learning: for a predefined Clusters, for these Each cluster constructs a contrastive loss because for one cluster, the rest... Each cluster is definitely a negative example; it encourages feature learning and feature consistency between two views, thereby maximizing inter-class distance; From embedded space The first view Obtain Each feature representation, and from the embedding space Another view China The proposed clustered scattering contrast loss is represented by several features. The definition is as follows: (3) (4) Among them, feature representation and The calculation is as follows: (5) (6) in It is the posterior probability of cluster assignment; in formula (3) It can be approximately divided into formula (4): cluster alignment and cluster uniformity; In addition, by replacing and To rewrite Alignment items, as shown below: (7) (8) (9) (10) in , , This represents the weight of each feature pair; Formula (10) shows that... The alignment term in the equation is represented as a reweighted contrast loss; From formula (10): when and When they belong to the same cluster, that is Then there is ;when and When they belong to different clusters, there are ;therefore, The comparison terms in the table only include sample pairs belonging to the same cluster; therefore, The comparison term is a generalized case that considers instance reweighted contrast loss for pseudo-labels; it was observed that... The consistency term in the code maximizes the distance between instances in different clusters. The alignment component further forces the two views to learn fine-grained spatiotemporal properties from each other, and the uniformity component further improves the differences between different kinds of spatiotemporal representations. During the training phase, the total loss for training both network branches simultaneously is: (11) When using the cross-view clustering level contrast loss in formula (2) During backpropagation, the second network branch A gradient-stopping operation was used; the parameters in the second network branch Only use and Update; additionally, for the extractor branch The extracted features are processed using the k-means algorithm instead of ,because More stable execution, producing more consistent clusters; Step 3, Anomaly Score Estimation: Use density estimation to score the samples as normal or abnormal, where a low estimated density indicates anomaly.

2. The video anomaly detection method based on contrastive clustering learning network according to claim 1, characterized in that, Step 1 specifically includes the following steps: All objects in each frame are detected using the YOLOv3 object detector; object detection is performed per frame. Generate a set containing object mapping Using optical flow as the initial stage for inferring object motion occurs between every pair of consecutive frames. and Calculated between frames; optical flow model extracts each frame. Optical flow; optical flow diagram is composed of The optical flow model is represented as a Siamese network, consisting of a feature extractor. and It consists of two branches that do not share parameters, where and These are the parameters from the two extractors, and a predictor layer is added after the second branch. ,in Represents the parameters in the predictor layer; feature extractor and Implemented using ResNet-50 for feature learning; the output features of the first and second feature extractors are respectively represented as... and And the output of the predictor layer in the second branch is expressed as follows: ,in .

3. The video anomaly detection method based on contrastive clustering learning network according to claim 2, characterized in that, Step 3 specifically includes the following steps: After training, spatiotemporal representation is used. and Perform tests; in each inference fragment Feeded frame by frame , ; Use density estimation to score samples as normal or abnormal, where low estimated density indicates abnormality; compute spatiotemporal representation. The L2 distance between the training cluster centers and the training cluster centers is expressed as their average value. ; Given a test framework Its score is simply the highest score among all objects: (12) The anomaly score for each frame in the segment is represented as: Because the abnormal event is expected to be prolonged, therefore... A one-dimensional Gaussian filter in the time domain is applied to smooth the results.

Images