Unsupervised video anomaly detection method based on multi-subcluster memory prototype

Abstract

This invention relates to an unsupervised video anomaly detection method based on multi-subcluster memory prototypes, belonging to the field of video anomaly detection technology. The method first uses a video feature clustering method based on contrastive learning to learn video segment features, avoiding the high difficulty and cost of acquiring labeled data. Then, a multi-subcluster memory prototype module is proposed, utilizing the fine-grained feature differences between normal videos to learn memory prototypes and reconstruct video features, effectively improving detection performance. Furthermore, a Gaussian filter feature transformation method is proposed to enhance the separability of normal and anomalous features during reconstruction, effectively alleviating the problem in previous methods where anomalous samples could also be effectively reconstructed.

Description

Technical Field

[0001] This invention belongs to the field of video anomaly detection technology, specifically relating to an unsupervised video anomaly detection method based on multi-subcluster memory prototypes. Background Technology

[0002] Video anomaly detection refers to the detection of unusual events in videos. Its applications are wide-ranging, including road traffic monitoring, surveillance of violent incidents, and crowd monitoring. In recent years, video anomaly detection technology has made significant progress. However, due to the unbounded nature of unusual events in practical applications and the difficulty in collecting large-scale annotated data, video anomaly detection still faces significant challenges.

[0003] Existing video anomaly detection technologies can be categorized into two types based on the annotations of the training videos required: weakly supervised video anomaly detection and unsupervised video anomaly detection. Weakly supervised video anomaly detection requires both normal and anomalous training videos with video-level annotations. In real-world scenarios, anomalies are often rare, making it extremely difficult and costly to acquire large amounts of anomalous data. Unsupervised video anomaly detection can be further divided into frame prediction-based methods and frame reconstruction-based methods. These methods do not rely on labels and only use normal videos to train the model. The model learns patterns from normal videos to distinguish between normal and anomalous frames, but they often suffer from high false negatives due to the accurate prediction or reconstruction of anomalous frames.

[0004] Weakly supervised video anomaly detection trains its model using a multi-instance learning strategy, requiring both normal and anomalous training videos with video-level annotations. Therefore, this method has a significant drawback:

[0005] 1. Data acquisition is difficult and costly. Training requires a large amount of labeled normal and abnormal video data. Currently, the number of publicly available video anomaly detection datasets that meet the standards is limited, and it is difficult to obtain abnormal videos in the real world, and it is also difficult to artificially create them.

[0006] Frame prediction-based methods and frame reconstruction-based unsupervised video anomaly detection methods, which train models using only normal videos, currently have two drawbacks:

[0007] 1. They only emphasize the commonalities of learning from normal videos and training, while ignoring the differences between normal videos due to factors such as geographical location, camera angle, and lighting conditions, which limits the performance of the model.

[0008] 2. Because only normal data is used for training, the discrimination boundary is not clear enough, and there is a problem that abnormal frames are also predicted or reconstructed well, resulting in a high false negative rate. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of existing technologies and provide an unsupervised video anomaly detection method based on multi-cluster memory prototypes. First, a video feature clustering method based on contrastive learning is used to learn video segment features, avoiding the high difficulty and cost of acquiring labeled data. Then, a multi-cluster memory prototype module is proposed, utilizing the fine-grained feature differences between normal videos to learn memory prototypes and reconstruct video features, effectively improving detection performance. Furthermore, a Gaussian filter feature transformation method is proposed to enhance the separability of normal and abnormal features during reconstruction, effectively alleviating the problem that abnormal samples can also be effectively reconstructed in previous methods.

[0010] To achieve the above objectives, the technical solution of the present invention is: an unsupervised video anomaly detection method based on a multi-subcluster memory prototype, comprising:

[0011] Video segment features are learned using a video feature clustering method based on contrastive learning;

[0012] A multi-subcluster memory prototype module is proposed, which learns the memory prototype by utilizing the fine-grained feature differences between normal videos and reconstructs the video features;

[0013] A Gaussian filter feature transformation method is proposed to enhance the separability of normal and abnormal features during the reconstruction process.

[0014] In one embodiment of the present invention, the method is implemented as follows: first, feature extraction is performed on the input video; then, the input video is processed by a Gaussian filter feature transformation module to enhance the video into a set of paired samples; next, video features are learned by a video feature clustering method based on contrastive learning and the video features are clustered in a high-dimensional vector space; then, the multi-cluster memory prototype module learns the video feature distribution and performs feature reconstruction; finally, anomaly determination is performed based on the reconstruction error of the feature reconstruction.

[0015] In one embodiment of the present invention, the input video data comes from the benchmark video datasets ShanghaiTech and UCF-Crime; ShanghaiTech is a challenging multi-scene dataset consisting of 13 campus scenes with different lighting conditions and camera angles; ShanghaiTech contains a total of 437 videos, including 238 training videos and 199 test videos; UCF-Crime is a large-scale anomaly detection dataset consisting of 1900 untrimmed videos captured from real-world street and indoor surveillance cameras; UCF-Crime includes 1610 training videos and 290 test videos.

[0016] In one embodiment of the present invention, the method for extracting features from the input video is as follows:

[0017] For each input video, it is first broken down into several segments, each consisting of 16 consecutive frames; then, for each video segment v i ∈V video V undergoes 10-corp data augmentation to obtain 10 augmented segments {v i0 ,v i1 ,...,v i9 Subsequently, the I3D network is used as the backbone network for feature extraction. The enhanced fragment is input into the pre-trained I3D network to obtain the fragment features {f}. i0 ,f i1 ,...,f i9 Finally, {f} i0 ,f i1 ,...,f i9 The data is randomly divided into two groups, and the average of the five features in each group is calculated to obtain the feature values. and Each video segment is constructed as a pair of video segment features.

[0018] In one embodiment of the present invention, the Gaussian filter feature conversion module performs feature transformation on video segments. Apply a Gaussian filter to enhance and The dissimilarity between them is specifically implemented as follows:

[0019] First, to ensure that the feature dimensions after Gaussian filtering feature transformation are the same as before transformation, the following steps are taken: Perform reflection filling; then, use a Gaussian kernel to perform Gaussian filtering and convolution operations on the features to obtain... It is expressed as follows:

[0020]

[0021] Where σ is the standard deviation of the Gaussian kernel;

[0022] Next, for ease of explanation, let At this time, from the same video clip and Forming positive sample pairs Features from different video clips and Enhanced to form negative sample pairs Where i≠j.

[0023] In one embodiment of the present invention, the specific implementation of learning video features by a video feature clustering method based on contrastive learning and clustering the video features in a high-dimensional vector space is as follows:

[0024] First, a video feature clustering module based on contrastive learning is constructed, which uses a multi-scale temporal network (MTN) as the encoder for contrastive learning; the two MTNs are defined as S... q and S k S q The input is S k The input is S q The output is called the discriminative feature, denoted as S k The output of is called the prototype feature, denoted as . and Represented as:

[0025]

[0026] Next, in S q S k Then, nonlinear projection heads G were added respectively. q (·) and G k (·); Discriminant embedding is obtained through a nonlinear projection head. and prototype embedding Represented as:

[0027]

[0028] To improve the effectiveness of contrastive learning, a data sample queue Q is defined; each mini-batch obtains... All samples are stored in a queue. When the queue is full, the first element enqueued is removed and discarded. This allows the model to reuse samples from the first few batches. Since the size of the queue Q can be much larger than the batch size, contrastive learning can have a large number of negative samples.

[0029] Then, a clustering process is performed:

[0030] (1) Sub-cluster aggregation loss function: To aggregate samples belonging to the same sub-cluster, a sub-cluster aggregation loss function is defined, which is expressed as follows:

[0031]

[0032] in This refers to the data stored in the queue in several batches. N represents the small batch size, N Q T represents the length of queue Q. g For temperature parameters;

[0033] Next, define a pair of MLP networks Φ with identical structures. q and Φ k ,Will and Mapping to K-dimensional weights and It is represented as:

[0034]

[0035] Among them, the weight allocation and Represents its corresponding discriminant features and prototype features The weight belonging to each cluster;

[0036] The sub-cluster weight vector of each mini-batch is described by matrix P. q and P k :

[0037]

[0038] Where T is the vector transpose symbol;

[0039] Then, matrix P q and P k Converting from row-major form to column-major form yields:

[0040]

[0041] (2) Inter-cluster separation loss function: To reduce diversity within sub-clusters and improve separation between sub-clusters, an inter-cluster separation loss function is defined. This function consists of weighted loss and cluster regularization loss. Weighted loss encourages sub-clusters to move away from each other, while cluster regularization loss mitigates the problem of the model easily assigning most samples to a few clusters during deep clustering. The inter-cluster separation loss function is expressed as:

[0042]

[0043] Where q r (j) represents the vector q r The element with index j in the middle, β is the weight parameter;

[0044] Clustering loss function L of multi-cluster video feature clustering module cluster From loss L agg and loss L sep Together they form a whole, in which the loss L agg Encourage reducing diversity within subclusters, while losing L sepThis encourages increased diversity among subclusters; L cluster Defined as:

[0045] L cluster =L agg +L sep

[0046] (3) Momentum update: During training, S q G q and Φ q As a regular network, parameters are updated via backpropagation, S k G k and Φ k As momentum networks, their parameters are not updated through backpropagation; their parameter update formula is:

[0047] θ k =μθ k +(1-μ)θ q :

[0048] Where θ q S represents q G q and Φ q Network parameters, θ k S represents k G k and Φ k The network parameters are μ∈[0,1), where μ is the momentum coefficient.

[0049] In one embodiment of the present invention, the multi-subcluster memory prototype module is specifically as follows:

[0050] Define two queues Q of the same size. s and Q p These are used to store the prototype features of video segments. and its corresponding allocation weight During training, the current batch of generated data will be processed in each round. and Add them to queue Q respectively s and Q p When the queue is full, the first element enqueued is removed and discarded; firstly, the local multi-subcluster memory prototype m local Defined as:

[0051]

[0052] in For the Softmax function, and These represent queues Q respectively. s and Q p The i-th element stored in it Queue Q represents s and Q p The size of the two is the same; m local The essence is several small batches of prototype features The matrix formed by the sub-cluster prototypes in the feature space, due to m local It is obtained from only a portion of the prototype features in all normal samples, hence it is called the local multi-subcluster memory prototype;

[0053] To further obtain the global multi-subcluster memory prototype m global Define a global multi-subcluster memory prototype queue Q. m Used to store m generated by each Step local Queue Q m Size Defined as:

[0054]

[0055] Where |F| is the total number of training samples, and bs is the mini-batch size; then, according to Q m To generate the global multi-subcluster memory prototype m global :

[0056]

[0057] in Indicates the current queue Q m The local multi-subcluster memory prototype with subscript i; the global multi-subcluster memory prototype m. global It contains K sub-cluster prototype vectors, corresponding to the K subclasses of normal patterns in normal video features; during training, each mini-batch will include m generated in the current mini-batch. local Add to queue Q m And calculate the m generated at that moment. global When the queue is full, the first element enqueued is removed from the queue and discarded.

[0058] Then, for discriminative features Perform feature reconstruction; reconstruct features Represented as:

[0059]

[0060] in For the Softmax function; After processing with the Softmax function, a weight vector of length K is obtained, representing the weights of the current video segment sample belonging to K subclusters; then, based on the weight vector and the global multi-subcluster memory prototype m... globalReconstructed features with fine-grained sub-cluster information

[0061] Finally, by optimizing the feature reconstruction loss function L... rebuild Make discriminative features In a high-dimensional feature space, it is close to its corresponding reconstructed feature. Feature reconstruction loss function L rebuild Defined as:

[0062]

[0063] In one embodiment of the present invention, the specific method for anomaly determination is as follows:

[0064] Using the discriminative features of the samples and its reconstruction features Calculate the anomaly score, which is defined as follows:

[0065]

[0066] When the input sample is a normal sample, the features are reconstructed. and discriminant features When the similarity between the input sample and the input sample is high, the anomaly score is low; however, when the input sample is anomaly, the score is lower due to the reconstruction features. Reconstructed from the global memory prototype representing normal sparse representation, the features are thus reconstructed at this time. and discriminant features The similarity between normal and abnormal samples is low, resulting in a higher abnormal score. Ultimately, the abnormal score is used to distinguish between normal and abnormal samples.

[0067] The present invention also provides an unsupervised video anomaly detection system based on a multi-subcluster memory prototype, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the steps described above.

[0068] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of the method described above.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] 1. This invention uses self-supervised contrastive learning to optimize the video anomaly detection model. The data used to train the model is unlabeled data, which greatly reduces the cost and difficulty of acquiring data. Compared with most weakly supervised video anomaly detection methods, this method has lower requirements for training data, avoids the huge cost of manually labeling data, and makes data acquisition easier.

[0071] 2. Previous unsupervised video anomaly detection methods only focused on the differences between normal and abnormal videos, neglecting the differences between normal videos due to variations in scene, lighting conditions, and camera angle. This invention proposes a video feature clustering method based on contrastive learning, using contrastive learning techniques to learn the relationships between normal video segments. This method aggregates features from similar video segments and separates features from dissimilar video segments, assigning video segments with similar semantic information to the same cluster. This method enables the model to learn the differences between normal video segments, thereby more precisely distinguishing between normal and abnormal patterns.

[0072] 3. Existing unsupervised video anomaly detection methods based on reconstruction have an unavoidable problem: because they are trained using only normal samples, some anomalous samples are also reconstructed with high quality, leading to a decrease in model performance. This invention proposes a Gaussian filter feature transformation module to enhance the separability between normal and anomalous features, making the boundary between normal and anomalous features clearer, effectively alleviating this problem and improving model performance. Attached Figure Description

[0073] Figure 1 This is a diagram illustrating the overall execution process of the present invention.

[0074] Figure 2 Examples of video data frames: (a) normal video frame; (b) abnormal video frame.

[0075] Figure 3 This is a diagram illustrating the feature extraction process of the present invention.

[0076] Figure 4 This is the Gaussian filter feature conversion module of the present invention.

[0077] Figure 5 This is a diagram illustrating the video feature clustering process based on contrastive learning in this invention.

[0078] Figure 6 This is the loss curve for ShanghaiTech.

[0079] Figure 7 This is the UCF-Crime loss curve.

[0080] Figure 8 To visualize the test results.

[0081] Figure 9 The AUC-ROC curves are the detection results from ShanghaiTech and UCF-Crime.

[0082] Figure 10 This is a flowchart illustrating the implementation of the method of the present invention. Detailed Implementation

[0083] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0084] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0085] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0086] This invention provides an unsupervised video anomaly detection method based on multi-subcluster memory prototypes, comprising:

[0087] Video segment features are learned using a video feature clustering method based on contrastive learning;

[0088] A multi-subcluster memory prototype module is proposed, which learns the memory prototype by utilizing the fine-grained feature differences between normal videos and reconstructs the video features;

[0089] A Gaussian filter feature transformation method is proposed to enhance the separability of normal and abnormal features during the reconstruction process.

[0090] The following is a detailed implementation process of the present invention.

[0091] This embodiment provides an unsupervised video anomaly detection method based on a multi-subcluster memory prototype. The overall execution flow of the technical solution is as follows: Figure 1 , 10As shown, the process first extracts features from the input video; then, the Gaussian filter feature transformation module processes the input video, enhancing it into a set of paired samples; next, a video feature clustering method based on contrastive learning learns the video features and clusters them in a high-dimensional vector space; then, a multi-cluster memory prototype module learns the video feature distribution and reconstructs the features; finally, anomaly detection is performed based on the reconstruction error of the feature reconstruction. The specific implementation steps are as follows:

[0092] Step 1: Video Data Input

[0093] The data used in this invention comes from the publicly available benchmark video datasets ShanghaiTech and UCF-Crime. ShanghaiTech is a challenging multi-scene dataset consisting of 13 campus scenes with varying lighting conditions and camera angles. This dataset contains 437 videos, including 238 training videos and 199 test videos. UCF-Crime is a large-scale anomaly detection dataset consisting of 1900 untrimmed videos captured from real-world street and indoor surveillance cameras. Compared to ShanghaiTech, UCF-Crime presents more complex and diverse backgrounds. This dataset includes 1610 training videos and 290 test videos. Both datasets contain diverse video surveillance scenarios, including unusual human behavior and anomalous events caused by vehicles or equipment. Examples of video data from both datasets are provided. Figure 2 As shown.

[0094] The method of this invention is trained using only normal videos from the training set and evaluated using all videos from the test set. Therefore, the statistical results of the actual dataset used are shown in Table 1:

[0095] Table 1. Statistical Results of ShanghaiTech and UCF-Crime Data

[0096]

[0097] Step 2, Feature Extraction

[0098] Feature extraction process as follows Figure 3 As shown. For each input normal video, the present invention first decomposes it into several segments, each segment consisting of 16 consecutive frames. Then, for each video segment v i Perform 10-corp data augmentation on ∈V to obtain 10 augmented fragments {v i0 ,v i1 ,...,v i9 Subsequently, this invention uses the I3D network as the backbone network for feature extraction. The enhanced vi The fragment is input into a pre-trained I3D network to obtain fragment features {f}. i0 ,f i1 ,...,f i9 Finally, {f} i0 ,f i1 ,...,f i9 The data is randomly divided into two groups, and the average of the five features in each group is calculated to obtain the feature values. and Now, each video segment is constructed as a pair of video segment features.

[0099] Step 3: Gaussian Filter Feature Transformation Module

[0100] Because the method of this invention only uses normal samples for training and lacks awareness of abnormal patterns, the model often reconstructs a portion of abnormal samples well during subsequent feature reconstruction, which adversely affects the model's performance. To enhance the separability of normal and abnormal features and alleviate the problem of effectively reconstructing even abnormal samples in existing methods, this invention proposes a simple and efficient technique called Gaussian filter feature transformation.

[0101] Specifically, this invention focuses on video segment features. A Gaussian filter was applied to enhance the input samples. and The dissimilarity between them. First, to ensure that the feature dimensions after Gaussian filtering feature transformation are the same as before transformation, this invention... Perform reflection filling; then, use a Gaussian kernel to perform Gaussian filtering and convolution operations on the features to obtain... It is expressed as follows:

[0102] Where σ is the standard deviation of the Gaussian kernel. The Gaussian filtering feature transformation process is as follows: Figure 4 As shown. Then, for the convenience of illustrating the present invention... At this time, from the same video clip and Forming positive sample pairs Features from different video clips and Enhanced to form negative sample pairs Where i≠j.

[0103] Step 4: Video feature clustering based on contrastive learning

[0104] The structure of the video feature clustering module based on contrastive learning is as follows: Figure 5As shown. This module uses a multi-scale temporal network (MTN) as the encoder for contrastive learning. The two MTN encoders are defined as S... q and S k encoder S q The input is Encoder S k The input is Encoder S q The output is called the discriminative feature, denoted as Encoder S k The output of is called the prototype feature, denoted as . and Represented as:

[0105]

[0106] Next, in encoder S q S k Then, nonlinear projection heads G were added respectively. q (·) and G k (·). Discriminative embedding is obtained through a nonlinear projection head. and prototype embedding Represented as:

[0107]

[0108] To improve the effectiveness of contrastive learning, a data sample queue Q is defined. Each mini-batch receives... All samples are stored in a queue. When the queue is full, the first element enqueued is removed and discarded, allowing the model to reuse samples from earlier batches. Since the size of the queue Q can be much larger than the mini-batch size, contrastive learning can have a large number of negative samples.

[0109] (1) Sub-cluster aggregation loss function. To aggregate samples belonging to the same sub-cluster, this invention defines a sub-cluster aggregation loss function, specifically expressed as follows:

[0110]

[0111] in This indicates the data stored in the queue from previous batches. N represents the mini-batch size, N Q T represents the length of queue Q. g This refers to the temperature parameter.

[0112] Next, this invention defines a pair of structurally identical MLP networks Φ q and Φ k ,Will and Mapping to K-dimensional weights and It is represented as:

[0113]

[0114] Among them, the weight allocation and Represents its corresponding discriminant features and prototype features The weight belonging to each cluster.

[0115] The sub-cluster weight vector of each mini-batch can be described as matrix P. q and P k :

[0116]

[0117] Where T is the vector transpose symbol. Then, matrix P... q and P k Converting from row-major form to column-major form yields:

[0118]

[0119] (2) Inter-cluster Separation Loss Function. To reduce diversity within sub-clusters and improve separation between them, this invention defines an inter-cluster separation loss function. This function consists of an assignment weight loss and a clustering regularization loss. The assignment weight loss encourages sub-clusters to move away from each other, while the clustering regularization loss mitigates the problem of the model easily assigning most samples to a few clusters during deep clustering. The inter-cluster separation loss function is expressed as:

[0120]

[0121] Where q r (j) represents the vector q r The element with index j is β, which is the weight parameter.

[0122] Clustering loss function L of multi-cluster video feature clustering module cluster From loss L agg and L sep Together they form, in which L agg The loss function encourages reducing diversity within subclusters, while L sep This encourages increased diversity among subclusters. cluster Defined as:

[0123] l cluster =L agg +L sep #(9)

[0124] (3) Momentum update. During training, S q G q and Φ q As a regular network, parameters are updated via backpropagation, S k G k and Φ k As momentum networks, their parameters are not updated through backpropagation; their parameter update formula is:

[0125] θ k =μθ k +(1-μ)θ q #(10)

[0126] Where θ q S represents q G q and Φ q Network parameters, θ k S represents k G k and Φ k The network parameters are μ∈[0,1), where μ is the momentum coefficient.

[0127] Step 5: Multi-subcluster memory prototype module

[0128] Define two queues Q of the same size. s and Q p These are used to store the prototype features of video segments. and its corresponding allocation weight During training, each step will generate the current mini-batch model. and Add them to queue Q respectively s and Q p When the queue is full, the element that was first enqueued is removed and discarded. First, the local multi-subcluster memory prototype m... local Defined as:

[0129]

[0130] in This refers to the Softmax function. and These represent queues Q respectively. s and Q p The i-th element stored in it Queue Q represents s and Q p The two are the same size. local The essence is a number of small batches of prototype features The matrix formed by the sub-cluster prototypes in the feature space, due to m localIt is obtained from only a portion of the prototype features in all normal samples, hence it is called the local multi-cluster memory prototype.

[0131] To further obtain the global multi-subcluster memory prototype m global Define a global multi-subcluster memory prototype queue Q. m This queue is used to store the m generated by each Step. local Queue Q m Size Defined as:

[0132]

[0133] Where |F| is the total number of training samples, and bs is the mini-batch size. Then, according to Q... m To generate the global multi-subcluster memory prototype m global :

[0134]

[0135] in Indicates the current queue Q m The local multi-subcluster memory prototype with subscript i. The global multi-subcluster memory prototype m. global It contains K sub-cluster prototype vectors, corresponding to the K subclasses of normal patterns in the normal video features. During training, each mini-batch generates m... local Add to queue Q m And calculate the m generated at that moment. global When the queue is full, the element that was first enqueued is removed from the queue and discarded.

[0136] Then, the model discriminates features. Perform feature reconstruction. Reconstruct features. Represented as:

[0137]

[0138] in This is the Softmax function. After processing using the Softmax function, a weighted vector of length K is obtained, representing the weights of the current video segment sample belonging to K subclusters. Then, based on the weighted vector and the global multi-subcluster memory prototype m... global Reconstructed features with fine-grained sub-cluster information

[0139] Finally, this invention optimizes the feature reconstruction loss function L. rebuild Make discriminative features In a high-dimensional feature space, it is close to its corresponding reconstructed feature. Feature reconstruction loss function L rebuild Defined as:

[0140]

[0141] Step 6: Abnormal Frame Determination

[0142] During model inference, discriminative features of the samples are used. and its reconstruction features Calculate the anomaly score, which is defined as follows:

[0143]

[0144] When the input sample is a normal sample, the features are reconstructed. and discriminant features When the similarity between the input samples is high, the anomaly score is low. However, when the input sample is an anomaly, due to the reconstruction features... Reconstructed from the global memory prototype representing normal sparse representation, the features are thus reconstructed at this time. and discriminant features The similarity between normal and abnormal data is low, resulting in a higher anomaly score. Ultimately, the model distinguishes between normal and abnormal data based on the anomaly score.

[0145] Step 7: Model Training and Evaluation

[0146] An unsupervised video anomaly detection model based on a multi-subcluster memory prototype was loaded; training video data from ShanghaiTech and UCF-Crime were input into the model for training. The model was optimized using the AdamW optimizer with a learning rate of 0.001 and a weight decay of 0.0005. For ShanghaiTech, the mini-batch size was set to 280, K to 16, and the queue size N. Q and Both are 4480, the standard deviation σ of Gaussian blur is 2, T g and T q The values ​​are 0.01 and 10, respectively. For UCF-Crime, this invention sets the mini-batch size to 1280, K to 20, and the queue size N. Q and Both are 102400, the standard deviation σ of the Gaussian blur is set to 2, T g and T q The values ​​are set to 0.1 and 0.1 respectively. This method is implemented in PyTorch. The training process is divided into two phases, totaling 800 iterations. The first 100 iterations are used for warm-up, utilizing only L... cluster The model was optimized, and a feature reconstruction loss L was added in the last 700 iterations. rebuild With Lcluster They jointly perform backpropagation. During the iteration process, the model's loss curve reflects the model's learning degree of video features and the effect of feature reconstruction. On the ShanghaiTech dataset, L... cluster Curve and L rebuild Curves Figure 6 As shown, L on the UCF-Crime dataset cluster Curve and L rebuild Curves Figure 7 As shown, L on the two datasets cluster The reason why the curves all show a significant upward fluctuation in the 101st iteration is that at this time, the Lrebuild loss begins to be added to the backpropagation, affecting the network parameters. Therefore, the curves continue to show a downward trend thereafter. In addition, L rebuild It does not participate in backpropagation in the first 100 iterations, so its default value is 0 in the first 100 iterations.

[0147] An unsupervised video anomaly detection model based on a multi-subcluster memory prototype was used to detect anomalies in the test set. ShanghaiTech (“(a)01_0051”, “(b)01_0053”, “(c)03_0032”) and UCF-Crime (“(d)Burglary079”, “(e)Explosion025”, “(f)Robbery 102”) were randomly selected for anomaly score and ground truth label visualization. Detection results are as follows: Figure 8 As shown in the figure, the curve represents the anomaly scoring curve, and the dark background area represents manually marked anomaly events.

[0148] The evaluation metric used in this method is the area under the frame-level ROC curve for video anomaly detection: AUC. The AUC-ROC curves of the proposed model on ShanghaiTech and UCF-Crime are shown below. Figure 9 As shown:

[0149] The test set is input into the model for video anomaly detection. Figure 9 The AUC-ROC curve shows that the detection performance is quite good. However, the AUC-ROC curve also indicates that the model may produce a small number of false positives (misclassifying normal samples as abnormal samples) and false negatives (misclassifying abnormal samples as normal samples) during the detection process. This is because the test set videos contain some more challenging samples, such as minor abnormal events or rare but normal events.

[0150] This invention defines a baseline model and conducts ablation experiments on the three key modules of the proposed model. In the baseline model, the number of sub-clusters K is set to 1, indicating that the model clusters features into a single cluster rather than multiple sub-clusters. Furthermore, the baseline model uses a traditional memory prototype generation method and does not perform Gaussian filtering feature transformation on the video segment features. The experimental results are shown in Table 2. The baseline model achieves a frame-level AUC of 80.08% on the ShanghaiTech dataset and 69.79% on the UCF-Crime dataset. The ablation experiment results demonstrate that the addition of each module proposed in this invention improves the model's performance.

[0151] Table 2. Experimental results of model ablation

[0152]

[0153] Table 3 presents the frame-level AUC results of the proposed method on the ShanghaiTech and UCF-Crime datasets and compares them with the current state-of-the-art (SOTA) methods. As shown in the figure, the proposed method outperforms existing SOTA unsupervised learning methods, achieving state-of-the-art frame-level AUC results of 86.26% and 77.60% on both the ShanghaiTech and UCF-Crime datasets, respectively.

[0154] Table 3. Model Performance Comparison Test

[0155]

[0156] To demonstrate the feasibility of deploying the proposed model in real-world video surveillance scenarios, this invention also conducted experiments on the model's inference speed. Specifically, under the same hardware environment, the same test samples were input into each model for inference, and the average inference time per frame for each model was calculated. The hardware environment consisted of an NVIDIA GeForce RTX 4090 (24GB) GPU and an Intel Core i7-13700KF CPU. The test samples were the entire test set from ShanghaiTech, and the inference time was captured using the "profile" tool in PyTorch. The results for each model corresponded to the average of 10 experiments. The experimental results are shown in Table 4. The proposed model achieved an average inference time of 13.28ms per frame during inference, second only to the S3R model with the best inference speed. Although the average inference speed per frame was 64ms slower than the S3R model, the model of this invention exhibited better AUC performance. The experimental data demonstrate the feasibility of deploying the proposed model in real-world video surveillance systems.

[0157] Table 4 Comparison of Model Inference Speed ​​Experiment

[0158]

[0159] Step 8: Model Invocation

[0160] The system reads a pre-trained video anomaly detection model stored in the cloud or locally to perform video anomaly detection. Specifically, it obtains the video stream from surveillance video and inputs it into the model, which is then analyzed by anomaly score. The value is compared with a pre-set threshold. If the abnormal score is greater than the threshold, it is determined to be abnormal and an alarm is generated. If the abnormal score is less than or equal to the threshold, it is determined to be normal.

[0161] The present invention also provides an unsupervised video anomaly detection system based on a multi-subcluster memory prototype, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the steps described above.

[0162] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of the method described above.

[0163] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0165] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0166] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0167] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An unsupervised video anomaly detection method based on multi-subcluster memory prototypes, characterized in that, include: Video segment features are learned using a video feature clustering method based on contrastive learning; A multi-subcluster memory prototype module is proposed, which learns the memory prototype by utilizing the fine-grained feature differences between normal videos and reconstructs the video features; A Gaussian filter feature transformation method is proposed to enhance the separability of normal and abnormal features during the reconstruction process; The method is implemented as follows: First, features are extracted from the input video; then, the input video is processed by a Gaussian filter feature transformation module to enhance the video into a set of paired samples; next, the video features are learned by a video feature clustering method based on contrastive learning and clustered in a high-dimensional vector space. Then, the multi-cluster memory prototype module learns the video feature distribution and performs feature reconstruction; finally, anomaly detection is performed based on the reconstruction error of the feature reconstruction. The specific implementation method of learning video features and clustering them in a high-dimensional vector space using a video feature clustering method based on contrastive learning is as follows: First, a video feature clustering module based on contrastive learning is constructed, which uses a multi-scale temporal network (MTN) as the encoder for contrastive learning; the two MTNs are defined as S... q and S k S q The input is S k The input is S q The output is called the discriminative feature, denoted as S k The output of is called the prototype feature, denoted as . and Represented as: Next, in S q S k Then, nonlinear projection heads G were added respectively. q (·) and G k (·); Discriminant embedding is obtained through a nonlinear projection head. and prototype embedding Represented as: To improve the effectiveness of contrastive learning, a data sample queue Q is defined; each mini-batch obtains... All samples are stored in a queue. When the queue is full, the first element enqueued is removed and discarded. This allows the model to reuse samples from the first few batches. Since the size of the queue Q can be much larger than the batch size, contrastive learning can have a large number of negative samples. Then, a clustering process is performed: (1) Sub-cluster aggregation loss function: To aggregate samples belonging to the same sub-cluster, a sub-cluster aggregation loss function is defined, which is expressed as follows: in This refers to the data stored in the queue in several batches. N represents the small batch size, N Q T represents the length of queue Q. g For temperature parameters; Next, define a pair of MLP networks Φ with identical structures. q and Φ k ,Will and Mapping to K-dimensional weights and It is represented as: Among them, the weight allocation and Represents its corresponding discriminant features and prototype features The weight belonging to each cluster; The sub-cluster weight vector of each mini-batch is described by matrix P. q and P k : Where T is the vector transpose symbol; Then, matrix P q and P k Converting from row-major form to column-major form yields: (2) Inter-cluster separation loss function: To reduce diversity within sub-clusters and improve separation between sub-clusters, an inter-cluster separation loss function is defined. This function consists of weight allocation loss and clustering regularization loss. The inter-cluster separation loss function is expressed as: Where q r (j) represents the vector q r The element with index j in the middle, β is the weight parameter; Clustering loss function L of multi-cluster video feature clustering module cluster From loss L agg and loss L sep Together they form a whole, in which the loss L agg Encourage reducing diversity within subclusters, while losing L sep This encourages increased diversity among subclusters; L cluster Defined as: L cluster =L agg +L sep (3) Momentum update: During training, S q G q and Φ q As a regular network, parameters are updated via backpropagation, S k G k and Φ k As momentum networks, their parameters are not updated through backpropagation; their parameter update formula is: Where θ q S represents q G q and Φ q Network parameters, θ k S represents k G k and Φ k The network parameters are μ∈[0,1), where μ is the momentum coefficient.

2. The unsupervised video anomaly detection method based on multi-subcluster memory prototype as described in claim 1, characterized in that, The input video data comes from the benchmark video datasets ShanghaiTech and UCF-Crime. ShanghaiTech is a challenging multi-scene dataset consisting of 13 campus scenes with different lighting conditions and camera angles. ShanghaiTech contains a total of 437 videos, including 238 training videos and 199 test videos. UCF-Crime is a large-scale anomaly detection dataset consisting of 1,900 untrimmed videos captured from real-world street and indoor surveillance cameras. UCF-Crime includes 1,610 training videos and 290 test videos.

3. The unsupervised video anomaly detection method based on multi-subcluster memory prototype according to claim 1, characterized in that, The method for feature extraction from the input video is as follows: For each input video, it is first broken down into several segments, each consisting of 16 consecutive frames; then, for each video segment v i ∈V video V undergoes 10-corp data augmentation to obtain 10 augmented segments {v i0 ,v i1 ,...,v i9 Subsequently, the I3D network is used as the backbone network for feature extraction. The enhanced fragment is input into the pre-trained I3D network to obtain the fragment features {f}. i0 ,f i1 ,...,f i9 Finally, {f} i0 ,f i1 ,...,f i9 The data is randomly divided into two groups, and the average of the five features in each group is calculated to obtain the feature values. and Each video segment is constructed as a pair of video segment features.

4. The unsupervised video anomaly detection method based on multi-subcluster memory prototype according to claim 3, characterized in that, The Gaussian filter feature conversion module performs feature transformation on video segments. Apply a Gaussian filter to enhance and The dissimilarity between them is specifically implemented as follows: First, to ensure that the feature dimensions after Gaussian filtering feature transformation are the same as before transformation, the following steps are taken: Perform a reflection fill operation; Then, Gaussian filtering and convolution operations are performed on the features using a Gaussian kernel to obtain... It is expressed as follows: Where σ is the standard deviation of the Gaussian kernel; Next, for ease of explanation, let At this time, from the same video clip and Forming positive sample pairs Features from different video clips and Enhanced to form negative sample pairs Where i≠j.

5. The unsupervised video anomaly detection method based on multi-subcluster memory prototype according to claim 1, characterized in that, The specific details of the multi-subcluster memory prototype module are as follows: Define two queues Q of the same size. s and Q p These are used to store the prototype features of video segments. and its corresponding allocation weight During training, the current batch of generated data will be processed in each round. and Add them to queue Q respectively s and Q p When the queue is full, the first element enqueued is removed and discarded; firstly, the local multi-subcluster memory prototype m local Defined as: in For the Softmax function, and These represent queues Q respectively. s and Q p The i-th element stored in it Queue Q represents s and Q p The size of the two is the same; m local The essence is several small batches of prototype features The matrix formed by the sub-cluster prototypes in the feature space, due to m local It is obtained from only a portion of the prototype features in all normal samples, hence it is called the local multi-subcluster memory prototype; To obtain the global multi-subcluster memory prototype m global Define a global multi-subcluster memory prototype queue Q. m Used to store m generated by each Step local Queue Q m Size Defined as: Where |F| is the total number of training samples, and bs is the mini-batch size; then, according to Q m To generate the global multi-subcluster memory prototype m global : in Indicates the current queue Q m The local multi-subcluster memory prototype with subscript i; the global multi-subcluster memory prototype m. global It contains K sub-cluster prototype vectors, corresponding to the K subclasses of normal patterns in normal video features; during training, each mini-batch will include m generated in the current mini-batch. local Add to queue Q m And calculate the m generated at that moment. global When the queue is full, the first element enqueued is removed from the queue and discarded. Then, for discriminative features Perform feature reconstruction; reconstruct features Represented as: in For the Softmax function; After processing with the Softmax function, a weight vector of length K is obtained, representing the weights of the current video segment sample belonging to K subclusters; then, based on the weight vector and the global multi-subcluster memory prototype m... global Reconstructed features with fine-grained sub-cluster information Finally, by optimizing the feature reconstruction loss function L... rebuild Make discriminative features In a high-dimensional feature space, it is close to its corresponding reconstructed feature. Feature reconstruction loss function L rebuild Defined as:

6. The unsupervised video anomaly detection method based on a multi-subcluster memory prototype according to claim 5, characterized in that, The specific method for anomaly detection is as follows: Using the discriminative features of the samples and its reconstruction features Calculate the anomaly score, which is defined as follows: When the input sample is a normal sample, the features are reconstructed. and discriminant features When the similarity is high, the anomaly score is low; however, when the input sample is an anomaly, the score is low due to the reconstruction features. Reconstructed from the global memory prototype representing normal sparse representation, the features are thus reconstructed at this time. and discriminant features The similarity is low, so the abnormal score is high; finally, the abnormal score is used to distinguish between normal and abnormal.

7. An unsupervised video anomaly detection system based on a multi-subcluster memory prototype, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the steps of the method as described in any one of claims 1-6.

8. A computer-readable storage medium having stored thereon computer program instructions executable by a processor, wherein when the processor executes the computer program instructions, it is able to implement the steps of the method as described in any one of claims 1-6.

Images