Video anomaly detection method based on clustering guided learning

Abstract

The application discloses a video anomaly detection method based on clustering guided learning, comprising the following steps: the method designs an efficient and excellent video automatic encoder on the basis of a U-Net framework, the encoder replaces convolution operation with self-attention operation based on a Swin Transformer (ST) to better capture image frame information, the U-Net self-encoder structure deletes the skip connection layer structure therein, the convolution structure is replaced with a Swin Transformer module, and the patch embed structure under the Swin Transformer framework is added to the encoder part to reduce the calculation amount; the learning ability and convergence speed of the ST framework are enhanced by using an I3D Block. Meanwhile, a double-scale deep clustering module is designed to compress and represent the feature information and spatial structure information of the image frame, and the generalization ability of the encoder is well limited. In addition, the model provides two kinds of anomaly detection methods of reconstruction and prediction.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a video anomaly detection method based on clustering-guided learning. Background Technology

[0002] With my country's economic development, the demand for daily safety is increasing. Currently, video surveillance systems are still largely traditional manual monitoring systems. Cameras are typically installed at key locations, and data is transmitted to a monitoring center and displayed in real-time on screens. Monitoring personnel then analyze the screens to assess and respond to any abnormal events. This system has several limitations: First, it is manpower-intensive, requiring personnel to take turns monitoring around the clock; second, it suffers from numerous missed alerts, as prolonged exposure to multiple screens can easily lead to fatigue and missed important information; and finally, it lacks predictive capabilities for abnormal events, serving primarily as a tool for post-event investigation. However, rapid advancements in computer technology, such as image processing, machine vision, and pattern recognition, have made it possible to overcome the limitations of traditional video surveillance systems and enable proactive monitoring of abnormal events within surveillance videos. Intelligent video surveillance systems can monitor abnormal events in real time. When an abnormal event or potential threat occurs, such as a vehicle darting across a sidewalk or people running in a shopping mall, the system will automatically notify security personnel to react and take appropriate measures to reduce losses from accidents or even prevent them from happening. This can greatly improve social security, enhance citizens' sense of security, reduce the workload of security personnel, and minimize public losses. Abnormal event detection in surveillance videos is a key component of intelligent video surveillance systems. Researching abnormal event detection technology and improving its detection performance and time efficiency has significant research and practical implications.

[0003] Video anomaly detection is a challenging task in most cases: First, video event anomalies can be decomposed into spatial and temporal anomalies. Some event anomalies cannot be determined by a single spatial or temporal clue; for example, skateboarders and pedestrians may look and move similarly, but they can be distinguished by changes in posture. Second, in datasets used for anomaly detection tasks, normal pattern data is readily available, but anomalous pattern data is difficult to collect due to high annotation costs. Therefore, most works describe the general distribution patterns of various handcrafted features of image frames from both spatiotemporal perspectives, and consider image frames that significantly deviate from the general distribution patterns as anomalous frames.

[0004] Current methods in this field are mostly based on deep learning, primarily using video autoencoders to model the spatiotemporal features of normal-mode data. These methods are largely based on the U-Net architecture, training the autoencoder to extract video features using only normal-mode data. When the input data significantly deviates from the normal mode, a large reconstruction error will exist between the decoder's output image frame and the original image frame, thus enabling the detection of abnormal frames. However, the powerful generalization ability of deep networks allows autoencoders to reconstruct some abnormal image frames well; therefore, how to suppress the generalization ability of deep networks has become a significant challenge in this research field. Summary of the Invention

[0005] The purpose of this invention is to provide a video anomaly detection method, GCL, based on clustering-guided learning. This method designs an efficient and high-performance video autoencoder on the U-Net framework. This encoder replaces convolutional operations with self-attention operations based on the Swing Transformer module to better capture image frame information, and utilizes the I3D module to enhance the learning ability and convergence speed of the ST framework. Simultaneously, a dual-scale deep clustering module is designed to compress and represent the feature information and spatial structure information of image frames, effectively limiting the generalization ability of the encoder.

[0006] To achieve the above objectives, a video anomaly detection method based on clustering-guided learning includes the following steps:

[0007] A basic U-Net autoencoder structure was built, the skip connection layer structure was removed, the convolutional structure was replaced with the Swing Transformer module, and the patch embedding structure under the Swing Transformer framework was added to the encoder part to reduce the amount of computation.

[0008] An I3D module is constructed, and the size of each convolutional kernel and the number of output channels in the I3D module are determined. This module is then fused with the Swin Transformer module in a residual form to obtain the IST module. Subsequently, 3D convolution is applied to spatially downsample the feature map output by the IST module.

[0009] Obtain the output feature map X of the Nth layer encoding. N ;

[0010] Output feature map X N Split into T×H×W C-dimensional feature vectors Where 1≤t≤T, 1≤h≤H, and 1≤ω≤W represent the time, spatial height, and spatial width indices of the video segment, respectively, and feature depth clustering is performed on the feature vector;

[0011] The feature vectors are obtained by using the cluster center points, and the feature vectors are concatenated to generate a feature map;

[0012] The feature map is input into the spatial clustering module, which has C cluster centers and a set of M cluster points for each cluster. The feature map is segmented into C clusters along the channel direction, with each cluster having T H×W dimensional feature sub-maps. Where 1≤c≤C represents the c-th cluster of feature subgraphs;

[0013] Obtain the objective function for deep clustering;

[0014] In the reconstruction phase, a deconvolution kernel is used for temporal upsampling to recover the original feature map; in the prediction phase, a convolution kernel is used for temporal downsampling to obtain the predicted feature map. Furthermore, the IST module employs the GELU function to prevent the network's feature extraction ability from being affected by the death of a large number of neurons.

[0015] Furthermore, obtain the output feature map X of the Nth layer encoding. N The specific method is as follows: This represents the I3D module in the i-th encoder stage. X represents the Swing Transformer module in the i-th encoder stage. i Let PE(·) represent the input feature map of the i-th layer, and let down represent the patch embedding structure. i (·) represents the spatial downsampling structure, and the encoder is formally expressed as follows:

[0016]

[0017] Where N is the number of coding layers.

[0018] Furthermore, feature depth clustering is performed on the feature vectors, as shown below:

[0019]

[0020] Among them cl k Let CL represent the k-th cluster center, 1≤k≤K, where CL={cl1,...,cl K} represents the set of cluster centroids; α is an adjustable hyperparameter, and this formula calculates the given feature vector. to each cluster center cl k The soft-assigned distance.

[0021] Furthermore, the output feature vector is obtained using the cluster center points, as follows:

[0022]

[0023] The output feature vectors are concatenated to obtain the feature map.

[0024]

[0025] Furthermore, the spatial clustering module clusters the spatial information of the feature map in each feature channel in the following way:

[0026]

[0027] Where γ is a hyperparameter.

[0028] Furthermore, the objective function for deep clustering is:

[0029]

[0030] Where, loss cl Clustering loss.

[0031] Furthermore, during the reconstruction phase, the decoder receives features output by the Nth layer encoder. Time upsampling The code is then input into the decoding layer; the complete decoder process is described below:

[0032]

[0033] In the prediction phase, a three-layer 3D convolutional network is used for output:

[0034]

[0035] Furthermore, the L2 norm is used as the reconstruction loss to measure the difference between the input X0 and the output X. recon Differences between video clips:

[0036]

[0037] The sum of the reconstruction loss and the clustering loss is used as the final loss function:

[0038] loss = loss recon +loss cl .

[0039] Compared with existing technologies, the solution adopted in this invention has the following advantages: the encoder using the Swing Transformer algorithm has better performance in complex monitoring scenarios; in addition, the encoder with I3D-assisted training has a faster training speed and higher efficiency; the memory-like structure designed based on clustering ideas is easier to train, and the memory parameters are contained in the network, making data migration easier; the compression degree of data features is higher, and the distinguishing boundary between abnormal and normal data features is amplified. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of the network structure of the present invention;

[0042] Figure 2 A comparison chart of the encoded features before and after adding the clustering module in this invention, visualized using T-SNE.

[0043] Figure 3 The images show the detection results of this invention on the three datasets Ped2, Avenue, and ShanghaiTech. Among them, (1) is the output image of Ped2, with the real image at the top, the reconstructed image in the middle, and the thermal error map of the two at the bottom; (2) is the output image of Avenue, with the real image at the top, the reconstructed image in the middle, and the thermal error map of the two at the bottom; (3) is the output image of ShanghaiTech, with the real image at the top, the reconstructed image in the middle, and the thermal error map of the two at the bottom.

[0044] Figure 4 The diagram shows the detection results of the present invention on three datasets; where (1) is the Ped2 anomaly score display diagram; (2) is the Avenue anomaly score display diagram; and (3) is the ShanghaiTech anomaly score display diagram. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit the application; that is, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0046] Example 1

[0047] This embodiment provides a video anomaly detection method based on clustering-guided learning, including:

[0048] Step 1: Build a basic U-Net autoencoder structure, remove the skip connection layer structure, replace the convolutional structure with the Swing Transformer module, and add a patch embedding structure under the Swing Transformer framework to the encoder part.

[0049] like Figure 1 As shown, to demonstrate the effectiveness of this invention, a U-Net network structure was built using the PyTorch deep learning library framework in Python.

[0050] Step 2: Build the I3D module, determine the size of each convolutional kernel and the number of output channels in the I3D module, and fuse this module with the Swin Transformer module in residual form to obtain the IST module; then apply 3D convolution to spatially downsample the feature map output by the IST module.

[0051] In this invention, the GELU function is used as the activation function for the IST module to prevent the death of a large number of neurons from affecting the network's ability to extract features.

[0052] Step 3: Obtain the output feature map X of the Nth layer encoding. N ;

[0053] Specifically, with This represents the I3D module in the i-th encoder stage. X represents the Swing Transformer module in the i-th encoder stage. i Let PE(·) represent the input feature map of the i-th layer, and let down represent the patch embedding structure. i (·) represents the spatial downsampling structure, then the encoder can be formally expressed as follows:

[0054]

[0055] Where N represents the number of encoding layers. In practical applications, downsampling of the video segment's temporal dimension is performed only once during the patch embedding stage to reduce computational complexity. No additional temporal downsampling is performed during either the encoding or decoding stages. This is because excessive temporal downsampling, while improving the network's inference efficiency, can cause a deep video encoder to gradually degenerate into an image encoder, reducing performance. Furthermore, in practical applications, due to the removal of skip connection layers, the model's recovery capability is limited. To better balance performance and computational cost, spatial downsampling is performed only in the first layer, and not in subsequent layers.

[0056] Step 4: Convert the output feature map X obtained in Step 3 into a single image. N Split into T×H×E C-dimensional feature vectors Where 1≤t≤T, 1≤h≤H, and 1≤w≤W represent the time, spatial height, and spatial width indices of the video segment, respectively, and feature depth clustering is performed on the feature vectors.

[0057] The goals of clustering are as follows:

[0058]

[0059] Among them cl k Let CL represent the k-th cluster center, 1≤k≤K, where CL={cl1,...,cl K} represents the set of cluster centroids. α is an adjustable hyperparameter, and this formula calculates the given feature vector. to each cluster center cl k The soft-assignment distance can also be understood as a global attention mechanism.

[0060] Step 5: Use the cluster center points to obtain the output feature vectors, and concatenate the feature vectors to generate a feature map;

[0061] Specifically, the cluster center obtained in step 4 can be understood as a kind of memory vector, which saves the feature vector in the normal pattern data. Therefore, the feature map generated based on the cluster points is more compact.

[0062]

[0063] The output feature vectors are concatenated to obtain the feature map.

[0064] Step 6: Input the feature map obtained in Step 5 into the spatial clustering module. This spatial clustering module has C cluster centers and a set of N cluster points for each cluster. The feature map is segmented into C clusters along the channel direction, with each cluster having T H×W dimensional feature sub-maps. Where 1≤c≤C represents the c-th cluster of feature subgraphs;

[0065]

[0066] Where γ is a hyperparameter, this spatial clustering module clusters the spatial information of the feature map on each feature channel, helping the network learn the structured information in the given feature map. However, this information carries strong prior knowledge, and relying on spatial clustering points to generate new feature maps will severely inhibit the network's generalization ability. Therefore, only clustering is performed.

[0067] Step 7: Based on steps 4-6, the objective function for deep clustering can be defined as:

[0068]

[0069] The initialization of cluster centers has a significant impact on model performance. Furthermore, in mini-batch training mode, the clustering algorithm cannot provide a reasonable initialization at the start of training. Therefore, at the beginning of training, we disconnect the gradient flow between the encoder and clusterer and allow them to train independently for a certain period. After the video features have been initially extracted, we reconnect the gradient flow between them. During testing, the cluster centers will no longer be learned.

[0070] Step 8: The decoder is largely consistent with the encoder in framework, but there are some differences in details. The features input to the decoder are temporally reduced once by PE(·). For reconstruction tasks, although this dimensionality reduction can significantly improve encoding efficiency, the timing of restoring the dimensionality-reduced features during the decoding stage will seriously affect the decoder's reconstruction performance. This invention chooses to use a 2×1×1 deconvolution kernel with a stride of (2, 1, 1) at the beginning of the decoding stage to perform temporal upsampling to restore the original image. For the prediction task, this invention uses a 2×1×1 convolution kernel with a stride of (2, 1, 1) to perform temporal downsampling once to obtain the predicted image.

[0071] Taking the reconstruction task as an example, the decoder receives the features output by the Nth layer encoder. Time upsampling After inputting into the decoding layer for computation, in the reconstructed state, the complete decoder process can be formally expressed as follows.

[0072]

[0073] In the image output stage, this invention abandons the symmetrical network structure and uses a three-layer 3D convolutional network for output, which helps to reconstruct higher-resolution images and makes training more stable.

[0074]

[0075] The decoder process in the prediction state is similar and will not be elaborated further.

[0076] Step 9: This invention has two loss functions: reconstruction loss and clustering loss. First, the model uses the L2 norm as the reconstruction loss to measure the relationship between the input X0 and the output X. recon Differences between video clips:

[0077]

[0078] Second, the clustering loss of the model is represented in step 7, and the final loss function is:

[0079] loss = loss recon +loss cl

[0080] Encoders using the Swing Transformer exhibit better performance in complex monitoring scenarios. Furthermore, encoders with I3D-assisted training offer faster training speeds and higher efficiency. Their memory-like structure, designed based on clustering principles, facilitates training, and the memory parameters are embedded within the network, making data transfer easier. Figure 2 This method achieves a higher degree of data feature compression, with various features being tightly encoded in a single space.

[0081] This invention uses the T-SNE algorithm to randomly select 10,000 encoded features for dimensionality reduction and visualization, and uses the cluster point closest to the feature point as the pseudo-label. The results are displayed in [the following text is incomplete and requires further context]. Figure 2 In this model, due to the attention mechanism, similar features are automatically grouped into one category, while features using dual-scale clustering are more evenly distributed into several clusters in the encoding space. Furthermore, feature points using only single-scale clustering are well divided into several clusters, while features using dual-scale clustering further cluster the cluster points themselves in the spatial dimension.

[0082] This invention selects three public anomaly detection datasets—Ped2, Avenue, and ShanghaiTech—to evaluate the performance of the GCL network model. Ped2 contains 16 training videos and 12 test videos with fixed viewpoints. Anomaly events include riding a bicycle, skateboarding, and driving a vehicle on a sidewalk. Avenue consists of 16 training videos and 21 test videos, containing 47 anomaly events, including throwing bags, approaching or moving away from the camera, and running on a sidewalk. ShanghaiTech contains 330 training videos and 107 test videos, with 130 anomaly events, such as gang fights, robberies, and brawls, distributed across 13 different scenarios.

[0083] During training, the model reads in 4 video frames at a time, resizing each frame to 224×224 pixels and normalizing pixel values ​​to the [0,1] range. An Adam optimizer with L2 weighted decay is used, with a decay momentum β = 0.02 set for training; the learning rate is set to 4×10⁻⁶. -4 The number of feature clustering points and spatial clustering points were 1024 and 128, respectively. Cosine annealing was used to gradually decay the learning rate. The depth N of each layer in the SW module was set to 6. 60, 60, and 80 epochs were run on the Ped2, Avenue, and ShanghaiTech datasets, respectively. The batch size for each dataset was set to 20, and the BN layers were frozen during training. Training was performed on four NVIDIA GTX2080TI GPUs with 11GB of VRAM each.

[0084] Based on the evaluation metrics widely used in the VAD field, this application uses the frame-level area under the curve (AUC) to evaluate the performance of the model. Table 1 shows the performance comparison between the method of this invention and existing methods on three public datasets.

[0085] Table 1 Performance comparison between GCL algorithm and existing algorithms

[0086]

[0087]

[0088] As can be seen from Table 1, the method of this invention exhibits leading performance on the larger, more complex dataset ShanghaiTech, but performs poorly on smaller datasets. This is because the self-attention mechanism requires a much larger dataset to train the network.

[0089] For detailed output images of this invention on three datasets, please refer to the actual effect diagrams. Figure 3 and Figure 4 , Figure 3 In the study, when three unusual behaviors occurred on the street—cycling, trucks passing by, and paper being thrown from the air—the reconstructed images showed a significant reconstruction error compared to the original images, demonstrating that the model can effectively identify anomalies. Figure 4 When an anomaly occurs, the anomaly score will rise rapidly; under normal circumstances, the anomaly score will remain at a low level.

[0090] To demonstrate the effectiveness of each module, this application designed several ablation experiments.

[0091] Table 2 Ablation experimental data from ShanghaiTech University

[0092]

[0093] With all other settings unchanged, the test results are shown in Table 2. It can be seen that the I3D module significantly enhances the performance of the self-attention mechanism, improving it by 2.2% and 3.5% respectively compared to using only the ST module and the I3D module. Furthermore, feature clustering also significantly improves model performance, increasing it by 0.8%, while the spatial clustering module improves performance by 0.4%. It is important to emphasize that using only the Swin Transformer and the memory-like feature clustering module will lead to training failure. This is because the normality of the initial cluster centroids disrupts the feature weight map adjusted by the self-attention mechanism. Discarding step 5 results in successful model training.

[0094] The above description is merely an optional embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

[0095] Although the claims in this application have been formulated for specific combinations of features, it should be understood that the scope of this disclosure also includes any novel feature or any novel combination of features, whether express or implied or generalized herein, whether or not it relates to the same scheme in any of the claims currently claimed.

Claims

1. A video anomaly detection method based on clustering-guided learning, characterized in that, The steps include the following: Build a basic U-Net autoencoder structure, remove the skip connection layer structure, replace the convolutional structure with the SwingTransformer module, and add a patch embedding structure under the Swing Transformer framework to the encoder part. An I3D module is constructed, and the size of each convolutional kernel and the number of output channels in the I3D module are determined. This module is then fused with the Swin Transformer module in a residual form to obtain the IST module. Subsequently, 3D convolution is applied to spatially downsample the feature map output by the IST module. Obtain the output feature map X of the Nth layer encoding. N ; Output feature map X N Split into T×H×W C-dimensional feature vectors Where 1≤t≤T, 1≤h≤H, and 1≤w≤W represent the time, spatial height, and spatial width indices of the video segment, respectively, and feature depth clustering is performed on the feature vector. The feature vectors are obtained by using the cluster center points, and the feature vectors are concatenated to generate a feature map; The feature map is input into the spatial clustering module, which has C cluster centers and a set of M cluster points for each cluster. The feature map is segmented into C clusters along the channel direction, with each cluster having T H×W dimensional feature sub-maps. Where 1≤c≤C represents the c-th cluster of feature subgraphs; Obtain the objective function for deep clustering; In the reconstruction phase, a deconvolution kernel is used for temporal upsampling to recover the original feature map; in the prediction phase, a convolution kernel is used for temporal downsampling to obtain the predicted feature map.

2. The video anomaly detection method based on clustering-guided learning according to claim 1, characterized in that, The IST module uses the GELU function.

3. The video anomaly detection method based on clustering-guided learning according to claim 1, characterized in that, Obtain the output feature map X of the Nth layer encoding. N The specific method is as follows: This represents the I3D module in the i-th encoder stage. X represents the Swing Transformer module in the i-th encoder stage. i Let PE(·) represent the input feature map of the i-th layer, and let down represent the patchbed structure. i (·) represents the spatial downsampling structure, and the encoder is formally expressed as follows: Where N is the number of coding layers.

4. The video anomaly detection method based on clustering-guided learning according to claim 1, characterized in that, The feature vectors are subjected to feature depth clustering as follows: Among them cl k Let CL represent the k-th cluster center, 1≤k≤K, where CL={cl1,...,cl K } represents the set of cluster centroids; α is an adjustable hyperparameter, and this formula calculates the given feature vector. to each cluster center cl k The soft-assignment distance.

5. The video anomaly detection method based on clustering-guided learning according to claim 1, characterized in that, The feature vector of the output is obtained using the cluster centroids, as follows: The output feature vectors are concatenated to obtain the feature map.

6. The video anomaly detection method based on clustering-guided learning according to claim 1, characterized in that, The spatial clustering module clusters the spatial information of the feature map in each feature channel in the following way: Where γ is a hyperparameter.

7. The video anomaly detection method based on clustering-guided learning according to claim 1, characterized in that, The objective function for deep clustering is: Where, loss cl This represents the clustering loss.

8. The video anomaly detection method based on clustering-guided learning according to claim 1, characterized in that, During the reconstruction phase, the decoder receives features output by the Nth layer encoder. Time upsampling The code is then input into the decoding layer; the complete decoder process is described below: In the prediction phase, a three-layer 3D convolutional network is used for output:

9. The video anomaly detection method based on clustering-guided learning according to claim 7, characterized in that, The L2 norm is used as the reconstruction loss to measure the difference between the input X0 and the output X. recon Differences between video clips: The sum of the reconstruction loss and the clustering loss is used as the final loss function: loss=loss recon +loss cl 。

Images