Pseudo-anomaly guided 3d convolutional auto-encoding video anomaly detection method

Abstract

The present application relates to a kind of pseudo-3D convolutional auto-encoding video anomaly detection methods of guiding exception, belong to video monitoring field.The method first uses normal data to simulate abnormal data distribution to generate pseudo-exception number, secondly, normal data and pseudo-exception data are used to train automatic encoder AE, form PA-3DCAE, i.e.pseudo-3D convolutional auto-encoding network of guiding exception;Then the reconstruction loss of pseudo-exception data is maximized, while minimizing the reconstruction loss of normal data;Finally, based on the peak signal-to-noise ratio PSNR of reconstruction, calculate abnormal score.The present application improves the performance of abnormal behavior detection model.

Description

Technical Field

[0001] This invention belongs to the field of video surveillance, and relates to technical fields such as video surveillance security, medical monitoring, and traffic supervision. Specifically, it relates to a pseudo-anomaly-guided 3D convolutional autoencoder video anomaly detection method. Background Technology

[0002] Surveillance systems are now widely deployed in public places, and anomaly detection plays a crucial role in maintaining public safety efficiently and cost-effectively within surveillance videos. Traditional manual anomaly detection methods cannot handle such massive amounts of surveillance video data, thus necessitating the use of video anomaly detection methods for automated anomaly detection. However, video anomaly detection differs from traditional classification tasks. In real life, abnormal behaviors are rare, and the definition of anomaly depends on different environments, making it difficult to collect all possible anomaly samples. Therefore, anomaly detection is often considered a semi-supervised problem, where only normal data is used during training, and anything that does not conform to normal data is considered anomaly.

[0003] Existing anomaly detection methods are mainly divided into two categories: handcrafted methods and deep learning-based methods. Early handcrafted methods (such as skeletons, trajectories, 3D spatiotemporal gradients, and dense spatiotemporal interest points) struggle to capture effective and comprehensive features, making them unsuitable for analyzing complex video surveillance scenes. In deep learning methods, autoencoders (AEs) have been widely studied in anomaly detection due to their powerful ability to represent high-dimensional data. However, some methods rely solely on 2D convolutional autoencoders (2D-CAEs), failing to utilize temporal features, severely impacting video anomaly detection performance. Although some literature adds convolutional LSTM operations to 2D-CAEs to extract temporal information from video events, the more layers a model has, the more complex it becomes. Furthermore, some methods use optical flow to learn the motion representation of targets, but optical flow estimation is computationally very expensive. 3D CNNs can learn both spatial and temporal information in videos; Zhao et al. used 3D CNNs to model normal video data from a spatiotemporal perspective.

[0004] Because Action Abnormalities (AEs) possess strong learning and generalization capabilities, some anomalous behaviors may also be generated, leading to missed detections. To limit the AE's ability to reconstruct anomalous data, some literature employs a memory mechanism in the latent space between the encoder and decoder. However, such networks are highly dependent on memory size; large memory capacities not only consume memory but also fail to effectively limit the AE's ability to reconstruct anomalous data, while small memory capacities may affect its ability to reconstruct normal data. Other methods use cascading amplification effects to address this problem, but these methods generally have complex model structures, requiring two or more reconstructions. Recently, a new approach in video anomalous behavior detection utilizes pseudo-anomaly data generated from normal training data to address the issue of the AE's "too strong" generalization ability. OGNet and G2D suggest using untrained and inversely learned generators to generate pseudo-anomaly data to train a binary classifier; however, these methods require two stages of training—one for adversarial training of the generator and another for training the binary classifier—which incurs significant computational costs.

[0005] Therefore, a new video anomaly detection method is urgently needed to solve the above problems. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a pseudo-anomaly-guided 3D convolutional autoencoder video anomaly detection method to improve the performance of anomaly behavior detection models.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A pseudo-anomaly-guided 3D convolutional autoencoder (PA-3DCAE) video anomaly detection method is proposed. First, pseudo-anomaly numbers are generated by simulating the distribution of anomalous data using normal data. Second, the autoencoder (AE) is trained using both normal and pseudo-anomaly data to form PA-3DCAE, which is a pseudo-anomaly-guided 3D convolutional autoencoder network. Then, the reconstruction loss of the pseudo-anomaly data is maximized while the reconstruction loss of the normal data is minimized. Finally, the anomaly score is calculated based on the peak signal-to-noise ratio (PSNR) of the reconstruction.

[0009] Furthermore, the PA-3DCAE uses 3DCNN in the AE to extract spatiotemporal features from the video. The encoder contains four 3D convolutional layers, all of which use the same 3×3×3 convolutional kernel. The stride of the first 3D convolutional kernel is set to 1×2×2 to prevent premature merging of temporal signals, and the stride of the other 3D convolutional kernels is set to 2×2×2. The output feature map of each kernel is a 3D tensor with a temporal dimension. A batch normalization layer and a LeakyReLU activation function are used after each convolutional layer.

[0010] The decoder section has a symmetrical structure relative to the encoder section. It uses a three-dimensional deconvolution layer with a span of 2×2×2 as a reconstruction branch to reconstruct the input signal from the hidden layer. The Tanh layer is used in the last layer to make the output range [-1,1].

[0011] During training, video frame X is used as the input sequence for PA-3DCAE, expressed as:

[0012]

[0013] Where, X P It is a frame sequence generated by the pseudo-anomaly synthesizer, X N It is a sequence of frames from normal training data, where p represents the ratio of pseudo-abnormal data used during training.

[0014] Furthermore, the pseudo-anomaly number is generated using a pseudo-anomaly synthesizer based on frame skipping or CutMix.

[0015] Furthermore, for frame sequences X generated by pseudo-anomaly synthesizers based on frame skipping or CutMix... P We need to maximize its reconstruction loss, which is expressed as:

[0016]

[0017] Among them, L P This represents the distance between maximizing the decoder output value and generating the pseudo-anomaly; T, C, H, and W are the number of frames, number of channels, height, and width of the input video frame, respectively. Indicates the decoder output value, I′ t This indicates a generated pseudo-exception.

[0018] Furthermore, anomaly score A is calculated based on the reconstructed peak signal-to-noise ratio (PSNR). t The expression:

[0019] A t =1-Q t

[0020]

[0021]

[0022] Among them, Q t This represents the normalized PSNR for all frames in the test video; P t R represents the PSNR of all frames in the test video, i.e., the model optimization objective; R is... The total number of pixels in the array, where t is the frame index. yes The maximum possible value.

[0023] The beneficial effects of this invention are as follows: This invention proposes two methods for generating pseudo-anomalies by simulating abnormal data distributions using normal data: frame skipping and CutMix-based methods. Based on these methods, pseudo-anomalies are used to assist in training the Action Encoder (AE), forming a pseudo-anomaly-guided 3D convolutional autoencoder network (PA-3DCAE). The network model of this invention uses 3D convolution to extract spatiotemporal features of the video and introduces the Limit Reconstruction (LRC) concept of the AE, which is encouraged during training to restrict the reconstruction of abnormal data. Comparative experiments demonstrate that the method of this invention can effectively improve the performance of the abnormal behavior detection model.

[0024] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0026] Figure 1 A 3D convolutional autoencoder network guided by pseudo-anomalies;

[0027] Figure 2 For normal data and pseudo-abnormal data generated by a frame skipping method;

[0028] Figure 3 This includes normal data and pseudo-abnormal data generated based on the CutMix method;

[0029] Figure 4 Reconstruction plots and error heatmaps from three datasets, Figure 4 (a) shows the reconstruction diagram and error heatmap of UCSD-Ped2. Figure 4 (b) shows the reconstruction graph and error heatmap of Avenue. Figure 4 (c) shows the reconstruction diagram and error heatmap of ShanghaiTech. Figure 4 (d) shows the reconstruction graph and error heatmap based on the frame skipping and CutMix models (difficult to detect). Detailed Implementation

[0030] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0031] Please see Figures 1-4 This invention proposes two methods for generating pseudo-anomalies by simulating anomalous data distributions using normal data: frame skipping and CutMix-based methods. Based on these methods, pseudo-anomalies are used to assist in training the Advanced Image Encoder (AE), which is termed the Pseudo-Anomaly Guided 3D Convolutional Autoencoder Network (PA-3DCAE). The model uses 3D convolution to extract spatiotemporal features from the video and introduces the Limit Reconstruction (LRC) concept of the AE. During training, it reconstructs normal data well and anomalous data poorly, thus encouraging the AE to be used for reconstructing restricted anomalous data.

[0032] 1. Model Design:

[0033] The network framework of this invention is as follows: Figure 1 As shown, the AE is trained using normal and pseudo-anomaly data, maximizing the reconstruction loss for pseudo-anomaly data while minimizing the reconstruction loss for normal data. This encourages the AE to perform reconstructions that are distinguishable between normal and anomaly data. After model training, the reconstruction error for normal frames is smaller, while the reconstruction error for anomaly frames is larger. Anomalies are detected based on the error gaps, and anomaly scores are calculated using frame-level reconstruction loss.

[0034] Pseudo-anomaly-guided 3D convolutional autoencoder network (PA-3DCAE):

[0035] This invention uses 3DCNN in After Effects (AE) to extract spatiotemporal features from video. AEs are typically designed to receive multiple frames of input. The input sequence of the model is defined as X, with a size of T×C×H×W, where T, C, H, and W are the number of frames, number of channels, height, and width of the input video, respectively. The reconstructed X can be expressed as:

[0036] X=D(ε(X)) (1)

[0037] Here, ε and D represent the encoder and decoder of the AE, respectively. The encoder contains four 3D convolutional layers. Tran et al. found that 3×3×3 convolutional kernels perform best in action recognition tasks. Therefore, this invention uses the same 3×3×3 convolutional kernels for all 3D convolutional layers in the model. The stride of the first 3D convolutional kernel is set to 1×2×2 to prevent premature merging of temporal signals, and the stride of the other 3D convolutional kernels is set to 2×2×2. The output feature map of each kernel is a 3D tensor with a temporal dimension. A batch normalization layer and the LeakyReLU activation function are used after each convolutional layer. This invention uses strided convolution to reduce the resolution of the feature map. This parameterization operation can support the network to find an information-rich way to reduce the spatial resolution of the feature map and learn further upsampling during the decoding stage. The decoder part has a symmetrical structure relative to the encoder part, using a 3D deconvolutional layer with a stride of 2×2×2 as the reconstruction branch to reconstruct the input signal from the hidden layer. A Tanh layer is used in the last layer to make the output range [-1,1].

[0038] Traditional methods use only normal data to train the Action Array (AE) for reconstruction. During testing, samples deviating from the normal data are considered anomalous. Due to the powerful generative capabilities of the AE, the model cannot guarantee a large reconstruction error when reconstructing anomalous data. Unlike traditional methods, this invention uses normal data to synthesize pseudo-anomalous data, and then uses this pseudo-anomalous data to limit the generative capabilities of the AE and encourage it to produce high reconstruction errors on any type of anomalous input. During training, video frame X is provided as input to the network, as follows:

[0039]

[0040] Among them, X P It is a frame sequence generated by the pseudo-anomaly synthesizer, X N It is a sequence of frames from normal training data, where p represents the ratio of pseudo-abnormal data used during training.

[0041] To make the normal frames reconstructed by the decoder similar to their ground truth values, the decoder output needs to be minimized. and ground true value I t The distance between ' is:

[0042]

[0043] 2. Calculate the anomaly score:

[0044] This invention utilizes the reconstructed Peak Signal-to-Noise Ratio (PSNR) to calculate anomaly scores. PSNR is a better indicator of image quality than Mean Squared Error (MSE), therefore the optimization objective of the model is as follows:

[0045]

[0046] Where R is The total number of pixels in the array, where t is the frame index. yes The maximum possible value.

[0047] Min-max normalization is applied to all frames in the test video to normalize the PSNR values ​​to the range [0,1], as shown below:

[0048]

[0049] In formula (12), the higher Q t A lower value represents a lower reconstruction loss, and vice versa. Therefore, the final anomaly score is:

[0050] A t =1-Q t (6)

[0051] Example 1:

[0052] A pseudo-anomaly synthesizer based on frame skipping:

[0053] Most surveillance videos are static, while abnormal behavior is more prone to significant motion changes. Therefore, effectively detecting rapid or sudden motion changes is particularly important. To simulate abnormal motion in normal data, this invention arbitrarily skips several frames to generate a pseudo-abnormal sequence, such as... Figure 2 As shown, the first row represents the normal motion pattern, where the human walks at a normal speed. The second row represents pseudo-abnormal frames generated by skipping two normal frames. It can be clearly seen that the pedestrian almost completely crosses the reference line within a few frames. The specific implementation process is shown below:

[0054] For X generated by a frame-skipping pseudo-anomaly synthesizer P To maximize its reconstruction loss, the loss can be similarly defined as:

[0055]

[0056] Example 2:

[0057] A pseudo-anomaly synthesizer based on CutMix:

[0058] Abnormal behavior can take many forms. In real-world scenarios, besides violent abnormal behaviors such as fighting and chasing, there may also be non-violent abnormal behaviors like fainting. Furthermore, abnormal objects may appear in surveillance videos. To simulate the abnormal appearance of normal data, this invention employs a data augmentation-based CutMix method to generate pseudo-anomalies (such as...) from normal data. Figure 3As shown in the figure, this method does not use the intrusion dataset, but only uses normal data to generate pseudo-anomaly data for training, following the conventional semi-supervised anomaly detection method. The main idea of ​​CutMix is ​​to crop out a part of the region, which is not filled with 0 pixels, but is randomly filled with the pixel values ​​of other data in the training set.

[0059] For the generated pseudo-exception I' n+t Similarly, we need to maximize its reconstruction loss, so the loss can also be defined similarly as:

[0060]

[0061] Comparative experiment:

[0062] Quantitative results analysis:

[0063] To verify the effectiveness of the proposed model, this invention compares it with many existing methods that perform well, including methods based on handcrafted features, such as hybrid probabilistic principal component analyzers (MPPCA); and deep learning-based methods, such as two-dimensional convolutional autoencoders (AE-Conv2D), convolutional LSTM-based autoencoders (AE-ConvLSTM), three-dimensional convolutional autoencoders (AE-Conv3D), object-centric and memory-guided residual spatiotemporal autoencoders (OM-RSTAE), memory-enhanced autoencoders (MemAE), memory-guided autoencoders (MNAD), cascaded reconstruction models (Cascade-Rec), adversarial learning-based classifiers (OGNet), and ensemble prediction and reconstruction (Pre+Rec).

[0064] Table 1 shows the AUC results of various methods on the UCSD-Ped2, Avenue, and ShanghaiTech datasets, in percentage form. It can be observed that the two models proposed in this invention achieve an AUC gain of nearly 20% compared to methods based on handcrafted features. OM-RSTAE, MemAE, and MNAD are all memory-based networks, and their highest AUC gains on the UCSD-Ped2, Avenue, and ShanghaiTech datasets are 94.1%, 84.0%, and 71.2%, respectively. The two models proposed in this invention surpass the highest gains of memory-based networks on all three different datasets without adding extra modules that increase network complexity, demonstrating the superiority of the proposed method over memory-based methods. Furthermore, it can be seen that the PA-3DCAE model outperforms the baseline on all three datasets. Specifically, the frame-skipping-based PA-3DCAE model achieves AUC gains of 6.3%, 3.7%, and 3.7% on the UCSD-Ped2, Avenue, and ShanghaiTech datasets, respectively. The CutMix-based PA-3DCAE model achieved AUC gains of 4.2%, 3.1%, and 1.2% on the UCSD-Ped2, Avenue, and ShanghaiTech datasets, respectively, demonstrating the ability of the proposed method to improve baselines across multiple datasets.

[0065] As shown in Table 1, the PA-3DCAE model based on frame skipping achieves the highest AUC on the UCSD-Ped2 and ShanghaiTech datasets, reaching 98.8% and 75.0%, respectively. Although its performance on the Avenue dataset is not as good as the Cascade-Rec model, the Cascade-Rec model uses a hybrid framework of frame reconstruction and flow prediction for anomalous behavior detection, requiring the training of two AEs sequentially and additional modules to estimate optical flow, which necessitates more network parameters and a more realistic ground flow field. In contrast, the model of this invention uses a simple AE to extract features and reconstruct frames, making its training speed more efficient than the Cascade-Rec model. For the CutMix-based PA-3DCAE model, its performance on the ShanghaiTech dataset is superior to other methods, while its performance on the Avenue dataset is inferior to the Cascade-Rec model. On the UCSD-Ped2 dataset, its performance is slightly lower than the OGNet and Cascade-Rec models, but the training difficulty of these two models is far greater than that of the model of this invention.

[0066] Table 1. AUC values ​​of different methods on the UCSD-Ped2, Avenue, and ShanghaiTech datasets.

[0067]

[0068] Note: The bolded parts in Table 1 are the optimal values ​​under the current conditions.

[0069] Qualitative results analysis:

[0070] Figure 4 The data displays the reconstruction results on three datasets: UCSD-Ped2, Avenue, and ShanghaiTech. The leftmost column shows the ground truth map, which includes anomalous behaviors such as cycling and running, highlighted with red borders. The second column shows the reconstruction map of the baseline and its reconstruction error heatmap. The third column shows the reconstruction map of the model based on the frame skipping method and its reconstruction error heatmap. The last column shows the reconstruction map of the model based on CutMix and its reconstruction error heatmap. These reconstruction error heatmaps are generated by calculating the squared error of each pixel between the reconstructed frame and the input frame, and then performing min-max normalization. Figure 4 (a) shows the reconstruction plot and error heatmap of the UCSD-Ped2 dataset. It can be seen that the baseline reconstructs the anomalies well, while the two methods proposed in this paper reconstruct the anomalies poorly. The error plot significantly highlights the anomalous events. Figure 4 Visualizations of Avenue and ShanghaiTech also show that the anomalies reconstructed by the PA-3DCAE model are more blurred compared to the baseline, which demonstrates the superiority of the proposed model in video anomaly detection.

[0071] In addition, from Figure 4 It can also be seen that the PA-3DCAE model based on frame skipping has more distorted reconstruction results compared with the PA-3DCAE model based on CutMix, and the abnormal events in the error map are more significant. This is because the abnormal behaviors in the map are mainly running people and bicycles, whose speed is significantly faster than the normal behavior around them. Therefore, the model based on frame skipping is more likely to detect the abnormalities. Figure 4 (a) Figure 4 The anomalous behavior in (c) is a bicycle, which exhibits not only speed anomalies but also obvious appearance anomalies. The CutMix-based model can easily detect these anomalies, but... Figure 4 The anomaly in (b) is a running person. This anomalous behavior is not only far from the camera but also obscured by other buildings, making it not very obvious. The CutMix-based model often fails to detect such anomalies as expected. Figure 4As shown in (d), the detection performance of the two models proposed in this invention is slightly improved compared to the baseline, but not significantly. This is because the bicycle in the image is too thin and its movement is normal, making it difficult for both models of this invention and the baseline to detect such anomalies. For some behaviors that appear abnormal but move normally, such as stones in the middle of the road and people fainting on the sidewalk, the detection performance based on the CutMix model is better than that based on frame skipping. In summary, choosing an appropriate method for generating pseudo-anomalies can effectively improve the performance of abnormal behavior detection models.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A pseudo-anomaly-guided 3D convolutional autoencoder video anomaly detection method, characterized in that, First, normal data is used to simulate the distribution of abnormal data to generate pseudo-anomaly numbers. Second, the autoencoder AE is trained using normal data and pseudo-anomaly data to form PA-3DCAE, which is a pseudo-anomaly-guided 3D convolutional autoencoder network. Then, the reconstruction loss of pseudo-anomaly data is maximized while the reconstruction loss of normal data is minimized. Finally, the anomaly score is calculated based on the peak signal-to-noise ratio (PSNR) of the reconstruction. The PA-3DCAE uses 3DCNN in AE to extract spatiotemporal features from video. The encoder contains four 3D convolutional layers, all of which use the same 3×3×3 convolutional kernel. The stride of the first 3D convolutional kernel is set to 1×2×2 to prevent premature merging of temporal signals, and the stride of the other 3D convolutional kernels is set to 2×2×2. The output feature map of each kernel is a 3D tensor with a temporal dimension. A batch normalization layer and the LeakyReLU activation function are used after each convolutional layer. The decoder section has a symmetrical structure relative to the encoder section. It uses a three-dimensional deconvolution layer with a span of 2×2×2 as a reconstruction branch to reconstruct the input signal from the hidden layer. The Tanh layer is used in the last layer to make the output range [-1,1]. During training, video frame X is used as the input sequence for PA-3DCAE, expressed as: in, It is a frame sequence generated by the pseudo-anomaly synthesizer. It is a sequence of frames from normal training data. This represents the ratio of pseudo-outlier data used during training. The pseudo-anomaly count is generated using a pseudo-anomaly synthesizer based on frame skipping or CutMix; for the frame sequence generated by the pseudo-anomaly synthesizer based on frame skipping or CutMix... We need to maximize its reconstruction loss, which is expressed as: in, This represents the distance between maximizing the decoder output value and generating the pseudo-anomaly; T, C, H, and W are the number of frames, number of channels, height, and width of the input video frame, respectively. This represents the decoder output value. This indicates a generated pseudo-exception.

2. The 3D convolutional autoencoder video anomaly detection method according to claim 1, characterized in that, Anomaly scores are calculated based on the reconstructed peak signal-to-noise ratio (PSNR). The expression: in, This represents the normalized PSNR for all frames in the test video; This represents the PSNR of all frames in the test video, which is the model optimization target. R yes The total number of pixels in t It is a frame index. yes The maximum possible value.

Images