A video anomaly detection method based on SSIM and STE-3D

Abstract

The present invention discloses a video anomaly detection method based on SSIM and STE-3D. The method comprises the following steps: first, replacing the mean square error (MSE) in the loss function and anomaly score calculation formula of a three-dimensional convolution-based video anomaly detection model to be enhanced with the structural similarity index (SSIM), inserting the spatiotemporal enhancement block (STE-3D), and obtaining an enhanced video anomaly detection model; preprocessing video frames in a training set into small video clips and inputting them into the enhanced video anomaly detection model for training, with the model input denoted as I; denoting the small video clips reconstructed from the preprocessed small video clips as O, calculating the loss value between I and O, and updating the model weights until training is complete; and in the prediction phase, calculating the reconstruction error between I and O to determine whether the current frame is an anomaly frame. The present invention utilizes the aforementioned video anomaly detection method based on SSIM and STE-3D, providing a metric more suitable for evaluating video anomalies and ensuring that the detected anomalies more accurately correspond to human observations.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a video anomaly detection method based on SSIM and STE-3D. Background Art

[0002] Before the advent of deep learning, traditional video anomaly detection methods relied on handcrafted features. For example, optical flow histograms, weighted statistics of optical flow, and motion boundary histograms treat the x and y directions of an optical flow image as two grayscale images. These features are then extracted from the gradient histograms of these grayscale images and used to detect anomalies. Dense trajectory methods use optical flow to obtain motion trajectories in videos and extract features along these trajectories for anomaly detection.

[0003] The rise of deep learning has led to the widespread application of supervised model training methods in vision tasks. However, labeling abnormal patterns in videos is time-consuming and difficult, especially in today's highly prevalent surveillance cameras. Therefore, video anomaly detection models typically follow a paradigm where only videos containing normal patterns are input during the training phase, while videos containing both normal and abnormal patterns are input during the testing phase. With the widespread application of deep learning in vision tasks, reconstruction-based video anomaly detection methods are becoming increasingly popular. These methods typically use autoencoders and generative adversarial networks to extract feature representations from videos.

[0004] Reconstruction-based methods extract features from video frames, assuming that normal frames can be reconstructed more accurately than anomalous frames. However, due to the strong generalization capabilities of deep learning models, anomalous frames can also be reconstructed effectively. Therefore, these methods typically seek ways to enable the model to reconstruct normal frames better and anomalous frames worse, thereby enabling the model to distinguish between normal and anomalous frames. For example, Gong et al. introduced a memory module into the bottleneck of an autoencoder, storing normal patterns extracted from normal video frames in this memory module. The model queries the memory module for similar vectors based on the encoder output and filters out dissimilar vectors. Subsequently, the video frame is reconstructed using the query vectors. This process helps to increase the difference between the reconstructed anomalous frames and the original anomalous frames. Building on this, Park et al. proposed a memory module that uses less video memory. Liu et al. also introduced the PDM and ICM blocks to enhance the model's ability to recognize various normal and unknown anomalous patterns. Marcella et al. constructed pseudo-anomalies from small segments of consecutive video frames by skipping one or several frames to train the autoencoder. The model then modeled the inverse of these pseudo-anomalies. Prediction-based methods typically take t consecutive video frames as input, then have the model predict the t+1 frame and compare the error between the predicted frame and the original frame to determine whether the video frame is abnormal. Wang et al. proposed using a frame prediction network based on a multi-path ConvGRU for modeling, which can better handle semantically informed objects and regions of different scales and capture the spatiotemporal correlations in normal videos.

[0005] In the field of video anomaly detection, many methods rely primarily on losses based on mean squared error (MSE), or use MSE as a primary component. However, MSE is calculated at the pixel level and has limited relevance to human perception. Since video anomaly detection aims to identify unusual events and objects in videos, the detected anomalies must be consistent with human perception. Models in the field of video anomaly detection do not aim to capture anomalies that are beyond or difficult for the human visual system to perceive. Summary of the Invention

[0006] The purpose of this invention is to provide a video anomaly detection method based on SSIM and STE-3D. Since SSIM is more consistent with human perception, the structural similarity index (SSIM) is used as the basis of the loss function and the anomaly score calculation formula, providing a more suitable indicator for evaluating video anomalies and ensuring that the detected anomalies correspond more accurately to human observations.

[0007] To achieve the above object, the present invention provides a video anomaly detection method based on SSIM and STE-3D, comprising the following steps:

[0008] S1. Replace the mean square error (MSE) in the loss function and anomaly score calculation formula of the 3D convolution-based video anomaly detection model to be enhanced with the structural similarity index (SSIM).

[0009] S2. Inserting the spatiotemporal enhancement block STE-3D at a specific position of the 3D convolution-based video anomaly detection model to be enhanced to obtain the enhanced video anomaly detection model;

[0010] S3. Preprocess the video frames in the training set into small video clips and input them into the enhanced video anomaly detection model for training. The model input is denoted as I.

[0011] S4: The enhanced video anomaly detection model is reconstructed based on the preprocessed small video clips to generate a small video clip, which is recorded as O. The loss value between I and O is calculated, and the model weight is updated after optimization by the optimizer until the training is completed.

[0012] S5, prediction stage, calculates the reconstruction error between I and O, and then determines whether the current frame is an abnormal frame;

[0013] Multiply the weight of the SSIM that replaces the MSE part by a specific coefficient , that is, the following formula (1) is upgraded to formula (2):

[0014] (1);

[0015] (2);

[0016] in, Formula (1) and Formula (2) represent the parts of the loss function and the anomaly score calculation formula except MSE and 1-SSIM;

[0017] In step S2, the spatiotemporal enhancement block STE-3D includes a 3D convolution with a kernel size of 1 and a stride of 1, a 3D convolution with a kernel size of 3 and a stride of 2, a 3D max pooling with a kernel size of 7, a 3D convolution with a kernel size of 3 and a stride of 1, a cubic linear interpolation upsampling module, a 3D convolution bypass with a kernel size of 1 and a stride of 1, a 3D convolution with a kernel size of 1 and a stride of 1, and a sigmoid activation function;

[0018] Inserting the spatiotemporal enhancement block STE-3D into the three-dimensional convolution-based video anomaly detection model to be enhanced includes adding the spatiotemporal enhancement block STE-3D to the encoder of the three-dimensional convolution-based video anomaly detection model to be enhanced and adding the spatiotemporal enhancement block STE-3D to the decoder of the three-dimensional convolution-based video anomaly detection model to be enhanced.

[0019] Preferably, in step S1, the structural similarity index SSIM is used for two given images A and B, and its expression is as shown in formula (3):

[0020] (3);

[0021] in, 、 and Represents brightness, contrast and structure respectively. 、 、 Control the relative importance of brightness, contrast and structural similarity in the SSIM index respectively. represents image A, Indicates a B picture;

[0022] Brightness is the average brightness level of the pixel values ​​in the image, and its expression is as shown in formula (4):

[0023] (4);

[0024] Contrast describes the brightness difference between adjacent pixels in an image, and its expression is as shown in formula (5):

[0025] (5);

[0026] The structure measures the spatial arrangement of pixel patterns in an image, which is expressed as formula (6):

[0027] (6);

[0028] The above formulas (4) to (6) involve Represent the pixel mean of images A and B respectively, represents the pixel variance of images A and B, represents the pixel covariance of images A and B, is a constant.

[0029] Preferably, the Gaussian weighted formula is used, and the size is 11×11 pixels and the standard deviation is The filter kernel is used to calculate the SSIM value between the original image and the reconstructed image, including the variance, mean and covariance, according to formula (4), formula (5) and formula (6);

[0030] According to formula (7), the average value of the SSIM of all windows of the two images is calculated as the SSIM value of the two images:

[0031] (7);

[0032] in, Mis the number of windows into which the image is divided, Indicates the current window;

[0033] make 、 , simplifying formula (7) to formula (8):

[0034] (8).

[0035] Preferably, in step S3, the preprocessing operation includes converting all video frames in the training set into grayscale image format, and dividing them into small video clips to input into the enhanced video anomaly detection model.

[0036] Preferably, in step S3, the features in the video frame are extracted by the 3D convolution-based video anomaly detection model to be enhanced. When the data stream reaches STE-3D, STE-3D performs temporal and spatial enhancement on the features in the video frame extracted by the 3D convolution-based video anomaly detection model to be enhanced, and then inputs the features back into other parts of the 3D convolution-based video anomaly detection model to be enhanced.

[0037] Therefore, the present invention adopts the above-mentioned video anomaly detection method based on SSIM and STE-3D, and the beneficial effects are as follows:

[0038] (1) The proposed method includes inserting STE-3D and replacing MSE with SSIM. SSIM helps the training of the enhanced 3D convolution-based video anomaly detection model to develop in a direction related to human vision; in particular, the insertion of STE-3D can help the baseline capture more spatiotemporal features, thereby better reconstructing normal patterns, while the reconstruction effect of abnormal patterns is poor.

[0039] (2) Experimental results verify the effectiveness of the proposed method in video anomaly detection. Numerous experimental results show that the proposed method can significantly improve the accuracy of the video anomaly detection model.

[0040] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a process of dividing the original image and the reconstructed image into windows to calculate the mean SSIM in an embodiment of a video anomaly detection method based on SSIM and STE-3D of the present invention, wherein (a) is a schematic diagram of the original image after being divided into windows, and (b) is a schematic diagram of the reconstructed image after being divided into windows.

[0042] Figure 21 is a schematic diagram of the structure of a spatiotemporal enhancement block STE-3D in a three-dimensional convolution in an embodiment of a video anomaly detection method based on SSIM and STE-3D according to the present invention;

[0043] Figure 3 It is a partial visualization feature map when the three-dimensional convolution-based video anomaly detection model to be enhanced is STEAL Net in an embodiment of a video anomaly detection method based on SSIM and STE-3D of the present invention;

[0044] Figure 4 This is an overall flow chart of an embodiment of a video anomaly detection method based on SSIM and STE-3D of the present invention;

[0045] Figure 5 This is the difference between the anomaly scores calculated using MSE and SSIM as the loss function and the anomaly score calculation formula in the Ped1 test video of the comparative example 1 of the video anomaly detection method based on SSIM and STE-3D of the present invention, where the first column in (a) is the original image at a certain time point, the second column is the reconstructed image at a certain time point, and the third column is the reconstruction error between the original image and the reconstructed image at a certain time point; (b) is a line graph showing the change of the video anomaly score over time;

[0046] Figure 6 It is a visualization diagram of weight parameters in STE-3D in comparative example 2 of a video anomaly detection method based on SSIM and STE-3D of the present invention. DETAILED DESCRIPTION

[0047] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0048] Unless otherwise defined, technical or scientific terms used in the present invention shall have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0049] Analyze the current field of video anomaly detection from the perspective of loss function and anomaly score calculation formula

[0050] A key component of deep learning is the loss function used for model training. The loss function measures how close the model is to the expected output, and choosing an appropriate loss function is crucial for achieving good performance in deep learning tasks. However, due to the diversity of available methods, practitioners can face challenges in selecting the most appropriate one. In the past few years, video anomaly detection has often used mean squared error (MSE) as the primary component of the model loss function. For example, Gong et al. used the sum of the MSE between the input and reconstructed video frames and the sum of the output cross-entropy computed by the memory module as the loss function, and only used this MSE value in the anomaly score calculation. Yang et al. used the sum of the Charbonnier loss (an enhanced version of the MSE) and the adjacent frame difference (AFD) loss between the input and reconstructed video frames as the loss function, and used PSNR (a variant of the MSE) as the primary component of the anomaly score calculation formula. Liu et al. used the MSE between the input anomaly vector and the reconstructed anomaly vector as the model loss function, in addition to other losses. However, these loss functions and anomaly score calculation formulas have failed to effectively align with human visual system evaluation results.

[0051] One approach to addressing the MSE's shortcomings is to incorporate image quality assessment algorithms that are more closely aligned with human vision into the core of the loss function and anomaly score calculation formula. Some image quality assessment algorithms exhibit high correlation with human perception. The Structural Similarity Index (SSIM) emphasizes structure-based image similarity, assuming that the human visual system excels at extracting structural information. In contrast, the Feature Similarity Index (FSIM) prioritizes phase congruence (PC) and gradient magnitude (GM), assuming that the human visual system primarily understands images through low-level features.

[0052] Attention mechanisms have achieved remarkable success in various visual tasks in recent years, including image classification, object detection, semantic segmentation, self-supervised learning, etc. The attention mechanism can be regarded as a process of dynamically adjusting weights based on the features of the input image. According to the data domain, attention mechanisms can be divided into several types: channel, spatial, event, and branch. Hu et al. proposed the Squeeze and Excite Network (SENet), which demonstrated the interdependence between modeling feature channels and significantly improved the performance of image classification tasks. Woo et al. proposed the Convolutional Block Attention Module (CBAM), which demonstrated the dependency between the channel dimension and the spatial dimension. Ristea et al. proposed a Self-Supervised Predictive Convolutional Attention Block (SSPCAB), but the block they constructed was based on two-dimensional convolution. Liu et al. proposed a spatial attention enhancement module ESA for image super-resolution.

[0053] This paper proposes using SSIM as the primary component of the model's loss function and the main formula for calculating the video frame anomaly score. This allows the model to learn structural features that are more consistent with human judgment of anomalies, rather than texture features that are insensitive to humans but sensitive to neural networks. To achieve higher-precision image reconstruction, this paper further proposes a spatiotemporal enhancement block (STE-3D).

[0054] STE-3D is applicable to all three-dimensional convolutional neural networks because STE-3D can be inserted into the structure of any three-dimensional convolutional neural network, and the insertion position is usually arbitrary. This module can provide more useful features for subsequent neural network layers. In theory, as long as the structure of the neural network is three-dimensional convolution, this module can be applied to any neural network field to enhance its ability to extract spatiotemporal features. The module of the present invention is similar to U-Net. The U-Net network is a backbone network commonly used in the field of video anomaly detection because it can extract normal patterns in the video while ignoring abnormal patterns. Similar to U-Net, the module of the present invention will also extract normal patterns in the video while ignoring abnormal patterns.

[0055] A video anomaly detection method based on SSIM and STE-3D is a general method based on 3D convolutional models in the field of enhanced video anomaly detection, comprising the following steps:

[0056] S1. Replace the mean square error (MSE) in the loss function and anomaly score calculation formula of the enhanced 3D convolution-based video anomaly detection model with the structural similarity index (SSIM) (taking the basic model AE as an example).

[0057] The structural similarity index SSIM is used for two given images A and B, and its expression is as shown in formula (3):

[0058] (3);

[0059] in, 、 and Represents brightness, contrast and structure respectively. 、 、 Control the relative importance of brightness, contrast and structural similarity in the SSIM index respectively. represents image A, Indicates a B picture.

[0060] Brightness is the average brightness level of pixel values ​​in an image, which directly affects the perception of light and dark in the image. The human visual system is very sensitive to changes in brightness, especially in areas where brightness changes sharply, which are usually where the details of the image are of concern. Its expression is as shown in formula (4):

[0061] (4);

[0062] Contrast describes the brightness difference between adjacent pixels in an image. The human visual system perceives edges and textures in images by identifying these contrast differences, which is crucial for understanding image content. Its expression is shown in formula (5):

[0063] (5);

[0064] Structure measures the spatial arrangement of pixel patterns in an image, representing its structural information. The human visual system tends to focus on these structural features because they are crucial for recognizing and interpreting objects and events in visual scenes. Its expression is shown in formula (6):

[0065] (6);

[0066] The above formulas (4) to (6) involve Represent the pixel mean of images A and B respectively, represents the pixel variance of images A and B, represents the pixel covariance of images A and B, is a constant.

[0067] To calculate the mean, variance, and covariance within a window around a pixel, the present invention uses a Gaussian weighted version of the formula and adopts a window size of 11×11 pixels with a standard deviation of The filter kernel of , according to formula (4), formula (5) and formula (6), the SSIM value between the original image and the reconstructed image can be calculated, including variance, mean and covariance.

[0068] However, these values ​​vary drastically across the entire image. Therefore, it is necessary to calculate the SSIM value within a window surrounding the pixel, and then calculate the average of the SSIM values ​​of all windows for both images A and B as the SSIM value for the two images according to formula (7):

[0069] (7);

[0070] in, M is the number of windows into which the image can be divided, Indicates the current window;

[0071] like Figure 1 The original image and the reconstructed image are divided into windows to calculate the mean SSIM. Figure 1As can be seen, when calculating the mean SSIM (MSSIM) of these two images, the original image and the reconstructed image are cut into windows of size k in the same manner. The window corresponding to the original image and the window of the reconstructed image are used together to calculate the SSIM value of each window. Finally, the SSIM values ​​of all windows are averaged to obtain the MSSIM of the entire image.

[0072] Therefore, in order to simplify the SSIM calculation formula, let 、 , formula (7) can be simplified to formula (8):

[0073] (8);

[0074] MSE compares the absolute error between two images, while Structural Similarity Index (SSIM) compares the relative error between two images. Therefore, when the value of the loss function MSE used for training is significantly different from the value of the loss function other than the MSE part, the weight of the SSIM replacing the MSE part should be multiplied by a specific coefficient. , that is, the following formula (1) is upgraded to formula (2):

[0075] (1);

[0076] (2);

[0077] in, Formula (1) and Formula (2) represent the parts of the loss function and the anomaly score calculation formula except MSE and 1-SSIM.

[0078] S2. Insert the spatiotemporal enhancement block STE-3D at a specific position of the 3D convolution-based video anomaly detection model to be enhanced to obtain the enhanced video anomaly detection model.

[0079] Ristea et al. introduced the SSPCAB module, significantly improving the performance of video anomaly detection models. However, this module relies on two-dimensional convolutions, while three-dimensional convolutions are more effective in capturing spatiotemporal features in videos. Furthermore, SSIM has limitations in guiding the extraction of temporal information. Therefore, a more effective method is needed to capture the temporal dynamics of video frames.

[0080] To improve the model's accuracy in detecting video anomalies, this paper proposes a spatiotemporal enhancement block (STE-3D), designed to be integrated into specific locations in the model to enhance the temporal and spatial representations of input features, enabling subsequent network layers to better process these enhanced features. First, STE-3D needs to meet lightweight criteria so that it can be inserted into various other three-dimensional convolutional networks without incurring a performance burden. Second, STE-3D requires a larger receptive field to better highlight important (normal pattern) portions of the features and suppress less important (abnormal pattern) portions.

[0081] When designing the STE-3D module, the present invention draws inspiration from the field of image super-resolution, such as Figure 2 The autoencoder on the left is the backbone network of the present invention that will be inserted into STE-3D. The blue rectangle indicates the insertion position, and the STE-3D on the right represents the internal structure of the block.

[0082] like Figure 2 As shown in Figure 3, the entire spatiotemporal enhancement block STE-3D includes a 3D convolution with a kernel size of 1 and a stride of 1, a 3D convolution with a kernel size of 3 and a stride of 2, a 3D maximum pooling with a kernel size of 7, a 3D convolution with a kernel size of 3 and a stride of 1, a cubic linear interpolation upsampling module, a 3D convolution bypass with a kernel size of 1 and a stride of 1, a 3D convolution with a kernel size of 1 and a stride of 1, and a sigmoid activation function.

[0083] Theoretically, STE-3D can be placed in any 3D convolution-based backbone network, with varying effectiveness depending on its placement. Inserting the STE-3D spatiotemporal enhancement block into the 3D convolution-based video anomaly detection model to be enhanced involves adding the STE-3D spatiotemporal enhancement block to the encoder and the decoder of the 3D convolution-based video anomaly detection model to be enhanced.

[0084] This embodiment chooses to insert the spatiotemporal enhancement block STE-3D into the baseline of STEAL Net, specifically adding the spatiotemporal enhancement block STE-3D to the encoder of STEAL Net and adding the spatiotemporal enhancement block STE-3D to the decoder of STEAL Net.

[0085] like Figure 3 As shown in Figure 2, the input and output images of the model are the first and last columns of the figure, respectively. The second and third columns show the feature map visualizations of the first convolutional layer of the encoder and the penultimate convolutional layer of the decoder before and after adding STE-3D.

[0086] Observed Figure 3It can be seen that: (1) Adding STE-3D to the encoder helps the model better capture and retain important features during the downsampling process. This helps to recover more detailed information in subsequent decoding. Since the encoder gradually reduces the spatial dimensions of the feature map, STE-3D helps the model focus on key information before the feature map becomes more abstract.

[0087] (2) Adding STE-3D to the decoder enhances the model’s ability to reconstruct details during upsampling. This helps locate regions of interest more precisely without anomalies, so the decoder is better able to reconstruct normal parts of the video frame.

[0088] S3. Preprocess the video frames in the training set into small video clips and input them into the enhanced video anomaly detection model for training. The model input is denoted as I.

[0089] The preprocessing operation includes converting all video frames in the training set into grayscale format and segmenting them into small video clips containing 16 consecutive frames to input into the enhanced video anomaly detection model.

[0090] The basic video anomaly detection model extracts features from video frames. When the data stream reaches STE-3D, STE-3D will enhance the features in the video frames extracted by the basic video anomaly detection model in terms of time and space, and then input them back into other parts of the basic video anomaly detection model.

[0091] S4. The output of the enhanced video anomaly detection model is recorded as O according to the preprocessed small video clip. In the training phase, the loss function between I and O is calculated according to formula (2). The model weight is updated after optimization by the Adam optimizer until the training is completed.

[0092] S5. In the prediction stage, the reconstruction error between I and O is calculated, and then it is determined whether the current frame is an abnormal frame.

[0093] Example

[0094] Selecting the experimental dataset

[0095] Pedestrian Dataset Ped1: This dataset is created based on videos from cameras monitoring pedestrian walkways at the University of California, San Diego. The main difficulty is that there are dense crowds of people, sometimes even forming congestion, which can block abnormal objects. The video resolution is The dataset contains 34 training videos and 36 test videos. Abnormal cases include non-walking human objects, such as carts and bicycles, and abnormal pedestrian behaviors, such as running.

[0096] Pedestrian Dataset Ped2: This dataset is the same as Ped1 and also monitors pedestrian walkways at the University of California, San Diego, but the camera positions and shooting locations are different. This dataset is the most commonly used and simplest for video anomaly detection. The video frame resolution is pixels, including 16 training videos and 12 testing videos.

[0097] CUHK Avenue dataset: This dataset is a single-scene video surveillance dataset. The main difficulty of the dataset is that the concrete columns in the video cause the target to be blocked and introduce human walking anomalies. The video frame resolution is There are 16 training videos and 21 testing videos in total.

[0098] ShanghaiTech dataset introduced by Shanghai University of Science and Technology: This dataset is a video anomaly detection dataset proposed by Shanghai University of Science and Technology, which contains video surveillance of multiple scenes. This dataset is a larger and more challenging dataset in video anomaly detection because it contains more scenes and richer anomaly information. The video frame resolution is pixels, including 330 training videos and 107 testing videos.

[0099] The selected baseline, i.e. the 3D convolution-based video anomaly detection model to be enhanced

[0100] AE-Conv3D: Zhao et al. introduced a spatiotemporal autoencoder (AE) that automatically learns video representations using deep neural networks. Features are extracted from both spatial and temporal dimensions through 3D convolution. Since no open-source code was available, a baseline from Mem-AE was used, which has a nearly identical architecture and slightly outperforms AE.

[0101] Mem-AE: Gong et al. proposed an improved memory-augmented autoencoder (Mem-AE), which first obtains encodings from the autoencoder, then queries to retrieve the most relevant content from the memory and sends it back to the autoencoder to reconstruct video frames containing only normal patterns.

[0102] STEAL Net: Astrid et al. proposed a temporal pseudo-anomaly synthesizer for training autoencoders (STEALNet). During training, STEAL Net constructs temporally related pseudo-anomalies by skipping N frames. Subsequently, STEAL Net minimizes the loss associated with normal patterns rather than maximizing the reconstruction loss of abnormal patterns.

[0103] Evaluation indicators

[0104] Building on previous research, this paper uses the area under the receiver operating characteristic (ROC) curve (AUC) at the video frame level as a performance metric for evaluating the accuracy of video anomaly detection models. This metric evaluates the model's ability to detect anomalies by considering the anomaly score assigned to each frame in the test dataset. A higher AUC value indicates a model's effectiveness in video anomaly detection.

[0105] When using the proposed method, including inserting STE-3D and replacing MSE with SSIM, SSIM helps the baseline training move towards a direction relevant to human vision. In particular, the insertion of STE-3D helps the baseline capture more spatiotemporal features, thereby better reconstructing normal patterns. At the same time, the reconstruction of abnormal patterns is less effective. The experimental results in Table 1 verify the effectiveness of the proposed method in video anomaly detection.

[0106] Table 1 AUC performance comparison on different baselines

[0107]

[0108] Table 1 shows the AUC performance comparison of the proposed method on several baselines: Ped1, Ped2, Avenue, and ShanghaiTech. The bold numbers represent the best results in the target dataset, while the underlined numbers represent the suboptimal results.

[0109] Numerous experimental results show that the method proposed in this invention can significantly improve the accuracy of video anomaly detection models.

[0110] Comparative Example 1:

[0111] Effect of using only SSIM: Theoretically, SSIM correlates better with the human visual system than MSE. Therefore, we expect that using SSIM will improve model performance. In our experiments, we only replaced MSE with SSIM in the loss function and anomaly score calculation formula of the baseline model; no other changes were made to the model.

[0112] Table 1 compares the AUC performance of the baseline model on Ped1, Ped2, Avenue, and ShanghaiTech with the baseline model after replacing the primary SSIM in the loss function and anomaly score calculation formula. Red marks indicate performance degradation. Bold numbers indicate the best results on the target dataset.

[0113] The experimental results in Table 2 highlight the superiority of SSIM over MSE.

[0114] Table 2 AUC performance comparison of baseline models on different datasets

[0115]

[0116] Figure 5 (a) Shows the difference between anomaly scores calculated for the Ped1 test video using MSE and SSIM as the loss function and anomaly score calculation formula. Specifically, the frame-level anomaly scores for test video 8 of the Ped1 dataset are compared before and after replacing MSE with SSIM in the autoencoder's loss function and anomaly score calculation formula (baseline and proposed model). Light blue represents the anomaly score curve using MSE, and light green represents the anomaly score curve using SSIM.

[0117] from Figure 5 The line graph in (b) shows that the model trained using SSIM is more accurate in calculating the anomaly score. The input, reconstructed output, and the difference between the two reconstructions are visualized. This visualization demonstrates that the enhanced autoencoder is better able to reconstruct normal patterns.

[0118] However, MSE tends to ignore high-level structural information, which is prioritized by the human visual system. In contrast, SSIM focuses on structural similarity by comparing brightness, contrast, and structure between images, which is more consistent with human visual perception. Although SSIM captures perceptually important patterns, it may ignore the subtle pixel-level differences that MSE is designed to penalize. This trade-off can lead to a slight decrease in accuracy in situations where pixel-level accuracy is more important than structural consistency, especially when subtle anomalies do not significantly change the overall structure of the video frame.

[0119] However, it is worth noting that this decrease is marginal and occurs less frequently than the improvement in accuracy. The superior performance of SSIM on some datasets, particularly the significant accuracy improvements observed for the AE model on Ped2 and STEAL Net on Ped1, suggests that SSIM's emphasis on structural consistency is beneficial in scenarios where anomalies are more structured. This suggests that SSIM provides a more perceptually meaningful metric in many anomaly detection tasks, improving model performance overall.

[0120] Comparative Example 2

[0121] The effect of using only STE-3D:

[0122] The experimental results in Table 3 demonstrate the effectiveness of the proposed STE-3D approach in improving model accuracy. The proposed approach compares the AUC performance with multiple baselines on Ped1, Ped2, Avenue, and ShanghaiTech. Bold numbers indicate the best results in their respective target datasets. STE-3D consistently improves model accuracy across all selected model datasets.

[0123] Table 3 AUC performance comparison with multiple baselines on different datasets

[0124]

[0125] from Figure 6 The heatmap shows that the module is indeed able to capture the spatial and temporal features in the video frame, where cooler colors indicate smaller weights and warmer colors indicate larger weights. Figure 6 All of these prove the effectiveness of the STE-3D proposed in this invention for video anomaly detection.

[0126] Therefore, the present invention adopts the above-mentioned video anomaly detection method based on SSIM and STE-3D, which can help the enhanced three-dimensional convolution-based video anomaly detection model capture more spatiotemporal features, thereby better reconstructing the normal mode, while the reconstruction effect of the abnormal mode is poor.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A video anomaly detection method based on SSIM and STE-3D, characterized in that: The following steps are involved: S1. Replace the mean square error (MSE) in the loss function and anomaly score calculation formula of the 3D convolution-based video anomaly detection model to be enhanced with the structural similarity index (SSIM). S2. Inserting the spatiotemporal enhancement block STE-3D at a specific position of the 3D convolution-based video anomaly detection model to be enhanced to obtain the enhanced video anomaly detection model; S3. Preprocess the video frames in the training set into small video clips and input them into the enhanced video anomaly detection model for training. The model input is denoted as I. S4: The enhanced video anomaly detection model is reconstructed based on the preprocessed small video clips to generate a small video clip, which is recorded as O. The loss value between I and O is calculated, and the model weight is updated after optimization by the optimizer until the training is completed. S5, prediction stage, calculates the reconstruction error between I and O, and then determines whether the current frame is an abnormal frame; Multiply the weight of the SSIM that replaces the MSE part by a specific coefficient , that is, the following formula (1) is upgraded to formula (2): （1）； （2）； in, Formula (1) and Formula (2) represent the parts of the loss function and the anomaly score calculation formula except MSE and 1-SSIM; In step S2, the spatiotemporal enhancement block STE-3D includes a 3D convolution with a kernel size of 1 and a stride of 1, a 3D convolution with a kernel size of 3 and a stride of 2, a 3D max pooling with a kernel size of 7, a 3D convolution with a kernel size of 3 and a stride of 1, a cubic linear interpolation upsampling module, a 3D convolution bypass with a kernel size of 1 and a stride of 1, a 3D convolution with a kernel size of 1 and a stride of 1, and a sigmoid activation function; Inserting the spatiotemporal enhancement block STE-3D into the three-dimensional convolution-based video anomaly detection model to be enhanced includes adding the spatiotemporal enhancement block STE-3D to the encoder of the three-dimensional convolution-based video anomaly detection model to be enhanced and adding the spatiotemporal enhancement block STE-3D to the decoder of the three-dimensional convolution-based video anomaly detection model to be enhanced.

2. The video anomaly detection method based on SSIM and STE-3D according to claim 1, characterized in that: In step S1, the structural similarity index SSIM is used for the given two images A and B, and its expression is as shown in formula (3): （3）； in, 、 and Represents brightness, contrast and structure respectively. 、 、 Control the relative importance of brightness, contrast and structural similarity in the SSIM index respectively. represents image A, Indicates a B picture; Brightness is the average brightness level of the pixel values ​​in the image, and its expression is as shown in formula (4): （4）； Contrast describes the brightness difference between adjacent pixels in an image, and its expression is as shown in formula (5): （5）； The structure measures the spatial arrangement of pixel patterns in an image, which is expressed as formula (6): （6）； The above formulas (4) to (6) involve Represent the pixel mean of images A and B respectively, represents the pixel variance of images A and B, represents the pixel covariance of images A and B, is a constant.

3. The video anomaly detection method based on SSIM and STE-3D according to claim 2, characterized in that: Using the Gaussian weighted formula, and using a size of 11×11 pixels and a standard deviation of The filter kernel is used to calculate the SSIM value between the original image and the reconstructed image, including the variance, mean and covariance, according to formula (4), formula (5) and formula (6); According to formula (7), the average value of the SSIM of all windows of the two images is calculated as the SSIM value of the two images: （7）； in, M is the number of windows into which the image is divided, Indicates the current window; make 、 , simplifying formula (7) to formula (8): （8）。 4. The video anomaly detection method based on SSIM and STE-3D according to claim 3, characterized in that: In step S3, the preprocessing operation includes converting all video frames in the training set into grayscale format and dividing them into small video clips to input into the enhanced video anomaly detection model.

5. The video anomaly detection method based on SSIM and STE-3D according to claim 4, characterized in that: In step S3, the 3D convolution-based video anomaly detection model to be enhanced extracts features from the video frame. When the data stream reaches STE-3D, STE-3D performs temporal and spatial enhancement on the features extracted from the video frame by the 3D convolution-based video anomaly detection model to be enhanced, and then re-inputs them into other parts of the 3D convolution-based video anomaly detection model to be enhanced.

Images