Video anomaly detection method based on SSIM and STE-3D

Abstract

The invention discloses a video anomaly detection method based on SSIM and STE-3D, and the method comprises the following steps: firstly replacing a loss function of a to-be-enhanced video anomaly detection model based on three-dimensional convolution and a mean square error MSE in an anomaly score calculation formula with a structural similarity index SSIM, and inserting a space-time enhancement block STE-3D, obtaining an enhanced video anomaly detection model; video frames in the training set are preprocessed into small video clips, the small video clips are input into the enhanced video anomaly detection model for training, and model input is marked as I; recording a small video clip generated by reconstruction according to the preprocessed small video clip as O, calculating a loss value between I and O, and updating the model weight until training is completed; and in the prediction stage, a reconstruction error between I and O is calculated, and whether the current frame is an abnormal frame is judged. According to the video anomaly detection method based on the SSIM and the STE-3D, indexes more suitable for evaluating video anomaly are provided, and it is ensured that the detected anomaly corresponds to human observation more accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a video anomaly detection method based on SSIM and STE-3D. Background Art

[0002] Before the emergence of deep learning, traditional video anomaly detection methods relied on manual features. For example, optical flow histogram, weighted statistics of optical flow, and motion boundary histogram treat the x and y directions of the optical flow image as two grayscale images, then extract the gradient histograms of these grayscale images and use these manual features to perform anomaly detection. Dense trajectory obtains the motion trajectory in the video through optical flow, and extracts features along the trajectory for anomaly detection.

[0003] The rise of deep learning has made supervised model training methods widely used in visual tasks. However, labeling abnormal patterns in videos is time-consuming and difficult, especially in the current scenario where surveillance cameras are already very popular. Therefore, video anomaly detection models usually follow the paradigm of only inputting videos with normal patterns in the training phase, and inputting videos with both normal and abnormal patterns in the testing phase. With the widespread application of deep learning in visual tasks, reconstruction-based video anomaly detection methods are becoming more and more popular. These methods usually use autoencoders and generative adversarial networks to extract feature representations in videos.

[0004] Reconstruction-based methods extract features from video frames, assuming that normal frames can be reconstructed more accurately than abnormal frames. However, due to the strong generalization ability of deep learning models, abnormal frames can also be effectively reconstructed. Therefore, these methods usually need to find some ways to make the model reconstruct normal frames better and reconstruct abnormal frames worse, so that the model can distinguish between normal and abnormal frames. For example, Gong et al. introduced a memory module into the bottleneck of the autoencoder, and the normal patterns extracted from normal video frames were stored in the memory module. The model queries similar vectors in the memory module based on the encoder output and filters out dissimilar vectors. Subsequently, the video frame is reconstructed using the queried vector. This process helps to expand the difference between the reconstructed abnormal frame and the original abnormal frame. On this basis, Park et al. proposed a memory module that occupies less video memory. Liu et al. also introduced PDM and ICM blocks to enhance the model's ability to recognize various normal patterns and unknown abnormal patterns. Marcella et al. used a method of skipping one or several frames to construct pseudo-anomalies from small fragments of continuous video frames to train the autoencoder, and then modeled the model in the opposite direction of these pseudo-anomalies. Prediction-based methods usually take continuous t-frame videos as input, then let the model predict the t+1 frame, and compare the error between the predicted frame and the original frame to determine whether the video frame is abnormal. Wang et al. proposed using a frame prediction network based on multi-path ConvGRU for modeling to better handle semantic information objects and regions of different scales and capture the spatiotemporal correlation in normal videos.

[0005] In the field of video anomaly detection, many methods mainly rely on mean squared error (MSE)-based losses or use MSE as the main component. However, MSE is calculated at the pixel level and has limited relevance to human perception. Since video anomaly detection aims to identify abnormal events and objects in videos, the detected anomalies must be consistent with human perception. The goal of models in the field of video anomaly detection is not to capture anomalies that are beyond or difficult to be perceived by the human visual system. Summary of the invention

[0006] The purpose of the present invention is to provide a video anomaly detection method based on SSIM and STE-3D. Since SSIM is more consistent with human perception, the structural similarity index (SSIM) is used as the basis of the loss function and the anomaly score calculation formula, which provides an indicator that is more suitable for evaluating video anomalies and ensures that the detected anomalies correspond more accurately to human observations.

[0007] To achieve the above object, the present invention provides a video anomaly detection method based on SSIM and STE-3D, comprising the following steps:

[0008] S1. Replace the mean square error MSE in the loss function and anomaly score calculation formula of the video anomaly detection model based on three-dimensional convolution to be enhanced with the structural similarity index SSIM;

[0009] S2, inserting the spatiotemporal enhancement block STE-3D at a specific position of the three-dimensional convolution-based video anomaly detection model to be enhanced to obtain an enhanced video anomaly detection model;

[0010] S3, preprocessing the video frames in the training set into small video clips and inputting them into the enhanced video anomaly detection model for training, and denoting the model input as I;

[0011] S4, reconstructing the small video clip generated by the enhanced video anomaly detection model according to the preprocessed small video clip as O, calculating the loss value between I and O, and updating the model weight after optimization by the optimizer until the training is completed;

[0012] S5, prediction stage, calculate the reconstruction error between I and O, and then determine whether the current frame is an abnormal frame.

[0013] Preferably, in step S1, the structural similarity index SSIM is used for two given images A and B, and its expression is as shown in formula (1):

[0014] SSIM(x,y)=(l(x,y)) α (c(x,y)) β (s(x,y)) γ (1);

[0015] Among them, l, c and s represent brightness, contrast and structure respectively, α, β and γ control the relative importance of the three components of brightness, contrast and structure similarity in the SSIM index respectively, x represents the A image, and y represents the B image;

[0016] Brightness is the average brightness level of the pixel values ​​in the image, and its expression is as shown in formula (2):

[0017]

[0018] Contrast describes the brightness difference between adjacent pixels in an image, and its expression is as shown in formula (3):

[0019]

[0020] The structure measures the spatial arrangement of pixel patterns in an image, and its expression is as follows:

[0021]

[0022] The μ involved in the above formulas (2) to (4) is A , μB Represent the pixel mean of images A and B respectively, represents the pixel variance of images A and B, σ AB represents the pixel covariance of images A and B, C 1 , C 2 , C 3 is a constant.

[0023] Preferably, a Gaussian weighted formula is used, and a filter kernel with a size of 11×11 pixels and a standard deviation of σ=1.5 is used to calculate the SSIM value between the original image and the reconstructed image according to formula (2), formula (3) and formula (4), including the variance, mean and covariance;

[0024] According to formula (5), the average value of SSIM of all windows of the two images is calculated as the SSIM value of the two images:

[0025]

[0026] Where M is the number of windows into which the image is divided, and i represents the current window;

[0027] Let α=β=γ=1, Simplify formula (5) to formula (6):

[0028]

[0029] The weight of the SSIM that replaces the MSE part is multiplied by a specific coefficient λ, that is, the following formula (7) is upgraded to formula (8):

[0030] loss = MSE + f(...) (7);

[0031] loss = λ(1-SSIM) + f(...) (8)

[0032] Wherein, f(...) in Formula (7) and Formula (8) represents the part of the loss function and the anomaly score calculation formula other than MSE and 1-SSIM.

[0033] Preferably, in step S2, the spatiotemporal enhancement block STE-3D includes a three-dimensional convolution with a kernel size of 1 and a stride of 1, a three-dimensional convolution with a kernel size of 3 and a stride of 2, a three-dimensional maximum pooling with a kernel size of 7, a three-dimensional convolution with a kernel size of 3 and a stride of 1, a cubic linear interpolation upsampling module, a three-dimensional convolution bypass with a kernel size of 1 and a stride of 1, a three-dimensional convolution with a kernel size of 1 and a stride of 1, and a sigmoid activation function;

[0034] Inserting the spatiotemporal enhancement block STE-3D into the three-dimensional convolution-based video anomaly detection model to be enhanced includes adding the spatiotemporal enhancement block STE-3D to the encoder of the three-dimensional convolution-based video anomaly detection model to be enhanced and adding the spatiotemporal enhancement block STE-3D to the decoder of the three-dimensional convolution-based video anomaly detection model to be enhanced.

[0035] Preferably, in step S3, the preprocessing operation includes converting all video frames in the training set into grayscale format, and dividing them into small video clips to input into the enhanced video anomaly detection model.

[0036] Preferably, in step S3, the features in the video frame are extracted by the three-dimensional convolution-based video anomaly detection model to be enhanced. When the data stream reaches STE-3D, STE-3D enhances the features in the video frame extracted from the three-dimensional convolution-based video anomaly detection model to be enhanced in terms of time and space, and then inputs them into other parts of the three-dimensional convolution-based video anomaly detection model to be enhanced again.

[0037] Therefore, the present invention adopts the above-mentioned video anomaly detection method based on SSIM and STE-3D, and the beneficial effects are as follows:

[0038] (1) The method proposed in the present invention includes inserting STE-3D and replacing MSE with SSIM. SSIM helps the training of the video anomaly detection model based on 3D convolution to be enhanced to develop in the direction related to human vision; in particular, the insertion of STE-3D can help the baseline capture more spatiotemporal features, thereby better reconstructing normal modes, while the reconstruction effect of abnormal modes is poor.

[0039] (2) Experimental results verify the effectiveness of the method proposed in the present invention in video anomaly detection. Numerous experimental results show that the method proposed in the present invention has the ability to significantly improve the accuracy in the video anomaly detection model.

[0040] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a process in which the original image and the reconstructed image of an embodiment of a video anomaly detection method based on SSIM and STE-3D of the present invention are divided into windows to calculate the mean SSIM, wherein (a) is a schematic diagram after the original image is divided into windows, and (b) is a schematic diagram of the reconstructed image being divided into windows.

[0042] Figure 2 It is a schematic diagram of the structure of a spatiotemporal enhancement block STE-3D in a three-dimensional convolution of an embodiment of a video anomaly detection method based on SSIM and STE-3D of the present invention;

[0043] Figure 3 It is a partial visualization feature map when the three-dimensional convolution-based video anomaly detection model to be enhanced in an embodiment of a video anomaly detection method based on SSIM and STE-3D of the present invention is STEAL Net;

[0044] Figure 4 It is an overall flow chart of an embodiment of a video anomaly detection method based on SSIM and STE-3D of the present invention;

[0045] Figure 5 It is the difference between the anomaly scores calculated by the model using MSE and SSIM as the loss function and the anomaly score calculation formula in the Ped1 test video of the comparative example 1 of the video anomaly detection method based on SSIM and STE-3D of the present invention, wherein the first column in (a) is the original image at a certain time point, the second column is the reconstructed image at a certain time point, the third column is the reconstruction error between the original image and the reconstructed image at a certain time point, and (b) is a line graph of the video anomaly score changing over time;

[0046] Figure 6 It is a visualized schematic diagram of weight parameters in STE-3D of comparative example 2 of a video anomaly detection method based on SSIM and STE-3D of the present invention. DETAILED DESCRIPTION

[0047] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0048] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0049] Analyze the current field of video anomaly detection from the perspective of loss function and anomaly score calculation formula

[0050] A key component of deep learning is the loss function used for model training. The loss function measures how close the model is to the expected output, and in deep learning tasks, choosing the right loss function is crucial to achieve good performance. However, due to the diversity of available methods, practitioners may face the challenge of choosing the most appropriate method. In the past few years, video anomaly detection usually uses mean squared error (MSE) as the main component of the model loss function. For example, Gong et al. used the sum of the MSE between the input video frame and the reconstructed video frame and the sum of the output cross entropy calculated by the memory module as the loss function, and only used the above MSE value in the anomaly score calculation. Yang et al. used the sum of Charbonnier loss (an enhanced version of MSE) and the adjacent frame difference (AFD) loss between the input video frame and the reconstructed video frame as the loss function, and used PSNR (a variant of MSE) as the main component of the anomaly score calculation formula. Liu et al. used the MSE between the input anomaly vector and the reconstructed anomaly vector as the model loss function, plus other losses. However, none of these loss functions and anomaly score calculation formulas have been able to effectively align with the results of human visual system evaluation.

[0051] One way to address the inadequacy of MSE is to incorporate image quality assessment algorithms that are more closely related to human vision into the core of the loss function and anomaly score calculation formula. Some image quality assessment algorithms show high correlation with human perception. The structural similarity index (SSIM) emphasizes structure-based image similarity, assuming that the human visual system excels in extracting structural information. In contrast, the feature similarity index (FSIM) prioritizes phase congruence (PC) and gradient magnitude (GM), assuming that the human visual system understands images primarily through low-level features.

[0052] Attention mechanisms have achieved remarkable success in various visual tasks in recent years, including image classification, object detection, semantic segmentation, self-supervised learning, etc. Attention mechanisms can be viewed as a process of dynamically adjusting weights based on input image features. Depending on the data domain, attention mechanisms can be divided into multiple types: channel, spatial, event, and branch. Hu et al. proposed the Squeeze and Excite Network (SENet), which demonstrated the interdependence between modeling feature channels and significantly improved the performance of image classification tasks. Woo et al. proposed the Convolutional Block Attention Module (CBAM), which demonstrated the dependency between the channel dimension and the spatial dimension. Ristea et al. proposed a Self-Supervised Predictive Convolutional Attention Block (SSPCAB), but the block they constructed was based on 2D convolution. Liu et al. proposed a Spatial Attention Enhancement Module ESA for image super-resolution.

[0053] The present invention proposes to use SSIM as the main component of the model loss function and the main body of the formula for calculating the anomaly score of the video frame. In this way, the model can learn structural features that are more consistent with abnormal performance judged by humans, rather than texture features that are insensitive to humans but sensitive to neural networks. In order to achieve higher-precision image reconstruction, the present invention further proposes a spatiotemporal enhancement block (STE-3D).

[0054] STE-3D is applicable to all three-dimensional convolutional neural networks because STE-3D can be inserted into the structure of any three-dimensional convolutional neural network, and the insertion position is usually arbitrary. This module can provide more useful features for subsequent neural network layers. In theory, as long as the structure of the neural network is three-dimensional convolutional, this module can be applied to any neural network field to enhance its ability to extract spatiotemporal features. The module of the present invention is similar to U-Net. The U-Net network is a commonly used backbone network in the field of video anomaly detection because it can extract normal patterns in the video while ignoring abnormal patterns. Similar to U-Net, the module of the present invention will also extract normal patterns in the video while ignoring abnormal patterns.

[0055] A video anomaly detection method based on SSIM and STE-3D is a general method based on a 3D convolutional model in the field of enhanced video anomaly detection, comprising the following steps:

[0056] S1. Replace the mean square error MSE in the loss function and anomaly score calculation formula of the video anomaly detection model based on three-dimensional convolution to be enhanced with the structural similarity index SSIM (taking the basic model AE as an example);

[0057] The structural similarity index SSIM is used for two given images A and B, and its expression is as shown in formula (1):

[0058] SSIM(x,y)=(l(x,y)) α (c(x,y)) β (s(x,y)) γ (1);

[0059] Among them, l, c and s represent brightness, contrast and structure respectively, α, β and γ control the relative importance of the three components of brightness, contrast and structure similarity in the SSIM index respectively, x represents image A and y represents image B.

[0060] Brightness is the average brightness level of pixel values ​​in an image, which directly affects the perception of light and dark in the image. The human visual system is very sensitive to changes in brightness, especially in areas where brightness changes sharply, which are usually where image details are of concern. Its expression is as shown in formula (2):

[0061]

[0062] Contrast describes the brightness difference between adjacent pixels in an image. The human visual system perceives the edges and textures in an image by identifying these contrast differences, which is crucial for understanding the content of an image. Its expression is shown in formula (3):

[0063]

[0064] The structure measures the spatial arrangement of pixel patterns in an image, representing its structural information. The human visual system tends to focus on these structural features because they are crucial for identifying and interpreting objects and events in visual scenes, as shown in formula (4):

[0065]

[0066] The μ involved in the above formulas (2) to (4) is A , μ B Represent the pixel mean of images A and B respectively, represents the pixel variance of images A and B, σ AB represents the pixel covariance of images A and B, C 1 , C 2 , C 3 is a constant.

[0067] In order to calculate the mean, variance and covariance within a window around a pixel, the present invention utilizes a Gaussian weighted version of the formula and adopts a filter kernel with a size of 11×11 pixels and a standard deviation of σ=1.5. According to formula (2), formula (3) and formula (4), the SSIM value between the original image and the reconstructed image, including the variance, mean and covariance, can be calculated.

[0068] However, these values ​​vary drastically throughout the image. Therefore, it is necessary to calculate the SSIM value within the window surrounding the pixel, and then calculate the average value of the SSIM of all windows of image A and image B as the SSIM value of the two images according to formula (5):

[0069]

[0070] Where M is the number of windows that the image can be divided into, and i represents the current window;

[0071] like Figure 1 The original image and the reconstructed image are shown as being divided into windows to calculate the mean SSIM. Figure 1It can be seen that when calculating the mean SSIM (MSSIM) of the two images, the original image and the reconstructed image are cut into windows of size k in the same way. The window corresponding to the original image and the window of the reconstructed image are used together to calculate the SSIM value of each window. Finally, the SSIM values ​​of all windows are averaged to obtain the MSSIM of the entire image.

[0072] Therefore, in order to simplify the SSIM calculation formula, let α=β=γ=1, Formula (5) can be simplified to formula (6):

[0073]

[0074] MSE compares the absolute error between two images, while the structural similarity index (SSIM) compares the relative error between two images. Therefore, when the value of the loss function MSE used for training is significantly different from the value of the loss function other than the MSE part, the weight of the SSIM replacing the MSE part should be multiplied by a specific coefficient λ, that is, the following formula (7) is upgraded to formula (8):

[0075] loss = MSE + f(...) (7);

[0076] loss=λ(1-SSIM)+f(...) (8);

[0077] Wherein, f(...) in Formula (7) and Formula (8) represents the part of the loss function and the anomaly score calculation formula other than MSE and 1-SSIM.

[0078] S2. Insert the spatiotemporal enhancement block STE-3D at a specific position of the three-dimensional convolution-based video anomaly detection model to be enhanced to obtain an enhanced video anomaly detection model.

[0079] Ristea et al. introduced the SSPCAB module, which significantly improved the performance of the video anomaly detection model. However, this module relies on two-dimensional convolution, while three-dimensional convolution is more effective in capturing the spatiotemporal characteristics of videos. In addition, SSIM has limitations in guiding the extraction of temporal information. Therefore, a more effective method is needed to capture the temporal dynamics in video frames.

[0080] In order to improve the accuracy of the model in detecting video anomalies, this paper proposes a spatiotemporal enhancement block (STE-3D), which is designed to be integrated into specific locations of the model to enhance the temporal and spatial representation of input features, so that subsequent network layers can better process these enhanced features. First, STE-3D needs to meet lightweight standards so that it can be inserted into various other 3D convolutional networks without incurring performance burden. Second, STE-3D needs a larger receptive field to better highlight the important (normal mode) parts of the features and suppress the less important (abnormal mode) parts.

[0081] When designing the STE-3D module, the present invention draws inspiration from the field of image super-resolution, such as Figure 2 The autoencoder on the left is the backbone network that the present invention will insert into STE-3D, the blue rectangle indicates the insertion position, and the STE-3D on the right indicates the internal structure of the block.

[0082] like Figure 2 As shown in Figure 3, the entire spatiotemporal enhancement block STE-3D includes a 3D convolution with a kernel size of 1 and a stride of 1, a 3D convolution with a kernel size of 3 and a stride of 2, a 3D maximum pooling with a kernel size of 7, a 3D convolution with a kernel size of 3 and a stride of 1, a cubic linear interpolation upsampling module, a 3D convolution bypass with a kernel size of 1 and a stride of 1, a 3D convolution with a kernel size of 1 and a stride of 1, and a sigmoid activation function.

[0083] Theoretically, STE-3D can be placed in any backbone network based on 3D convolution, and the effect will vary depending on its placement. Inserting the spatiotemporal enhancement block STE-3D into the 3D convolution-based video anomaly detection model to be enhanced includes adding the spatiotemporal enhancement block STE-3D to the encoder of the 3D convolution-based video anomaly detection model to be enhanced and adding the spatiotemporal enhancement block STE-3D to the decoder of the 3D convolution-based video anomaly detection model to be enhanced.

[0084] This embodiment chooses to insert the spatiotemporal enhancement block STE-3D into the baseline of STEAL Net, specifically adding the spatiotemporal enhancement block STE-3D to the encoder of STEAL Net and adding the spatiotemporal enhancement block STE-3D to the decoder of STEAL Net.

[0085] like Figure 3 As shown, the input and output images of the model are the first and last columns of the figure, respectively. The second and third columns show the feature map visualizations of the first convolutional layer of the encoder and the penultimate convolutional layer of the decoder before and after adding STE-3D.

[0086] Observed Figure 3It can be seen that: (1) Adding STE-3D to the encoder helps the model better capture and retain important features during downsampling. This helps to recover more detailed information in subsequent decoding. Since the encoder gradually reduces the spatial dimensions of the feature map, STE-3D helps the model focus on key information before the feature map becomes more abstract.

[0087] (2) Adding STE-3D to the decoder enhances the model’s ability to reconstruct details during upsampling. This helps locate regions of interest more accurately without anomalies, so the decoder is better able to reconstruct normal parts of video frames.

[0088] S3, preprocessing the video frames in the training set into small video clips and inputting them into the enhanced video anomaly detection model for training, and denoting the model input as I;

[0089] The preprocessing operation includes converting all video frames in the training set into grayscale format and segmenting them into small video clips containing 16 consecutive frames and inputting them into the enhanced video anomaly detection model.

[0090] The basic video anomaly detection model extracts features from video frames. When the data stream reaches STE-3D, STE-3D will enhance the features in the video frames extracted by the basic video anomaly detection model in terms of time and space, and then input them into other parts of the basic video anomaly detection model again.

[0091] S4, the output of the enhanced video anomaly detection model is recorded as O according to the preprocessed small video clip. In the training stage, the loss function between I and O is calculated according to formula (8), and the model weight is updated after optimization by the Adam optimizer until the training is completed;

[0092] S5. In the prediction stage, the reconstruction error between I and O is calculated, and then it is determined whether the current frame is an abnormal frame.

[0093] Example

[0094] Selecting the experimental dataset

[0095] Pedestrian Dataset Ped1: This dataset is made from videos from cameras monitoring pedestrian walkways at the University of California, San Diego. The main difficulty is that there are dense crowds of people, sometimes even congested, which obscures abnormal objects. The video resolution is 158×238 pixels, including 34 training videos and 36 test videos. Abnormal situations include non-walking human objects, such as carts, bicycles, etc., and abnormal pedestrian behaviors, such as running.

[0096] Pedestrian Dataset Ped2: This dataset is the same as Ped1, and also monitors pedestrian walkways at the University of California, San Diego, but the camera positions and shooting locations are different. This dataset is the most commonly used and simplest in video anomaly detection, with a video frame resolution of 240×360 pixels, including 16 training videos and 12 test videos.

[0097] CUHKAvenue dataset: This dataset is a single-scene video surveillance dataset. The main difficulty of the dataset is that the concrete columns in the video cause the target to be blocked and introduce human walking anomalies. The video frame resolution is 640×360 pixels, with a total of 16 training videos and 21 test videos.

[0098] Dataset introduced by ShanghaiTech University ShanghaiTech: This dataset is a video anomaly detection dataset proposed by ShanghaiTech University, which contains video surveillance of multiple scenes. This dataset is a larger and more challenging dataset in video anomaly detection because it contains more scenes and richer anomaly information. The video frame resolution is 846×480 pixels, including 330 training videos and 107 test videos.

[0099] The selected baseline, i.e. the 3D convolution-based video anomaly detection model to be enhanced

[0100] AE-Conv3D: Zhao et al. introduced a spatiotemporal autoencoder (AE) that uses deep neural networks to automatically learn video representations. Features are extracted from the spatial and temporal dimensions through 3D convolution. Since no open source code was found, the baseline in Mem-AE was used, which has almost the same structure as AE and performs slightly better than AE.

[0101] Mem-AE: Gong et al. proposed an improved memory-augmented autoencoder (Mem-AE) that first obtains encodings from the autoencoder and then queries to retrieve the most relevant content from the memory and sends it back to the autoencoder to reconstruct video frames containing only normal patterns.

[0102] STEAL Net: Astrid et al. proposed a temporal pseudo-anomaly synthesizer for training autoencoders (STEALNet). During the training phase, STEALNet constructs time-related pseudo-anomalies by skipping N frames. Subsequently, STEAL Net minimizes the loss associated with normal patterns instead of maximizing the reconstruction loss of abnormal patterns.

[0103] Evaluation indicators

[0104] Based on previous research, this paper adopts the area under the receiver operating characteristic curve (ROC) (AUC) at the video frame level as a performance metric to evaluate the accuracy of video anomaly detection models. This metric evaluates the ability of the model to detect anomalies by considering the anomaly score assigned to each frame in the test dataset. A higher AUC value indicates that the model is more effective in video anomaly detection.

[0105] When using the method proposed in the present invention, including inserting STE-3D and replacing MSE with SSIM. SSIM helps the baseline training to develop in the direction related to human vision. In particular, the insertion of STE-3D can help the baseline capture more spatiotemporal features, thereby better reconstructing normal modes, while the reconstruction effect of abnormal modes is poor. The experimental results in Table 1 verify the effectiveness of the method proposed in the present invention in video anomaly detection.

[0106] Table 1 Comparison of AUC performance on different baselines

[0107]

[0108] Table 1 shows the AUC performance comparison of the present invention on several baselines: Ped1, Ped2, Avenue and ShanghaiTech. The bold numbers represent the best results in the target dataset, while the underlined numbers represent the suboptimal results.

[0109] Numerous experimental results show that the method proposed in the present invention can significantly improve the accuracy of the video anomaly detection model.

[0110] Comparative Example 1:

[0111] Effect of using only SSIM: In theory, SSIM is better correlated with the human visual system than MSE. Therefore, it is expected that using SSIM can improve the performance of the model. In the experiment, only MSE was replaced with SSIM in the loss function and anomaly score calculation formula of the baseline model, and no other changes were made to the model.

[0112] Table 2 compares the AUC performance of the baseline model on Ped1, Ped2, Avenue, and ShanghaiTech with the baseline model after replacing the main SSIM into the loss function and anomaly score calculation formula. The red marks indicate performance degradation. The bold numbers indicate the best results on the target dataset.

[0113] The experimental results in Table 2 highlight the superiority of SSIM over MSE.

[0114] Table 2 Comparison of AUC performance of baseline models on different datasets

[0115]

[0116] Figure 5 (a) shows the difference between the anomaly scores calculated by the model using MSE and SSIM as the loss function and anomaly score calculation formula in the Ped1 test video. Specifically, the frame-level anomaly score comparison before and after replacing the MSE in the loss function and anomaly score calculation formula of the autoencoder with SSIM in the test video 8 of the Ped1 dataset (baseline and model of the present invention). Light blue represents the anomaly score curve using MSE, and light green represents the anomaly score curve using SSIM.

[0117] from Figure 5 As can be seen from the line graph in (b), the model trained with SSIM is more accurate in calculating the anomaly score. The input, reconstructed output, and the difference between the two are visualized. The visualization results show that the enhanced autoencoder of the present invention can better reconstruct the normal mode.

[0118] However, MSE tends to ignore high-level structural information that is prioritized by the human visual system. In contrast, SSIM focuses on structural similarity by comparing brightness, contrast, and structure between images, which is more consistent with human visual perception. Although SSIM captures perceptually important patterns, it may ignore subtle pixel-level differences that MSE is designed to penalize. This trade-off may lead to a slight decrease in accuracy in cases where pixel-level accuracy is more important than structural consistency, especially when subtle anomalies do not significantly change the overall structure of the video frame.

[0119] However, it is worth noting that this decrease is marginal and occurs less frequently than the accuracy improvement. The superior performance of SSIM on some datasets, especially the significant accuracy improvement observed for AE models on Ped2 and STEALNet on Ped1, suggests that SSIM's emphasis on structural consistency is beneficial in scenarios where anomalies are more structured. This suggests that SSIM provides a more perceptually meaningful metric in many anomaly detection tasks, improving model performance overall.

[0120] Comparative Example 2

[0121] The effect of using only STE-3D:

[0122] As shown in the experimental results in Table 3, the proposed STE-3D is effective in improving model accuracy. The AUC performance of the proposed method is compared with multiple baselines on Ped1, Ped2, Avenue, and ShanghaiTech. The bold numbers represent the best results in their respective target datasets. When testing the datasets of all selected models, STE-3D can improve model accuracy.

[0123] Table 3 AUC performance comparison with multiple baselines on different datasets

[0124]

[0125] from Figure 6 The heat map shows that the module is indeed able to capture the spatial and temporal features in the video frame, where cooler colors represent smaller weights and warmer colors represent larger weights. Figure 6 All of these prove the effectiveness of the STE-3D proposed in this invention for video anomaly detection.

[0126] Therefore, the present invention adopts the above-mentioned video anomaly detection method based on SSIM and STE-3D, which can help the enhanced three-dimensional convolution-based video anomaly detection model to capture more spatiotemporal features, thereby better reconstructing the normal mode, while the reconstruction effect of the abnormal mode is poor.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A video anomaly detection method based on SSIM and STE-3D, characterized in that: The following steps are involved: S1. Replace the mean square error MSE in the loss function and anomaly score calculation formula of the video anomaly detection model based on three-dimensional convolution to be enhanced with the structural similarity index SSIM; S2, inserting the spatiotemporal enhancement block STE-3D at a specific position of the three-dimensional convolution-based video anomaly detection model to be enhanced to obtain an enhanced video anomaly detection model; S3, preprocessing the video frames in the training set into small video clips and inputting them into the enhanced video anomaly detection model for training, and denoting the model input as I; S4, reconstructing the small video clip generated by the enhanced video anomaly detection model according to the preprocessed small video clip as O, calculating the loss value between I and O, and updating the model weight after optimization by the optimizer until the training is completed; S5, prediction stage, calculate the reconstruction error between I and O, and then determine whether the current frame is an abnormal frame.

2. The video anomaly detection method based on SSIM and STE-3D according to claim 1, characterized in that: In step S1, the structural similarity index SSIM is used for two given images A and B, and its expression is as shown in formula (1): SSIM(x,y)=(l(x,y)) α (c(x,y)) β (s(x,y)) γ (1): Among them, l, c and s represent brightness, contrast and structure respectively, α, β and γ control the relative importance of the three components of brightness, contrast and structure similarity in the SSIM index respectively, x represents the A image, and y represents the B image; Brightness is the average brightness level of the pixel values ​​in the image, and its expression is as shown in formula (2): Contrast describes the brightness difference between adjacent pixels in an image, and its expression is as shown in formula (3): The structure measures the spatial arrangement of pixel patterns in an image, and its expression is as follows: The μ involved in the above formulas (2) to (4) is A , μ B Represent the pixel mean of images A and B respectively, represents the pixel variance of images A and B, σ AB represents the pixel covariance of images A and B, C1, C2, C3 are constants.

3. The video anomaly detection method based on SSIM and STE-3D according to claim 2, characterized in that: Using the Gaussian weighted formula and a filter kernel with a size of 11 × 11 pixels and a standard deviation of σ = 1.5, the SSIM values ​​between the original image and the reconstructed image, including the variance, mean, and covariance, are calculated according to formula (2), formula (3), and formula (4); According to formula (5), the average value of SSIM of all windows of the two images is calculated as the SSIM value of the two images: Where M is the number of windows into which the image is divided, and i represents the current window; Let α=β=γ=1, Simplify formula (5) to formula (6): The weight of the SSIM that replaces the MSE part is multiplied by a specific coefficient λ, that is, the following formula (7) is upgraded to formula (8): loss = MSE + f(...) (7); loss=λ(1-SSIM)+f(...) (8); Wherein, f(...) in Formula (7) and Formula (8) represents the part of the loss function and the anomaly score calculation formula other than MSE and 1-SSIM.

4. The video anomaly detection method based on SSIM and STE-3D according to claim 3, characterized in that: In step S2, the spatiotemporal enhancement block STE-3D includes a 3D convolution with a kernel size of 1 and a stride of 1, a 3D convolution with a kernel size of 3 and a stride of 2, a 3D maximum pooling with a kernel size of 7, a 3D convolution with a kernel size of 3 and a stride of 1, a cubic linear interpolation upsampling module, a 3D convolution bypass with a kernel size of 1 and a stride of 1, a 3D convolution with a kernel size of 1 and a stride of 1, and a sigmoid activation function; Inserting the spatiotemporal enhancement block STE-3D into the three-dimensional convolution-based video anomaly detection model to be enhanced includes adding the spatiotemporal enhancement block STE-3D to the encoder of the three-dimensional convolution-based video anomaly detection model to be enhanced and adding the spatiotemporal enhancement block STE-3D to the decoder of the three-dimensional convolution-based video anomaly detection model to be enhanced.

5. The video anomaly detection method based on SSIM and STE-3D according to claim 4, characterized in that: In step S3, the preprocessing operation includes converting all video frames in the training set into grayscale format, and dividing them into small video clips to input into the enhanced video anomaly detection model.

6. The video anomaly detection method based on SSIM and STE-3D according to claim 5, characterized in that: In step S3, the features in the video frame are extracted by the 3D convolution-based video anomaly detection model to be enhanced. When the data stream reaches STE-3D, STE-3D enhances the features in the video frame extracted from the 3D convolution-based video anomaly detection model to be enhanced in terms of time and space, and then inputs them into other parts of the 3D convolution-based video anomaly detection model to be enhanced again.

Images