An algorithm for video abnormal event detection based on STMUnet

Abstract

The application discloses a video anomaly detection algorithm based on STMUnet. The application proposes an STMUnet network aiming at the strong correlation of time and space of the video anomaly detection algorithm based on frame prediction, the network utilizes the excellent reconstruction capability of the Unet network and combines the space-time modeling capability of the space-time conversion module of the application; in order to strengthen the reconstruction capability of the network to normal video frames and inhibit the reconstruction capability of the network to abnormal video frames, the application inserts a self-supervised mask convolutional Transformer variance block (Self-Supervised Masked Convolutional Transformer Variance Block for Anomaly Detection) after the penultimate upsampling of the original decoder D of the Unet network, the module uses a mask convolution to integrate local feature information into global feature information, and simultaneously fuses variance attention and Transformer channel attention, so as to strengthen the distinguishing capability of the network to abnormal frames. The video anomaly detection algorithm based on STMUnet has the characteristics of strong generalization and high precision.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition, specifically relating to a video anomaly event detection algorithm based on STMUnet. Background Technology

[0002] With the continuous development of surveillance and security technologies, video data is growing exponentially. In addressing the challenge of this massive amount of video data, manually screening for abnormal events in videos has become costly and inefficient. Therefore, there is an urgent need for an intelligent and efficient algorithm to automatically identify abnormal events in videos to improve the overall effectiveness of surveillance systems.

[0003] Currently, many video anomaly detection algorithms use the One Class Classification (OCC) method, which only trains on video frames that do not contain anomalies. OCC can be mainly divided into three types for video anomaly detection: reconstruction-based, prediction-based, and hybrid methods.

[0004] In reconstruction-based video anomaly detection methods, Gong et al. (D.Gong et al., "MemorizingNormality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection," 2019 IEEE / CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp.1705-1714, doi:10.1109 / ICCV.2019.00179.) proposed a memory-enhanced autoencoder (Mem-AE). The memory module is used to increase the reconstruction error of video anomaly frames, effectively enhancing the distinction between normal and abnormal video frames. However, this method does not consider the contextual relationship of the reconstructed video frames.

[0005] In prediction-based video anomaly detection methods, future frames are predicted by inputting the previous few frames. Yan et al. (C.Yan, S.Zhang, Y.Liu, G.Pang and W.Wang, "Feature Prediction Diffusion Model for Video Anomaly Detection," 2023 IEEE / CVF International Conference on Computer Vision (ICCV), Paris, France, 2023, pp.5504-5514, doi:10.1109 / ICCV51070.2023.00509.) proposed a video frame feature prediction method based on a diffusion model for video anomaly detection. Although this method considers the contextual relationship of video frames, it does not consider the relationship between local and global information in the feature map.

[0006] Among hybrid approaches, Chang et al. (Yunpeng Chang, Zhigang Tu, Wei Xie, BinLuo, Shifu Zhang, Haigang Sui, and Junsong Yuan, "Video anomaly detection with spatio-temporal dissociation," Pattern Recogn. 122, C (Feb 2022), doi: https: / / doi.org / 10.1016 / j.patcog.2021.108213) combined the advantages of reconstruction-based and prediction-based methods. They captured appearance information by reconstructing a single frame and captured motion information by predicting future frames. This method also used a variance attention mechanism to improve the performance of the autoencoder. However, this method uses a relatively singular attention mechanism and does not fully consider the relationship between feature map channels.

[0007] In summary, although OCC-based video anomaly detection methods have made some progress, in order to better meet real-world needs, a video anomaly detection algorithm that fully considers the contextual relationships of video frames, the role of attention mechanisms in video anomaly detection, and integrates local and global information is required. Summary of the Invention

[0008] The purpose of this invention is to provide a video anomaly event detection algorithm based on STMUnet. This invention strengthens the temporal and spatial information dependence of the network by reconstructing the encoder and decoder of the Unet network, thereby improving the ability to distinguish between video anomaly events and normal video events.

[0009] To achieve the above-mentioned invention, the present invention specifically includes the following steps:

[0010] Step 1: Collect normal and abnormal video clips and perform preprocessing;

[0011] Step 2: Construct the encoder for the STMUnet network;

[0012] Step 3: Construct the decoder for the STMUnet network;

[0013] Step 4: Train the STMUnet network using video frame sequences that do not contain anomalous events;

[0014] Step 5: Input the video frame sequence into the STMUnet network and determine whether the video frames are abnormal based on a given threshold.

[0015] Furthermore, the video anomaly event detection algorithm based on STMUnet is characterized in that, in step one, the acquisition and preprocessing of normal and abnormal video segments refers to acquiring 640×480 resolution video data, which includes 60 video segments, each with a duration of 1 minute, covering different scenes and situations. For video segments with anomalies, this invention will label each frame in the video segment to clearly indicate the start and end times of the anomaly. Specifically, for each video segment with anomalies, an array is used to label the video segment, where a value of 0 in the array indicates that the corresponding frame is a video frame without anomalies, and a value of 1 in the array indicates that the corresponding frame is a video frame with video anomalies.

[0016] Furthermore, the video anomaly event detection algorithm based on STMUnet is characterized in that, in step two, the encoder for constructing the STMUnet network refers to sequentially adding a pre-trained deep and broad convolutional neural network WR (WiderResNet) and a space-time conversion module STCM before the original encoder E of the Unet network. The space-time conversion module STCM has two parallel branches, namely a spatial conversion branch S and a temporal conversion branch T. The original encoder E, the pre-trained WR, the space-time conversion module STCM, and the encoder E together constitute the STMUnet encoder E. STM .

[0017] Furthermore, the aforementioned video anomaly detection algorithm based on STMUnet is characterized in that, in step three, constructing the STMUnet network decoder refers to inserting a self-supervised masked convolutional transformer variance block (SSMCTVB) for anomaly detection after the penultimate upsampling of the original decoder D of the Unet network. These two modules together constitute the STMUnet decoder D. STM .

[0018] Furthermore, the aforementioned STMUnet-based video anomaly event detection algorithm is characterized in that, in step four, training the STMUnet network using a video frame sequence that does not contain anomaly events refers to training the STMUnet network using a video frame sequence that does not contain anomaly events: the input to the STMUnet network is a sequence of frames in the video; given four consecutive frames {I1, I2, I3, I4}, the predicted frame is... The network is trained using a sequence of video frames that do not contain abnormal events, and the predicted frames are used. The gradient loss and Huber loss between the real frame I5 and the actual frame I5, as well as the Huber loss after the feature map passes through the masked convolution module, are used as the loss functions of STMUnet during training.

[0019] Furthermore, the video anomaly detection algorithm based on STMUnet is characterized in that, in step five, inputting a video frame sequence into the STMUnet network means determining whether a video frame is abnormal based on a given threshold, which involves using predicted frames. The gradient loss and Huber loss between the real frame i5 and the Huber loss after the feature map is processed by the masked convolution module are used to calculate the anomaly score. The anomaly score of each video segment Seq is processed by the maximum and minimum normalization, that is, converted to the range [0,1]. An anomaly threshold T is set. The predicted frame with an anomaly score greater than T is an anomaly frame. When there is at least one frame in the video segment Seq that is judged to be an anomaly frame, it means that there is an anomaly event in the video segment Seq. Attached Figure Description

[0020] Figure 1 The main steps of a video anomaly event detection algorithm based on STMUnet, provided as an example of the present invention, are as follows.

[0021] Figure 2This is a diagram of the STMUnet network structure as an example of the present invention.

[0022] Figure 3 This is a structural diagram of the time-space conversion module described in an example of the present invention.

[0023] Figure 4 This is a structural diagram of the SSMCCTVB module as described in an example of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0025] The application principle of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0026] like Figure 1 As shown, the present invention implements a video anomaly event detection algorithm based on STMUnet, which includes the following steps:

[0027] S101, Collect normal and abnormal video clips and perform preprocessing;

[0028] S102, the encoder for building the STMUnet network;

[0029] S103, the decoder for building the STMUnet network;

[0030] S104, Train the STMUnet network using video frame sequences that do not contain anomalous events;

[0031] S105: Input a video frame sequence into the STMUnet network and determine whether the video frame is abnormal based on a given threshold.

[0032] The step S101, which involves collecting normal and abnormal video segments and performing preprocessing, refers to collecting 640×480 resolution video data. The video data includes 60 video segments, each with a duration of 1 minute, covering different scenes and situations. For video segments with abnormal events, this invention will label each frame in the video segment to clearly indicate the start and end times of the abnormality. Specifically, for each video segment with abnormal events, an array is used to label the video segment. The value 0 in the array indicates that the corresponding frame is a video frame without abnormal events, and the value 1 in the array indicates that the corresponding frame is a video frame with abnormal video events.

[0033] The encoder for constructing the STMUnet network in step S102 refers to sequentially adding a pre-trained deep and wide convolutional neural network WR and a spatiotemporal transformation module STCM before the original encoder E of the Unet network. The spatiotemporal transformation module STCM has two parallel branches, namely a spatial transformation branch S and a temporal transformation branch T. The original encoder E, the pre-trained WR, the spatiotemporal transformation module STCM, and the encoder E together constitute the STMUnet encoder E. STM .

[0034] Step S103 describes constructing the STMUnet network decoder by inserting a self-supervised masked convolutional Transformer variance module (SSMCTVB) after upsampling the original decoder D of the Unet network. The SMCTVB consists of a masked full convolution, a Transformer, and a variance fusion attention module. The original decoder D and the SMCTVB together constitute the STMUnet decoder D. STM .

[0035] Step S104, which describes training the STMUnet network using a video frame sequence that does not contain anomalous events, refers to training the STMUnet network using a video frame sequence that does not contain anomalous events. The input to the STMUnet network is a sequence of frames from a video. Given four consecutive frames {I1, I2, I3, I4}, the predicted frame is... The network is trained using a sequence of video frames that do not contain abnormal events, and the predicted frames are used. The gradient loss and Huber loss between the real frame I5 and the Huber loss, as well as the Huber loss after the feature map passes through the masked convolution module, are used as the loss functions of STMUnet during training.

[0036] Step S105 involves inputting a video frame sequence into the STMUnet network and determining whether the video frames are abnormal based on a given threshold, using predicted frames. The anomaly score is calculated using the gradient loss and Huber loss between the real frame I5 and the feature map, as well as the Huber loss after the feature map passes through the masked convolution module. The anomaly score is then calculated for each video segment Seq.

[0037] The maximum and minimum normalization process converts the data to the range [0,1]. An anomaly threshold T is set, and the predicted frame with an anomaly score greater than T is considered an anomaly frame. When at least one frame in the video segment Seq is determined to be an anomaly frame, it means that there is an anomaly event in the video segment Seq.

[0038] Specific embodiments of the present invention:

[0039] The main body of the method of the present invention comprises five parts: 1) acquiring normal and abnormal video segments and preprocessing them; 2) constructing the encoder of the STMUnet network; 3) constructing the decoder of the STMUnet network; 4) training the STMUnet network using video frame sequences that do not contain abnormal events; 5) inputting the video frame sequences into the STMUnet network and judging whether the video frames are abnormal according to a given threshold.

[0040] 1. Acquire normal and abnormal video clips and perform preprocessing.

[0041] The system collects 640×480 resolution video data, which includes 60 video segments, each 1 minute long, covering different scenes and situations. For video segments with abnormal events, the invention tags each frame in the video segment to clearly indicate the start and end times of the abnormality. Specifically, for each video segment with an abnormal event, an array is used to tag the video segment, where a value of 0 indicates that the corresponding frame is a video frame without an abnormal event, and a value of 1 indicates that the corresponding frame is a video frame with a video abnormal event.

[0042] 2. Construct the encoder for the STMUnet network.

[0043] like Figure 2 The diagram shows the STMUnet network structure of this invention. The STMUnet encoder consists of a pre-trained WR, a spatiotemporal transformation module, convolutional layers, and downsampling. The WR and spatiotemporal transformation module refer to adding a pre-trained WR and a spatiotemporal transformation module (STCM) sequentially before the original encoder E of the Unet network. This example uses WiderResnet38 as the feature pre-extraction module. WiderResnet38 is used to extract feature maps F from four consecutive frames {I1, I2, I3, I4}, resulting in a set F = {f1, f2, f3, f4}. F is then input into the spatiotemporal transformation module (STCM), as shown below. Figure 3 The diagram shown is a structural diagram of the spatiotemporal conversion module described in this invention. The spatiotemporal conversion module STCM has two parallel branches, namely, a spatial conversion branch S and a temporal conversion branch T. The spatial conversion branch S takes the input feature map A = {a1, a2, a3, a4} and obtains a new spatial feature map f according to formula (1). S .

[0044]

[0045] Where f S ∈R C×H×W , This represents the first part after dividing a1 into four equal parts from high to low. This represents the second part after dividing a2 into four equal parts from high to low; This represents the third part after dividing a3 into four equal parts from high to low. This represents the fourth part after dividing a4 into four equal parts from high to low; Concat is a concatenation operation that concatenates the four tensors along a specified dimension. dim=1 indicates that the features are concatenated sequentially along the height direction, resulting in a spatial feature map f. S The time transformation branch T transforms the input feature map B = {b1, b2, b3, b4} into a new time feature map f according to formula (2). T .

[0046]

[0047] Where f T ∈R C×H×W , This represents the first part after dividing b1 into four equal parts from left to right; This represents the second part after dividing b2 into four equal parts from left to right; This represents the third part after dividing b3 into four equal parts from left to right; This represents the fourth part after dividing b4 into four equal parts from left to right; Concat is a concatenation operation that concatenates the four tensors along a specified dimension. dim=0 indicates that the features are concatenated sequentially from left to right, resulting in the temporal feature map f. T Finally, the vectors obtained from the time and space branches are vector-added according to formula (3).

[0048]

[0049] 3. Constructing a decoder for the STMUnet network

[0050] like Figure 2 As shown, the STMUnet decoder consists of convolutional layers, upsampling operations, and the SSMCVB module. To avoid gradient vanishing, each upsampling operation has a skip connection to its corresponding downsampling operation. Figure 4 The diagram shown illustrates the structure of the SSMCVB module as described in this invention. The SSMCVB module is inserted after the penultimate upsampling of the original decoder D in the Unet network. The SSMCVB consists of a masked full convolution, a Transformer, and a variance fusion attention module. The masked full convolution comprises eight sub-convolutions k1, k2, ..., k8, and the masked region is represented by M, where M ∈ R. C×1×1Eight sub-convolutions are performed on the input feature map, and then the sum of the values ​​from each sub-convolution is used to replace the value of M to obtain a new feature map F. M The Transformer and variance fusion attention module will convert the result F after mask convolution into a variable. M As input, firstly, for F M Perform variance attention calculation to obtain the variance attention feature map att, att∈R H×W Simultaneously, Transformer channel attention is calculated on FM to obtain channel attention feature maps T, where T∈R. C×1×1 The variance attention feature map and the Transformer channel attention feature map are fused according to formula (4).

[0051] F TV (c,i,j)=T(c,1,1)×att(i,j) (4)

[0052] Where F TV ∈R C×H×W For the result of fusion, F TV In (c,i,j) and T(c,1,1), c is the number of channels, i and j are the spatial index positions of channel c, and finally the final output of SSMCCTVB is obtained according to formula (5).

[0053]

[0054] Where ⊙ represents matrix multiplication.

[0055] 4. Train the STMUnet network using video frame sequences that do not contain anomalous events.

[0056] Training the STMUnet network using a sequence of video frames that do not contain anomalous events: The input to the STMUnet network is a sequence of frames from a video. Given four consecutive frames {I1, I2, I3, I4}, the predicted frame is... The network is trained using a sequence of video frames that do not contain abnormal events, and the predicted frames are used. The gradient loss and Huber loss between the real frame I5 and the feature map after passing through the mask convolution module are used as the loss function of STMUnet during training. Specifically, the gradient loss is measured by comparing the gradient difference between the reconstructed future frame and the real future frame, and the gradient loss is calculated according to formula (6).

[0057]

[0058] in, Let x represent the predicted future frame, i and j represent the spatial index of the video frame. Furthermore, the Huber loss is calculated according to formula (7).

[0059]

[0060] in δ represents the predicted future frame, x is the actual future frame, and δ is a threshold parameter. When the difference between the actual value and the predicted value is less than or equal to δ, the squared loss is used, and when the difference is greater than δ, the absolute loss is used. Furthermore, the Huber loss is also calculated for the SSMCCTVB module according to formula (8).

[0061]

[0062] Where F SSMCTVB For the input of the SSMCCTVB module, As the output of the SSMCVB module, combining formulas (6) to (8), the training loss function of the STMUnet network is shown in formula (9):

[0063]

[0064] Where λ gd , λ δ , λ δ,SSMCTVB It is a balancing hyperparameter.

[0065] The Adam optimizer was used during training, with a decay rate of 0.9 for the first moment estimate, 0.999 for the second moment estimate, 60 epochs, a batch size of 4 training data points, a learning rate of 0.0001, and λ... gd , λ δ , λ δ,SSMCTVB The values ​​were set to 0.3, 0.3, and 0.4 respectively, and the graphics card used was an NVIDIA GeForce RTX 3090.

[0066] 5. Input a video frame sequence into the STMUnet network and determine whether the video frames are abnormal based on a given threshold.

[0067] Use prediction frames The anomaly score is calculated using the gradient loss and Huber loss between the real frame I5 and the Huber loss after the feature map passes through the mask convolution module. Specifically, the anomaly score of the predicted frame is calculated according to formula (10).

[0068]

[0069] Where 0 < α + β ≤ 1, H(t) represents the max-min normalization operation, which transforms the outlier scores to the range [0, 1] according to formula (11).

[0070]

[0071] During the test, α was set to 0.3, β was set to 0.4, and the anomaly threshold T was set to 0.8. Predicted frames with an anomaly score greater than T were considered anomalous frames. When at least one frame in the video segment Seq was determined to be an anomalous frame, it meant that there was an anomalous event in the video segment Seq.

[0072] Compared with the prior art, the present invention has the following beneficial effects:

[0073] (1) This invention provides a spatiotemporal transformation module that enhances the contextual dependency of anomalous events, enabling the network to improve its ability to distinguish anomalous events in videos. This invention adds a pre-trained WR and a spatiotemporal transformation module STCM sequentially before the original encoder E of the Unet network. The WR improves feature extraction by increasing the network width and using residual connections, giving the network stronger representation and generalization capabilities. The two parallel branches of the spatiotemporal transformation module STCM enhance the temporal and spatial dependencies between consecutive video frames, thereby strengthening the network's ability to distinguish anomalous events in videos.

[0074] (2) This invention provides an improved self-supervised masked convolution Transformer variance module SSMTVB. SSMTVB performs convolution operations on the input feature map through 8 sub-convolutions. Then, the sum of the values ​​obtained from each sub-convolution is used to replace the value of the intermediate mask to integrate local information into global information. This helps the model to understand the input data more comprehensively, capture global context information better, improve the model's sensitivity to details and local features, and enhance the network's ability to identify abnormal events. At the same time, it integrates variance attention and Transformer channel attention, enhances the model's global and local attention to abnormal events, and improves the model's performance in complex scenarios.

Claims

1. A video anomaly event detection method based on STMUnet, characterized in that, Step 1: Acquire and preprocess normal and abnormal video clips: Acquire 60 video clips at a resolution of 640×480, each 1 minute long, covering different scenes and situations. For video clips with abnormal events, label each frame to indicate the start and end times of the abnormality, using 0 to represent a normal frame and 1 to represent an abnormal frame. Step 2: Construct the encoder of the STMUnet network. Step 3: Construct the decoder of the STMUnet network. Step 4: Train the STMUnet network using video frame sequences that do not contain abnormal events: The input to the STMUnet network is four consecutive frames. The predicted frame is During training, gradient difference loss, Huber loss, and the Huber loss of the masked convolution module are used as loss functions; Step 5: Input the video frame sequence into the STMUnet network, and determine whether the video frame is abnormal based on a given threshold: this refers to using predicted frames. and real frames The gradient loss, Huber loss, and Huber loss after the feature map passes through the masked convolution module are used to calculate the anomaly score. The anomaly score of each video segment (Seq) is then subjected to min-max normalization, converting it to the range [0,1]. An anomaly threshold is then set. T abnormal score greater than T The predicted frame is the anomalous frame. When at least one frame in the video segment Seq is determined to be an anomalous frame, it means that there is an anomalous event in the video segment Seq. The encoder of the STMUnet network mentioned in step two refers to adding a pre-trained WR and a spatiotemporal transformation module STCM sequentially before the original encoder E of the Unet network. The WR is used to extract four consecutive frames. Feature map ,Will The input is fed into the space-time transformation module STCM, which has two parallel branches: a spatial transformation branch S and a temporal transformation branch T. The spatial transformation branch S processes the input feature map. The new spatial feature map is obtained according to formula (1). , ， in Indicates to The first part after dividing the material into four equal parts from high to low; Indicates to The second part after dividing the material into four equal parts from high to low; Indicates to The third part after dividing the material into four equal parts from high to low; Indicates to The fourth part after dividing the material into four equal parts from high to low; This is a concatenation operation that concatenates four tensors along a specified dimension. dim=1 indicates that features are concatenated sequentially along the height direction, resulting in a spatial feature map. The time transformation branch T will input the feature map. The new time feature map is obtained according to formula (2). , ， in Indicates to The first part after dividing the material into four equal parts from left to right; Indicates to Indicates to The third part after dividing the material into four equal parts from left to right; Indicates to The fourth part after dividing the material into four equal parts from left to right; This is a concatenation operation that concatenates four tensors along a specified dimension. `dim=0` indicates that the features are concatenated sequentially from left to right, resulting in a temporal feature map. Finally, the vectors obtained from the time and space branches are added together according to formula (3). ’ Received The feature map obtained after vector addition is used as input to the subsequent Unet network.

2. The video anomaly event detection method based on STMUnet according to claim 1, characterized in that, Step 3, constructing the decoder for the STMUnet network, refers to inserting a self-supervised masked convolutional Transformer variance module (SSMCTVB) after the penultimate upsampling of the original decoder D of the Unet network. The SSMCCTVB consists of a masked full convolution, a Transformer, and a variance fusion attention module. The masked full convolution consists of eight sub-convolutions k1, k2, ..., k8, and the masked region is represented by M. Eight sub-convolutions are performed on the input feature map, and then the sum of the values ​​from each sub-convolution is used to replace the value of M to obtain a new feature map F. M The Transformer and variance fusion attention module will convert the result F after mask convolution into a variable. M As input, firstly, for F M Perform variance attention calculation to obtain the variance attention feature map. , Meanwhile, for F M Perform Transformer channel attention calculation to obtain channel attention feature maps. , The variance attention feature map and the Transformer channel attention feature map are fused according to formula (4). ， in The result after fusion c is the number of channels. i , j Let c be the spatial index position of channel c. Finally, the final output of SSMCCTVB is obtained according to formula (5). ， in This represents matrix multiplication.

Images