Video anomaly detection method based on multi-task learning

Abstract

The invention discloses a video anomaly detection method based on multi-task learning, a model comprises a space-time encoder, a prototype memory network, a double-decoder architecture and a reconstruction decoder, and the method comprises the following steps: carrying out hierarchical feature extraction on a training video frame sequence by using the space-time encoder; performing memory enhancement processing on the spatial-temporal characteristics through a prototype memory network; respectively carrying out future frame prediction and current frame reconstruction through a double-decoder architecture; calculating a multi-task loss function, and updating parameters of the space-time encoder, the prototype memory network and the double decoders through back propagation according to the multi-task loss function; according to the method, a more comprehensive self-supervised learning framework is constructed, and the generalization ability of the model is enhanced.

Description

Technical Field

[0001] This invention relates to a video anomaly detection model based on multi-task learning, and also to a memory enhancement method based on multi-task learning for video anomaly detection, belonging to the field of image processing technology. Background Technology

[0002] Video anomaly detection is a core technology in computer vision and intelligent surveillance, designed to automatically identify events in videos that do not conform to normal behavioral patterns, such as traffic accidents, violent acts, and trespassing. With the widespread deployment of surveillance cameras in public places, automated anomaly detection technology has significant application value in public safety, traffic management, and industrial monitoring.

[0003] Existing video anomaly detection methods are mainly divided into two categories: weakly supervised methods and unsupervised methods. Weakly supervised methods rely on video-level label information, and although they perform well in certain scenarios, they are limited by the scarcity and high cost of labeled data. Therefore, unsupervised methods have become a research hotspot, as they are trained using only normal samples and identify anomalous events that deviate from the normal pattern by modeling normal patterns.

[0004] Typical unsupervised methods include reconstruction-based methods, such as autoencoders, which learn to reconstruct normal samples but produce large reconstruction errors for anomalous samples; prediction-based methods, which use temporal models to predict future frames, but anomalous samples can increase prediction errors; and memory-based methods, such as memory-enhanced autoencoders, which use prototype memory networks to store normal pattern prototypes, limiting the model's ability to reconstruct anomalous samples.

[0005] In recent years, the Transformer architecture has been introduced into video anomaly detection tasks due to its powerful sequence modeling capabilities. For example, methods such as TransAnomaly and HSTforU utilize VisionTransformer or its variants for spatiotemporal feature modeling, achieving some progress. However, these methods still have the following problems: 1. Transformer overgeneralization: It still has a strong ability to represent anomalous samples, leading to an increased false positive rate for normal samples; 2. Task singularity: Most methods only adopt one task, reconstruction or prediction, failing to fully utilize the complementarity of spatiotemporal features; 3. Insufficient constraint mechanism: There is a lack of effective mechanisms to limit the model's ability to generalize to anomalous samples; 4. Insufficient feature utilization: It fails to effectively combine original features with memory-enhanced features, resulting in limited detection capabilities for complex anomaly types. Summary of the Invention

[0006] The purpose of this invention is to overcome the problems existing in the prior art and provide a video anomaly detection method based on multi-task learning, which improves the detection accuracy and robustness of the original Transformer architecture.

[0007] To address the above technical problems, this invention provides a video anomaly detection method based on multi-task learning, wherein the anomaly detection model includes: Spatiotemporal Transformer encoder: Extracts hierarchical spatiotemporal features from the input video frame sequence; Prototype memory network: Connected to the encoder, used to store normal pattern prototypes and enhance the memory of encoded features through similarity matching; Dual-decoder architecture: connecting the encoder and the prototype memory network, including a predictive decoder and a reconstruction decoder; The prediction decoder performs a future frame prediction task based on the original temporal features output by the encoder. The reconstruction decoder performs the current frame reconstruction task based on the original temporal features and the memory enhancement features output by the prototype memory network.

[0008] Furthermore, the spatiotemporal Transformer encoder includes a spatial encoder for processing single-frame images to extract spatial features and a temporal encoder for processing continuous frame sequences to extract temporal features.

[0009] Furthermore, the spatial encoder includes a patch embedding module, which patches N consecutive video frame sequences. ; The processing steps of the spatial encoder are as follows: S1.1.1 Patch Embedding: For each input frame image The image is divided into P non-overlapping image blocks of size p×p and embedded. S1.1.2: Add a learnable spatial classification token to each patch sequence. And positional encoding: the initial sequence Input to a by In a module consisting of stacked identical spatial Transformer coding layers; S1.1.3: Spatial Feature Vectors pass Each spatial Transformer layer handles the capture of intra-frame spatial dependencies.

[0010] Furthermore, the processing steps of the time encoder are as follows: S1.2.1: Collect spatial CLS tokens of all video frames, arrange them in chronological order, and construct the initial time series. ; S1.2.2: Add a time classification token at the beginning of the time series. It aggregates global context information of the entire segment in the time dimension; and generates a location encoding vector for each position in the time series. Finally, an initial representation of the time series that incorporates temporal location information is obtained. ; S1.2.3: Initial time series Input to a by In a module composed of stacked time Transformer encoding layers, the time series is obtained through... A time-varying Transformer layer models the inter-frame temporal dynamics.

[0011] Furthermore, the prototype memory network maintains a learnable memory bank containing M prototype features. Each prototype This indicates a normal mode for a feature; The specific steps for memory enhancement using the prototype memory network are as follows: S2.1: Determine input features through memory addressing Which or more normal pattern prototypes in the memory bank are most similar to; S2.2: During training, the memory bank is updated using an exponential moving average. In the formula It's the update rate. It is an indicator function that selects the prototype with the highest attention weight; S2.3: By introducing a memory shrinkage loss that combines entropy regularization and diversity constraints, we encourage the use of sparse memory and prevent over-smoothing.

[0012] Furthermore, the specific steps in S2.1 are as follows: S2.1.1: Calculate the similarity-based attention weights between encoded temporal features and stored prototypes; S2.1.2: Given coding time characteristics Attention weights are computed using softmax normalization with temperature scaling. S2.1.3: Perform memory retrieval based on the addressing result, read relevant normal mode information from the memory bank, and calculate the retrieved memory representation as a weighted combination of all prototypes.

[0013] Furthermore, the dual-decoder architecture utilizes the original temporal feature f and the memory-enhanced feature... Perform complementary tasks, where the predictive decoder directly processes the raw temporal features f through an upsampling layer to predict the next frame. The reconstruction decoder reconstructs the input frame by concatenating and combining original and memory-enhanced features. This allows the reconstruction task to benefit from memory-constrained representations, while the prediction task retains direct access to temporal dynamics.

[0014] Furthermore, during the model training phase, the loss functions of multiple tasks are integrated into a total training loss through weighted summation to coordinate the optimization process and balance the contribution of each task to the model parameters. The loss includes reconstruction loss and future loss.

[0015] This invention also relates to a video anomaly detection method based on multi-task learning, comprising a model training phase and an anomaly detection phase, wherein the model training phase includes the following steps: S1: Use a spatiotemporal Transformer encoder to perform hierarchical feature extraction on the training video frame sequence to obtain spatiotemporal features; S2: The spatiotemporal features are enhanced through a prototype memory network; S3: Future frame prediction and current frame reconstruction are performed separately using a dual-decoder architecture; S4: Calculate the multi-task loss function, which includes prediction loss, reconstruction loss and memory shrinkage loss; S5: Update the parameters of the spatiotemporal Transformer encoder, prototype memory network, and dual decoder through backpropagation according to the multi-task loss function; The anomaly detection phase includes the following steps: 1) Obtain the video frame sequence to be detected; 2) Use a spatiotemporal Transformer encoder to extract spatiotemporal features from the video frame sequence; 3) Obtain attention weights by performing memory addressing on the spatiotemporal features through a prototype memory network; 4) Use dual decoders to output future frames and reconstructed frames separately; 5) Calculate the comprehensive anomaly score based on the future frame, the reconstructed frame, and the attention weights; 6) Determine whether there are any abnormal events in the video based on the comprehensive anomaly score.

[0016] Compared with existing technologies, this invention achieves the following beneficial effects: 1. For the first time, it deeply integrates the prototype memory network with the Transformer model architecture, effectively constraining the model's overgeneralization of anomalous samples by storing normal mode prototypes; 2. It innovatively designs a dual decoder for prediction and reconstruction, making full use of the complementarity of original temporal features and memory-enhanced features to achieve comprehensive detection of different types of anomalies; 3. It organically combines future frame prediction and current frame reconstruction tasks to build a more comprehensive self-supervised learning framework, enhancing the model's generalization ability. Attached Figure Description

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The drawings are provided for reference and illustration only and are not intended to limit the present invention.

[0018] Figure 1 This is a diagram illustrating the overall architecture of the Transformer model in this invention. Figure 2 This is a flowchart of the Transformer model in this invention; Figure 3 This is a flowchart of the training phase of a memory enhancement method model based on multi-task learning for video anomaly detection. Detailed Implementation

[0019] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific illustrations.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the invention.

[0021] This invention aims to provide a video anomaly detection method based on multi-task learning, such as... Figure 1 As shown, the video anomaly detection model includes: Spatiotemporal Transformer encoder: Extracts hierarchical spatiotemporal features from input video frame sequences; the spatiotemporal Transformer encoder includes a spatial encoder that processes single-frame images to extract spatial features and a temporal encoder that processes continuous frame sequences to extract temporal features; Prototype Memory Network: Connected to the encoder, it stores normal pattern prototypes and enhances the memory of encoded features through similarity matching; The prototype memory network is a memory network that stores representative features of normal patterns; The prototype memory network maintains a learnable memory containing M prototype features. Each prototype This represents a normal mode for a feature. The size M of the memory bank includes 512 prototype features. The learning rate of the memory bank is scheduled with an initial learning rate of 5×10^-4 and cosine annealing decay. Dual-decoder architecture: connects the encoder and the prototype memory network, including a prediction decoder and a reconstruction decoder; the dual-decoder is a parallel decoder architecture that performs prediction and reconstruction tasks separately. The predictive decoder performs the task of predicting future frames based on the raw temporal features of the encoder output; The reconstruction decoder performs the current frame reconstruction task based on the original temporal features and the memory enhancement features output by the prototype memory network.

[0022] The dual-decoder architecture utilizes the original temporal feature f and memory-enhanced features. Perform complementary tasks, where the predictive decoder directly processes the raw temporal features f through an upsampling layer to predict the next frame: The reconstruction decoder reconstructs the input frame by concatenating and combining original and memory-enhanced features: ,in The representation of feature concatenation allows the reconstruction task to benefit from memory-constrained representations, while the prediction task retains direct access to temporal dynamics.

[0023] The Spatiotemporal Transformer encoder employs a hierarchical structure, processing video sequences sequentially through a spatial encoder and a temporal encoder.

[0024] The spatial encoder includes a patch embedding module, which embeds N consecutive video frame sequences. Processing is carried out, among which ; The processing steps of the spatial encoder are as follows: S1.1.1 Patch Embedding: For each input frame image The image is divided into P non-overlapping image blocks of size p×p and embedded as follows: , in, Indicates the number of patches. , For patch size, For the embedded dimension; S1.1.2: Add a learnable spatial classification token to each patch sequence. And positional encoding: the initial sequence Input to a by In a module consisting of stacked identical spatial Transformer coding layers: ,in This indicates a splicing operation. Indicates spatial location encoding; S1.1.3: Spatial features through A spatial Transformer layer is used to capture intra-frame spatial dependencies: In the formula, These are spatial feature vectors.

[0025] The processing steps of a time encoder are as follows: S1.2.1: Collect spatial CLS tokens of all video frames, arrange them in chronological order, and construct the initial time series. N is the number of frames in the input video segment, and each vector in this sequence represents the global spatial content of a frame of image; S1.2.2: Add a time classification token at the beginning of the time series. It aggregates global context information of the entire segment in the time dimension; and generates a location encoding vector for each position in the time series. Finally, an initial representation of the time series that incorporates temporal location information is obtained. ; S1.2.3: Initial time series Input to a by In a module composed of stacked time Transformer encoding layers, the time series is obtained through... Modeling inter-frame temporal dynamics using a Transformer layer at different times: In the formula, For the first A time series.

[0026] The specific steps for memory enhancement using prototype memory networks are as follows: S2.1: Determine input features through memory addressing Which or more normal pattern prototypes in the memory bank are most similar to; The specific steps for S2.1 are as follows: S2.1.1: Calculate the similarity-based attention weights between encoded temporal features and stored prototypes; S2.1.2: Given coding time characteristics Attention weights are computed using softmax normalization with temperature scaling. ,in It is a temperature parameter that controls the sharpness of attention distribution; S2.1.3: Based on the addressing result, perform memory retrieval, read relevant normal mode information from the memory bank, and calculate the retrieved memory representation as a weighted combination of all prototypes: In the formula, For weighted time features, The normal prototype is characterized by... S2.2: During training, the memory bank is updated using an exponential moving average. In the formula It's the update rate. It is an indicator function that selects the prototype with the highest attention weight; S2.3: To encourage the use of sparse memory and prevent over-smoothing, a memory shrinkage loss combining entropy regularization and diversity constraints is introduced: ,in Encourage sparse attention distribution. Enhancing the diversity of prototype usage constraints These are the weighting coefficients.

[0027] During the model training phase, the loss functions of multiple tasks are integrated into a single total training loss through weighted summation to coordinate the optimization process and balance the contributions of each task to the model parameters. In the formula, =1.0、 =0.8 and =0.3 is the weighting coefficient that balances the different loss components. The prediction loss and reconstruction loss in the formula are calculated using the L2 norm between the future frame or reconstructed frame and the ground truth frame. Prediction loss: Reconstruction loss: ,in Indicates the input frame for reconstruction .

[0028] The Transformer model of this invention will also perform the following operations during the decoding process, based on the normal operating mode determined during the encoding process: First, calculate the peak signal-to-noise ratio (PSNR) for both the prediction and reconstruction tasks: ,in It represents the maximum pixel value, and MSE represents the mean square error. Then, the concentration of attention weights is measured using memory entropy: By combining the intermediate outlier score with the normalized PSNR value and memory entropy, the intermediate outlier score... ,in =0.4 is the control coefficient for the contribution of memory entropy to the overall outlier score. For memory entropy, That is, taking the logarithm of M, which is used to normalize the memory entropy. ; The final outlier scores are normalized to the range [0,1]: In the formula These are outlier scores in the middle. The minimum abnormal score, The maximum outlier score.

[0029] This invention also relates to a video anomaly detection method based on multi-task learning, including a model training phase and an anomaly detection phase, such as... Figure 2 As shown, the model training phase includes the following steps: S1: Use a spatiotemporal Transformer encoder to extract hierarchical features from the training video frame sequence to obtain spatiotemporal features; decompose the massive video data into manageable spatiotemporal units through a hierarchical space-time processing strategy, enabling the model to efficiently process long sequence video data.

[0030] S2: Spatiotemporal features are enhanced through a prototype memory network. Because the memory can dynamically learn and compress the essential features of normal patterns, a compact and representative memory containing only normal prototypes is formed. This allows the framework to quickly adapt to new monitoring scenarios, remaining robust to unseen normal behavior while maintaining high detection sensitivity to unknown types of anomalies.

[0031] S3: A dual-decoder architecture is used to perform future frame prediction and current frame reconstruction separately. By performing both tasks simultaneously, it can cover a variety of anomaly types. Whether it is an appearance anomaly or a motion anomaly, at least one decoder can generate a significant detection signal, thereby comprehensively improving the overall detection capability for different anomaly types.

[0032] S4: Calculate the multi-task loss function, which includes prediction loss, reconstruction loss, and memory shrinkage loss; S5: Update the parameters of the spatiotemporal Transformer encoder, prototype memory network, and dual decoder through backpropagation based on the multi-task loss function.

[0033] Through a memory-enhanced prototype memory network, a spatiotemporal Transformer encoder, and a dual-decoder collaborative mechanism, the framework can more accurately identify and locate anomalous events in videos. It learns more deeply from normal patterns, thus being more sensitive to anomalies that deviate from them, thereby maintaining high detection accuracy and a low false alarm rate even in complex scenes.

[0034] like Figure 3 As shown, the anomaly detection phase includes the following steps: 1) Obtain the video frame sequence to be detected; 2) Use a spatiotemporal Transformer encoder to extract spatiotemporal features from the video frame sequence; 3) Obtain attention weights by memory addressing spatiotemporal features through a prototype memory network; 4) Use dual decoders to output future frames and reconstructed frames separately; 5) Calculate the comprehensive anomaly score based on future frames, reconstructed frames, and attention weights; 6) Determine whether there are any abnormal events in the video based on the comprehensive anomaly score.

[0035] Because the Transformer model of this invention separates and collaborates in learning normal patterns and detecting anomalies, the model no longer needs to learn what anomalies are, thus avoiding the problem of overgeneralization of anomaly samples due to the unpredictability of anomalies.

[0036] The overall architecture and training algorithm of the model ensure the coordinated operation of the three core components: the memory encoder, the prototype memory, and the dual decoders. The carefully designed multi-task loss function, through a weight balancing strategy, guides the model to train stably and converge to the optimal solution, ultimately forming a powerful, reliable, and easy-to-train complete system.

[0037] Experiments show that the model proposed in this invention achieves frame-level AUC of 97.7% and 88.9% on the UCSD Ped2 and CUHK Avenue datasets, respectively, which is 0.5% and 1.1% higher than the existing best methods, respectively. This verifies the effectiveness of the memory-constrained learning mechanism and can be widely applied in the environmental anomaly recognition of smart lawnmowers or other drones.

[0038] The above are merely preferred embodiments of the present invention, illustrating and describing the basic principles, main features, and advantages of the invention. They are not intended to limit the scope of patent protection of the invention. Those skilled in the art should understand that the invention is not limited to the above embodiments. Besides the above embodiments, the invention can have other implementations without departing from its spirit and scope. Various changes and modifications to the invention are possible, and all technical solutions formed by equivalent substitutions or transformations fall within the scope of protection claimed by the invention. The scope of protection of the invention is defined by the appended claims and their equivalents. Technical features not described in the invention can be implemented using existing technology and will not be elaborated upon here.

Claims

1. A video anomaly detection method based on multi-task learning, characterized in that, The anomaly detection model includes: Spatiotemporal Transformer encoder: Extracts hierarchical spatiotemporal features from the input video frame sequence; Prototype memory network: Connected to the encoder, used to store normal pattern prototypes and enhance the memory of encoded features through similarity matching; Dual-decoder architecture: connecting the encoder and the prototype memory network, including a predictive decoder and a reconstruction decoder; The prediction decoder performs a future frame prediction task based on the original temporal features output by the encoder. The reconstruction decoder performs the current frame reconstruction task based on the original temporal features and the memory enhancement features output by the prototype memory network.

2. The video anomaly detection method based on multi-task learning according to claim 1, characterized in that, The spatiotemporal Transformer encoder includes a spatial encoder that processes single-frame images to extract spatial features and a temporal encoder that processes continuous frame sequences to extract temporal features.

3. The video anomaly detection method based on multi-task learning according to claim 2, characterized in that, The spatial encoder includes a patch embedding module, which performs patch embedding on N consecutive video frame sequences. ; The processing steps of the spatial encoder are as follows: S1.1.1 Patch Embedding: For each input frame image The image is divided into P non-overlapping image blocks of size p×p and embedded. S1.1.2: Add a learnable spatial classification token to each patch sequence. And positional encoding: the initial sequence Input to a by In a module consisting of stacked identical spatial Transformer coding layers; S1.1.3: Spatial Feature Vectors pass Each spatial Transformer layer handles the capture of intra-frame spatial dependencies.

4. The video anomaly detection method based on multi-task learning according to claim 3, characterized in that, The processing steps of the time encoder are as follows: S1.2.1: Collect spatial CLS tokens of all video frames, arrange them in chronological order, and construct the initial time series. ; S1.2.2: Add a time classification token at the beginning of the time series. It aggregates global context information of the entire segment in the time dimension; and generates a location encoding vector for each position in the time series. Finally, an initial representation of the time series is obtained by incorporating temporal location information; S1.2.3: Initial time series Input to a by In a module composed of stacked time Transformer encoding layers, the time series is obtained through... A time-varying Transformer layer models the inter-frame temporal dynamics.

5. The video anomaly detection method based on multi-task learning according to claim 3, characterized in that, The prototype memory network maintains a learnable memory bank containing M prototype features. Each prototype This indicates a normal mode for a feature; The specific steps for memory enhancement using the prototype memory network are as follows: S2.1: Determine input features through memory addressing Which or more normal pattern prototypes in the memory bank are most similar to; S2.2: During training, the memory bank is updated using an exponential moving average. In the formula It's the update rate. It is an indicator function that selects the prototype with the highest attention weight; S2.3: By introducing a memory shrinkage loss that combines entropy regularization and diversity constraints, we encourage the use of sparse memory and prevent over-smoothing.

6. The video anomaly detection method based on multi-task learning according to claim 5, characterized in that, The specific steps for S2.1 are as follows: S2.1.1: Calculate the similarity-based attention weights between encoded temporal features and stored prototypes; S2.1.2: Given coding time characteristics Attention weights are computed using softmax normalization with temperature scaling. S2.1.3: Perform memory retrieval based on the addressing result, read relevant normal mode information from the memory bank, and calculate the retrieved memory representation as a weighted combination of all prototypes.

7. The video anomaly detection method based on multi-task learning according to claim 6, characterized in that, The dual-decoder architecture utilizes original temporal features and memory-enhanced features. Perform complementary tasks, where the predictive decoder directly processes the raw temporal features f through an upsampling layer to predict the next frame. The reconstruction decoder reconstructs the input frame by concatenating and combining original and memory-enhanced features. This allows the reconstruction task to benefit from memory-constrained representations, while the prediction task retains direct access to temporal dynamics.

8. The video anomaly detection method based on multi-task learning according to claim 7, characterized in that: During the model training phase, the loss functions of multiple tasks are integrated into a total training loss through weighted summation to coordinate the optimization process and balance the contribution of each task to the model parameters.

9. The video anomaly detection method based on multi-task learning according to claim 7, characterized in that, Anomaly detection will also be performed during the decoding process: First, calculate the PSNR values ​​for the prediction and reconstruction tasks; Then, the concentration of attention weights is measured by memory entropy, and the intermediate outlier score is obtained by combining the normalized PSNR value and memory entropy with the intermediate outlier score. The final outlier scores are normalized to the range [0,1].

10. A video anomaly detection method based on multi-task learning according to any one of claims 1-8, characterized in that, It includes a model training phase and an anomaly detection phase, wherein the model training phase Includes the following steps: S1: Use a spatiotemporal Transformer encoder to perform hierarchical feature extraction on the training video frame sequence to obtain spatiotemporal features; S2: The spatiotemporal features are enhanced through a prototype memory network; S3: Future frame prediction and current frame reconstruction are performed separately using a dual-decoder architecture; S4: Calculate the multi-task loss function, which includes prediction loss, reconstruction loss and memory shrinkage loss; S5: Update the parameters of the spatiotemporal Transformer encoder, prototype memory network, and dual decoder through backpropagation according to the multi-task loss function; The anomaly detection phase includes the following steps: 1) Obtain the video frame sequence to be detected; 2) Use the spatiotemporal Transformer encoder to extract spatiotemporal features from the video frame sequence; 3) Obtain attention weights by performing memory addressing on the spatiotemporal features through the prototype memory network; 4) Use the dual decoders to output future frames and reconstructed frames respectively; 5) Calculate the comprehensive anomaly score based on the future frame, the reconstructed frame, and the attention weights; 6) Determine whether there are any abnormal events in the video based on the comprehensive anomaly score.

Images