Video anomaly detection method based on motion feature diffusion prediction

Abstract

This invention belongs to the fields of computer vision and video processing technology. It proposes a video anomaly detection algorithm based on motion feature diffusion prediction, which improves the detection accuracy and reduces model complexity. The method is based on motion feature prediction and appearance reconstruction using a generative model. Specifically, the technical solution involves acquiring video data samples and preprocessing them. These samples are then input into the encoder of a variational autoencoder (VAE) network for processing. The compressed features from the encoder are input into a motion prediction model. The predicted features output by the motion prediction model are then interpolated and fused with the compressed features from the encoder before being input into the VAE network decoder. The VAE network decoder outputs the appearance reconstruction result, which is used to determine whether an anomaly has occurred. This invention is primarily applied to the design and manufacturing of video surveillance systems.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and video processing technology, and relates to an efficient video anomaly detection method for complex scenes. It aims to improve the real-time performance and accuracy of abnormal behavior detection, and is particularly suitable for scenarios such as public safety monitoring and intelligent security. Specifically, it involves a video anomaly detection algorithm based on motion feature diffusion prediction. Background Technology

[0002] With the rapid development of smart cities and public safety, the demand for video surveillance systems has increased significantly. Traditional video anomaly detection methods rely on manual monitoring, but these methods are inefficient and consume enormous resources in high-density monitoring scenarios. In recent years, deep learning methods have shown potential in automated video anomaly detection, especially the application of generative models, which has made unsupervised or semi-supervised anomaly detection possible.

[0003] Currently, research methods in the field of video anomaly detection are mainly divided into supervised methods based on feature extraction, weakly supervised methods based on pseudo-label generation, and unsupervised methods based on generative models. Unsupervised methods are particularly important because they can achieve anomaly detection by learning the feature distribution of normal samples, even with limited data labeling.

[0004] However, despite the significant progress made by deep learning methods in video anomaly detection, several challenges remain. First, deep learning methods are typically based on complex neural network architectures, such as generative adversarial networks (GANs) and variational autoencoders (VAEs). These models often have high computational complexity and resource consumption when processing video data, making it difficult to meet the speed requirements of real-time video surveillance systems. Furthermore, they suffer from issues related to generation quality and training stability. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention aims to propose a video anomaly detection method based on motion feature diffusion prediction. This method can improve the detection accuracy of anomalies and reduce model complexity. The method is based on motion feature prediction and appearance reconstruction using a generative model. The technical solution adopted in this invention is as follows: A video anomaly detection method based on motion feature diffusion prediction collects video data samples and preprocesses them. These samples are then input into the encoder of a variational autoencoder (VAE) network for processing. The features compressed by the encoder are input into a motion prediction model. The predicted features output by the motion prediction model are then interpolated and fused with the features compressed by the encoder before being input into the VAE network decoder. The VAE network decoder outputs the appearance reconstruction result, which is used to determine whether an anomaly has occurred.

[0006] The specific steps are as follows:

[0007] Step 1: Collect video data samples and preprocess them. The video data samples include normal samples and abnormal samples. Preprocess the video to obtain continuous video frames of the training sample set and the test sample set.

[0008] Step 2, Appearance Reconstruction Model: Appearance reconstruction is implemented based on a variational autoencoder (VAE) network architecture, including an appearance reconstruction model encoder and an appearance reconstruction model decoder. The appearance reconstruction model encoder extracts features from the input continuous video frames to obtain a sample feature set, maps the sample feature set into the latent space v, and then obtains the compressed appearance features z through a downsampling operation. k ;

[0009] Step 3, Motion Prediction Model: The motion prediction network architecture includes a diffusion process and a U-Net network for denoising. The sample feature set obtained by feature extraction from the encoder described in Step 2 for consecutive video frames is input into the motion prediction network. The motion prediction network uses the features of the k-th frame through a diffusion process to obtain the noise features of the k-th frame. These noise features, along with the features from the previous (k-1) frames, are used as input to the U-Net network to predict the sample features of the k-th frame. The feature z obtained in step 2 k With the The final feature z is obtained by fusing the features using interpolation methods. r , will z r The reconstructed image is generated by the appearance reconstruction model decoder as input to the appearance reconstruction model decoder.

[0010] Step 4: Minimize the prediction loss and reconstruction loss of normal data samples to train the appearance reconstruction model and motion prediction model, and perform anomaly detection inference. By jointly training the motion prediction model and appearance reconstruction model, the loss function is optimized to improve the detection accuracy of each model. Continuous video frames are input to the trained models, and the reconstructed image is output by the decoder of the appearance reconstruction model. Determine if any abnormal situation has occurred.

[0011] The video data samples were collected and preprocessed. The video data samples included normal samples and abnormal samples. The preprocessing of the videos yielded a training sample set and a test sample set: Specifically, the self-collected dataset was obtained by processing the surveillance video data, which included the daytime dataset SADAY and the nighttime dataset SANight. SADAY consists of video clips under visible light conditions extracted from daytime surveillance videos, containing 23 training videos and 25 test videos, with a total of 40 abnormal events, including people carrying firearms, illegal trespassing, and throwing objects. SANight consists of video clips extracted from surveillance videos obtained through infrared cameras at night, containing 10 training videos and 16 test videos, with a total of 30 abnormal events, including people loitering, robbery, and throwing objects.

[0012] In step 2, feature vectors of the input data samples are extracted by the appearance reconstruction model encoder. Feature extraction is performed on the input continuous video frames to obtain the latent space v. The latent space v is downsampled to obtain z. k Motion information features are introduced during the reconstruction process to ensure the distinguishable reconstruction of normal and abnormal samples. Then, the latent features generated in step 3 are... With the latent features z of the predicted sample input k Interpolation is performed to train the appearance reconstruction model decoder to generate diversity.

[0013] The appearance reconstruction model uses a continuous video frame sequence as input to an improved U-Net network encoder. The improved U-Net network introduces a spatial attention mechanism during the upsampling stage: before each skip connection operation, the upsampled feature map and the feature map in the corresponding encoding path are multiplied by a spatial attention weight map to obtain global spatial information of the image as accurately as possible. The improved U-Net network has 3 layers to retain some global information. At the same time, a Dropout strategy with a probability of 0.4 is introduced into the output feature map of each layer of the network. During training, some feature maps are randomly deactivated with a probability of 40% to prevent overfitting. It focuses on capturing the internal correlation of the image appearance and enhances the U-Net network's ability to extract effective features and spatial information.

[0014] The improved U-Net network inherits the symmetrical U-shaped structure containing compression and expansion paths. The compression path corresponds to the downsampling stage, which gradually reduces the spatial resolution of the feature map and extracts high-level abstract features. The expansion path corresponds to the upsampling stage, which gradually restores the spatial resolution of the feature map, eventually restoring it to the same or close size as the input image, achieving pixel-level prediction. The network input is a 256×256 image. The compression path consists of 3 units, each using 3 effective convolutions and 1 max pooling. After each pooling, the number of feature vectors is multiplied by 2, resulting in a feature vector of size 32×32. The expansion path consists of 3 units. Before each unit begins, the feature vector size is multiplied by 2 and its number is halved through deconvolution. Then, it is merged with the feature vector from the symmetrical compression path on the left. The convolution operation in the expansion path uses effective convolution operations.

[0015] The motion prediction model uses the same improved U-Net network architecture as the appearance reconstruction model as the variational autoencoder (VAE). The output of the motion prediction model is input into the encoder of the VAE, and the motion prediction features and appearance features are fused by interpolation.

[0016] In step 3, the feature vectors of the training video are used to perform multi-step motion feature prediction through a motion prediction model, and the sample features of the k-th frame are obtained using a diffusion model. The final state feature z is obtained through an interpolation strategy. r , will z r As input to the module upsampling process, a reconstructed image is generated.

[0017] Predicting the (k+1)th frame sample using the first k frames of samples, for a given video consecutive frames X = {x1, x2, ..., x...} k},use This represents the feature of the k-th frame with a time step of t. This represents the original features extracted by the encoder;

[0018] Use the features of the first k-1 samples and noise features of the k-th frame The goal is to obtain the sample features of the k-th frame as input to the feature prediction template.

[0019] We utilize an improved U-Net denoising network for feature prediction. The cross-attention layer, which simultaneously focuses on conditional information and latent representation, is modified to a self-attention mechanism that focuses only on latent representation. The self-attention mechanism performs a linear transformation on the input feature map, resulting in Query, Key, and Value. Query is the query vector representing the information requirement at the current feature location; Key is the key vector representing the content features at each location; and Value is the value vector carrying the actual feature information to be aggregated. An attention weight matrix is ​​then calculated based on this matrix. Finally, the output feature map is a weighted fusion of the weight matrix and the feature map. This forces the improved U-Net denoising network to learn the feature distribution from normal samples as much as possible without involving other latent conditions.

[0020] The goal of motion prediction models is to learn a distribution q(x) that approximates the original data distribution. 0 The distribution p) θ (x 0 In the diffusion process described in step 3, the posterior distribution q(z) 1:T ∣z 0 Fixed to a Markov chain:

[0021]

[0022] Among them, z 0 For the original sample features, z t For the sample features after adding noise at step t, z 1:T Let q(z) be the set of all noisy features from step 1 to step T, where T is the step size of the noisy process, t∈[1,T] is the time step, and q(z) is the set of all noisy features from step 1 to step T. t ∣z t-1 ) indicates that given the previous feature z t-1 Under the condition that feature z in this step t Given a noise probability distribution, the forward diffusion process is defined as follows:

[0023]

[0024] Where I is the identity matrix, This represents a normal Gaussian distribution with a mean of 1 / 2. Covariance is β t I, select β t ∈(0,1) is used as a hyperparameter for adding Gaussian noise, satisfying β1<β2<...<β t The sampling step is represented using the reparameterization technique, let α t =1-β t , z t You can use z 0 The linear combination of the standard Gaussian distribution is represented as:

[0025]

[0026] in, This indicates that the sampling noise at each time step is independent and follows a Gaussian distribution;

[0027] Using reconstructed feature z k The k-th feature k obtained from the forward diffusion process, along with the first k-1 original features, provides motion information to the improved U-Net network for denoising, enabling it to focus on learning motion features.

[0028] Motion prediction models directly address noisy latent variables. The above training denoising process improves the parameters of the U-Net network through training. θ To optimize, the goal of the denoising loss is to minimize the difference between the denoised latent variables and the original latent variables. According to the diffusion model, the objective function simplifies to:

[0029]

[0030] Where ∈ represents the original noise, ∈ θ For the predicted denoising noise, θ represents the network parameters. Let represent the expectation of the sampled latent features z, noise ∈, and time step t. The square of the L2 norm is used to measure the difference between predicted noise and actual noise. By minimizing the denoising error, this loss function ensures that the network can effectively recover the underlying representation of normal motion information from the noise.

[0031] The reverse process utilizes an improved U-Net network for denoising: it generates the features of the k-th sample at time t-1 by progressively denoising. as follows:

[0032]

[0033] Where σ t By controlling the added noise at each step and continuously supervising the process, motion prediction is effectively guaranteed, ultimately yielding the denoised latent vector.

[0034] The latent feature space v obtained after the image is encoded is downsampled to obtain latent variables, which are then compared with the denoised latent vector. Interpolation is performed to obtain the final state:

[0035]

[0036] Here, λ is a hyperparameter used to control the weighting ratio of the two latent vectors. This interpolation method can maintain the diversity of the generated data in the latent space and help the decoder generate more diverse images.

[0037] Ultimately, z r As input to the module upsampling process, a reconstructed image is generated.

[0038] In step 4, the prediction loss and reconstruction loss of normal data samples are minimized to train the model and perform anomaly detection inference. By jointly training the motion prediction module and the appearance reconstruction module, the loss function is optimized to improve the model's detection accuracy. Continuous video frames are input into the trained model to determine whether an anomaly has occurred. The specific process is as follows:

[0039] The objective function consists of motion prediction loss and appearance reconstruction loss;

[0040] In motion prediction models, training a VAE network on latent variables... Denoising is performed by introducing the first k-1 noisy features to ensure the model can effectively recover and predict motion features from noise. Denoising training is conducted on the noisy latent variables to ultimately predict the denoised latent features. The goal of the denoising loss is to minimize the difference between the denoised latent variables and the original latent variables. Based on the diffusion model, the objective function simplifies to:

[0041]

[0042] Where ∈ represents the original noise, ∈ θ For the predicted denoising noise, θ represents the network parameters. Let represent the expectation of the sampled latent features z, noise ∈, and time step t. It represents the square of the L2 norm, which measures the difference between predicted noise and actual noise;

[0043] In the appearance reconstruction model, the objective function is defined using the reconstruction error, and the calculation formula is as follows:

[0044]

[0045] Where, x k The original input image, To reconstruct the image, the final overall objective function is:

[0046]

[0047] Where, λ pre and λ rec This is a hyperparameter used to control the weight of each loss term in the total loss.

[0048] The features and beneficial effects of this invention are:

[0049] (1) The method proposed in this invention avoids the reliance on optical flow calculation and other complex preprocessing processes in traditional methods. It introduces a diffusion model to obtain image motion information while reducing computational load. The model only uses the continuous features of the first few frames to learn the feature distribution of normal samples without external latent conditions, and replaces the cross-attention layer with a traditional attention layer to focus on motion feature learning. The proposed algorithm model only needs to train a shared U-Net network, effectively reducing the computational load and training time of the model without affecting the detection accuracy.

[0050] (2) An improved U-Net network is proposed, which introduces a spatial attention mechanism in the model upsampling process to obtain global information of the image as accurately as possible, focuses on capturing the internal correlation of the image appearance, and enhances the ability of the U-Net network to extract effective features and spatial information. Attached image description:

[0051] Figure 1 Example image of a self-collected dataset.

[0052] Figure 2 : A schematic diagram of a global video anomaly detection method based on frame prediction.

[0053] Figure 3 : Diagram of the improved U-Net architecture. Detailed Implementation

[0054] To overcome the shortcomings of existing technologies, this invention proposes a video anomaly detection algorithm based on motion feature diffusion prediction. By leveraging the advantages of diffusion models in generating high-quality images, it achieves high-precision and low-complexity anomaly detection.

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0056] The video anomaly detection method based on motion feature diffusion prediction collects video data samples and preprocesses them. Then, the samples are input into the encoder of a variational autoencoder (VAE) network for processing. The features compressed by the encoder are input into a motion prediction model. The predicted features output by the motion prediction model are then interpolated and fused with the features compressed by the encoder and input into the VAE network decoder. The VAE network decoder outputs the appearance reconstruction result and determines whether an anomaly has occurred.

[0057] The specific implementation steps are as follows:

[0058] Step 1: Collect video data samples and preprocess them. The video data samples include normal samples and abnormal samples. Preprocess the video to obtain continuous video frames of the training sample set and the test sample set.

[0059] Step 2, Appearance Reconstruction Model: Appearance reconstruction is implemented based on a variational autoencoder (VAE) network architecture, including an appearance reconstruction model encoder and an appearance reconstruction model decoder. The appearance reconstruction model encoder extracts features from the input continuous video frames to obtain a sample feature set, maps the sample feature set into the latent space v, and then obtains the compressed appearance features z through a downsampling operation. k ;

[0060] Step 3, Motion Prediction Model: The motion prediction network architecture includes a diffusion process and a U-Net network for denoising. The sample feature set obtained by feature extraction from the encoder described in Step 2 for consecutive video frames is input into the motion prediction network. The motion prediction network uses the features of the k-th frame through a diffusion process to obtain the noise features of the k-th frame. These noise features, along with the features from the previous (k-1) frames, are used as input to the U-Net network to predict the sample features of the k-th frame. The feature z obtained in step 2 k With the The final feature z is obtained by fusing the features using interpolation methods. r , will z r The reconstructed image is generated by the appearance reconstruction model decoder as input to the appearance reconstruction model decoder.

[0061] Step 4: Minimize the prediction loss and reconstruction loss of normal data samples to train the appearance reconstruction model and motion prediction model, and perform anomaly detection inference. By jointly training the motion prediction model and appearance reconstruction model, the loss function is optimized to improve the detection accuracy of each model. Continuous video frames are input to the trained models, and the reconstructed image is output by the decoder of the appearance reconstruction model. Determine if any abnormal situation has occurred.

[0062] The video data samples were collected and preprocessed. The video data samples included normal samples and abnormal samples. The preprocessing of the videos yielded a training sample set and a test sample set: Specifically, the self-collected dataset was obtained by processing the surveillance video data, which included the daytime dataset SADAY and the nighttime dataset SANight. SADAY consists of video clips under visible light conditions extracted from daytime surveillance videos, containing 23 training videos and 25 test videos, with a total of 40 abnormal events, including people carrying firearms, illegal trespassing, and throwing objects. SANight consists of video clips extracted from surveillance videos obtained through infrared cameras at night, containing 10 training videos and 16 test videos, with a total of 30 abnormal events, including people loitering, robbery, and throwing objects.

[0063] In step 2, feature vectors of the input data samples are extracted by the appearance reconstruction model encoder. Feature extraction is performed on the input continuous video frames to obtain the latent space v. The latent space v is downsampled to obtain z. k Motion information features are introduced during the reconstruction process to ensure the distinguishable reconstruction of normal and abnormal samples. Then, the latent features generated in step 3 are... With the latent features z of the predicted sample input k Interpolation is performed to train the appearance reconstruction model decoder to generate diversity.

[0064] The appearance reconstruction model uses a continuous video frame sequence as input to an improved U-Net network encoder. The improved U-Net network introduces a spatial attention mechanism during the upsampling stage: before each skip connection operation, the upsampled feature map and the feature map in the corresponding encoding path are multiplied by a spatial attention weight map to obtain global spatial information of the image as accurately as possible. The improved U-Net network has 3 layers to retain some global information. At the same time, a Dropout strategy with a probability of 0.4 is introduced into the output feature map of each layer of the network. During training, some feature maps are randomly deactivated with a probability of 40% to prevent overfitting. It focuses on capturing the internal correlation of the image appearance and enhances the U-Net network's ability to extract effective features and spatial information.

[0065] The improved U-Net network inherits the symmetrical U-shaped structure containing compression and expansion paths. The compression path corresponds to the downsampling stage, which gradually reduces the spatial resolution of the feature map and extracts high-level abstract features. The expansion path corresponds to the upsampling stage, which gradually restores the spatial resolution of the feature map, eventually restoring it to the same or close size as the input image, achieving pixel-level prediction. The network input is a 256×256 image. The compression path consists of 3 units, each using 3 effective convolutions and 1 max pooling. After each pooling, the number of feature vectors is multiplied by 2, resulting in a feature vector of size 32×32. The expansion path consists of 3 units. Before each unit begins, the feature vector size is multiplied by 2 and its number is halved through deconvolution. Then, it is merged with the feature vector from the symmetrical compression path on the left. The convolution operation in the expansion path uses effective convolution operations.

[0066] The motion prediction model uses the same improved U-Net network architecture as the appearance reconstruction model as the variational autoencoder (VAE). The output of the motion prediction model is input into the encoder of the VAE, and the motion prediction features and appearance features are fused by interpolation.

[0067] In step 3, the feature vectors of the training video are used to perform multi-step motion feature prediction through a motion prediction model, and the sample features of the k-th frame are obtained using a diffusion model. The final state feature z is obtained through an interpolation strategy. r , will z r As input to the module upsampling process, a reconstructed image is generated.

[0068] Predicting the (k+1)th frame sample using the first k frames of samples, for a given video consecutive frames X = {x1, x2, ..., x...} k},use This represents the feature of the k-th frame with a time step of t. This represents the original features extracted by the encoder;

[0069] Use the features of the first k-1 samples and noise features of the k-th frame The goal is to obtain the sample features of the k-th frame as input to the feature prediction template.

[0070] We utilize an improved U-Net denoising network for feature prediction. The cross-attention layer, which simultaneously focuses on conditional information and latent representation, is modified to a self-attention mechanism that focuses only on latent representation. The self-attention mechanism performs a linear transformation on the input feature map, resulting in Query, Key, and Value. Query is the query vector representing the information requirement at the current feature location; Key is the key vector representing the content features at each location; and Value is the value vector carrying the actual feature information to be aggregated. An attention weight matrix is ​​then calculated based on this matrix. Finally, the output feature map is a weighted fusion of the weight matrix and the feature map. This forces the improved U-Net denoising network to learn the feature distribution from normal samples as much as possible without involving other latent conditions.

[0071] The goal of motion prediction models is to learn a distribution q(x) that approximates the original data distribution. 0 The distribution p) θ (x 0 In the diffusion process described in step 3, the posterior distribution q(z) 1:T ∣z 0 Fixed to a Markov chain:

[0072]

[0073] Among them, z 0 For the original sample features, z t For the sample features after adding noise at step t, z 1:T Let q(z) be the set of all noisy features from step 1 to step T, where T is the step size of the noisy process, t∈[1,T] is the time step, and q(z) is the set of all noisy features from step 1 to step T. t ∣z t-1 ) indicates that given the previous feature z t-1 Under the condition that feature z in this step t Given a noise probability distribution, the forward diffusion process is defined as follows:

[0074]

[0075] Where I is the identity matrix, This represents a normal Gaussian distribution with a mean of 1 / 2. Covariance is β t I, select β t ∈(0,1) is used as a hyperparameter for adding Gaussian noise, satisfying β1<β2<...<β t The sampling step is represented using the reparameterization technique, let α t =1-β t , z t You can use z 0 The linear combination of the standard Gaussian distribution is represented as:

[0076]

[0077] in, This indicates that the sampling noise at each time step is independent and follows a Gaussian distribution;

[0078] Using reconstructed feature z k The k-th feature k obtained from the forward diffusion process, along with the first k-1 original features, provides motion information to the improved U-Net network for denoising, enabling it to focus on learning motion features.

[0079] The motion prediction module directly addresses noisy latent variables. The above training denoising process improves the parameters of the U-Net network through training. θ To optimize, the goal of the denoising loss is to minimize the difference between the denoised latent variables and the original latent variables. According to the diffusion model, the objective function simplifies to:

[0080]

[0081] Where ∈ represents the original noise, ∈ θ For the predicted denoising noise, θ represents the network parameters. Let represent the expectation of the sampled latent features z, noise ∈, and time step t. The square of the L2 norm is used to measure the difference between predicted noise and actual noise. By minimizing the denoising error, this loss function ensures that the network can effectively recover the underlying representation of normal motion information from the noise.

[0082] The reverse process utilizes an improved U-Net network for denoising: it generates the features of the k-th sample at time t-1 by progressively denoising. as follows:

[0083]

[0084] Where σ t By controlling the added noise at each step and continuously supervising the process, motion prediction is effectively guaranteed, ultimately yielding the denoised latent vector.

[0085] The latent feature space v obtained after the image is encoded is downsampled to obtain latent variables, which are then compared with the denoised latent vector. Interpolation is performed to obtain the final state:

[0086]

[0087] Here, λ is a hyperparameter used to control the weighting ratio of the two latent vectors. This interpolation method can maintain the diversity of the generated data in the latent space and help the decoder generate more diverse images.

[0088] Ultimately, z r As input to the module upsampling process, a reconstructed image is generated.

[0089] In step 4, the prediction loss and reconstruction loss of normal data samples are minimized to train the model and perform anomaly detection inference. By jointly training the motion prediction module and the appearance reconstruction module, the loss function is optimized to improve the model's detection accuracy. Continuous video frames are input into the trained model to determine whether an anomaly has occurred. The specific process is as follows:

[0090] The objective function consists of motion prediction loss and appearance reconstruction loss;

[0091] In motion prediction models, training a VAE network on latent variables... Denoising is performed by introducing the first k-1 noisy features to ensure the model can effectively recover and predict motion features from noise. Denoising training is conducted on the noisy latent variables to ultimately predict the denoised latent features. The goal of the denoising loss is to minimize the difference between the denoised latent variables and the original latent variables. Based on the diffusion model, the objective function simplifies to:

[0092]

[0093] Where ∈ represents the original noise, ∈ θ For the predicted denoising noise, θ represents the network parameters. Let represent the expectation of the sampled latent features z, noise ∈, and time step t. It represents the square of the L2 norm, which measures the difference between predicted noise and actual noise;

[0094] In the appearance reconstruction model, the objective function is defined using the reconstruction error, and the calculation formula is as follows:

[0095]

[0096] Where, x k The original input image, To reconstruct the image, the final overall objective function is:

[0097]

[0098] Where, λ pre and λ rec This is a hyperparameter used to control the weight of each loss term in the total loss.

[0099] The present invention will be further described in detail below with reference to the accompanying drawings and specific examples.

[0100] The purpose of this invention is to propose a video anomaly detection algorithm based on motion feature diffusion prediction. This method can improve the detection accuracy and reduce the model complexity. The method is based on motion feature prediction and appearance reconstruction using a generative model, and the specific implementation steps are as follows:

[0101] Step 1: Collect video data samples and preprocess them. The video data samples include normal samples and abnormal samples. Preprocessing the videos yields training sample sets and test sample sets.

[0102] Step 2: Extract feature vectors from the input data samples using the appearance reconstruction module encoder. Feature extraction is performed on the input continuous video frames to obtain a sample feature set Z. New features are then obtained by downsampling in the feature space v and reparameterized to obtain z. k .

[0103] Step 3: Utilize the feature vectors of the training video to perform multi-step motion feature prediction using the motion prediction module. Obtain the sample features of the k-th frame using a diffusion model. The intermediate state features z are obtained through an interpolation strategy. r , will z r As input to the module upsampling process, a reconstructed image is generated.

[0104] Step 4: Minimize the prediction loss and reconstruction loss of normal data samples to train the model and perform anomaly detection inference. The loss function is optimized by jointly training the motion prediction module and the appearance reconstruction module to improve the model's detection accuracy. Continuous video frames are input to the trained model to determine if any anomalies have occurred.

[0105] Further, in step 1, video data samples are collected and preprocessed. The video data samples include normal and abnormal samples. Preprocessing the videos yields training and testing sample sets. The specific process is as follows:

[0106] This invention obtains self-collected datasets by processing surveillance video data, namely, the Seaside Anolamy Day (SADay) dataset and the Seaside Anolamy Night (SANight) dataset for daytime anomaly detection in open coastal scenes.

[0107] SADay consists of video clips taken from daytime surveillance footage under visible light conditions. It includes 23 training videos and 25 test videos, totaling 40 abnormal events, such as people carrying firearms, illegal border crossings, and throwing objects.

[0108] SANight extracts video clips from surveillance videos obtained by infrared cameras at night. It includes 10 training videos and 16 test videos, totaling 30 abnormal events, including people loitering, robbery, and objects being thrown.

[0109] Further, in step 2, the feature vectors of the input data samples are extracted by the appearance reconstruction module encoder. Feature extraction is performed on the input continuous video frames to obtain the sample feature set Z. New features are then obtained by downsampling in the feature space v and reparameterized to obtain z. k The specific process is as follows:

[0110] The VAE reconstruction process incorporates predicted sample features to ensure the distinguishable reconstruction of normal and abnormal samples. Specifically, the latent features generated by the diffusion model are interpolated with the latent features of the predicted sample input to train the decoder's generative diversity.

[0111] The appearance reconstruction module uses the output image obtained from motion prediction as the encoder input. The encoder adopts the improved U-Net network in the motion prediction module. The improved U-Net network has 3 layers to retain certain global information, and Dropout is introduced to prevent the model from overfitting.

[0112] The improved U-Net network inherits the symmetrical U-shaped structure containing compression and expansion paths. The compression path corresponds to the downsampling stage, which gradually reduces the spatial resolution of the feature map and extracts high-level abstract features. The expansion path corresponds to the upsampling stage, which gradually restores the spatial resolution of the feature map, eventually restoring it to the same or close size as the input image, achieving pixel-level prediction. The network input is a 256×256 image. The compression path consists of 3 units, each using 3 effective convolutions and 1 max pooling. After each pooling, the number of feature vectors is multiplied by 2, resulting in a feature vector of size 32×32. The expansion path consists of 3 units. Before each unit begins, the feature vector size is multiplied by 2 and its number is halved through deconvolution. Then, it is merged with the feature vector from the symmetrical compression path on the left. The convolution operation in the expansion path uses effective convolution operations.

[0113] The output of the motion prediction model is input into the encoder of the VAE, and the motion prediction features and appearance features are fused by interpolation.

[0114] Further, in step 3, the feature vectors of the training video are used to perform multi-step motion feature prediction through the motion prediction module. The sample features of the k-th frame are obtained using a diffusion model. The intermediate state features z are obtained through an interpolation strategy. r , will z r As input to the module upsampling process, a reconstructed image is generated. The specific process is as follows:

[0115] The (k+1)th frame sample is predicted using samples from the first k frames. For a given series of video frames X = {x1, x2, ..., x...} k}, using the features of the k-th frame with time step t, This represents the original features extracted by the encoder.

[0116] Use the features of the first k-1 samples and noise features of the k-th frame The goal is to obtain the sample features of the k-th frame as input to the feature prediction template.

[0117] A U-Net diffusion network was built for feature prediction. The cross-attention layer was modified to a self-attention mechanism, focusing only on the latent representation. The self-attention mechanism performs a linear transformation on the input feature map, which serves as the Query, Key, and Value. The Query is a query vector representing the information requirement at the current feature location; the Key is a key vector representing the content features at each location; and the Value is a value vector carrying the actual feature information to be aggregated. An attention weight matrix is ​​then calculated accordingly, and finally, the output feature map is a weighted fusion of the weight matrix and the feature map. This forces the model to learn the feature distribution from normal samples as much as possible without involving other latent conditions.

[0118] The appearance reconstruction model focuses on extracting the appearance information of the first k frames, while the motion prediction model only focuses on extracting the motion of the k-th frame based on the previous (k-1)-th frames, that is, the changes.

[0119] The goal of motion prediction models is to learn a distribution q(z) that approximates the original data distribution. 0 The distribution p) θ (z 0 ).

[0120] In the forward process, the posterior distribution q(z) 1:T ∣z 0 Fixed to a Markov chain:

[0121]

[0122] Among them, z 0 For the original sample features, z t For the sample features after adding noise at step t, z 1:T Let q(z) be the set of all noisy features from step 1 to step T, where T is the step size of the noisy process, t∈[1,T] is the time step, and q(z) is the set of all noisy features from step 1 to step T. t ∣z t-1 ) indicates that given the previous feature z t-1Under the condition that feature z in this step t The probability distribution with added noise. For a given data sample, the forward diffusion process is defined as:

[0123]

[0124] Where I is the identity matrix, This represents a Gaussian (normal) distribution with a mean of 1 / 2. Covariance is β t I, select β t ∈(0,1) is used as a hyperparameter for adding Gaussian noise, satisfying β1<β2<...<β t The sampling step is represented using a reparameterization technique, let α t =1-β t , z t You can use z 0 The linear combination of the standard Gaussian distribution is represented as:

[0125]

[0126] in, This indicates that the sampling noise at each time step is independent and follows a Gaussian distribution.

[0127] Using reconstructed feature z k The k-th feature obtained from the forward diffusion process, along with the first k-1 original features, provides motion information to the θ(.) network, enabling the diffusion module to focus on learning motion features.

[0128] Motion prediction models (diffusion models) directly address noisy latent variables. The above training denoising process improves the parameters of the U-Net denoising network through training. θ The optimization goal of the denoising loss is to minimize the difference between the denoised latent variables and the original latent variables. Based on the diffusion model, the objective function can be simplified to:

[0129]

[0130] Where ∈ represents the original noise, ∈ θ For the predicted denoising noise, θ represents the network parameters. Let represent the expectation of the sampled latent features z, noise ∈, and time step t. The square of the L2 norm is used to measure the difference between predicted and actual noise. By minimizing the denoising error, this loss function ensures that the network can effectively recover the underlying representation of normal motion information from the noise.

[0131] The reverse process generates the features of the k-th sample at time t-1 by progressively denoising. as follows:

[0132]

[0133] Where σ t By controlling the added noise at each step and continuously supervising the process, motion prediction is effectively guaranteed, ultimately yielding the denoised latent vector.

[0134] The latent feature space v obtained after the image is encoded is downsampled to obtain latent variables, which are then compared with the denoised latent vector. Interpolation is performed to obtain the final state:

[0135]

[0136] Here, λ is a hyperparameter used to control the weighting ratio of the two latent vectors. This interpolation method allows us to preserve the diversity of the generated data in the latent space and helps the decoder generate more diverse images.

[0137] z r As input to the module upsampling process, a reconstructed image is generated.

[0138] Further, in step 4, the model is trained by minimizing the prediction loss and reconstruction loss of normal data samples, and anomaly detection inference is performed. The loss function is optimized by jointly training the motion prediction module and the appearance reconstruction module to improve the model's detection accuracy. Continuous video frames are input to the trained model to determine if anomalies have occurred. Appearance constraints on image data are imposed by minimizing the spatial distance between the reconstructed image and the real image. Specifically, intensity loss is used to ensure the similarity of spatial pixel intensity, and gradient loss is used to ensure the similarity of pixel details in adjacent images. The specific process is as follows:

[0139] The objective function of the proposed algorithm consists of motion prediction loss and appearance reconstruction loss.

[0140] In the motion prediction module, the network is trained on latent variables. Denoising is performed by introducing the first k-1 noisy features to ensure the model can effectively recover and predict motion features from noise. This module predicts the denoised latent features by training on the noisy latent variables. The goal of the denoising loss is to minimize the difference between the denoised latent variables and the original latent variables. Based on the diffusion model, the objective function can be simplified to:

[0141]

[0142] Where ∈ represents the original noise, ∈ θ For the predicted denoising noise, θ represents the network parameters. Let represent the expectation of the sampled latent features z, noise ∈, and time step t. The square of the L2 norm is used to measure the difference between predicted and actual noise. By minimizing the denoising error, this loss function ensures that the network can effectively recover the underlying representation of normal motion information from the noise.

[0143] In the appearance reconstruction module, the objective function is defined using reconstruction error, and the calculation formula is as follows:

[0144]

[0145] Where, x k The original input image, To reconstruct the image, the final overall objective function is:

[0146]

[0147] Where, λ pre and λ rec is a hyperparameter used to control the weight of each loss term in the total loss. The objective function combines reconstruction loss and prediction loss. By introducing motion prediction loss, it enhances the consistency of appearance and motion during image reconstruction, expands the reconstruction difference between normal and abnormal samples, and ensures that the model can accurately reconstruct normal behavior while also identifying abnormal behavior that is significantly different from normal behavior.

[0148] The complete training process of this invention is as follows:

[0149] a. Data Loading: Read the dataset used for training, traverse the folders under the specified path and read the image files, and preprocess the images: ① Resize each frame to 256×256; ② Normalize the pixel intensity in all frames to [-1,1]. Construct video segments by randomly sampling frames, and use these segments as training data, with the image channel order being BGR;

[0150] b. Parameter initialization: Given the initial values ​​of the encoding network, the obtained continuous image frame sequence is input into the encoding network. Each sequence has k frames. Feature samples are extracted sequentially, and the first k-1 sample features are used for motion feature prediction.

[0151] c. Motion feature prediction. Initialize the denoising network parameters, using the features of the first k-1 samples. The downsampled feature z k and noise features of the kth frame. The input is fed into a denoising network for motion feature prediction, using downsampled features.

[0152] d. Appearance Feature Reconstruction. Feature extraction is performed on the input continuous video frames to obtain a sample feature set Z. New features are then obtained by downsampling in the feature space v and reparameterized to obtain z. k The predicted output samples will be... The input is fed into the encoder, along with the denoised latent vector. Interpolating the normal vector to obtain z r As input to the module upsampling process, it generates the reconstructed image.

[0153] e. By minimizing the reconstruction loss Motion Prediction Loss The common constraint model is trained.

[0154] g. Except for the initialization step, iterate through each batch of the training dataset and repeat the above process for iterative training. The generator and discriminator are trained using different optimizers until all training iterations are completed and the training model is saved.

[0155] The testing process iterates through each video in the test dataset, making predictions frame by frame and calculating the PSNR of the prediction results. It also calculates the performance metrics between the generated and ground truth sequences. The prediction results and PSNR values ​​are used to generate prediction curves and heatmaps. During this process, the optical flow of the generated and ground truth sequences can also be visualized. After predicting all videos, the AUC is calculated based on the prediction results and ground truth labels to measure model performance.

[0156] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A video anomaly detection method based on motion feature diffusion prediction, characterized by the following steps: as follows: Step 1: Collect video data samples and preprocess them. The video data samples include normal samples and abnormal samples. Preprocess the video to obtain continuous video frames of the training sample set and the test sample set. Step 2, Appearance Reconstruction Model: Appearance reconstruction is implemented based on a variational autoencoder (VAE) network architecture, including an appearance reconstruction model encoder and an appearance reconstruction model decoder. The appearance reconstruction model encoder extracts features from the input continuous video frames to obtain a sample feature set, maps the sample feature set into the latent space v, and then obtains the compressed appearance features through a downsampling operation. ; Step 3, Motion Prediction Model: The motion prediction network architecture includes a diffusion process and a U-Net network for denoising. The sample feature set obtained by feature extraction from the encoder described in Step 2 for consecutive video frames is input into the motion prediction network. The motion prediction network uses the features of the k-th frame through a diffusion process to obtain the noise features of the k-th frame. These noise features, along with the features from the previous (k-1) frames, are used as input to the U-Net network to predict the sample features of the k-th frame. The features obtained in step 2 With the The final features are obtained by fusing them using interpolation methods. ,Will The reconstructed image is generated by the appearance reconstruction model decoder as input to the appearance reconstruction model decoder. ; Step 4: Minimize the prediction loss and reconstruction loss of normal data samples to train the appearance reconstruction model and motion prediction model, and perform anomaly detection inference. By jointly training the motion prediction model and appearance reconstruction model, the loss function is optimized to improve the detection accuracy of each model. Continuous video frames are input to the trained models, and the reconstructed image is output by the decoder of the appearance reconstruction model. Determine if any abnormal situation has occurred; among which: The appearance reconstruction model uses a continuous video frame sequence as input to an improved U-Net network encoder. The improved U-Net network introduces a spatial attention mechanism during the upsampling stage: before each skip connection operation, the upsampled feature map and the feature map in the corresponding encoding path are multiplied by a spatial attention weight map to obtain global spatial information of the image as accurately as possible. The improved U-Net network has three layers to retain some global information. Simultaneously, a Dropout strategy with a probability of 0.4 is introduced into the output feature map of each layer of the network, randomly deactivating some feature maps with a 40% probability during training to prevent overfitting. This focuses on capturing the internal correlations of the image appearance, enhancing the U-Net network's ability to extract effective features and spatial information. The improved U-Net network inherits the symmetrical U-shaped structure containing compression and expansion paths. The compression path corresponds to the downsampling stage, which gradually reduces the spatial resolution of the feature map and extracts high-level abstract features. The expansion path corresponds to the upsampling stage, which gradually restores the spatial resolution of the feature map, eventually restoring it to the same or close size as the input image, achieving pixel-level prediction. The network input is a 256×256 image. The compression path consists of 3 units, each using 3 effective convolutions and 1 max pooling. After each pooling, the number of feature vectors is multiplied by 2, resulting in a feature vector of size 32×32. The expansion path consists of 3 units. Before each unit begins, the feature vector size is multiplied by 2 and its number is halved through deconvolution. Then, it is merged with the feature vector from the symmetrical compression path on the left. The convolution operation in the expansion path uses effective convolution operations.

2. The video anomaly detection method based on motion feature diffusion prediction as described in claim 1, characterized in that, The video data samples were collected and preprocessed. The video data samples included normal samples and abnormal samples. The preprocessing of the videos yielded a training sample set and a test sample set: Specifically, the self-collected dataset was obtained by processing the surveillance video data, which included the daytime dataset SADAY and the nighttime dataset SANight. SADAY consists of video clips under visible light conditions extracted from daytime surveillance videos, containing 23 training videos and 25 test videos, with a total of 40 abnormal events, including people carrying firearms, illegal trespassing, and throwing objects. SANight consists of video clips extracted from surveillance videos obtained through infrared cameras at night, containing 10 training videos and 16 test videos, with a total of 30 abnormal events, including people loitering, robbery, and throwing objects.

3. The video anomaly detection method based on motion feature diffusion prediction as described in claim 1, characterized in that, In step 2, feature vectors of the input data samples are extracted using an appearance reconstruction model encoder. Feature extraction is performed on consecutive input video frames to obtain a latent space v. The latent space v is then downsampled to obtain... Motion information features are introduced during the reconstruction process to ensure the distinguishable reconstruction of normal and abnormal samples. Then, the latent features generated in step 3 are... Latent features of the predicted sample input Interpolation is performed to train the appearance reconstruction model decoder to generate diversity.

4. The video anomaly detection method based on motion feature diffusion prediction as described in claim 1, characterized in that, The motion prediction model uses the same improved U-Net network architecture as the appearance reconstruction model as the variational autoencoder (VAE). The output of the motion prediction model is input into the encoder of the VAE, and the motion prediction features and appearance features are fused by interpolation. In step 3, the feature vectors of the training video are used to perform multi-step motion feature prediction through a motion prediction model, and the sample features of the k-th frame are obtained using a diffusion model. The final state features are obtained through an interpolation strategy. ,Will As input to the module upsampling process, a reconstructed image is generated. ; Before adoption Frame Sample Prediction Frame samples, for a given series of video frames ,use Indicates the time step as The Frame features, This represents the original features extracted by the encoder; Before adoption Individual sample features and the Frame noise characteristics As input to the feature prediction template, the goal is to obtain the first... Frame sample features ; We utilize an improved U-Net denoising network for feature prediction. The cross-attention layer, which simultaneously focuses on conditional information and latent representation, is modified to a self-attention mechanism that focuses only on latent representation. The self-attention mechanism performs a linear transformation on the input feature map, resulting in Query, Key, and Value. Query is the query vector representing the information requirement at the current feature location; Key is the key vector representing the content features at each location; and Value is the value vector carrying the actual feature information to be aggregated. An attention weight matrix is ​​then calculated based on this matrix. Finally, the output feature map is a weighted fusion of the weight matrix and the feature map. This forces the improved U-Net denoising network to learn the feature distribution from normal samples as much as possible without involving other latent conditions. The goal of motion prediction models is to learn a distribution that approximates the original data distribution. Distribution In the diffusion process described in step 3, the posterior distribution Fixed to a Markov chain: in, Features of the original sample For the sample features after adding noise at step t, This is the set of all noisy features from step 1 to step T. This is the step size for the noise addition process. For time step, Indicates that given the features from the previous step Under the conditions, this step features Given a noise probability distribution, the forward diffusion process is defined as follows: in, It is the identity matrix. This represents a normal Gaussian distribution with a mean of 1 / 2. covariance is Select As a hyperparameter for adding Gaussian noise, it satisfies The sampling step is represented using a reparameterization technique, letting , It can be used The linear combination of the standard Gaussian distribution is represented as: in, , indicating that the sampling noise at each time step is independent and follows a Gaussian distribution; Employing Reconstruction Features The first obtained from the forward diffusion process Features and the former The original features provide motion information to the improved U-Net network for denoising, enabling it to focus on learning motion features; Motion prediction models directly address noisy latent variables. The above training denoising process improves the parameters of the U-Net network through training. To optimize, the goal of the denoising loss is to minimize the difference between the denoised latent variables and the original latent variables. According to the diffusion model, the objective function simplifies to: in, The original noise, For the predicted denoising noise, For network parameters, Represents the latent features sampled ,noise and time step Expectations The square of the L2 norm is used to measure the difference between predicted noise and actual noise. By minimizing the denoising error, this loss function ensures that the network can effectively recover the underlying representation of normal motion information from the noise. The reverse process utilizes an improved U-Net network for denoising: it generates the first denoised version by progressively denoising the network. One sample in Characteristics of a moment as follows: in By controlling the added noise at each step and continuously supervising the process, motion prediction is effectively guaranteed, ultimately yielding the denoised latent vector. ; The latent feature space obtained after the image is encoded Downsampling is performed to obtain latent variables, and the denoised latent vectors are then compared. Interpolation is performed to obtain the final state: in, It is a hyperparameter used to control the weighting ratio of two latent vectors. This interpolation method can maintain the diversity of the generated data in the latent space and help the decoder generate more diverse images. Ultimately, As input to the module upsampling process, a reconstructed image is generated. .

5. The video anomaly detection method based on motion feature diffusion prediction as described in claim 1, characterized in that, In step 4, the prediction loss and reconstruction loss of normal data samples are minimized to train the model and perform anomaly detection inference. By jointly training the motion prediction module and the appearance reconstruction module, the loss function is optimized to improve the model's detection accuracy. Continuous video frames are input into the trained model to determine whether an anomaly has occurred. The specific process is as follows: The objective function consists of motion prediction loss and appearance reconstruction loss; In motion prediction models, training a VAE network on latent variables... Denoising is performed by introducing the first k-1 noisy features to ensure that the model can effectively recover and predict motion features from noise. The model is trained on noisy latent variables and finally predicts the denoised latent features. The objective of the denoising loss is to minimize the difference between the denoised latent variables and the original latent variables. Based on the diffusion model, the objective function simplifies to: in, The original noise, For the predicted denoising noise, For network parameters, Represents the latent features sampled ,noise and time step Expectations It represents the square of the L2 norm, which measures the difference between predicted noise and actual noise; In the appearance reconstruction model, the objective function is defined using the reconstruction error, and the calculation formula is as follows: in, The original input image, To reconstruct the image; the final overall objective function is: in, and This is a hyperparameter used to control the weight of each loss term in the total loss.

Images