A Video Anomaly Detection Method Based on a Multimodal Conditional Diffusion Model

Abstract

This invention discloses a video anomaly detection method based on a multimodal conditional diffusion model, belonging to the field of video anomaly detection technology. The method involves: preprocessing the video to generate a video spatiotemporal cube and an optical flow spatiotemporal cube; constructing a diffusion model to obtain anomaly scores; weighting the reconstructed anomaly scores and predicted anomaly scores; and evaluating the final anomaly scores to complete video anomaly detection. This invention combines optical flow reconstruction and video frame prediction into a multi-task approach, making anomalies easier to distinguish; it integrates multimodal data to capture more dimensional feature information, improving the accuracy of anomaly detection; it makes the model more stable when dealing with noise and complex backgrounds, improving the stability of anomaly detection; and it uses the reconstructed optical flow as a condition to guide future frame prediction, widening the prediction error between anomaly prediction frames and original frames while narrowing the prediction error between normal prediction frames and original frames, ensuring training effectiveness and performance.

Description

Technical Field

[0001] This invention belongs to the field of video anomaly detection technology, specifically relating to a video anomaly detection method based on a multimodal conditional diffusion model. Background Technology

[0002] Video anomaly detection is a crucial computer vision task, widely applied in security monitoring, intelligent transportation systems, and public safety. Traditional anomaly detection methods typically rely on manual feature extraction and rule-based models, which have limited performance when handling complex and varied scenes. In recent years, the rapid development of deep learning technology has provided new solutions for anomaly detection, with methods based on deep neural networks significantly outperforming traditional methods in terms of accuracy and robustness. However, effectively utilizing multimodal data and improving anomaly detection performance remains a challenging problem.

[0003] Multimodal data fusion technology leverages the complementary nature of information from different data sources to enhance a system's perception and understanding capabilities. In video anomaly detection, common multimodal data includes video frames, optical flow, audio, and sensor data. By combining multimodal data, dynamic changes and environmental information within the video can be captured more comprehensively, thereby improving the accuracy of anomaly detection.

[0004] Diffusion models are a class of generative models that generate high-quality data samples from noise through a progressive denoising process. In recent years, diffusion models have made significant progress in image generation, video generation, and signal processing. The advantages of diffusion models lie in the stability of their generation process and their high fitting ability to data distributions. This makes diffusion models perform exceptionally well when handling complex and high-dimensional data. However, applying diffusion models to video anomaly detection, especially combining them with multimodal data for conditional generation and anomaly evaluation, remains an emerging research direction.

[0005] In summary, the technical problem that this invention aims to solve is how to address the noise and complex background issues in video anomaly processing after fusing multimodal data, thereby improving the accuracy of anomaly detection. Summary of the Invention

[0006] The purpose of this invention is to provide a video anomaly detection method based on a multimodal conditional diffusion model to solve the problems mentioned in the background art.

[0007] The objective of this invention is achieved as follows: a video anomaly detection method based on a multimodal conditional diffusion model, characterized by the following steps:

[0008] Step S1: Preprocess the video to generate a video spatiotemporal cube and an optical flow spatiotemporal cube;

[0009] Step S2: Construct a diffusion model and obtain anomaly scores;

[0010] The diffusion model includes a reconstruction diffusion network module and a conditional prediction diffusion network module;

[0011] Step S2-1: Input the optical flow spacetime cube into the reconstruction diffusion network module for training to obtain the reconstructed optical flow spacetime cube and obtain the reconstruction anomaly score;

[0012] Step S2-2: Input the video spatiotemporal cube and its corresponding reconstructed optical flow spatiotemporal cube into the conditional prediction diffusion network module to predict the next frame and obtain the prediction anomaly score;

[0013] Step S3: Weight the reconstructed anomaly score and the predicted anomaly score to obtain the final anomaly score;

[0014] Step S4: Evaluate the obtained final anomaly score to complete the video anomaly detection.

[0015] Preferably, in step S1, the video is preprocessed to generate a video spatiotemporal cube and an optical flow spatiotemporal cube. Specifically, the operation is as follows:

[0016] The input video segment is decomposed into a frame sequence, and the YOLOv8 object detector is used to perform object detection on the frame sequence.

[0017] Based on the object detection results, continuous original images and optical flow maps are cropped to construct a video spatiotemporal cube V and an optical flow spatiotemporal cube F centered on the object, where V is {v0, v1, ..., v...}. k-1 ,v k ,v k+1}, F is {f0,f1,…,f k-1 ,f k};

[0018] Optical flow extraction is performed on the frame sequence using the pre-trained optical flow extractor Flownet 2.0.

[0019] Preferably, the reconstruction diffusion network module and the conditional prediction diffusion network use the U-net architecture, which includes linear layers, 2D convolutional layers, self-attention layers, and group normalization layers.

[0020] The reconstruction diffusion network module adds two independent branch structures to the downsampling path and upsampling path on the U-net architecture, including a global branch and a local branch; the global branch focuses on extracting global motion information through larger convolution kernels and more pooling operations.

[0021] The local branches preserve local motion details through smaller convolution kernels and fewer pooling operations;

[0022] The conditional prediction diffusion network incorporates a dynamic interaction path and a temporal feature aggregation module into the U-net architecture. The dynamic interaction path utilizes a cross-attention mechanism to calculate the similarity between video frames and reconstructed optical flow features. The temporal feature aggregation module uses a self-attention mechanism to perform feature fusion on each frame in the time series.

[0023] Preferably, in step S2-1, the optical flow spatiotemporal cube is input into the reconstruction diffusion network module for training to obtain the reconstructed optical flow spatiotemporal cube and the reconstruction anomaly score. Specifically, the operation is as follows:

[0024] Step S2-1-1: Input the optical flow spatiotemporal cube F into the reconstruction diffusion network module for training. The goal of the training process is to learn an approximation of the original data distribution q(F). 0 p) θ (F 0 );

[0025] Define the forward diffusion process as follows:

[0026]

[0027] Where t∈[1,T], it represents the diffusion time step; q(F t |F t-1 The distribution follows the normal distribution;

[0028]

[0029] Where, α t ∈(0,1) represents the control F t-1 The hyperparameter α, which is the percentage weight of noise, increases with time step t. t Gradually decrease; I is the identity matrix;

[0030]

[0031] in, F represents a standard Gaussian noise. t Represented as

[0032] Step S2-1-2: Optimize our reconstructed diffusion network module θ by minimizing the variational bound L;

[0033] The definition is as follows:

[0034] Where, ∈ θ (F t,t) represents the predicted noise at time step t; To obtain the average value in the formula;

[0035] Step S2-1-3: Input F into the reconstructed diffusion model during the backsampling process. t Calculate

[0036]

[0037] Where, σ t The noise intensity parameter representing the time step is ∈ random noise with a standard normal distribution; through continuous iteration of the formula, the diffusion model outputs the reconstructed optical flow cube;

[0038] Step S2-1-4: Use the mean squared error (MSE) to calculate the reconstructed outlier score, as follows:

[0039]

[0040] in, Output the reconstructed optical flow cube to reconstruct the diffusion model; F 0 Let F be an optical flow cube: {f0,f1,…,f k-1 ,f k}

[0041] Preferably, in step S3, the reconstructed anomaly score and the predicted anomaly score are weighted and calculated to obtain the final anomaly score. Specifically, the operation is as follows:

[0042] Step S3-1: For the input video spatiotemporal cube V, perform calculations on its v-th... k+1 Frames are executed with forward propagation; Keep the original form and compare it with the reconstructed form. Connect them in the corresponding channel dimension;

[0043] The forward diffusion process is defined as follows:

[0044]

[0045] Where t∈[1,T] represents the diffusion time step, α t ∈(0,1) represents control and the percentage of noise ∈ This represents a standard Gaussian noise;

[0046] Step S3-2: Train and optimize the conditional prediction diffusion network module β by minimizing the variational lower bound L;

[0047] Step S3-3: Calculate the original value using the mean squared error (MSE). The predictions obtained by the conditional prediction diffusion network module The difference between them is used as the predicted anomaly score.

[0048] Preferably, step S3-2 involves training and optimizing the conditional prediction diffusion network module β by minimizing the variational lower bound L, with the specific operation formula as follows:

[0049]

[0050] For a trained conditional prediction diffusion network module, the input... and And samples were taken to obtain in, The diffusion state of the (k+1)th frame in the input video spatiotemporal cube at time step t; Represents the original video spatiotemporal cube from frame 0 to frame k; The optical flow spacetime cube obtained by reconstructing the diffusion network; For the conditional prediction diffusion network module, at time step t-1, it performs a conditional prediction on frame v. k+1 The predicted value;

[0051] Through continuous iteration of this process, the model predictions are finally calculated. Its formula is defined as follows:

[0052]

[0053] Where, σ t The noise intensity parameter represents the time step t, which is random noise distributed in a standard normal distribution.

[0054] Preferably, in step S3-3, the original mean square error (MSE) is used to calculate the original value. The predictions obtained by the conditional prediction diffusion network module The difference between them is used as the predicted anomaly score, and the specific formula is as follows:

[0055]

[0056] Among them, S p To predict outlier scores; The input is the raw image data of the (k+1)th frame in the video spatiotemporal cube; The image is the (k+1)th frame predicted by the conditional diffusion prediction network model.

[0057] Preferably, in step S4, the obtained final anomaly score is evaluated to complete the video anomaly detection. The specific operation is as follows:

[0058] The reconstruction anomaly score obtained from the reconstruction diffusion network and the prediction anomaly score obtained from the conditional prediction diffusion network module are normalized to obtain the following result. and Then, a weighted calculation is performed to obtain the final anomaly score S, which is used for anomaly assessment. The specific formula is defined as follows:

[0059]

[0060] Among them, w p The weighting parameter represents the predicted outlier score; w r w represents the weighting parameter for reconstructing outlier scores. r ,w p ∈[0,1],w r +w p =1.

[0061] Compared with the prior art, the present invention has the following improvements and advantages:

[0062] 1. By adopting a multi-task approach combining optical flow reconstruction and video frame prediction, anomalies are more easily distinguished. At the same time, by fusing multimodal data, more dimensional feature information is captured, significantly improving the accuracy of anomaly detection. The stepwise denoising process using the improved diffusion model makes the model more stable when dealing with noise and complex backgrounds, improving the stability of anomaly detection.

[0063] 2. Using the reconstructed optical flow as a condition to guide the prediction of future frames will further amplify the prediction error between abnormal prediction frames and the original frames, and reduce the prediction error between normal prediction frames and the original frames. This helps the network better distinguish between normal and abnormal modes in the video, and can ensure the training effect and performance of the diffusion model. Attached Figure Description

[0064] Figure 1 This is a flowchart of the method of the present invention.

[0065] Figure 2 This is a network structure diagram of the diffusion model of this invention.

[0066] Figure 3 This is a schematic diagram showing the frame-level AUC comparison results based on Avenue.

[0067] Figure 4 This is a schematic diagram showing the frame-level AUC comparison results based on Ped2.

[0068] Figure 5 This is a schematic diagram showing the frame-level AUC comparison results based on ShanghaiTech. Detailed Implementation

[0069] The invention will be further summarized below with reference to the accompanying drawings.

[0070] like Figure 1 As shown, a video anomaly detection method based on a multimodal conditional diffusion model is presented. The method includes the following steps:

[0071] Step S1: Preprocess the video to generate a video spatiotemporal cube and an optical flow spatiotemporal cube;

[0072] The input video segment is decomposed into a frame sequence, and the YOLOv8 object detector is used to perform object detection on the frame sequence.

[0073] Based on the object detection results, continuous original images and optical flow maps are cropped to construct a video spatiotemporal cube V and an optical flow spatiotemporal cube F centered on the object, where V is {v0, v1, ..., v...}. k-1 ,v k ,v k+1}, F is {f0,f1,…,f k-1 ,f k We choose a value of k of 3, meaning V uses 5 frames of images and F uses 4 frames of optical flow.

[0074] Optical flow extraction is performed on the frame sequence using the pre-trained optical flow extractor Flownet 2.0.

[0075] Step S2: Construct a diffusion model and obtain anomaly scores;

[0076] The diffusion model comprises a reconstruction diffusion network module and a conditional prediction diffusion network module. The reconstruction diffusion network module uses a dual-branch structure based on an improved U-Net architecture. This module includes linear layers, 2D convolutional layers, self-attention layers, and group normalization layers. To improve reconstruction accuracy and robustness in handling different anomaly patterns, the improved U-Net adds two independent branches to both the downsampling and upsampling paths. The global branch, using larger convolutional kernels and more pooling operations, focuses on extracting global motion information, such as large-scale motion changes. This branch focuses on the global dynamics of the scene to capture the overall motion patterns during anomalies. The local branch, using smaller convolutional kernels and fewer pooling operations, preserves local motion details, such as gestures or changes in local objects. This branch pays more attention to the dynamics of local regions, thereby improving the reconstruction capability for subtle movements. This dual-branch structure dynamically fuses this information through an attention mechanism, ensuring comprehensive utilization of multi-scale features during the reconstruction process.

[0077] Step S2-1: Input the optical flow spacetime cube into the reconstruction diffusion network module for training to obtain the reconstructed optical flow spacetime cube and obtain the reconstruction anomaly score;

[0078] Step S2-1-1: Input the optical flow spatiotemporal cube F into the reconstruction diffusion network module for training. The goal of the training process is to learn an approximation of the original data distribution q(F). 0 p) θ (F 0 );

[0079] Define the forward diffusion process as follows:

[0080]

[0081] Where t∈[1,T], it represents the diffusion time step; q(F t |F t-1 The distribution follows the normal distribution;

[0082]

[0083] Where, α t ∈(0,1) represents the control F t-1 The hyperparameter α, which is the percentage weight of noise, increases with time step t. t Gradually decrease; I is the identity matrix;

[0084]

[0085] in, F represents a standard Gaussian noise. t Represented as

[0086] Step S2-1-2: Optimize our reconstructed diffusion network module θ by minimizing the variational bound L;

[0087] The definition is as follows:

[0088] Where, ∈ θ (F t ,t) represents the predicted noise at time step t; To obtain the average value in the formula;

[0089] Step S2-1-3: Input F into the reconstructed diffusion model during the backsampling process. t Calculate

[0090]

[0091] Where, σ t The noise intensity parameter representing the time step is ∈ random noise with a standard normal distribution; through continuous iteration of the formula, the diffusion model outputs the reconstructed optical flow cube;

[0092] Step S2-1-4: Use the mean squared error (MSE) to calculate the reconstructed outlier score, as follows:

[0093]

[0094] in, Output the reconstructed optical flow cube to reconstruct the diffusion model; F 0 Let F be an optical flow cube: {f0,f1,…,f k-1 ,f k}

[0095] Step S2-2: Input the video spatiotemporal cube and its corresponding reconstructed optical flow spatiotemporal cube into the conditional prediction diffusion network module to predict the next frame and obtain the prediction anomaly score;

[0096] like Figure 2 As shown, step S3: The reconstructed anomaly score and the predicted anomaly score are weighted and calculated to obtain the final anomaly score;

[0097] A conditional prediction diffusion network module is used to perform weighted calculations on the reconstructed anomaly scores and the predicted anomaly scores. The conditional prediction diffusion network module uses an improved U-net architecture, which incorporates the following two modules:

[0098] Dynamic Interaction Path: This path utilizes a cross-attention mechanism to calculate the similarity between video frames and reconstructed optical flow features. Specifically, a similarity matrix is ​​generated through dot product, and the influence of reconstructed optical flow features is dynamically adjusted according to their weights. This process enables the model to adaptively emphasize reconstructed optical flow information related to the motion of the current frame at each time step, enhancing its understanding of motion patterns.

[0099] Temporal Feature Aggregation Module: This module uses a self-attention mechanism to fuse features for each frame in the time series. By calculating the importance of features in each frame, it selectively aggregates historical information, thereby enhancing attention to key time steps. This design enables the model to more accurately capture long-term and short-term dependencies, improving the sensitivity of anomaly detection.

[0100] The specific steps are as follows:

[0101] Step S3-1: For the input video spatiotemporal cube V, perform calculations on its v-th... k+1 Frames are executed with forward propagation; Keep the original form and compare it with the reconstructed form. Connect them in the corresponding channel dimension;

[0102] The forward diffusion process is defined as follows:

[0103]

[0104] Where t∈[1,T] represents the diffusion time step, α t ∈(0,1) represents control and the percentage of noise ∈ This represents a standard Gaussian noise;

[0105] Step S3-2: Train and optimize the conditional prediction diffusion network module β by minimizing the variational lower bound L. The specific operation formula is as follows:

[0106]

[0107] For a trained conditional prediction diffusion network module, the input... and And samples were taken to obtain in, The diffusion state of the (k+1)th frame in the input video spatiotemporal cube at time step t; Represents the original video spatiotemporal cube from frame 0 to frame k; The optical flow spacetime cube obtained by reconstructing the diffusion network; For the conditional prediction diffusion network module, at time step t-1, it performs a conditional prediction on frame v. k+1 The predicted value;

[0108] Through continuous iteration of this process, the model predictions are finally calculated. Its formula is defined as follows:

[0109]

[0110] Where, σ t The noise intensity parameter represents the time step t, ∈ random noise with a standard normal distribution;

[0111] In step S3-3, the original mean square error (MSE) is used to calculate the original value. The predictions obtained by the conditional prediction diffusion network module The difference between them is used as the predicted anomaly score, and the specific formula is as follows:

[0112]

[0113] Among them, S p To predict outlier scores; The input is the raw image data of the (k+1)th frame in the video spatiotemporal cube; The image is the (k+1)th frame predicted by the conditional diffusion prediction network model.

[0114] Step S4: Evaluate the obtained final anomaly score to complete the video anomaly detection.

[0115] In step S4, the final anomaly score is evaluated to complete the video anomaly detection. The specific operation is as follows:

[0116] The reconstruction anomaly score obtained from the reconstruction diffusion network and the prediction anomaly score obtained from the conditional prediction diffusion network module are normalized to obtain the following result. and Then, a weighted calculation is performed to obtain the final anomaly score S, which is used for anomaly assessment. The specific formula is defined as follows:

[0117]

[0118] Among them, w p The weighting parameter represents the predicted outlier score; w r w represents the weighting parameter for reconstructing outlier scores. r ,w p ∈[0,1],w r +w p =1.

[0119] This invention application uses the above method to calculate the anomaly score of the anomaly detection model to evaluate the model performance, thereby ensuring the training effect and performance of the anomaly detection model.

[0120] To better illustrate the advantages of the technical solution of the present invention, the following experiments are disclosed in this embodiment:

[0121] Experimental setup

[0122] Datasets: The performance of this method was evaluated using three benchmark datasets widely used in video anomaly detection: Ped2, Avenue, and ShanghaiTech.

[0123] Evaluation Metric: Frame-level AUC (area under the ROC curve) is used as the primary evaluation metric to objectively measure algorithm performance. A higher frame-level AUC value indicates a better performance of the algorithm in distinguishing between abnormal and normal frames.

[0124] Experimental details: When performing object detection with YOLOv8, different confidence thresholds were set for the Ped2, Avenue, and ShanghaiTech datasets, at values ​​of 0.3, 0.5, and 0.5, respectively. The spatiotemporal cube is composed of tensors of size 3(2)×T×32×32, where T represents the number of frames. We set the diffusion time step t of the diffusion model to 1000, and the other diffusion model parameters were set the same as in DDIM. Our framework was implemented using PyTorch, and the parameters were optimized using the Adam optimizer with a learning rate of 0.0002 and a batch size of 128.

[0125] Experimental results

[0126] Table 1 compares our proposed method with state-of-the-art methods from recent years. Performance data for these benchmark methods are derived from their respective original papers. We report the micro-average AUC on three commonly used public datasets (Ped2, Avenue, and ShanghaiTech). Table 1 shows that our proposed method achieves state-of-the-art performance on all three datasets, demonstrating its effectiveness.

[0127] Table 1. Comparison of frame-level AUC (%) on three benchmark datasets. The best performance on each dataset is indicated in bold:

[0128]

[0129]

[0130] Combining Table 1 and Figure 3 , Figure 4 , Figure 5 To elaborate:

[0131] The AUC of the three widely used public datasets mentioned above: Ped2, Avenue, and ShanghaiTech. (See Table 1 and...) Figure 3 , Figure 4 , Figure 5 As can be seen, compared with other methods, our method achieves the best performance on all three datasets, which proves the effectiveness of our method.

[0132] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A video anomaly detection method based on a multimodal conditional diffusion model, characterized in that: The method includes the following steps: Step S1: Preprocess the video to generate a video spatiotemporal cube and an optical flow spatiotemporal cube; Step S2: Construct a diffusion model and obtain anomaly scores; The diffusion model includes a reconstruction diffusion network module and a conditional prediction diffusion network module; the reconstruction diffusion network module and the conditional prediction diffusion network use the U-net architecture, which includes linear layers, 2D convolutional layers, self-attention layers, and group normalization layers. The reconstruction diffusion network module adds two independent branch structures to the downsampling path and upsampling path on the U-net architecture, including a global branch and a local branch; the global branch focuses on extracting global motion information through larger convolution kernels and more pooling operations. The local branches preserve local motion details through smaller convolution kernels and fewer pooling operations; The conditional prediction diffusion network incorporates a dynamic interaction path and a temporal feature aggregation module into the U-net architecture. The dynamic interaction path utilizes a cross-attention mechanism to calculate the similarity between video frames and reconstructed optical flow features. The temporal feature aggregation module uses a self-attention mechanism to perform feature fusion on each frame in the time series. Step S2-1: Input the optical flow spacetime cube into the reconstruction diffusion network module for training to obtain the reconstructed optical flow spacetime cube and obtain the reconstruction anomaly score. The specific operation is as follows: Step S2-1-1: Input the optical flow spatiotemporal cube F into the reconstruction diffusion network module for training. The goal of the training process is to learn an approximation of the original data distribution q(F). 0 p) θ (F 0 ); Define the forward diffusion process as follows: Where t∈[1,T], it represents the diffusion time step; q(F t |F t-1 The distribution follows the normal distribution; Where, α t ∈(0,1) represents the control F t-1 The hyperparameter α, which is the percentage weight of noise, increases with time step t. t Gradually decrease; I is the identity matrix; in, F represents a standard Gaussian noise. t Represented as Step S2-1-2: Optimize our reconstructed diffusion network module θ by minimizing the variational bound L; The definition is as follows: Where, ∈ θ (F t ,t) represents the predicted noise at time step t; To obtain the average value in the formula; Step S2-1-3: Input F into the reconstructed diffusion model during the backsampling process. t Calculate Where, σ t The noise intensity parameter represents the time step, and ∈ represents random noise with a standard normal distribution; through continuous iteration of the formula, the diffusion model outputs the reconstructed optical flow cube; Step S2-1-4: Use the mean squared error (MSE) to calculate the reconstructed outlier score, as follows: in, The reconstructed optical flow spacetime cube is output to reconstruct the diffusion model; F 0 Let F be an optical flow spacetime cube: {f0,f1,…,f k-1 ,f k } Step S2-2: Input the video spatiotemporal cube and its corresponding reconstructed optical flow spatiotemporal cube into the conditional prediction diffusion network module to predict the next frame and obtain the prediction anomaly score; Step S3: Weight the reconstructed anomaly score and the predicted anomaly score to obtain the final anomaly score; Step S4: Evaluate the obtained final anomaly score to complete the video anomaly detection.

2. The video anomaly detection method based on a multimodal conditional diffusion model according to claim 1, characterized in that: In step S1, the video is preprocessed to generate a video spatiotemporal cube and an optical flow spatiotemporal cube. The specific operations are as follows: The input video segment is decomposed into a frame sequence, and the YOLOv8 object detector is used to perform object detection on the frame sequence. Based on the object detection results, continuous original images and optical flow maps are cropped to construct a video spatiotemporal cube V and an optical flow spatiotemporal cube F centered on the object, where V is {v0, v1, ..., v...}. k-1 ,v k ,v k+1 }, F is {f0,f1,…,f k-1 ,f k }; Optical flow extraction is performed on the frame sequence using the pre-trained optical flow extractor Flownet 2.

0.

3. The video anomaly detection method based on a multimodal conditional diffusion model according to claim 1, characterized in that: In step S3, the reconstructed anomaly score and the predicted anomaly score are weighted and calculated to obtain the final anomaly score. The specific operation is as follows: Step S3-1: For the input video spatiotemporal cube V, perform calculations on its v-th... k+1 Frames are executed with forward propagation; Keep the original form and compare it with the reconstructed form. Connect them in the corresponding channel dimension; The forward diffusion process is defined as follows: Where t∈[1,T] represents the diffusion time step, α t ∈(0,1) represents control The hyperparameters for the percentage weights of noise ∈ This represents a standard Gaussian noise; The diffusion state of the (k+1)th frame in the input video spatiotemporal cube at time step t; For the conditional prediction diffusion network, at time step t-1, the frame v k+1 The predicted value; The input is the raw image data of the (k+1)th frame in the video spatiotemporal cube; Step S3-2: Train and optimize the conditional prediction diffusion network β by minimizing the variational lower bound L; Step S3-3: Calculate the original value using the mean squared error (MSE). The prediction obtained by the conditional prediction diffusion network The difference between them is used as the predicted anomaly score.

4. The video anomaly detection method based on a multimodal conditional diffusion model according to claim 3, characterized in that: Step S3-2: The conditional prediction diffusion network β is trained and optimized by minimizing the variational lower bound L. The specific operation formula is as follows: For a trained conditional prediction diffusion network, the input... and And samples were taken to obtain in, The diffusion state of the (k+1)th frame in the input video spatiotemporal cube at time step t; Represents the original video spatiotemporal cube from frame 0 to frame k; The optical flow spacetime cube obtained by reconstructing the diffusion network; For the conditional prediction diffusion network, at time step t-1, the frame v k+1 The predicted value; Through continuous iteration of this process, the model predictions are finally calculated. Its formula is defined as follows: Where, σ t The hyperparameter represents the percentage weight of the time step t and the noise, ∈ random noise with a standard normal distribution.

5. The video anomaly detection method based on a multimodal conditional diffusion model according to claim 3, characterized in that: In step S3-3, the original mean square error (MSE) is used to calculate the original value. The prediction obtained by the conditional prediction diffusion network The difference between them is used as the predicted anomaly score, and the specific formula is as follows: Among them, S p To predict outlier scores; The input is the raw image data of the (k+1)th frame in the video spatiotemporal cube; The image is the (k+1)th frame predicted by the conditional diffusion prediction network model.

6. The video anomaly detection method based on a multimodal conditional diffusion model according to claim 1, characterized in that: In step S4, the final anomaly score is evaluated to complete the video anomaly detection. The specific operation is as follows: The reconstruction anomaly score obtained from the reconstruction diffusion network and the prediction anomaly score obtained from the conditional prediction diffusion network are normalized to obtain the following: and Then, a weighted calculation is performed to obtain the final anomaly score S, which is used for anomaly assessment. The specific formula is defined as follows: Among them, w p The weighting parameter represents the predicted outlier score; w r w represents the weighting parameter for reconstructing outlier scores. r ,w p ∈[0,1],w r +w p =1.

Images