Video anomaly detection method based on multi-modal conditional diffusion model

Abstract

The invention discloses a video anomaly detection method based on a multi-modal condition diffusion model, and belongs to the technical field of video anomaly detection. The method comprises the following steps: preprocessing a video to generate a video space-time cube and an optical flow space-time cube; constructing a diffusion model, and obtaining an abnormal score; performing weighted calculation on the reconstruction anomaly score and the prediction anomaly score; and evaluating the obtained final anomaly score to complete video anomaly detection. According to the invention, through a multi-task combination mode of optical flow reconstruction and video frame prediction, abnormity is easier to distinguish; multi-modal data are fused, feature information of more dimensions is captured, and the accuracy of anomaly detection is improved; the model is more stable when processing noise and complex backgrounds, and the stability of anomaly detection is improved; the reconstructed optical flow is used as a condition for guiding future frame prediction, the prediction error between an abnormal prediction frame and an original frame is expanded, the prediction error between a normal prediction frame and the original frame is reduced, and the training effect and performance are ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of video anomaly detection, and in particular relates to a video anomaly detection method based on a multimodal conditional diffusion model. Background Art

[0002] Video anomaly detection is a critical computer vision task that is widely used in security surveillance, intelligent transportation systems, and public safety. Traditional anomaly detection methods usually rely on manual feature extraction and rule-based models, which have limited performance in dealing with complex and diverse scenarios. In recent years, the rapid development of deep learning technology has provided new solutions for anomaly detection, among which methods based on deep neural networks significantly outperform traditional methods in accuracy and robustness. However, how to effectively utilize multimodal data and improve the performance of anomaly detection remains a challenging problem.

[0003] Multimodal data fusion technology can utilize the information complementarity of different data sources to improve the system's perception and understanding capabilities. In video anomaly detection, common multimodal data includes video frames, optical flow, audio, and sensor data. By combining multimodal data, dynamic changes and environmental information in the video can be more comprehensively captured, thereby improving the accuracy of anomaly detection.

[0004] Diffusion Model is a kind of generative model that can generate high-quality data samples from noise through a gradual denoising process. In recent years, diffusion models have made significant progress in the fields of image generation, video generation, and signal processing. The advantages of diffusion models lie in the stability of their generation process and their high fitting ability to data distribution. This makes diffusion models perform well in processing complex and high-dimensional data. However, applying diffusion models to video anomaly detection, especially combining multimodal data for conditional generation and anomaly assessment, is still an emerging research direction.

[0005] In summary, how to solve the processing noise and complex background problems of video anomalies after fusing multimodal data, thereby improving the accuracy of anomaly detection is the technical problem that the present invention aims to solve. Summary of the invention

[0006] The purpose of the present invention is to provide a video anomaly detection method based on a multimodal conditional diffusion model to solve the problems raised in the above background technology.

[0007] The object of the present invention is achieved by: a video anomaly detection method based on a multimodal conditional diffusion model, characterized in that the method comprises the following steps:

[0008] Step S1: preprocessing the video to generate a video space-time cube and an optical flow space-time cube;

[0009] Step S2: construct a diffusion model and obtain anomaly scores;

[0010] The diffusion model includes a reconstruction diffusion network module and a conditional prediction diffusion network module;

[0011] Step S2-1: Input the optical flow space-time cube into the reconstruction diffusion network module for training, obtain the reconstructed optical flow space-time cube, and obtain the reconstruction anomaly score;

[0012] Step S2-2: Input the video space-time cube and its corresponding reconstructed optical flow space-time cube into the conditional prediction diffusion network module to predict the next frame and obtain the predicted anomaly score;

[0013] Step S3: Perform weighted calculation on the reconstructed anomaly score and the predicted anomaly score to obtain the final anomaly score;

[0014] Step S4: Evaluate the final anomaly score obtained to complete video anomaly detection.

[0015] Preferably, in step S1, the video is preprocessed to generate a video space-time cube and an optical flow space-time cube, and the specific operations are:

[0016] The input video clip is decomposed into a frame sequence, and the object detector YOLOv8 is used to perform object detection on the frame sequence;

[0017] Based on the object detection results, the continuous original images and optical flow maps are cropped to form the object-centered video spatiotemporal cube V and optical flow spatiotemporal cube F, where V is {v 0 ,v 1 ,…,v k-1 ,v k ,v k+1}, F is {f 0 ,f 1 ,…,f k-1 ,f k};

[0018] Optical flow extraction is performed on the frame sequence using the pre-trained optical flow extractor Flownet 2.0.

[0019] Preferably, the reconstruction diffusion network module and the conditional prediction diffusion network use a U-net architecture, which includes a linear layer, a 2D convolutional layer, a self-attention layer, and a group normalization layer;

[0020] The reconstruction diffusion network module adds two independent branch structures to the downsampling path and upsampling path on the U-net architecture, including a global branch and a local branch; the global branch focuses on extracting global motion information through larger convolution kernels and more pooling operations;

[0021] The local branch retains local motion detail information through smaller convolution kernels and fewer pooling operations;

[0022] The conditional prediction diffusion network adds a dynamic interaction path and a temporal feature aggregation module to the U-net architecture. The dynamic interaction path uses a cross-attention mechanism to calculate the similarity between video frames and reconstructed optical flow features, and the temporal feature aggregation module uses a self-attention mechanism to perform feature fusion on each frame in the time series.

[0023] Preferably, in step S2-1, the optical flow space-time cube is input into the reconstruction diffusion network module for training to obtain the reconstructed optical flow space-time cube and the reconstruction anomaly score, and the specific operation is:

[0024] Step S2-1-1: Input the optical flow space-time cube F into the reconstruction diffusion network module for training. The goal of the training process is to learn to approximate the original data distribution q(F 0 ) θ (F 0 );

[0025] Define the diffusion forward process, the diffusion forward process is:

[0026]

[0027] Where t∈[1,T] represents the diffusion time step; q(F t ∣F t-1 ) is the following normal distribution;

[0028]

[0029] Among them, α t ∈(0,1) represents the control F t-1 and the percentage weight of the noise, as time step t increases α t Gradually decreases; I is the unit matrix;

[0030]

[0031] in, represents a standard Gaussian noise, F t Expressed as

[0032] Step S2-1-2: Optimize our reconstruction diffusion network module θ by minimizing the variational bound L;

[0033] The definition is as follows:

[0034] Among them, ∈ θ (F t ,t) represents the prediction noise at time step t; To obtain the average value in the formula;

[0035] Step S2-1-3: Input F to the reconstructed diffusion model during the reverse sampling process t , calculate

[0036]

[0037] Among them, σ t The noise intensity parameter representing the time step is ∈ the random noise of the standard normal distribution; through the continuous iteration of the formula, the diffusion model is reconstructed to output the reconstructed optical flow cube;

[0038] Step S2-1-4: Use the mean square error MSE to calculate the reconstruction anomaly score, which is calculated as follows:

[0039]

[0040] in, Output the reconstructed optical flow cube for the reconstruction diffusion model; F 0 For the optical flow cube F: {f 0 ,f 1 ,…,f k-1 ,f k}.

[0041] Preferably, in step S3, the reconstructed anomaly score and the predicted anomaly score are weightedly calculated to obtain a final anomaly score, and the specific operation is:

[0042] Step S3-1: For the input video space-time cube V, perform the v-th k+1 The frame performs forward diffusion; Keep the original form and compare it with the reconstructed Connect them correspondingly in the channel dimension;

[0043] The forward diffusion process is defined as follows:

[0044]

[0045] Where t∈[1,T] represents the diffusion time step, α t ∈(0,1) indicates control and the percentage of noise ∈, represents a standard Gaussian noise;

[0046] Step S3-2: training the optimization conditional prediction diffusion network module β by minimizing the variational lower bound L;

[0047] Step S3-3: Calculate the original And the conditional prediction diffusion network module predicts The difference between them is used as the predicted anomaly score.

[0048] Preferably, the step S3-2: training the optimization condition prediction diffusion network module β by minimizing the variational lower bound L, the specific operation formula is as follows:

[0049]

[0050] For the trained conditional prediction diffusion network module, input and And sample to get in, is the diffusion state of the k+1th frame in the input video space-time cube at time step t; Represents the original video space-time cube from frame 0 to frame k; is the optical flow space-time cube obtained by reconstructing the diffusion network; The conditional prediction diffusion network module predicts frame v at time step t-1 k+1 The predicted value of

[0051] Through the continuous iteration of this process, the model prediction is finally calculated. The formula definition is as follows:

[0052]

[0053] Among them, σ t represents the noise intensity parameter at time step t, ∈ random noise from standard normal distribution.

[0054] Preferably, in step S3-3, the original And the conditional prediction diffusion network module predicts The difference between them is used as the predicted anomaly score. The specific operation formula is as follows:

[0055]

[0056] Among them, S p To predict the anomaly score; is the original image data of the k+1th frame in the input video space-time cube; The k+1th frame image predicted by the conditional diffusion prediction network model.

[0057] Preferably, in step S4, the obtained final anomaly score is evaluated to complete the video anomaly detection, and the specific operations are as follows:

[0058] The reconstruction anomaly score obtained by the reconstruction diffusion network and the predicted anomaly score obtained by the conditional prediction diffusion network module are normalized to obtain and Then perform weighted calculation to obtain the final anomaly score S for anomaly assessment. The specific formula is defined as follows:

[0059]

[0060] Among them, w p represents the weight parameter for predicting the anomaly score; w r represents the weight parameter for reconstructing the anomaly score, w r ,w p ∈[0,1],w r +w p =1.

[0061] Compared with the prior art, the present invention has the following improvements and advantages:

[0062] 1. By adopting a multi-task combination of optical flow reconstruction and video frame prediction, anomalies are easier to distinguish; at the same time, multimodal data is integrated to capture feature information in more dimensions, significantly improving the accuracy of anomaly detection; the gradual denoising process of the improved diffusion model makes the model more stable when dealing with noise and complex backgrounds, improving the stability of anomaly detection.

[0063] 2. The reconstructed optical flow is used as a guide for future frame prediction, which will further increase the prediction error between the abnormal prediction frame and the original frame, and reduce the prediction error between the normal prediction frame and the original frame, which will help the network better distinguish the normal mode and abnormal mode of the video, and ensure the training effect and performance of the diffusion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Flow chart of the method of the present invention.

[0065] Figure 2 This is a network structure diagram of the diffusion model of the present invention.

[0066] Figure 3 The figure is a schematic diagram of the frame-level AUC comparison results based on Avenue.

[0067] Figure 4Schematic diagram of frame-level AUC comparison results based on Ped2.

[0068] Figure 5 This is a schematic diagram of the frame-level AUC comparison results based on ShanghaiTech. DETAILED DESCRIPTION

[0069] The present invention is further summarized below with reference to the accompanying drawings.

[0070] like Figure 1 As shown, a video anomaly detection method based on a multimodal conditional diffusion model comprises the following steps:

[0071] Step S1: preprocessing the video to generate a video space-time cube and an optical flow space-time cube;

[0072] The input video clip is decomposed into a frame sequence, and the object detector YOLOv8 is used to perform object detection on the frame sequence;

[0073] Based on the object detection results, the continuous original images and optical flow maps are cropped to form a video spatiotemporal cube V and an optical flow spatiotemporal cube F centered on the object, where V is {v 0 ,v 1 ,…,v k-1 ,v k ,v k+1}, F is {f 0 ,f 1 ,…,f k-1 ,f k}; We choose the size of k to be 3, that is, V uses 5 frames of images and F uses 4 frames of optical flow;

[0074] Optical flow extraction is performed on the frame sequence using the pre-trained optical flow extractor Flownet 2.0.

[0075] Step S2: construct a diffusion model and obtain anomaly scores;

[0076] The diffusion model includes a reconstruction diffusion network module and a conditional prediction diffusion network module. The reconstruction diffusion network module uses a dual-branch structure based on the improved U-Net architecture; the reconstruction diffusion network module includes a linear layer, a 2D convolution layer, a self-attention layer, and a group normalization layer; in order to improve the reconstruction accuracy and the robustness of processing different abnormal modes, the improved U-Net downsampling path and upsampling path add two independent branch structures respectively. The global branch focuses on extracting global motion information, such as large-scale motion changes, through larger convolution kernels and more pooling operations. This branch focuses on the global dynamics of the scene to capture the overall motion pattern during abnormalities; the local branch retains local motion detail information, such as changes in gestures or local objects, through smaller convolution kernels and fewer pooling operations. This branch pays more attention to the dynamics of local areas, thereby improving the reconstruction ability of subtle motion. The dual-branch structure dynamically fuses this information through the attention mechanism to ensure the comprehensive utilization of multi-scale features during the reconstruction process.

[0077] Step S2-1: Input the optical flow space-time cube into the reconstruction diffusion network module for training, obtain the reconstructed optical flow space-time cube, and obtain the reconstruction anomaly score;

[0078] Step S2-1-1: Input the optical flow space-time cube F into the reconstruction diffusion network module for training. The goal of the training process is to learn to approximate the original data distribution q(F 0 ) θ (F 0 );

[0079] Define the diffusion forward process, the diffusion forward process is:

[0080]

[0081] Where t∈[1,T] represents the diffusion time step; q(F t ∣F t-1 ) is the following normal distribution;

[0082]

[0083] Among them, α t ∈(0,1) represents the control F t-1 and the percentage weight of the noise, as time step t increases α t Gradually decreases; I is the unit matrix;

[0084]

[0085] in, represents a standard Gaussian noise, F t Expressed as

[0086] Step S2-1-2: Optimize our reconstruction diffusion network module θ by minimizing the variational bound L;

[0087] The definition is as follows:

[0088] Among them, ∈ θ (F t ,t) represents the prediction noise at time step t; To obtain the average value in the formula;

[0089] Step S2-1-3: Input F to the reconstructed diffusion model during the reverse sampling process t , calculate

[0090]

[0091] Among them, σ t The noise intensity parameter representing the time step is ∈ the random noise of the standard normal distribution; through the continuous iteration of the formula, the diffusion model is reconstructed to output the reconstructed optical flow cube;

[0092] Step S2-1-4: Use the mean square error MSE to calculate the reconstruction anomaly score, which is calculated as follows:

[0093]

[0094] in, Output the reconstructed optical flow cube for the reconstruction diffusion model; F 0 For the optical flow cube F: {f 0 ,f 1 ,…,f k-1 ,f k}

[0095] Step S2-2: Input the video space-time cube and its corresponding reconstructed optical flow space-time cube into the conditional prediction diffusion network module to predict the next frame and obtain the predicted anomaly score;

[0096] like Figure 2 As shown, step S3: weighted calculation of the reconstructed anomaly score and the predicted anomaly score to obtain a final anomaly score;

[0097] The conditional prediction diffusion network module is used to perform weighted calculations on the reconstructed anomaly score and the predicted anomaly score. The conditional prediction diffusion network module uses an improved U-net architecture, in which the following two modules are added:

[0098] Dynamic interaction path: This path uses the cross-attention mechanism to calculate the similarity between the video frame and the reconstructed optical flow features. Specifically, a similarity matrix is ​​generated through dot multiplication, and the influence of the reconstructed optical flow features is dynamically adjusted according to its weight. This process enables the model to adaptively emphasize the reconstructed optical flow information related to the current frame motion at each time step, enhancing the ability to understand motion patterns;

[0099] Temporal feature aggregation module: This module uses the self-attention mechanism to fuse features of each frame in the time series. By calculating the importance of each frame feature, historical information is selectively aggregated to increase attention to key time steps. This design enables the model to more accurately capture long-term and short-term dependencies and improve the sensitivity of anomaly detection.

[0100] The specific operations are:

[0101] Step S3-1: For the input video space-time cube V, perform the v-th k+1 The frame performs forward diffusion; Keep the original form and compare it with the reconstructed Connect them correspondingly in the channel dimension;

[0102] The forward diffusion process is defined as follows:

[0103]

[0104] Where t∈[1,T] represents the diffusion time step, α t ∈(0,1) indicates control and the percentage of noise ∈, represents a standard Gaussian noise;

[0105] Step S3-2: Train the optimized conditional prediction diffusion network module β by minimizing the variational lower bound L. The specific operation formula is as follows:

[0106]

[0107] For the trained conditional prediction diffusion network module, input and And sample to get in, is the diffusion state of the k+1th frame in the input video space-time cube at time step t; Represents the original video space-time cube from frame 0 to frame k; is the optical flow space-time cube obtained by reconstructing the diffusion network; The conditional prediction diffusion network module predicts frame v at time step t-1 k+1 The predicted value of

[0108] Through the continuous iteration of this process, the model prediction is finally calculated. The formula definition is as follows:

[0109]

[0110] Among them, σ t represents the noise intensity parameter of time step t, ∈ random noise of standard normal distribution;

[0111] In step S3-3, the original And the conditional prediction diffusion network module predicts The difference between them is used as the predicted anomaly score. The specific operation formula is as follows:

[0112]

[0113] Among them, S p To predict the anomaly score; is the original image data of the k+1th frame in the input video space-time cube; The k+1th frame image predicted by the conditional diffusion prediction network model.

[0114] Step S4: Evaluate the final anomaly score obtained to complete video anomaly detection.

[0115] In step S4, the final anomaly score obtained is evaluated to complete the video anomaly detection. The specific operations are as follows:

[0116] The reconstruction anomaly score obtained by the reconstruction diffusion network and the predicted anomaly score obtained by the conditional prediction diffusion network module are normalized to obtain and Then perform weighted calculation to obtain the final anomaly score S for anomaly assessment. The specific formula is defined as follows:

[0117]

[0118] Among them, w p represents the weight parameter for predicting the anomaly score; w r represents the weight parameter for reconstructing the anomaly score, w r ,w p ∈[0,1],w r +w p =1.

[0119] The present invention applies the method to calculate the anomaly score of the anomaly detection model to evaluate the model performance, so as to ensure the training effect and performance of the anomaly detection model.

[0120] In order to better illustrate the advantages of the technical solution of the present invention, the following experiments are disclosed in this embodiment:

[0121] Experimental setup

[0122] Datasets: Three benchmark datasets widely used for video anomaly detection, Ped2, Avenue, and ShanghaiTech, are used to evaluate the performance of our method.

[0123] Evaluation indicators: Frame-level AUC (area under the ROC curve) is used as the main evaluation indicator to objectively measure the performance of the algorithm. The higher the frame-level AUC value, the better the algorithm is in distinguishing abnormal and normal frames.

[0124] Experimental details: When performing object detection with YOLOv8, different confidence thresholds are set to 0.3, 0.5, and 0.5 in the Ped2, Avenue, and ShanghaiTech datasets, respectively. The spatiotemporal cube is composed of a 3(2)×T×32×32 tensor, where T represents the number of frames. We set the diffusion time step t of the diffusion model to 1000, and other diffusion model-related parameter settings are the same as in DDIM. Our framework is implemented in PyTorch and uses the Adam optimizer for parameter optimization, with the learning rate set to 0.0002 and the batch size to 128.

[0125] Experimental Results

[0126] In Table 1, our method is compared with the most advanced methods in recent years. The performance data of these baseline methods are all from their respective original papers. We report the micro-average AUC on three commonly used public datasets (Ped2, Avenue, and ShanghaiTech). Table 1 shows that compared with other methods, our method achieves the best performance on all three datasets, demonstrating the effectiveness of our method.

[0127] Table 1. Comparison of frame-level AUC (%) on three benchmark datasets. The best performance on each dataset is in bold:

[0128]

[0129]

[0130] Combining Table 1 and Figure 3 , Figure 4 , Figure 5 To elaborate:

[0131] The AUCs of the three widely used public datasets Ped2, Avenue, and ShanghaiTech are shown in Table 1 and Figure 3 , Figure 4 , Figure 5 It can be seen that compared with other methods, our method achieves the best performance on all three datasets, which proves the effectiveness of our method.

[0132] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A video anomaly detection method based on a multimodal conditional diffusion model, characterized in that: The method comprises the following steps: Step S1: preprocessing the video to generate a video space-time cube and an optical flow space-time cube; Step S2: construct a diffusion model and obtain anomaly scores; The diffusion model includes a reconstruction diffusion network module and a conditional prediction diffusion network module; Step S2-1: Input the optical flow space-time cube into the reconstruction diffusion network module for training, obtain the reconstructed optical flow space-time cube, and obtain the reconstruction anomaly score; Step S2-2: Input the video space-time cube and its corresponding reconstructed optical flow space-time cube into the conditional prediction diffusion network module to predict the next frame and obtain the predicted anomaly score; Step S3: Perform weighted calculation on the reconstructed anomaly score and the predicted anomaly score to obtain the final anomaly score; Step S4: Evaluate the final anomaly score obtained to complete video anomaly detection.

2. The video anomaly detection method based on a multimodal conditional diffusion model according to claim 1, characterized in that: In step S1, the video is preprocessed to generate a video space-time cube and an optical flow space-time cube. The specific operations are as follows: The input video clip is decomposed into a frame sequence, and the object detector YOLOv8 is used to perform object detection on the frame sequence; Based on the object detection results, the continuous original images and optical flow maps are cropped to form the object-centered video spatiotemporal cube V and optical flow spatiotemporal cube F, where V is {v0,v1,…,v k-1 ,v k ,v k+1 }, F is {f0,f1,…,f k-1 ,f k }; Optical flow extraction is performed on the frame sequence using the pre-trained optical flow extractor Flownet 2.

0.

3. The video anomaly detection method based on a multimodal conditional diffusion model according to claim 1, characterized in that: The reconstruction diffusion network module and the conditional prediction diffusion network use a U-net architecture, which includes a linear layer, a 2D convolutional layer, a self-attention layer, and a group normalization layer; The reconstruction diffusion network module adds two independent branch structures to the downsampling path and the upsampling path on the U-net architecture, including a global branch and a local branch; The global branch focuses on extracting global motion information through larger convolution kernels and more pooling operations; The local branch retains local motion detail information through smaller convolution kernels and fewer pooling operations; The conditional prediction diffusion network adds a dynamic interaction path and a temporal feature aggregation module to the U-net architecture. The dynamic interaction path uses a cross-attention mechanism to calculate the similarity between video frames and reconstructed optical flow features, and the temporal feature aggregation module uses a self-attention mechanism to perform feature fusion on each frame in the time series.

4. The video anomaly detection method based on a multimodal conditional diffusion model according to claim 1, characterized in that: In step S2-1, the optical flow space-time cube is input into the reconstruction diffusion network module for training to obtain the reconstructed optical flow space-time cube and the reconstruction anomaly score. The specific operation is: Step S2-1-1: Input the optical flow space-time cube F into the reconstruction diffusion network module for training. The goal of the training process is to learn to approximate the original data distribution q(F 0 ) θ (F 0 ); Define the diffusion forward process, the diffusion forward process is: Where t∈[1,T] represents the diffusion time step; q(F t ∣F t-1 ) is the following normal distribution; Among them, α t ∈(0,1) represents the control F t-1 and the percentage weight of the noise, as time step t increases α t Gradually decreases; I is the unit matrix; in, represents a standard Gaussian noise, F t Expressed as Step S2-1-2: Optimize our reconstruction diffusion network module θ by minimizing the variational bound L; The definition is as follows: Among them, ∈ θ (F t ,t) represents the prediction noise at time step t; To obtain the average value in the formula; Step S2-1-3: Input F to the reconstructed diffusion model during the reverse sampling process t , calculate Among them, σ t The noise intensity parameter representing the time step is ∈ the random noise of the standard normal distribution; through the continuous iteration of the formula, the diffusion model is reconstructed to output the reconstructed optical flow cube; Step S2-1-4: Use the mean square error MSE to calculate the reconstruction anomaly score, which is calculated as follows: in, Output the reconstructed optical flow cube for the reconstruction diffusion model; F 0 is the optical flow cube F: {f0,f1,…,f k-1 ,f k }.

5. The video anomaly detection method based on a multimodal conditional diffusion model according to claim 1, characterized in that: In step S3, the reconstructed anomaly score and the predicted anomaly score are weightedly calculated to obtain the final anomaly score. The specific operation is: Step S3-1: For the input video space-time cube V, perform the v-th k+1 The frame performs forward diffusion; Keep the original form and compare it with the reconstructed Connect them correspondingly in the channel dimension; The forward diffusion process is defined as follows: Where t∈[1,T] represents the diffusion time step, α t ∈(0,1) indicates control and the percentage of noise ∈, represents a standard Gaussian noise; Step S3-2: training the optimization conditional prediction diffusion network β by minimizing the variational lower bound L; Step S3-3: Calculate the original and the conditional prediction diffusion network prediction The difference between them is used as the predicted anomaly score.

6. The video anomaly detection method based on a multimodal conditional diffusion model according to claim 5, characterized in that: The step S3-2: training the optimization condition prediction diffusion network β by minimizing the variational lower bound L, the specific operation formula is as follows: For the trained conditional prediction diffusion network, input and And sample to get in, is the diffusion state of the k+1th frame in the input video space-time cube at time step t; Represents the original video space-time cube from frame 0 to frame k; is the optical flow space-time cube obtained by reconstructing the diffusion network; The conditional prediction diffusion network predicts frame v at time step t-1 k+1 The predicted value of Through the continuous iteration of this process, the model prediction is finally calculated. The formula definition is as follows: Among them, σ t represents the noise intensity parameter at time step t, ∈ random noise from standard normal distribution.

7. The video anomaly detection method based on a multimodal conditional diffusion model according to claim 5, characterized in that: In step S3-3, the original and the conditional prediction diffusion network prediction The difference between them is used as the predicted anomaly score. The specific operation formula is as follows: Among them, S p To predict the anomaly score; is the original image data of the k+1th frame in the input video space-time cube; The k+1th frame image predicted by the conditional diffusion prediction network model.

8. The video anomaly detection method based on a multimodal conditional diffusion model according to claim 1, characterized in that: In step S4, the obtained final anomaly score is evaluated to complete the video anomaly detection. The specific operations are as follows: The reconstruction anomaly score obtained by the reconstruction diffusion network and the prediction anomaly score obtained by the conditional prediction diffusion network are normalized to obtain and Then perform weighted calculation to obtain the final anomaly score S for anomaly assessment. The specific formula is defined as follows: Among them, w p represents the weight parameter for predicting the anomaly score; w r represents the weight parameter for reconstructing the anomaly score, w r ,w p ∈[0,1],w r +w p =1.

Images