Video anomaly detection method based on diffusion model and text embedding

Abstract

The invention discloses a video anomaly detection method based on a diffusion model and text embedding, and relates to the technical field of video anomaly detection. The invention provides a video anomaly detection method based on a diffusion model and text embedding, and the method comprises the steps: firstly extracting the text information of each video frame through a graphic and text generation model, and obtaining semantic features through a pre-trained CLIP text encoder; a video anomaly detection model is constructed based on an image diffusion model and a text diffusion model, an image diffusion denoising network and a text diffusion denoising network are respectively guided to reconstruct image features and text features by using a cross attention mechanism, model training is completed through overall optimization, and then video anomaly is detected. Therefore, by the adoption of the method, the problem of excessive generalization in a specific scene can be relieved by means of the powerful generation capacity of the diffusion model and the semantic understanding capacity of the graph-to-text model, the interpretability of video anomaly detection is enhanced, and the performance of the video anomaly detection model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video anomaly detection, and in particular to a video anomaly detection method based on diffusion model and text embedding. Background Art

[0002] Video anomaly detection aims to analyze the dominant patterns (normal) in the data, so as to detect a few "unusual" samples (abnormal). There are two main paradigms for the research on video anomaly detection, one is video anomaly detection based on weak supervision, and the other is video anomaly detection based on unsupervised supervision. Since abnormal samples are usually difficult to observe and have the characteristics of scarcity, the unsupervised video anomaly detection method is more suitable for the "abnormal" problem.

[0003] The field of unsupervised anomaly detection has a long research history. According to the chronological order of algorithm design and development, unsupervised anomaly detection algorithms can be roughly divided into methods based on manual features and methods based on deep learning.

[0004] (1) Methods based on handcrafted features. Traditional anomaly detection algorithms are based on designed handcrafted features. According to the different designed features, they can be divided into two categories: trajectory features and spatiotemporal features. Most of the methods based on handcrafted features are based on statistical models. They collect and count the distribution of normal patterns through training sets, and then regard samples that deviate from the distribution as anomalies during the test process. In addition, sparse coding or dictionary learning are also commonly used methods to encode normal patterns. The basic assumption of these methods is that after the dictionary encodes the normal pattern on the training set, any normal pattern can be linearly represented as a linear combination of the basic dictionary elements. Therefore, if the reconstruction error of a pattern is high, the pattern is considered anomaly. Although methods based on handcrafted features are robust in dealing with complex scenes, a common disadvantage of these methods is that they may not be able to detect long-term activities such as wandering. This is because wandering is related to a person's long-term global movement rather than local spatiotemporal movement patterns.

[0005] (2) Deep learning-based methods use deep learning methods such as autoencoders, recurrent neural networks, long short-term memory networks, and generative adversarial networks to extract video features and establish a prediction model for video anomaly detection. The assumption of this type of method is that abnormal frames should be more difficult to predict than normal frames. Therefore, for video frames containing anomalies, the peak signal-to-noise ratio (quality) between the model's predicted frames and the true frames is expected to be lower. However, all of these anomaly detections are based on the reconstruction / prediction of rule-based training data. Although all of these methods assume that abnormal events will lead to large reconstruction / prediction errors, this assumption is not necessarily true due to the good generalization ability of deep neural networks. This results in the reconstruction / prediction errors of some abnormal events being similar to those of normal events, thereby reducing the model's discriminability for anomalies.

[0006] In the unsupervised video anomaly detection task, the algorithm model only uses video samples containing a single (normal) pattern to train the model, without the need for abnormal samples. Researchers often propose some hypotheses to detect anomalies, and make normal samples meet the hypothesis through model training. During testing, samples that cannot meet the hypothesis are considered abnormal. Common hypothesis-based frameworks include: predicting future frames, reconstructing current frames, etc. However, simply modeling the patterns in normal samples has a large anomaly miss rate. Although in the unsupervised setting, there is only a single pattern (normal) data to train the model, the normal pattern still contains a variety of elements. How to encode these normal element information to comprehensively and compactly model the complete normal pattern is the key to solving the problem. Therefore, developing algorithms to model normal patterns from different aspects and angles and encode a variety of elements in the pattern is an effective means to alleviate the "over-generalization" phenomenon and improve the performance of video anomaly detection.

[0007] While ensuring accuracy, the video anomaly detection task has high requirements for the efficiency, real-time performance, and robustness of the algorithm. In actual applications, edge monitoring devices with limited computing power often require algorithms to quickly and effectively implement video anomaly detection functions while occupying small computing resources. At the same time, the complexity of abnormal events in real life requires the algorithm to perform a comprehensive analysis to detect all possible anomalies. In addition, a robust video anomaly detection algorithm also requires the model to be able to flexibly and quickly adapt to different scenarios. When monitoring equipment is installed in a brand new scenario, retraining and learning a new model is often time-consuming, labor-intensive, and impractical.

[0008] To this end, it is necessary to propose a new method that can alleviate the problem of model over-generalization and enhance interpretability by constraining the model training process. Summary of the invention

[0009] The purpose of the present invention is to provide a video anomaly detection method based on diffusion model and text embedding, so as to solve the problems of over-generalization and poor interpretability of existing video anomaly detection methods.

[0010] To achieve the above object, the present invention provides a video anomaly detection method based on diffusion model and text embedding, comprising:

[0011] S1. First, the text information of each video frame is extracted using the image-text generation model, and then the semantic features are obtained through the pre-trained CLIP text encoder;

[0012] S2. Based on the image diffusion model and the text diffusion model, a video anomaly detection model is constructed, and the model training is completed through overall optimization to detect video anomalies.

[0013] Preferably, the video anomaly detection model includes an image encoder, an image diffusion module, a text diffusion module and an image decoder.

[0014] Preferably, model training comprises the following steps:

[0015] S21, using the image encoder to obtain the hidden features of the input video frame image, and then processing it through the image diffusion module, and introducing semantic features to guide the image diffusion denoising network using the cross attention mechanism to obtain the reconstructed image features;

[0016] S22. Use the text diffusion model to process the acquired semantic features, introduce the true value image, and use the cross-attention mechanism to guide the text diffusion denoising network to obtain the reconstructed text features.

[0017] Preferably, image feature reconstruction loss:

[0018]

[0019] Where, L pred-i represents the image feature reconstruction loss, I E represents the input image features, Represents the reconstructed image features;

[0020] Text feature reconstruction loss:

[0021]

[0022] Where, L pred-t represents the text feature reconstruction loss, T E is the input text feature, is the reconstructed text feature.

[0023] Preferably, the loss L of the image diffusion module is LDM-I for:

[0024]

[0025] In the formula, ∈ I is the image denoising network, ε represents noise, ε I represents the image denoising autoencoder, ε~N(0,1) indicates that the noise conforms to the Gaussian distribution, represents the hidden layer image features after adding noise in the tth step;

[0026] The loss L of the text diffusion module LDM-T for:

[0027]

[0028] Among them, ∈T is the text denoising network, Represents the text features after noise addition in the tth step.

[0029] Preferably, the total loss function of the model training is:

[0030] L all =L pred-i +λ1L pred-t +λ2L LDM-I +λ3L LDM-T ;

[0031] Where, L all represents the total loss function, λ1, λ2 and λ3 represent the weights of text feature reconstruction loss, image diffusion module loss and text diffusion module loss respectively.

[0032] Therefore, the present invention adopts the above-mentioned video anomaly detection method based on diffusion model and text embedding, which has the following technical effects:

[0033] (1) In the image denoising network, the semantic information of the image is used as a condition to guide the image reconstruction process through the cross-attention mechanism, so as to ensure the temporal consistency of the semantic information of the reconstructed video frame and avoid generating images that are too different from the video frame, which would cause the normal video frame to be reconstructed into an abnormal one. Similarly, in the text denoising network, the true value map of the predicted image is used as a condition to guide the reconstruction process of the text features through the cross-attention mechanism, so that the generated semantic information is closer to the features of the real video frame.

[0034] (2) The semantic features of video frame images are extracted and fused with image features, which alleviates the over-generalization problem in specific scenarios and can significantly enhance the interpretability of video anomaly detection. At the same time, the image diffusion module and the text diffusion module are combined and optimized as a whole to improve the performance of the video anomaly detection model.

[0035] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of the video anomaly detection method based on diffusion model and text embedding;

[0037] Figure 2 It is a network diagram of the video anomaly detection method based on diffusion model and text embedding. DETAILED DESCRIPTION

[0038] The present invention can be explained in more detail by the following examples. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following examples.

[0039] like Figure 1As shown, the present invention provides a video anomaly detection method based on diffusion model and text embedding, comprising the following steps:

[0040] S1. First, the pre-trained graph-to-text model BLIP-2 is used to extract the text description of the video frame by frame. Then the pre-trained CLIP text encoder is used to encode the video frame description to obtain semantic features. The semantic features of each frame of all videos in the training set and the test set are saved separately for subsequent model training.

[0041] Among them, fusing semantic features with image features can significantly improve the interpretability of video anomaly detection. Specifically, the task of unsupervised video anomaly detection stipulates that the model only inputs normal videos during training. By reconstructing the last frame of continuous video frames, the model can learn the normal mode of a specific scene. During testing, for abnormal videos, the effect of the last frame of the image reconstructed by the model is close to the normal mode, and the difference with the real abnormal video frame is large. In this case, the reconstructed video frame can be judged as abnormal. However, reconstruction error alone cannot explain why an abnormal judgment is made. The existing large image-to-text model can describe the main content of the image. The text description of normal video frames and abnormal video frames can intuitively distinguish between normal and abnormal, thereby improving the interpretability of video anomaly detection.

[0042] S2. The quality of video frame modeling directly affects the performance of video anomaly detection. Existing detection methods are mainly based on the autoencoder to encode continuous video frames, then model the normal mode on the intermediate layer features, and then decode the decoder to obtain the reconstructed image. However, whether it is based on the autoencoder or the memory mechanism, it will lead to the problem of over-generalization of specific scenes. The reason is that the powerful generalization ability of the convolutional neural network makes the normal mode learned by the model too single.

[0043] The diffusion model has a strong generation capability and the characteristics of generating sample diversity. Therefore, this embodiment uses the diffusion model to model normal videos in order to learn more diverse and complete normal modes. In addition, in the denoising process of the diffusion model, the previously extracted text features are fused with the image features after noise addition, and the cross-attention mechanism is used to guide the denoising network to generate images that are closer to the semantic information. In this way, the diversity of normal mode modeling can be met, while ensuring the consistency of the time sequence of the semantic information of the reconstructed video frame, avoiding the denoising network from generating images that are too different from the previous video frames, thereby reconstructing the normal video frame into an abnormality. Similarly, for the semantic features of the input continuous video frames, the diffusion model can be used to reconstruct the semantic features of the last frame. At this time, the true value image of the next frame of the input continuous video frame can be used as a condition to guide the reconstruction of the text features in the denoising process of text diffusion, so that the generated semantic information is closer to the features of the real video frame.

[0044] like Figure 2 As shown in Figure 1, the model of video anomaly detection includes an image encoder E, an image diffusion denoising network ∈I, a text diffusion denoising network ∈T, and an image decoder D. The input of the model is continuous video frame images i1, i2, ..., i k-1 .

[0045] S21, firstly, k-1 consecutive video frame images i1, i2, ..., i k-1 It is sent to the image feature encoder E to extract the corresponding features and obtain the hidden layer features of the input video frame image, which is represented by IE.

[0046] Then, I E Input into the diffusion model for noise addition. After T steps of noise addition, a noise image that conforms to the Gaussian distribution is obtained. Where T is the time step of the forward noise addition process.

[0047] After that, the image diffusion denoising network ∈ I Perform reverse denoising to predict The denoised image. Among them, ∈ I It is a network based on U-Net with semantic information as condition. The loss L of the image diffusion model LDM-I It is expressed as:

[0048]

[0049] In the formula, LDM-I represents the image diffusion model, ∈I is the image denoising network, ε represents noise, and ε I represents the image denoising autoencoder, ε~N(0,1) indicates that the noise conforms to the Gaussian distribution, represents the hidden layer image features after noise addition in the tth step, where t is obtained by uniformly sampling in (1, 2, …, T).

[0050] In the denoising process, the introduced text features enhance the reconstruction ability of the U-Net network through the cross-attention mechanism. The cross-attention is expressed as:

[0051]

[0052] Among them, Q, K, and V represent query, key, and value respectively. Q comes from the image hidden layer features, K and V come from text feature embedding, and d represents the feature dimension.

[0053] Where T K It is the semantic feature of the next frame image of the continuous input video frame. and are all trainable parameter matrices.

[0054] After denoising, the reconstructed image features are obtained Image feature reconstruction loss L pred-i It is expressed as:

[0055]

[0056] Among them, I E is the input image feature, is the reconstructed image feature.

[0057] S22. The process of text diffusion is similar to that of image diffusion. The text denoising network is a U-Net-based network ∈ T conditioned on the next frame of continuous video frames. Among them, the loss L of the text diffusion model is LDM-T It is expressed as:

[0058]

[0059] Among them, LDM-T represents the text diffusion model, ∈ T For the text denoising network, Represents the text features after adding noise in step t, and is predicted by the above objective function Denoised text.

[0060] In the denoising process, the introduced image features enhance the reconstruction ability of the U-Net network through the cross-attention mechanism. The cross-attention is expressed as:

[0061]

[0062] Among them, Q comes from text features, and K and V come from image hidden layer features.

[0063] in is the semantic feature after adding noise, I K is the hidden layer feature of the next frame image of the continuous input video frame.

[0064] After denoising, the reconstructed text features are obtained Among them, the text feature reconstruction loss L pred-t It is expressed as:

[0065]

[0066] Among them, T E is the input text feature, is the reconstructed text feature.

[0067] The image diffusion module and the text diffusion module are incorporated into an overall neural network to construct the video anomaly detection model of this embodiment, and its overall optimization objective function is:

[0068] L all =Lpred-i +λ1L pred-t +λ2L LDM-I +λ3L LDM-T ;

[0069] Where, L all is the total loss function, λ1, λ2 and λ3 are weight parameters.

[0070] The adamW optimizer is used to update the overall weight of the model, and several rounds of iterative training are performed to obtain encoding and decoding capabilities and image text reconstruction capabilities, and to ensure the accuracy of video anomaly detection, thereby realizing the training and testing of the video anomaly detection model.

[0071] Therefore, the present invention adopts the above-mentioned video anomaly detection method based on diffusion model and text embedding, which can alleviate the over-generalization problem in specific scenarios with the help of the powerful generation ability of the diffusion model and the semantic understanding ability of the graph-to-text model, enhance the interpretability of video anomaly detection, and improve the performance of the video anomaly detection model.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A video anomaly detection method based on diffusion model and text embedding, characterized in that: include: S1. First, the text information of each video frame is extracted using the image-text generation model, and then the semantic features are obtained through the pre-trained CLIP text encoder; S2. Based on the image diffusion model and the text diffusion model, a video anomaly detection model is constructed, and the model training is completed through overall optimization to detect video anomalies.

2. The video anomaly detection method based on diffusion model and text embedding according to claim 1 is characterized in that: The video anomaly detection model includes an image encoder, an image diffusion module, a text diffusion module and an image decoder.

3. The video anomaly detection method based on diffusion model and text embedding according to claim 2 is characterized in that: Model training includes the following steps: S21, using the image encoder to obtain the hidden features of the input video frame image, and then processing it through the image diffusion module, and introducing semantic features to guide the image diffusion denoising network using the cross attention mechanism to obtain the reconstructed image features; S22. Use the text diffusion model to process the acquired semantic features, introduce the true value image, and use the cross-attention mechanism to guide the text diffusion denoising network to obtain the reconstructed text features.

4. The video anomaly detection method based on diffusion model and text embedding according to claim 3 is characterized in that: Image feature reconstruction loss: Where, L pred-i represents the image feature reconstruction loss, I E represents the input image features, Represents the reconstructed image features; Text feature reconstruction loss: Where, L pred-t represents the text feature reconstruction loss, T E is the input text feature, is the reconstructed text feature.

5. The video anomaly detection method based on diffusion model and text embedding according to claim 3 is characterized in that: The loss L of the image diffusion module LDM-I for: In the formula, ∈ I is the image denoising network, ε represents noise, ε I represents the image denoising autoencoder, ε~N(0,1) indicates that the noise conforms to the Gaussian distribution, represents the hidden layer image features after adding noise in the tth step; The loss L of the text diffusion module LDM-T for: Among them, ∈ T is the text denoising network, Represents the text features after noise addition in the tth step.

6. The video anomaly detection method based on diffusion model and text embedding according to claim 3 is characterized in that: The total loss function for model training is: L all =L pred-i +λ1L pred-t +λ2L LDM-I +λ3L LDM-T ; Where, L all represents the total loss function, λ1, λ2 and λ3 represent the weights of text feature reconstruction loss, image diffusion module loss and text diffusion module loss respectively.

Images