Video anomaly detection method based on diffusion model and text embedding

Abstract

The application discloses a video anomaly detection method based on a diffusion model and text embedding, and relates to the technical field of video anomaly detection.The video anomaly detection method based on the diffusion model and the text embedding comprises the following steps: first, text information of each video frame is extracted by using a picture-text generation model; and then, semantic features are obtained by using a pre-trained CLIP text encoder; based on an image diffusion model and a text diffusion model, a video anomaly detection model is constructed; cross attention mechanisms are used to guide an image diffusion denoising network and a text diffusion denoising network to reconstruct image features and text features, respectively; and finally, model training is completed through overall optimization, and then, video anomaly detection is performed.Therefore, by using the method, the powerful generation capability of the diffusion model and the semantic understanding capability of the picture-text generation model are used, the overgeneralization problem in a specific scene is alleviated, the explainability of video anomaly detection is enhanced, and the performance of the video anomaly detection model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video anomaly detection, and particularly to a video anomaly detection method based on diffusion model and text embedding. BACKGROUND

[0002] Video anomaly detection aims to analyze the dominant patterns (normal) in the data, so as to detect the individual minority "unusual" samples (anomaly). There are mainly two paradigms for the research of video anomaly detection, one is weakly supervised video anomaly detection, and the other is unsupervised video anomaly detection. Since the abnormal samples are usually difficult to observe and have the characteristics of scarcity, the unsupervised video anomaly detection method is more suitable for the "abnormal" problem.

[0003] The field of unsupervised anomaly detection has a long research history. According to the time sequence of algorithm design and development, the unsupervised anomaly detection algorithm can be roughly divided into hand-crafted feature-based method and deep learning-based method.

[0004] (1) Hand-crafted feature-based method, traditional anomaly detection algorithms are based on designed hand-crafted features, and can be divided into two categories according to the designed features: trajectory features and spatio-temporal features. Most of the hand-crafted feature-based methods are based on statistical models, which collect and count the distribution of normal patterns through the training set, and then in the testing process, the samples deviating from the distribution are regarded as anomalies. In addition, sparse coding or dictionary learning is also a common method for encoding normal patterns. The basic assumption of these methods is that after the dictionary encodes the normal patterns on the training set, any normal pattern can be linearly represented as a linear combination of the basic dictionary elements. Therefore, if the reconstruction error of a pattern is high, the pattern is regarded as an anomaly. Although the hand-crafted feature-based method is robust in dealing with complex scenes, a common shortcoming of these methods is that they may not be able to detect long-term activities such as wandering. This is because wandering is related to a person's long-term global motion rather than local spatio-temporal motion patterns.

[0005] (2) Deep learning-based method, using deep learning methods such as autoencoder, recurrent neural network, long short-term memory network, and generative adversarial network to extract video features and establish a prediction model for video anomaly detection. The assumption of this kind of method is that the abnormal frame should be more difficult to predict than the normal frame. Therefore, for the video frame containing anomalies, the peak signal-to-noise ratio (quality) between the predicted frame of the model and the real frame is expected to be lower. However, all these anomaly detections are based on the reconstruction / prediction of the rule training data, and although all these methods assume that abnormal events will cause larger reconstruction / prediction errors, this assumption does not necessarily hold due to the good generalization ability of deep neural networks. This leads to the reconstruction / prediction error of part of the abnormal events similar to normal events, thereby reducing the discriminability of the model to anomalies.

[0006] In the unsupervised video anomaly detection task, the algorithm model only uses the video samples containing a single (normal) mode to train the model without abnormal samples. Researchers often propose some assumptions to detect anomalies, and make the normal samples satisfy the assumptions through model training. In the test, the samples that do not satisfy the proposed assumptions are regarded as anomalies. Common assumption-based frameworks include predicting future frames, reconstructing current frames, etc. However, simply modeling the mode in the normal sample has a high anomaly omission rate. Although in the unsupervised setting, only single mode (normal) data is used to train the model, the normal mode still contains a variety of elements. How to encode the information of these normal elements to comprehensively and compactly model the complete normal mode is the key to solving the problem. Therefore, developing algorithms to model the normal mode from different aspects and different angles and encode the diverse elements in the mode is an effective means to alleviate the "overgeneralization" phenomenon and improve the performance of video anomaly detection.

[0007] While ensuring accuracy, the video anomaly detection task has high requirements for the efficiency, real-time performance, and robustness of the algorithm. In the actual application process, edge monitoring devices with limited computing power often require the algorithm to quickly and effectively realize the video anomaly detection function under the premise of occupying small computing resources. At the same time, the complexity of abnormal events in real life requires the algorithm to conduct comprehensive analysis to detect all possible anomalies. In addition, a robust video anomaly detection algorithm also requires the model to be flexible and quickly adaptable to different scenarios. When the monitoring device is installed in a brand new scene, retraining and learning a new model is often time-consuming and impractical.

[0008] Therefore, it is necessary to propose a new method that can alleviate the model overgeneralization problem and enhance the explainability by constraining the training process of the model. SUMMARY

[0009] The purpose of the present application is to provide a video anomaly detection method based on diffusion model and text embedding, which solves the problem of overgeneralization and weak explainability of existing video anomaly detection methods.

[0010] To achieve the above purpose, the present application provides a video anomaly detection method based on diffusion model and text embedding, comprising:

[0011] S1, first extract the text information of each video frame using a picture-text generation model, and then obtain semantic features through a pre-trained CLIP text encoder;

[0012] S2, based on an image diffusion model and a text diffusion model, a video anomaly detection model is constructed, and the model training is completed through overall optimization, and then the video anomaly is detected.

[0013] Preferably, the video anomaly detection model comprises an image encoder, an image diffusion module, a text diffusion module and an image decoder.

[0014] Preferably, the model training comprises the following steps:

[0015] S21, hidden layer features of input video frame images are obtained by using an image encoder, and then processed by an image diffusion module, and guided by a cross attention mechanism to guide an image diffusion denoising network by introducing semantic features, to obtain reconstructed image features;

[0016] S22, the obtained semantic features are processed by using a text diffusion model, and guided by a cross attention mechanism to guide a text diffusion denoising network by introducing ground truth images, to obtain reconstructed text features.

[0017] Preferably, the image feature reconstruction loss is:

[0018]

[0019] In the formula, L pred-i represents the image feature reconstruction loss, I E represents the input image feature, represents the reconstructed image feature.

[0020] The text feature reconstruction loss is:

[0021]

[0022] In the formula, L pred-t represents the text feature reconstruction loss, T E is the input text feature, is the reconstructed text feature.

[0023] Preferably, the loss L LDM-I of the image diffusion module is:

[0024]

[0025] In the formula, ∈ I is an image denoising network, ε represents noise, and ε I represents an image denoising autoencoder, and ε~N(0, 1) represents that the noise conforms to a Gaussian distribution, represents the hidden layer image feature after adding noise at the t-th step;

[0026] The loss L LDM-T of the text diffusion module is:

[0027]

[0028] wherein ∈T is a text denoising network, denotes the text feature after the t-th step of adding noise.

[0029] Preferably, the total loss function of model training is:

[0030] L all = L pred-i + λ 1 L pred-t + λ 2 L LDM-I + λ 3 L LDM-T .

[0031] In the formula, L all denotes the total loss function, and λ 1, λ 2 and λ 3 respectively denote the weight of the text feature reconstruction loss, the image diffusion module loss and the text diffusion module loss.

[0032] Therefore, the video anomaly detection method based on the diffusion model and the text embedding has the following technical effects:

[0033] (1) In the image denoising network, the semantic information of the image is taken as the condition, the cross-attention mechanism is used to guide the image reconstruction process, the consistency of the semantic information of the reconstructed video frame in time sequence is ensured, the situation that the normal video frame is reconstructed as an anomaly is avoided, and the situation that the generated image is too different from the video frame is avoided; similarly, in the text denoising network, the true value image of the predicted image is taken as the condition, the cross-attention mechanism is used to guide the reconstruction process of the text feature, and the generated semantic information is closer to the feature of the real video frame.

[0034] (2) The semantic features of the video frame image are extracted, the semantic features and the image features are fused, the over-generalization problem in a specific scene is alleviated, and the explainability of the video anomaly detection can be significantly enhanced; meanwhile, the image diffusion module and the text diffusion module are optimized as a whole, and the performance of the video anomaly detection model can be improved.

[0035] The technical solutions of the present application will be further described in detail below with the aid of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a flowchart of the video anomaly detection method based on the diffusion model and the text embedding;

[0037] Figure 2 is a network schematic diagram of the video anomaly detection method based on the diffusion model and the text embedding. DETAILED DESCRIPTION

[0038] The present application can be explained in more detail by the following embodiments, and the purpose of the present application is to protect all changes and improvements within the scope of the present application, and the present application is not limited to the following embodiments.

[0039] As Figure 1As shown, the present application provides a video anomaly detection method based on diffusion model and text embedding, comprising the following steps:

[0040] S1, first, the pre-trained graph-to-text model BLIP-2 is used to extract the text description of the video frame by frame, then the pre-trained CLIP text encoder is used to encode the video frame description to obtain the semantic features, and the semantic features of all video frames of the training set and the test set are saved respectively for subsequent model training.

[0041] Among them, the fusion of semantic features and image features can significantly improve the explainability of video anomaly detection. Specifically, the task of unsupervised video anomaly detection requires that the model only input normal videos during training. By reconstructing the last frame of consecutive video frames, the model can learn the normal mode of a specific scene. During testing, for abnormal videos, the reconstructed last frame image of the model is close to the normal mode, and the difference with the real abnormal video frame is large, so the reconstructed video frame can be judged as abnormal. However, only the reconstruction error cannot well explain why the abnormal judgment is made. The existing graph-to-text large model can describe the main content of the image, and the text description of the normal video frame and the abnormal video frame can intuitively distinguish normal and abnormal, thereby improving the explainability of video anomaly detection.

[0042] S2, the modeling of video frames directly affects the performance of video anomaly detection. Existing detection methods mainly encode consecutive video frames based on autoencoders, then model the normal mode on the intermediate layer features, and then decode to obtain the reconstructed image. However, both methods based on autoencoders and methods based on memory mechanisms will cause overgeneralization problems in specific scenes, because the powerful generalization ability of convolutional neural networks makes the normal mode learned by the model too single.

[0043] The diffusion model has strong generation ability and the characteristics of generating diverse samples, therefore, the embodiment adopts the diffusion model to model the normal video to learn more diverse and complete normal modes. Moreover, in the denoising process of the diffusion model, the previously extracted text features are fused with the noisy image features, and the cross-attention mechanism is used to guide the denoising network to generate images closer to the semantic information. In this way, the diversity of normal mode modeling can be met while ensuring the consistency of the semantic information of the reconstructed video frame in time sequence, avoiding the generation of images by the denoising network that are too different from the previous video frames, thereby reconstructing the normal video frame as abnormal. Similarly, for the semantic features of the input consecutive video frames, the semantic features of the last frame can be reconstructed using the diffusion model, at this time, the true value image of the next frame of the input consecutive video frames can be used as a condition to guide the reconstruction of the text features in the text diffusion denoising process, so that the generated semantic information is closer to the features of the real video frame.

[0044] As Figure 2 shown in FIG. 1, the model of video anomaly detection includes an image encoder E, an image diffusion denoising network ∈I, a text diffusion denoising network ∈T, and an image decoder D. The input of the model is a sequence of video frame images i1, i2, …, ik. k-1 .

[0045] S21, the first k-1 video frame images i1, i2, …, ik-1 are sent to the image feature encoder E to extract the corresponding features, and the hidden layer features of the input video frame images are obtained, denoted as IE. k-1

[0046] Then, I E is input into the diffusion model for noise addition. After T steps of noise addition, the noise image complying with the Gaussian distribution is obtained.

[0047] After that, the image diffusion denoising network ∈ I is used for reverse denoising to predict the denoised image. Wherein, ∈ I is a U-Net-based network conditioned on semantic information, and the loss L LDM-I of the image diffusion model is denoted as:

[0048]

[0049] In the formula, LDM-I represents the image diffusion model, ∈I is the image denoising network, ε represents noise, and ε I represents the image denoising autoencoder, and ε ~ N(0, 1) represents that the noise complies with the Gaussian distribution, denotes the hidden layer image feature after t-step noise addition, and t is uniformly sampled in (1, 2, …, T).

[0050] In the denoising process, the introduced text features enhance the reconstruction ability of the U-Net network through the cross-attention mechanism, and the cross-attention is denoted as:

[0051]

[0052] Wherein, Q, K, and V represent query, key, and value respectively, Q comes from the image hidden layer feature, K and V come from the text feature embedding, and d represents the feature dimension.

[0053] Wherein T K is the semantic feature of the next frame image of the continuous input video frame. and are trainable parameter matrices.

[0054] After denoising, the reconstructed image features are obtained​​ Image feature reconstruction loss L pred-i is expressed as:

[0055]

[0056] wherein I E is the input image feature, is the reconstructed image feature.

[0057] S22, the process of text diffusion is similar to image diffusion, and the text denoising network is a U-Net-based network conditioned on the image of the next frame of continuous video frames ∈T. Wherein the loss L of the text diffusion model LDM-T is expressed as:

[0058]

[0059] wherein LDM-T represents the text diffusion model, ∈ T is the text denoising network, represents the text feature after adding noise at the t-th step, and the above objective function is used to predict the denoised text.

[0060] In the denoising process, the introduced image feature enhances the reconstruction ability of the U-Net network through the cross-attention mechanism, and the cross-attention is expressed as:

[0061]

[0062] wherein Q comes from the text feature, and K and V come from the image hidden layer feature.

[0063] wherein is the noisy semantic feature, I K is the hidden layer feature of the image of the next frame of continuous input video frames.

[0064] After denoising, the reconstructed text feature is obtained, wherein the text feature reconstruction loss L pred-t is expressed as:

[0065]

[0066] wherein T E is the input text feature, is the reconstructed text feature.

[0067] The image diffusion module and the text diffusion module are included in an overall neural network to construct the video anomaly detection model of the embodiment, and the overall optimization objective function is:

[0068] L all = Lpred-i + lambda1 * L pred-t + lambda2 * L LDM-I + lambda3 * L LDM-T ;

[0069] In the formula, L all is the total loss function, lambda1, lambda2 and lambda3 are weight parameters.

[0070] The adamW optimizer is used to update the weights of the whole model, and the model is iteratively trained for several rounds to obtain the encoding and decoding capabilities and the image-text reconstruction capability, and to ensure the accuracy of the video anomaly detection, so as to realize the training and testing of the video anomaly detection model.

[0071] Therefore, the video anomaly detection method based on the diffusion model and the text embedding can relieve the over-generalization problem in specific scenarios, enhance the explainability of the video anomaly detection, and improve the performance of the video anomaly detection model by means of the powerful generation capability of the diffusion model and the semantic understanding capability of the image-to-text model.

[0072] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application but not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that: the technical solutions of the present application can still be modified or replaced by equivalents, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A method for video anomaly detection based on diffusion model and text embedding, characterized in that, The method comprises the steps of: S1, first extracting the text information of each video frame by using a picture-text generation model, and then obtaining semantic features by using a pre-trained CLIP text encoder; S2, constructing a video anomaly detection model based on an image diffusion model and a text diffusion model, and completing model training through overall optimization, and then detecting video anomalies; The video anomaly detection model comprises an image encoder, an image diffusion module, a text diffusion module and an image decoder. The model training comprises the following steps: S21, obtaining the hidden layer features of the input video frame image by using the image encoder, then processing the hidden layer features by using the image diffusion module, introducing the semantic features, guiding the image diffusion denoising network by using the cross attention mechanism, and obtaining the reconstructed image features; S22, processing the obtained semantic features by using the text diffusion model, introducing the true value image, guiding the text diffusion denoising network by using the cross attention mechanism, and obtaining the reconstructed text features; Image feature reconstruction loss: ； In the formula, denotes the image feature reconstruction loss, denotes the input image feature, denotes the reconstructed image feature; Text feature reconstruction loss: ； In the formula, denotes the text feature reconstruction loss, is the input text feature, is the reconstructed text feature; Loss of the image diffusion module is: ； In the formula, is an image denoising network, represents noise, represents an image denoising autoencoder, represents that the noise conforms to a Gaussian distribution, represents the first step hidden layer image feature after adding noise; Loss of the text diffusion module is: ； wherein, is a text denoising network, denotes the step of adding noise to the text features; The total loss function of the model training is: ； In the formula, denotes the total loss function, , and respectively denote the weight of the text feature reconstruction loss, the image diffusion module loss, and the text diffusion module loss.

Images