Video abnormal event detection method and device, computer equipment and storage medium

Abstract

The invention discloses a video abnormal event detection method and apparatus, a computer device and a storage medium. The method comprises the steps of obtaining a to-be-detected video frame and a historical video frame; respectively inputting the historical video frame into a dynamic feature prediction model and a time feature prediction model to obtain first image feature data generated by the dynamic feature prediction model and second image feature data generated by the time feature prediction model; generating a predicted video frame based on the first image feature data and the second image feature data; and according to the difference between the predicted video frame and the to-be-detected video frame, determining that an abnormal event exists in the to-be-detected video frame, or determining that no abnormal event exists in the to-be-detected video frame. Therefore, more accurate and more comprehensive understanding of normal samples is constructed, and the accuracy of video abnormal event detection is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of video anomaly detection technology, and in particular to a method, apparatus, computer device, and storage medium for detecting abnormal video events. Background Technology

[0002] Video anomaly detection aims to detect rare, unusual events in videos that deviate from normal behavior. Unsupervised video anomaly detection focuses on detecting video frames where anomalies occur from videos with only video-level labels. It is suitable for real-world situations where the probability of anomalies is low and frame-level labels are difficult to obtain.

[0003] In related technologies, generative adversarial networks (GANs) are used to perform video anomaly detection tasks. However, these technologies have poor ability to distinguish between normal and abnormal samples, and the accuracy of anomaly detection needs to be improved. Summary of the Invention

[0004] This invention aims to at least partially address one of the technical problems in related technologies. To this end, this invention proposes a method, apparatus, computer device, and storage medium for detecting abnormal events in videos, thereby constructing a more accurate and comprehensive understanding of normal samples to better distinguish between normal and abnormal samples, effectively improving the accuracy of abnormal event detection in videos.

[0005] To achieve the above objectives, a first aspect of the present invention provides a video anomaly event detection method, the method comprising: acquiring a video frame to be detected and historical video frames; wherein the time corresponding to the historical video frames is earlier than the time corresponding to the video frame to be detected; inputting the historical video frames into a dynamic feature prediction model and a temporal feature prediction model respectively to obtain first image feature data generated by the dynamic feature prediction model and second image feature data generated by the temporal feature prediction model; wherein the first image feature data and the second image feature data correspond to the time of the video frame to be detected; generating a predicted video frame based on the first image feature data and the second image feature data; and determining, based on the difference between the predicted video frame and the video frame to be detected, whether an anomaly event exists in the video frame to be detected, or whether an anomaly event does not exist in the video frame to be detected.

[0006] According to one embodiment of the present invention, the dynamic feature prediction model includes a predictive coding network module and a first feature coding module; the step of inputting the historical video frame into the dynamic feature prediction model to obtain the first image feature data generated by the dynamic feature prediction model includes: inputting the historical video frame into the predictive coding network module to obtain a video frame generated by the predictive coding network module; wherein the video frame generated by the predictive coding network module corresponds to the time of the video frame to be detected; and using the first feature coding module to extract features from the video frame generated by the predictive coding network module to obtain the first image feature data.

[0007] According to one embodiment of the present invention, the time feature prediction model includes a second feature encoding module and a diffusion module; the step of inputting the historical video frame into the time feature prediction model to obtain the second image feature data generated by the time feature prediction model includes: inputting the historical video frame into the second feature encoding module to obtain the predicted features output by the second feature encoding module; wherein, the predicted features correspond to the time of the video frame to be detected; and inputting the predicted features into the diffusion module to obtain the second image feature data generated by the diffusion module.

[0008] According to one embodiment of the present invention, the diffusion module includes a temporal diffusion processing unit and a temporal sampling processing unit; the step of inputting the predicted features into the diffusion module to obtain the second image feature data generated by the diffusion module includes: inputting the predicted features into the temporal diffusion processing unit to obtain the noisy features output by the temporal diffusion processing unit; and using the temporal sampling processing unit to perform sampling processing on the noisy features to obtain the second image feature data output by the temporal sampling processing unit.

[0009] According to one embodiment of the present invention, the diffusion module further includes a temporal relationship encoding unit; before the sampling processing of the noisy features using the temporal sampling processing unit, the video anomaly detection method further includes: inputting the historical video frame to the temporal relationship encoding unit to obtain temporal relationship constraint data output by the temporal relationship encoding unit; the sampling processing of the noisy features using the temporal sampling processing unit to obtain the second image feature data output by the temporal sampling processing unit includes: using the temporal sampling processing unit to sample the noisy features based on the temporal relationship constraint data to obtain the second image feature data.

[0010] According to one embodiment of the present invention, generating a predicted video frame based on the first image feature data and the second image feature data includes: fusing the first image feature data and the second image feature data to obtain fused predicted feature data; and generating the predicted video frame based on the fused predicted feature data.

[0011] According to one embodiment of the present invention, determining whether an abnormal event exists in the video frame to be detected, or whether an abnormal event does not exist in the video frame to be detected, based on the difference between the predicted video frame and the video frame to be detected, includes: calculating the peak signal-to-noise ratio (PSNR) of the predicted video frame and the video frame to be detected to obtain PNR data between the predicted video frame and the video frame to be detected; and determining whether an abnormal event exists in the video frame to be detected, or whether an abnormal event does not exist in the video frame to be detected, based on the PNR data.

[0012] To achieve the above objectives, a second aspect of the present invention provides a video anomaly event detection device, the device comprising: an original video frame acquisition module, configured to acquire a video frame to be detected and historical video frames; wherein the time corresponding to the historical video frames is earlier than the time corresponding to the video frame to be detected; a feature data acquisition module, configured to input the historical video frames into a dynamic feature prediction model and a temporal feature prediction model respectively, to obtain first image feature data generated by the dynamic feature prediction model and second image feature data generated by the temporal feature prediction model; wherein the first image feature data and the second image feature data correspond to the time of the video frame to be detected; a predicted video frame generation module, configured to generate a predicted video frame based on the first image feature data and the second image feature data; and an anomaly event detection module, configured to determine, based on the difference between the predicted video frame and the video frame to be detected, whether an anomaly event exists in the video frame to be detected, or whether an anomaly event does not exist in the video frame to be detected.

[0013] To achieve the above objectives, a third aspect of the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the video anomaly event detection method described in any of the foregoing embodiments.

[0014] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the video anomaly detection method described in any of the preceding embodiments.

[0015] According to multiple embodiments provided by the present invention, by utilizing a dynamic feature prediction model to capture dynamic feature information in normal mode and a temporal feature prediction model to capture temporal feature information in normal mode, the combination of the two achieves a multi-dimensional feature complementarity and error correction mechanism, constructing a more accurate and comprehensive understanding of normal samples. This enables the generated predicted video frames to simultaneously satisfy the motion patterns and temporal logic under normal mode and / or scene conditions. This allows for better differentiation between normal and abnormal samples, effectively improving the accuracy and robustness of video anomaly event detection.

[0016] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a video anomaly event detection method provided according to one embodiment of this specification.

[0018] Figure 2a This is a schematic diagram of a process for generating first image feature data according to one embodiment of this specification.

[0019] Figure 2b This is a schematic diagram of a PredNet network provided according to one embodiment of this specification.

[0020] Figure 3 This is a schematic diagram of a process for generating second image feature data according to one embodiment of this specification.

[0021] Figure 4a This is a schematic diagram of the process for generating second image feature data according to another embodiment of this specification.

[0022] Figure 4b This is a schematic diagram of a time diffusion process provided according to one embodiment of this specification.

[0023] Figure 4c This is a schematic diagram of a single sampling process provided according to one embodiment of this specification.

[0024] Figure 5a This is a flowchart illustrating a process for determining whether an abnormal event exists in a video frame to be detected, according to one embodiment of this specification.

[0025] Figure 5b This is a schematic diagram of an unsupervised video anomaly detection method provided according to one embodiment of this specification.

[0026] Figure 6This is a structural block diagram of a video anomaly detection device provided according to one embodiment of this specification. Detailed Implementation

[0027] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0028] Video anomaly detection aims to detect rare, unusual events in videos that deviate from normal behavior. However, anomalous events are typically rare, complex, and unbounded in nature, making it difficult to obtain large amounts of labeled data in the natural world. Since frame-level labels for anomalous events are hard to obtain, using video-level labels would make it difficult to distinguish between normal and anomalous frames in the video. Therefore, unsupervised methods are more suitable for the reality that anomalous events occur with low probability and that frame-level labels are difficult to obtain.

[0029] Unsupervised video anomaly detection focuses on detecting video frames where anomalies occur from videos with only video-level labels. Because there are no video frame labels, it is difficult to distinguish between normal and abnormal video frames. In related technologies, generative adversarial networks (GANs) are used to perform video anomaly detection tasks. Although GAN-based methods have good performance, they suffer from two problems: (1) GANs lack distribution modeling of generated samples, making it difficult to ensure that generated samples can effectively represent various normal sample situations, thus reducing the accuracy of video anomaly detection; (2) Anomalies often appear in new anomaly patterns and / or scenarios, increasing the difficulty for the generative model to capture normal / abnormal data, resulting in poor model differentiation between normal and abnormal samples.

[0030] To improve the ability to distinguish between normal and abnormal samples in video anomaly detection and to enhance the accuracy of anomaly detection, it is necessary to propose a video anomaly detection method, apparatus, computer equipment, and storage medium. The video anomaly detection method provided in this specification inputs historical video frames into a dynamic feature prediction model and a temporal feature prediction model, respectively. The dynamic feature prediction model explicitly models dynamic features and generates first image feature data, while the temporal feature prediction model learns the long-term dependencies and evolutionary patterns of the video frame sequence in the temporal dimension and generates second image feature data. Based on the first and second image feature data, a predicted video frame is generated, with the predicted video frame's time being the same as the time of the video frame to be detected. By analyzing the differences between the predicted video frame and the video frame to be detected, it can be determined whether an anomaly event exists in the video frame to be detected, thus completing the video anomaly detection task.

[0031] Therefore, by utilizing a dynamic feature prediction model to capture dynamic feature information in normal mode and a temporal feature prediction model to capture temporal feature information in normal mode, the combination of these two approaches achieves multi-dimensional feature complementarity and error correction mechanisms. This constructs a more accurate and comprehensive understanding of normal samples, enabling the generated predicted video frames to simultaneously satisfy the motion patterns and temporal logic under normal mode and / or scene conditions. This allows for better differentiation between normal and abnormal samples, effectively improving the accuracy and robustness of video anomaly event detection.

[0032] This specification provides a video abnormal event detection method, which is described in the embodiments below. Figure 1 As shown, the video abnormal event detection method may include the following steps.

[0033] S110. Obtain the video frame to be detected and historical video frames; wherein, the time corresponding to the historical video frame is earlier than the time corresponding to the video frame to be detected.

[0034] S120. Input the historical video frames into the dynamic feature prediction model and the temporal feature prediction model respectively to obtain the first image feature data generated by the dynamic feature prediction model and the second image feature data generated by the temporal feature prediction model; wherein, the first image feature data and the second image feature data correspond to the time of the video frame to be detected.

[0035] S130. Generate a predicted video frame based on the first image feature data and the second image feature data.

[0036] S140. Based on the difference between the predicted video frame and the video frame to be detected, determine whether there is an abnormal event in the video frame to be detected, or determine whether there is no abnormal event in the video frame to be detected.

[0037] Among them, historical video frames and video frames to be detected are video frames from the same video.

[0038] Dynamic feature prediction models can be used to learn the motion patterns (such as optical flow, trajectory, short-term displacement, etc.) of objects or regions in videos, capturing spatial changes and dynamic interactions between objects. The first image feature data consists of image features predicted by the dynamic feature prediction model based on the dynamic features of historical video frames.

[0039] Temporal feature prediction models can be used to learn the long-term dependencies and evolutionary patterns of video frame sequences over time (such as appearance changes, scene state transitions, periodic behaviors, etc.), and can model contextual information within a longer time window to understand the temporal logic and evolutionary process of events. The second image feature data consists of image features predicted by the temporal feature prediction model based on the temporal features of historical video frames.

[0040] Specifically, the process involves acquiring the video frame to be detected and historical video frames, with the historical video frames occurring before the time of the video frame to be detected. The historical video frames are then input into a trained dynamic feature prediction model, which generates first image feature data based on the dynamic feature information in the historical video frames. This first image feature data reflects the image or video frame feature content predicted from the historical video frames. Additionally, the historical video frames are input into a trained temporal feature prediction model, which generates second image feature data based on the temporal feature information in the historical video frames. This second temporal feature data reflects the image or video frame feature content predicted from the temporal dimension.

[0041] The two types of models analyze historical video frames from different perspectives, extract feature information from different aspects, and the image feature data generated by the models through prediction modeling all correspond to the time of the video frame to be detected.

[0042] Predicted video frames are generated based on the obtained first and second image feature data. The predicted video frames are video frames in normal mode and / or scene that should appear at the corresponding time of the video frame to be detected, based on the feature information of different dimensions of historical video frames.

[0043] The predicted video frames are used as normal samples. By comparing the differences between the predicted video frames and the actual video frames to be detected, it is determined whether there are abnormal events in the video frames to be detected. If the difference is greater than or equal to a certain set threshold, it can be determined that there are abnormal events in the video frames to be detected; if the difference is less than the set threshold, it can be determined that there are no abnormal events in the video frames to be detected.

[0044] For example, the dynamic feature prediction model can employ any one or more of the following: a model based on 3D convolutional neural networks (3D CNNs), a model based on recurrent neural networks, or a PredNet (Predictive Coding Network) model. The temporal feature prediction model can employ any one or more of the following: an autoencoder model, a temporal prediction model based on generative adversarial networks, a temporal modeling model based on Transformer, or a diffusion model.

[0045] In some embodiments, generating a predicted video frame based on first image feature data and second image feature data may include: obtaining a first predicted image based on the first image feature data, obtaining a second predicted image based on the second image feature data, and fusing the first predicted image and the second predicted image to obtain a predicted video frame.

[0046] In other embodiments, generating a predicted video frame based on first image feature data and second image feature data may include: fusing the first image feature data and second image feature data to obtain fused image feature data, and obtaining a predicted video frame based on the fused image feature data.

[0047] It should be noted that if the time corresponding to the video frame to be detected is recorded as the kth time in the video, then the historical video frame can be the video frame of the previous k-1 times in the video, or it can be a part of the video frames selected from the video frames of the previous k-1 times in the video.

[0048] In the above embodiments, a dynamic feature prediction model is used to capture dynamic feature information in normal mode, and a temporal feature prediction model is used to capture temporal feature information in normal mode. The dual-branch model combining these two approaches enables multi-dimensional feature complementarity and error correction mechanisms, building a more accurate and comprehensive understanding of normal samples. This allows the generated predicted video frames to simultaneously satisfy the motion patterns and temporal logic under normal mode and / or scene conditions. Therefore, compared to methods in related technologies, the video anomaly event detection method provided in this specification can ensure that the predicted video frames accurately and effectively represent the situation of normal samples, achieving effective differentiation between normal and anomaly samples. The generated predicted video frames, as normal samples in both dynamic and temporal feature dimensions that are closer to the real mode and / or scene, can be used to more realistically and accurately describe the characteristics of normal events, thereby effectively improving the accuracy and robustness of video anomaly event detection.

[0049] In some implementations, the dynamic feature prediction model includes a predictive coding network module and a first feature coding module; Reference Figure 2a As shown, inputting historical video frames into a dynamic feature prediction model to obtain the first image feature data generated by the dynamic feature prediction model can include the following steps.

[0050] S210. Input the historical video frames into the predictive coding network module to obtain the video frames generated by the predictive coding network module; wherein, the video frames generated by the predictive coding network module correspond to the time of the video frames to be detected.

[0051] S220. The first feature encoding module is used to extract features from the video frames generated by the predictive encoding network module to obtain the first image feature data.

[0052] The predictive coding network module is the PredNet network module. The first feature coding module is used to convert the video frames predicted by the predictive coding network module into feature vectors.

[0053] Specifically, the dynamic feature prediction model includes a predictive coding network module and a first feature coding module. The main function of the predictive coding network module is to use the principle of predictive coding to predict based on the input historical video frames and generate predicted video frames. The time of this predicted video frame corresponds to the time of the video frame to be detected.

[0054] The video frames generated by the predictive coding network module are input to the first feature coding module, which extracts features from them to obtain the first image feature data, thereby converting the video frames predicted by the predictive coding network module into feature vectors.

[0055] In some embodiments, the first feature encoding module employs a Transformer encoder.

[0056] For example, the predictive coding network module, also known as the PredNet network module, consists of a series of repeatedly stacked modules that make local predictions about the input, subtract the prediction from the actual input, and pass it to the next layer. The PredNet network module first performs downward predictions from the upper layers, then improves the accuracy of the predictions through upward error correction from the lower layers. This design allows the network to progressively learn and understand dynamic information in image sequences.

[0057] refer to Figure 2b As shown, each module in the PredNet network consists of four basic parts: an input convolutional layer (A... l ), recursive representation layer (R) l ), prediction layer and error representation layer (E l R l Composed of recurrent convolutional networks, used to generate A l Prediction in the next frame The network obtains A l and The difference between them is calculated, and the error is expressed in the form E. l This includes separate positive error populations and negative error populations.

[0058] E l It is passed forward through the convolutional layer to become the next layer (A) l+1 The input for R. l Receive E l The copy and from the next level network (R l+1 The top-down input of the representation layer.

[0059] Suppose the video frame to be detected is the video frame at time k in the video, denoted as X. k The video frames from the first k-1 time steps are taken as historical video frames, and these historical video frames are denoted as X.{1,k-1} Inputting historical video frames into the PredNet network module yields the video frame X predicted by the PredNet network module. 'k The video frame is used as input to the Transformer encoder. The Transformer encoder is used to extract features from the predicted frame to obtain the feature data y0” corresponding to the k-th time step, which is the first image feature data.

[0060] It should be noted that the first feature encoding module can also use other encoders such as graph neural network encoders, which can be determined according to the actual application scenario or requirements.

[0061] In some implementations, the temporal feature prediction model includes a second feature encoding module and a diffusion module. (See reference...) Figure 3 As shown, inputting historical video frames into a temporal feature prediction model to obtain second image feature data generated by the temporal feature prediction model can include the following steps.

[0062] S310. Input the historical video frames into the second feature encoding module to obtain the predicted features output by the second feature encoding module; wherein, the predicted features correspond to the time of the video frame to be detected.

[0063] S320. Input the predicted features into the diffusion module to obtain the second image feature data generated by the diffusion module.

[0064] The second feature encoding module can be used to extract features from the input historical video frames and obtain corresponding predicted features based on the extracted features.

[0065] Specifically, the time feature prediction model includes a second feature encoding module and a diffusion module. Historical video frames are input into the second feature encoding module, which extracts and encodes features from the historical video frames to generate predicted features corresponding to the time of the video frame to be detected.

[0066] The predicted feature is input into the diffusion module, which generates second image feature data. By simulating the diffusion and propagation process of features over time, the diffusion module enhances the expressive power and robustness of the features, enabling the generated feature data to more accurately represent the temporal feature distribution under normal conditions.

[0067] In some embodiments, the second feature encoding module may be any one of a Transformer encoder, a graph neural network encoder, etc., and the diffusion module may be any one of a Denoising Diffusion Probabilistic Model (DDPM), a Denoising Diffusion Implicit Model (DDIM), a Score-Based Generative Model, a Conditional Diffusion Model, etc.

[0068] In some implementations, the diffusion module includes a time-diffusion processing unit and a time-sampling processing unit. (See reference...) Figure 4a As shown, inputting the predicted features into the diffusion module to obtain the second image feature data generated by the diffusion module can include the following steps.

[0069] S410. Input the predicted features into the time diffusion processing unit to obtain the noisy features output by the time diffusion processing unit.

[0070] S420. The time sampling processing unit is used to sample and process the noise-added features to obtain the second image feature data output by the time sampling processing unit.

[0071] The time diffusion processing unit is used to gradually add noise to the predicted features according to a preset noise schedule.

[0072] The time sampling processing unit is used to perform inverse diffusion (denoising) processing on the noisy features output by the time diffusion processing unit.

[0073] Specifically, the diffusion module includes a temporal diffusion processing unit and a temporal sampling processing unit. The temporal diffusion processing unit performs temporal diffusion processing on the input predicted features to simulate the changes and propagation of features over time. By gradually adding noise to the predicted features, their original stable structure is disrupted, allowing the features to diffuse and evolve over time. This diffusion processing enhances the features' adaptability to dynamic temporal changes and captures hidden time-related information and potential patterns within the predicted features.

[0074] Then, the temporal sampling processing unit is responsible for sampling the noisy features output by the temporal diffusion processing unit, extracting representative and key image feature data from the noisy features. The sampling process can be seen as a filtering and reconstruction of the noisy features, which can remove redundant information in the noise, retain feature components useful for subsequent tasks, and further enhance the expression of features in the temporal dimension.

[0075] For example, the temporal diffusion processing unit can add Gaussian-distributed noise to the predicted features at each time step. As the time step increases, the predicted features are gradually submerged in noise, eventually becoming a state close to pure noise. In this process, the temporal information of the features interacts with the noise in a complex way, thereby achieving feature diffusion.

[0076] The temporal sampling processing unit can employ sampling strategies such as probability-based sampling and importance-based sampling. For example, in probability-based sampling, the temporal sampling processing unit can assign different sampling probabilities to each element in the noisy feature based on the degree of noise contamination and its importance to the overall feature. Then, the noisy feature is sampled according to these probabilities to obtain the second image feature data. In this way, it is ensured that the sampled feature data not only contains the main information of the original predicted features but also has better temporal stability and discriminative power.

[0077] In some implementations, the diffusion module further includes a temporal relationship encoding unit. Before sampling the noise features using the temporal sampling processing unit, the video anomaly detection method may further include: inputting historical video frames into the temporal relationship encoding unit to obtain temporal relationship constraint data output by the temporal relationship encoding unit.

[0078] Accordingly, the time sampling processing unit is used to sample the noise-added features to obtain the second image feature data output by the time sampling processing unit. This may include: using the time sampling processing unit to sample the noise-added features based on the time relationship constraint data to obtain the second image feature data.

[0079] The temporal relationship coding unit is used to learn the temporal information in historical video frames to guide the denoising sampling process in the diffusion module.

[0080] Temporal relationship constraint data is used to reflect temporal relationship information in historical video frames.

[0081] Specifically, a temporal relationship encoding unit is introduced into the diffusion module to more deeply mine and utilize the temporal relationship information between historical video frames. Historical video frames are input into the temporal relationship encoding unit, which analyzes and processes them, outputting temporal relationship constraint data. After introducing this constraint data, the temporal sampling processing unit samples the noisy features based on this data. Specifically, the temporal relationship constraint data serves as prior knowledge or a constraint condition to guide the sampling process. After sampling processing based on the temporal relationship constraint data, the temporal sampling processing unit outputs second image feature data. This second image feature data not only contains useful information from the noisy features but also fully considers the temporal relationships between historical video frames, resulting in stronger temporal representativeness and discriminative power.

[0082] For example, suppose the video frame to be detected is the video frame at time k in the video, denoted as X. k The video frames from the first k-1 time steps are taken as historical video frames, and these historical video frames are denoted as X. {1,k-1} . refer to Figure 4b The diagram shows a schematic of the time-diffusion processing unit performing the time-diffusion process. This process can be represented by the following formula:

[0083]

[0084] The above formula is a single-step Monte Carlo sampling calculation formula, which can be used to calculate the Monte Carlo sampling in one step. Generate time sampling start point in, β is the predicted feature at time k output by the second feature encoding module; w is standard Gaussian noise, and w ~ N(0, I), following a normal distribution with mean 0 and covariance matrix I (identity matrix); t Related to the noise level, this setting adjusts the proportion of noise in the diffused sample and can be configured according to specific application requirements. Achieving a multi-step noise addition process through single-step Monte Carlo sampling effectively reduces the number of iterations, while cross-step sampling enables rapid and efficient denoising of the added samples.

[0085] refer to Figure 4c As shown, the diffusion module from arrive The single sampling process. h is the step size of the noise reduction process.

[0086] During this sampling process, the input is the first k-1 frames X. {1,k-1} As supervision, the temporal relation encoding unit allows the model to learn temporal information from continuous features. Each time, another noisy sample is obtained for this round of sampling using the formula for the aforementioned diffusion process. s represents the time interval from time t to time 0 during the noise reduction process. In some cases, the temporal relationship coding unit can further utilize the temporal relationship information from the previous k-1 frames and the noisy samples. Generate the corresponding sample y s .

[0087] The self-attention layer is used to learn temporal information, allowing the model to retain motion information, making the boundary distribution of normal samples more robust, and using cosine similarity to guide the reduction of the deviation between generated samples and original samples. The energy function corresponding to cosine similarity is denoted as...

[0088] Simultaneously, high-frequency information in the features is removed through downsampling and upsampling to ensure sufficient similarity between the features of the input / output time-diffusion module, and mean squared error is used to guide the reduction of the deviation between the generated samples and the original samples. The corresponding energy function is:

[0089]

[0090] in, Samples are obtained through downsampling and upsampling. The result obtained after removing high-frequency information from the image. Samples are obtained through downsampling and upsampling. The result is obtained after removing high-frequency information from the image.

[0091] Using λ s and λ i Two weighted hyperparameters greater than 0 are used to weight and sum the two energy functions, i.e.:

[0092]

[0093] The difference between the obtained cosine similarity and mean square error is calculated, and then... The derivative is used as an offset to reduce the noise level of the current sampling features. Therefore, a reliable energy judgment function based on the cosine similarity and mean square error between samples is designed to guide the reliable denoising sampling process for noisy samples.

[0094] Then, the features generated by the UNet network are used as the gradient of the energy function subtracted from the mean, thereby modifying the proportion of noise and samples in the mean. That is:

[0095]

[0096] The final result Represented as:

[0097]

[0098] In the above formula, the energy function guides the model to modify the denoising features of the previous sampling by jumping h steps each time. When the above iteration process reaches t=0, w is 0, and y0' can be obtained as the generated feature of the diffusion module.

[0099] In this specification, the diffusion module is also referred to as the time diffusion module.

[0100] In some implementations, generating a predicted video frame based on first image feature data and second image feature data may include: fusing the first image feature data and second image feature data to obtain fused predicted feature data; and generating a predicted video frame based on the fused predicted feature data.

[0101] Specifically, after obtaining the first image feature data generated by the dynamic feature prediction model and the second image feature data generated by the temporal feature prediction model, the first and second image feature data are fused to obtain fused prediction feature data. This fused prediction feature data can then be converted into corresponding images, i.e., predicted video frames, using an image decoder or a generative network.

[0102] For example, the fusion of the first image feature data and the second image feature data can be achieved by any of the following methods: feature splicing, weighted fusion, attention mechanism fusion, etc.; the generation of predicted video frames based on the fused predicted feature data can be achieved by any of the following methods: CNN (Convolutional Neural Network) decoder, GAN (Generative Adversarial Network) generator, diffusion model decoder, etc.

[0103] In some implementations, reference Figure 5a As shown, determining whether an abnormal event exists in the video frame to be detected, or whether no abnormal event exists in the video frame to be detected, based on the difference between the predicted video frame and the video frame to be detected, may include the following steps.

[0104] S510. Calculate the peak signal-to-noise ratio (PSNR) between the predicted video frame and the video frame to be detected to obtain the PSNR data between the predicted video frame and the video frame to be detected.

[0105] S520. Determine whether there is an abnormal event in the video frame to be detected based on the peak signal-to-noise ratio data, or determine whether there is no abnormal event in the video frame to be detected.

[0106] Specifically, the difference between the predicted video frame and the video frame to be detected is compared. This is achieved by calculating the peak signal-to-noise ratio (PSNR) between the two and comparing it with a preset threshold. If the PSNR is greater than or equal to the preset threshold, it indicates that the difference between the video frame to be detected and the predicted video frame is small, and it can be determined that there are no abnormal events in the video frame to be detected. If the PSNR is less than the preset threshold, it indicates that the difference between the video frame to be detected and the predicted video frame is large, and it can be determined that there are abnormal events in the video frame to be detected.

[0107] For example, to perform appearance and feature analysis on normal and abnormal video frames, this specification proposes an unsupervised video anomaly detection method. This method employs a model with two branches: a direct prediction branch and a temporal diffusion branch. The direct prediction branch uses a PredNet network to directly predict the next frame of the video sequence and uses a Transformer encoder to extract features from the predicted frame; the temporal diffusion branch uses a Transformer encoder to extract video frame features.

[0108] The temporal diffusion branch primarily employs a temporal diffusion module. This module uses Gaussian noise to diffuse temporal features, generating a noisy set of temporal features. It uses a single-step Monte Carlo sampling method to select noisy samples and determines their reliability based on the cosine similarity and mean square error between the noisy and original samples. This module further incorporates a single-iteration noisy process and multiple-step sampling to fully learn the more complex temporal variations of the sample features. The temporal diffusion module describes reliable temporal features of video frames and complements the video features directly predicted by the PredNet network, effectively enhancing the ability to distinguish between normal and abnormal samples.

[0109] refer to Figure 5b As shown, in the architecture of the unsupervised video anomaly detection method provided in this specification, the direct prediction branch of the PredNet network directly predicts the first k-1 frames (denoted as X). {1,k-1} The next frame (i.e., the k-th frame) is predicted, and the predicted video frame is denoted as X. 'k Then, the encoder (i.e., the first feature encoding module, which is a Transformer encoder in this embodiment) is used to obtain the predicted feature y0 of the k-th frame.

[0110] The subsequent temporal diffusion branch uses an encoder (i.e., the second feature encoding module, which in this embodiment uses a Transformer encoder) to obtain the predicted features of the k-th frame from the features of the first k-1 frames. Noise features generated in one step through a time diffusion process The sampling process uses features from the first k-1 consecutive frames as supervision to gradually reduce the noise features generated during the diffusion process. The timing information noise in the data is used to generate y0'.

[0111] By superimposing y0' and y0”, the features predicted by the two branches are deeply fused to generate fused prediction features. These features are then decoded (in this embodiment, a Transformer decoder is used) to obtain the final predicted image X. ”k .

[0112] The anomaly score is determined by the final predicted image X. ”k and the original video frame X of frame k k It is obtained by calculating the peak signal-to-noise ratio.

[0113] Therefore, a two-branch model combining a PredNet network and temporal diffusion is constructed to improve the ability to distinguish between normal and abnormal samples. This model includes a Transformer encoder, a temporal diffusion module, a PredNet network, and a Transformer decoder. The PredNet network is used to construct the edge distribution of normal samples, making the distribution of normal patterns more robust. The temporal diffusion module uses a self-attention layer to enhance the model's ability to learn temporal information and guides the model to learn the temporal feature distribution of normal patterns through an energy function. The temporal diffusion module uses the input video frames as supervision, employs a self-attention mechanism to ensure the authenticity of the generated predicted samples, and finally calculates the anomaly score based on the peak signal-to-noise ratio (PSNR) of the predicted image and the original image. Because the two-branch model achieves a more accurate and comprehensive understanding of normal samples, the generated predicted video frames can simultaneously satisfy the motion patterns and temporal logic of normal patterns and / or scenes, thus effectively improving the accuracy of video anomaly event detection.

[0114] This specification provides a video abnormal event detection device, with reference to... Figure 6 As shown, the video abnormal event detection device 600 may include: an original video frame acquisition module 610, a feature data acquisition module 620, a predicted video frame generation module 630, and an abnormal event detection module 640.

[0115] The original video frame acquisition module 610 is used to acquire the video frame to be detected and historical video frames; wherein, the time corresponding to the historical video frame is earlier than the time corresponding to the video frame to be detected.

[0116] The feature data acquisition module 620 is used to input historical video frames into the dynamic feature prediction model and the temporal feature prediction model respectively to obtain the first image feature data generated by the dynamic feature prediction model and the second image feature data generated by the temporal feature prediction model; wherein the first image feature data and the second image feature data correspond to the time of the video frame to be detected.

[0117] The predicted video frame generation module 630 is used to generate predicted video frames based on the first image feature data and the second image feature data.

[0118] The abnormal event detection module 640 is used to determine whether an abnormal event exists in the video frame to be detected, or to determine whether an abnormal event does not exist in the video frame to be detected, based on the difference between the predicted video frame and the video frame to be detected.

[0119] Specific limitations regarding the video anomaly detection device can be found in the limitations of the video anomaly detection method described above, and will not be repeated here. Each module in the aforementioned video anomaly detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0120] This specification also provides a computer device, which may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the video abnormal event detection method in any of the foregoing embodiments.

[0121] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the video anomaly detection method in any of the foregoing embodiments.

[0122] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0123] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0124] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0125] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0126] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0127] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for detecting abnormal events in video, characterized in that, The method includes: Acquire the video frame to be detected and historical video frames; wherein the time corresponding to the historical video frame is earlier than the time corresponding to the video frame to be detected; The historical video frames are respectively input into the dynamic feature prediction model and the temporal feature prediction model to obtain the first image feature data generated by the dynamic feature prediction model and the second image feature data generated by the temporal feature prediction model; wherein, the first image feature data and the second image feature data correspond to the time of the video frame to be detected; Predicted video frames are generated based on the first image feature data and the second image feature data; Based on the difference between the predicted video frame and the video frame to be detected, it is determined that there is an abnormal event in the video frame to be detected, or that there is no abnormal event in the video frame to be detected.

2. The method according to claim 1, characterized in that, The dynamic feature prediction model includes a predictive coding network module and a first feature coding module; the step of inputting the historical video frames into the dynamic feature prediction model to obtain the first image feature data generated by the dynamic feature prediction model includes: The historical video frames are input into the predictive coding network module to obtain the video frames generated by the predictive coding network module; wherein the video frames generated by the predictive coding network module correspond to the time of the video frame to be detected. The first image feature data is obtained by extracting features from the video frames generated by the predictive coding network module using the first feature encoding module.

3. The method according to claim 1, characterized in that, The temporal feature prediction model includes a second feature encoding module and a diffusion module; the step of inputting the historical video frames into the temporal feature prediction model to obtain the second image feature data generated by the temporal feature prediction model includes: The historical video frames are input into the second feature encoding module to obtain the predicted features output by the second feature encoding module; wherein, the predicted features correspond to the time of the video frame to be detected; The predicted features are input into the diffusion module to obtain the second image feature data generated by the diffusion module.

4. The method according to claim 3, characterized in that, The diffusion module includes a time diffusion processing unit and a time sampling processing unit; the step of inputting the predicted features into the diffusion module to obtain the second image feature data generated by the diffusion module includes: The predicted features are input into the time diffusion processing unit to obtain the noisy features output by the time diffusion processing unit; The noisy features are sampled using the time sampling processing unit to obtain the second image feature data output by the time sampling processing unit.

5. The method according to claim 4, characterized in that, The diffusion module further includes a time relation encoding unit; before the time sampling processing unit samples the noisy features, the method further includes: The historical video frames are input into the temporal relationship encoding unit to obtain temporal relationship constraint data output by the temporal relationship encoding unit; The step of sampling the noisy features using the time sampling processing unit to obtain the second image feature data output by the time sampling processing unit includes: The time sampling processing unit performs sampling processing on the noisy features based on the time relationship constraint data to obtain the second image feature data.

6. The method according to any one of claims 1 to 5, characterized in that, The step of generating predicted video frames based on the first image feature data and the second image feature data includes: The first image feature data and the second image feature data are fused together to obtain fused prediction feature data; The predicted video frame is generated based on the fused prediction feature data.

7. The method according to any one of claims 1 to 5, characterized in that, The step of determining whether an abnormal event exists in the video frame to be detected, or determining whether an abnormal event does not exist in the video frame to be detected, based on the difference between the predicted video frame and the video frame to be detected, includes: Peak signal-to-noise ratio (PSNR) is calculated for the predicted video frame and the video frame to be detected to obtain the PSNR data between the predicted video frame and the video frame to be detected. Based on the peak signal-to-noise ratio data, it can be determined whether an abnormal event exists in the video frame to be detected, or whether no abnormal event exists in the video frame to be detected.

8. A video anomaly event detection device, characterized in that, The device includes: The original video frame acquisition module is used to acquire the video frame to be detected and historical video frames; wherein, the time corresponding to the historical video frame is earlier than the time corresponding to the video frame to be detected. The feature data acquisition module is used to input the historical video frames into the dynamic feature prediction model and the temporal feature prediction model respectively to obtain the first image feature data generated by the dynamic feature prediction model and the second image feature data generated by the temporal feature prediction model; wherein, the first image feature data and the second image feature data correspond to the time of the video frame to be detected; A predicted video frame generation module is used to generate predicted video frames based on the first image feature data and the second image feature data; An abnormal event detection module is used to determine, based on the difference between the predicted video frame and the video frame to be detected, whether an abnormal event exists in the video frame to be detected, or whether an abnormal event does not exist in the video frame to be detected.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images