Scene-dependent video anomaly detection and prediction method based on hidden space prediction model

Abstract

The invention discloses a scene-dependent video anomaly detection and prediction method based on a hidden space prediction model, and the method comprises the steps: firstly obtaining a preset number of video frames in a to-be-processed video, and forming an input video; an input video clip is processed by a forward network, a predicted video frame is output, and the predicted video frame is regarded as a current frame; whether an abnormal event occurs or not is judged by comparing the difference between a predicted value and a true value of a current frame, so that a video anomaly detection task is solved; after the predicted current frame is supplemented to the input video clip, a predicted future frame is output; after a preset number of future frames are predicted by the forward network in an autoregression mode, the future frames are input into the backward network; whether an abnormal event occurs in a preset number of frame time in the future is judged by comparing the difference between the predicted value and the true value of the observation frame, so that a video abnormal prediction task is solved. According to the method, scene-related anomaly detection and prediction can be well processed.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a scene-dependent video anomaly detection and prediction method based on a latent space prediction model. Background Art

[0002] Video anomaly detection is a basic and key problem in computer vision. Its purpose is to detect unexpected abnormal events in the video. As one of the important application technologies of intelligent monitoring systems, it is of great significance to prevent dangerous events and ensure public safety.

[0003] Since the types of abnormal events are not exhaustive and relatively rare, related technologies usually set video anomaly detection as a semi-supervised task. In the training data, only normal events without specific labels are included. The goal of the task is to train a single classifier that can faithfully learn the distribution of normal data and avoid incorrect generalization of abnormal events. Specifically, existing methods usually build deep autoencoders based on the method of reconstructing video frames or predicting video frames to learn the characteristics of normal events in the video. Any event that deviates significantly from the learned normal features is considered abnormal. However, these methods have limited understanding of the scene and it is difficult to detect abnormal events that depend on the existence of specific scenes, such as "cyclists on the sidewalk". In addition to detecting various types of ongoing abnormal events, predicting abnormal events that are about to occur in the future is also a practical task worthy of researchers' attention. If an early warning can be given before an abnormal event occurs, it is possible to avoid huge losses of life and property. However, this task has not yet attracted widespread attention. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a scene-dependent video anomaly detection and prediction method based on a latent space prediction model. First, a preset number of video frames in the video to be processed are obtained to form an input video clip, which is input into a forward and backward frame prediction network framework; a score-based scene-dependent autoencoder model is used to construct the method to solve the scene dependency problem of abnormal events; the input video clip is processed by a forward network, and a predicted video frame is output, and the predicted video frame is regarded as the current frame; by comparing the difference between the predicted value and the true value of the current frame, it is determined whether the abnormal event occurs, thereby solving the video anomaly detection task; after the predicted current frame is added to the input video clip, And delete the first frame of the video clip to keep the number of input video frames consistent, form a new input video clip, input it into the forward network, and output the predicted future frames; after the forward network predicts a preset number of future frames in an autoregressive manner, these preset number of predicted future frames are added to the original input video clip, and the preset number of frames at the head of the video clip are deleted to keep the number of input video frames consistent, and then input it into the backward network, the backward network reversely predicts the observed frame based on the frame predicted by the forward network and a part of the observed frame; by comparing the difference between the predicted value and the true value of the observed frame, it is judged whether an abnormal event occurs within the preset number of frames in the future, thereby solving the video abnormality prediction task. The method of the present invention can well handle scene-related abnormality detection and prediction.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] Step 1: Obtain a preset number of video frames in the video to be processed to form an input video clip, and input it into the forward and backward frame prediction network framework;

[0007] Step 2: The forward and backward frame prediction network framework consists of a forward network and a backward network. Both have the same network architecture and are constructed using a score-based scene-dependent autoencoder model to solve the scene-dependent problem of abnormal events.

[0008] Step 3: The input video clip is processed by the forward network and the predicted video frame is output, which is regarded as the current frame;

[0009] Step 4: By comparing the difference between the predicted value and the true value of the current frame, it is determined whether an abnormal event has occurred, thereby solving the video anomaly detection task;

[0010] Step 5: Add the predicted current frame to the input video segment and delete the first frame of the video segment to keep the number of input video frames consistent, form a new input video segment, input it into the forward network, and output the predicted future frame;

[0011] Step 6: After the forward network predicts a preset number of future frames in an autoregressive manner, the preset number of predicted future frames are added to the original input video clip, and the preset number of frames at the head of the video clip are deleted to keep the number of input video frames consistent, and then input into the backward network, which reversely predicts the observed frame based on the frame predicted by the forward network and a part of the observed frame;

[0012] Step 7: By comparing the difference between the predicted value and the true value of the observed frame, it is determined whether an abnormal event will occur within a preset number of frames in the future, thereby solving the video anomaly prediction task.

[0013] Furthermore, the score-based scene-dependent autoencoder model includes a scene-general hierarchical variational autoencoder model, a score-based diffusion model and a scene condition extraction module; after the video clip is input into the model, the scene-general hierarchical variational autoencoder first encodes the input into scene-general features, and the scene condition extraction module extracts the input scene feature information; then, the scene-general features and the scene feature information are fed into the score-based diffusion model; then, sampling is performed through a denoiser, and the scene conditions are embedded in the denoiser to obtain scene-dependent features by conditional sampling from the perturbation distribution; finally, the generated scene-dependent features are decoded by the scene-general hierarchical variational autoencoder to obtain a predicted target frame;

[0014] Furthermore, the scenario-general hierarchical variational autoencoder model includes a likelihood function Prior p θ (z) and the approximate posterior q φ (z|x in ); define φ and θ as the scene-general hierarchical variational encoder and scene-general hierarchical variational decoder respectively, z is the feature in the latent space; given an input video clip x consisting of n frames in =f t-n ,…,f -1 , the scene-general hierarchical variational encoder encodes the input group into latent features z1,z2,…,z N , N represents the number of layers and groups of the hierarchical autoencoder; the latent variables are represented in the form of feature maps of different resolutions; z1 represents the set of variables with the lowest resolution at the top of the network, z N Denote the set of latent variables at the bottom of the network, corresponding to the highest resolution, and each latent variable in the prior and approximate posterior is represented by a multiplicative normal distribution, so that the prior and approximate posterior are expressed as:

[0015]

[0016] Furthermore, the scene-general hierarchical variational encoder φ is used to transform the input video sequence x inEncoding; the encoding process is performed through three layers of encoder units, each encoder unit contains a residual encoder block and a downsampling layer; in each encoder block, the input x in First, the spatiotemporal features are initially encoded through convolution operations; then the correlation between feature channels is obtained through squeezing and excitation operations; then the connection relationship in the horizontal, vertical and depth directions of the features is obtained through multi-head self-attention operations; then the connection relationship is summed through aggregation operations to obtain the spatiotemporal features that integrate the long-distance information of the video; finally, the downsampling block reduces the dimension of the spatiotemporal features encoded by the encoder through convolution operations to obtain the encoded spatiotemporal features z, which follow the approximate posterior distribution q φ (z|x in ).

[0017] Furthermore, the scene-general hierarchical variational decoder θ decodes the encoded spatiotemporal features z; the decoding process is performed by three layers of decoder units, each of which includes a residual decoder block and an upsampling layer; in each decoder block, the input z is first decoded preliminarily by a convolution operation; then, the correlation between feature channels is decoded by squeezing and excitation operations, and the attention between channels is screened; then, the connection relationship between each pixel of the feature map and its horizontal, vertical and depth directions is decoded by a multi-head self-attention operation; then, the connection relationship is summed by an aggregation operation, and the refined spatiotemporal features that integrate the long-distance information of the video are decoded; finally, the upsampling block performs a convolution operation to increase the dimension of the spatiotemporal features decoded by the decoder to obtain the predicted target frame

[0018] Furthermore, the scene-general hierarchical variational encoder φ and the scene-general hierarchical variational decoder θ can be trained end-to-end, and their loss is written as:

[0019]

[0020] Using negative log-likelihood To estimate the prediction loss Aims to maintain consistency between the target frame and the predicted frame; the KL divergence term of the scene-general hierarchical variational autoencoder can be further written as:

[0021]

[0022] Furthermore, the score-based diffusion model transforms the scene-general prior p obtained by the scene-general scene variational autoencoder into θ (z) is refined into a scene-dependent prior p ψ (z); The score-based diffusion model includes a forward diffusion process and a reverse denoising process; After training the scene-general hierarchical variational encoder φ and decoder θ, the time-dependent score function is trained using a reverse stochastic differential equation Sampling is performed from z(s); the scene-general hierarchical variational decoder θ then decodes the sampled scene-dependent features into video frames, as described by:

[0023]

[0024] Furthermore, the forward diffusion process adds noise to the spatiotemporal features encoded by the scene-general hierarchical variational encoder φ, which is indexed by a continuous time variable s∈[0,1] and represented as where z(0) is the initial state of feature z and z(s) is the perturbation of feature z at time s; this diffusion process is modeled as the solution of a stochastic differential equation:

[0025] dz=f(z,s)ds+g(s)d

[0026] Among them, f(·,s):R→R is a vector-valued function, which is regarded as the drift coefficient of z(s); and g(·):R→R is a scalar function, which is regarded as the diffusion coefficient of z(s); ω represents the standard Wiener process of a time-continuous random process; f(z,s) and g(s) make the state z(1) after the feature is fully perturbed follow a normal distribution at the end of the diffusion process; by sampling from z(1) and then inverting the stochastic differential equation, an inverse denoising model εψ is trained, which is expressed as:

[0027]

[0028] in is a time-reverse Wiener process, ds is an infinitesimal negative time step, is the score function of the characteristic marginal distribution under the forward diffusion process time s.

[0029] Furthermore, the inverse denoising model is implemented by a noise conditional score network, which is composed of a residual block; in the residual block, the samples sampled in the input z(s) are first downsampled by a convolution operation, then intermediately processed by a convolution operation, and finally upsampled by a convolution operation, and the output predicted noise is denoised.

[0030] Furthermore, the generation process of the sampled scene-dependent features can be equivalently regarded as upsampling the scene-general hierarchical variational autoencoder on the prior of the score-based diffusion model; the scene-general hierarchical variational autoencoder and the score-based diffusion model are jointly trained by minimizing the variational upper bound of the negative log-likelihood:

[0031]

[0032] in is the frame prediction loss, which is optimized by decomposing the KL divergence term of the score-based diffusion model into its entropy term and cross entropy term, which can be written as:

[0033]

[0034] Furthermore, the scene condition extraction module embeds the condition factor into the denoiser ∈ ψ In order to achieve ∈ ψ (z(s))→∈ ψ (z(s),c), where c is the combination of the time step s and the scene condition extracted by the trainable scene information extractor; the scene condition extraction module includes a scene information extractor E sce and a scene information decoder D sce ; Scene information extractor E sce Pass the condition to the denoiser ∈ ψ All hidden layers of , thus forcing the denoiser ∈ ψ Depends on E sce (x in ) to generate scene dependencies; the scene information decoder is responsible for reconstructing the scene information D to which the input belongs sce (E sce (x in ))→scene(x in ).

[0035] Furthermore, the scene information extractor E sce It consists of three convolutional layers. In each convolutional layer, the input x in The scene features are obtained by processing through convolution operation and then activating through LeakyReLU function; the scene information decoder D sce It consists of five convolutional layers. In each convolutional layer, the input scene features are processed by convolution operation and then activated by LeakyReLU function to obtain the reconstructed scene graph. This micro autoencoder is pre-trained by reconstruction loss, which can be written as:

[0036]

[0037] Furthermore, the denoiser ∈ is trained using the classifier-free diffusion guided strategy ψ , including learning from conditional and unconditional distributions, and implementing the following linear combinations:

[0038]

[0039] Where λ>1 is the guidance scale, which can enhance the guidance effect; c is the time step s and the extracted scene information E sce (x in); after the interactive reverse process of conditional denoising, the target frame can be effectively predicted from the denoised frame.

[0040] Furthermore, an additional temporal coherence term is proposed to ensure that the predicted frame maintains temporal consistency with the real video sequence; in the training stage, let X = {f t-n ,…,f t-1 ,f t ,f t+1 ,…,f t+n} is a video clip, where f t is the target frame to be predicted; select f t-n ,f t-1 As f t-n ,…,f t-1 In the key frame, select f t+1 ,f t+n As f t+1 ,…,f t+n The time loss based on the key frame difference is written as follows:

[0041]

[0042] Furthermore, a stage flag ρ∈{1,2,3} is set to represent different training steps, namely the scene condition extraction module, the general hierarchical variational autoencoder model, and the score-based diffusion model; formally, the overall loss function is described as follows:

[0043]

[0044] where λ kfd is the temporal loss based on keyframe differences The weight of .

[0045] Furthermore, in the anomaly detection inference process, the video is divided into n video clips with T frames. In each T-frame segment in the video, the fraction of frames with the minimum peak signal-to-noise ratio (PSNR) value is regarded as the score of the video clip. In order to quantify the probability of anomaly occurrence, each PSNR value score(t) is normalized to obtain an anomaly score S(t) in the range of [0,1], written as:

[0046]

[0047] Furthermore, during the anomaly prediction inference process, the backward network uses the future frames predicted by the forward network and observation frame f t-1 ,…,f t+i+1-n Take as input the predicted observation frame f t+i-n ; First use the predicted frame and the observed true frame ft+i-n The PSNR value between is used as the anomaly score of the i-th future frame, i∈[1,α], α is the prediction time; then, the minimum value among the α PSNR values ​​is selected as the prediction score score(t+1:t+α); finally, each expected value is normalized to obtain the anomaly score S(t+1:t+α) in the range [0,1].

[0048] The beneficial effects of the present invention are as follows:

[0049] The present invention provides a scene-dependent video anomaly detection and prediction method, which obtains a preset number of video frames in the video to be processed and inputs them into a forward and backward frame prediction framework for video anomaly detection and prediction. Based on the framework, a new score-based scene-related autoencoder is designed, which is a diffusion-based model for detecting and predicting scene-related abnormal events. In the model, a more powerful hierarchical variational autoencoder is introduced to extract scene-general features, and a score-based diffusion model in a latent space is introduced to refine these features and generate scene-related features. An effective conditional adjustment strategy is applied to the diffusion model to generate scene-related samples. In addition, a temporal loss is proposed to constrain the motion consistency of objects in video sequences. The method of the present invention can handle scene-related anomaly detection and prediction well. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic flow diagram of the method of the present invention;

[0051] Figure 2 Schematic diagram of a score-based scene-dependent autoencoder model of the present invention;

[0052] Figure 3 It is a schematic diagram of a universal hierarchical variational autoencoder model for the scenario of the present invention. DETAILED DESCRIPTION

[0053] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0054] like Figure 1 As shown, the scene-dependent video anomaly detection and prediction method provided by the embodiment of the present invention includes:

[0055] A preset number of video frames in the video to be processed are obtained, and the preset number of video frames are scaled, cropped, and mean normalized to obtain video data with a dimension of 3×T×224×224. Optionally, T represents a preset number of video frames selected from the video to be processed. In this embodiment, T=8. The video data is sent to a forward and backward frame prediction network framework.

[0056] The forward and backward frame prediction network framework consists of a forward network and a backward network. Both have the same network architecture and are constructed using a score-based scene-dependent autoencoder model to solve the scene-dependent problem of abnormal events. The input video clip is processed by the forward network and the predicted video frame is output, which is regarded as the current frame. By comparing the difference between the predicted value and the true value of the current frame, it is determined whether an abnormal event has occurred, thereby solving the video anomaly detection task.

[0057] Specifically, in the anomaly detection inference process, the video is divided into n video clips with T frames. In each T-frame segment in the video, the fraction of frames with the minimum peak signal-to-noise ratio (PSNR) value is regarded as the score of the video clip. In order to quantify the probability of anomaly occurrence, each PSNR value score(t) is normalized to obtain an anomaly score S(t) in the range of [0,1], which can be written as:

[0058]

[0059] Then the predicted current frame is added to the input video clip, and the first frame of the video clip is deleted to keep the number of input video frames consistent, forming a new input video clip, which is input into the forward network, and the predicted future frames are output. After the forward network predicts the preset number of future frames in this autoregressive manner, these preset number of predicted future frames are added to the original input video clip, and the preset number of frames at the head of the video clip are deleted to keep the number of input video frames consistent, and then they are input into the backward network. The backward network reversely predicts the observed frame based on the frame predicted by the forward network and part of the observed frame. By comparing the difference between the predicted value and the true value of the observed frame, it is determined whether an abnormal event occurs within the preset number of frames in the future, thereby solving the video anomaly prediction task.

[0060] Specifically, during the anomaly prediction inference process, the backward network uses the future frames predicted by the forward network and observation frame f t-1 ,…,f t+i+1-n Take as input the predicted observation frame f t+i-n First, use the predicted frame and the observed true frame f t+i-n The PSNR value between is used as the anomaly score of the i-th future frame (i∈[1,α], α is the prediction time). Then, the minimum value among the α PSNR values ​​is selected as the prediction score score(t+1:t+α). Finally, each expected value is normalized to obtain the anomaly score S(t+1:t+α) in the range [0,1].

[0061] Figure 2It is a schematic diagram of a score-based scene-dependent autoencoder model provided by an embodiment of the present invention. The score-based scene-dependent autoencoder model includes a scene-general hierarchical variational autoencoder model, a score-based diffusion model and a scene condition extraction module. After the video clip is input into the model, the scene-general hierarchical variational autoencoder first encodes the input into high-quality scene-general features, and the scene condition extraction module extracts the input scene feature information. Then, the scene-general features and the scene feature information are sent to the score-based diffusion model. In the latent space, the scene-general features are perturbed by a diffusion process. Next, sampling is performed through a denoiser, and the scene conditions are embedded in the denoiser to obtain scene-dependent features by conditional sampling from the perturbation distribution. Finally, the generated scene-dependent features are decoded by a scene-general hierarchical variational autoencoder to obtain a predicted target frame.

[0062] The considered models cover different training stages, and a stage flag ρ∈{1,2,3} is set to represent different training steps (i.e., scene condition extraction module, general hierarchical variational autoencoder model, and score-based diffusion model). Formally, the overall loss function can be described as follows:

[0063]

[0064] where λ kfd is the temporal loss based on keyframe differences The formula can also be used to train the reverse network. In this way, the reverse network can use the reverse input and observed information to predict video anomalies.

[0065] Figure 3 is a schematic diagram of a scenario-general hierarchical variational autoencoder model provided by an embodiment of the present invention. Specifically, the scenario-general hierarchical variational autoencoder includes a likelihood function Prior p θ (z) and the approximate posterior q φ (z|x in ). Let φ and θ be the scene-universal hierarchical variational encoder and scene-universal hierarchical variational decoder, respectively, and z is a feature in the latent space. Given an input video clip x consisting of n frames in =f t-n ,…,f t-1 , the scene-general hierarchical variational encoder encodes the input group into latent features z1,z2,…,z N , N represents the number of layers and groups of the hierarchical autoencoder. The latent variables are represented in the form of feature maps of different resolutions. z1 represents the set of variables with the lowest resolution at the top of the network, z NRepresenting a set of latent variables at the bottom of the network, corresponding to the highest resolution, each latent variable in the prior and approximate posterior can be represented by a multiplicative normal distribution. In this way, the prior and approximate posterior can be expressed as:

[0066]

[0067] The scene-general hierarchical variational encoder φ takes an input video sequence x in The encoding process is performed through three layers of encoder units, each of which contains a residual encoder block and a downsampling layer. In each encoder block, the input x in First, the spatiotemporal features are initially encoded through convolution operations. Then, the correlation between feature channels is obtained through squeezing and excitation operations. Then, the connection relationship in the horizontal, vertical and depth directions of the features is obtained through multi-head self-attention operations. Then, the connection relationship is summed through aggregation operations to obtain spatiotemporal features that integrate long-distance information of the video. Finally, the downsampling block reduces the dimension of the spatiotemporal features encoded by the encoder through convolution operations to obtain the encoded spatiotemporal features z, which follow the approximate posterior distribution q φ (z|x in ).

[0068] The scene-general hierarchical variational decoder θ decodes the encoded spatiotemporal features z. The decoding process is performed through three layers of decoder units, each of which contains a residual decoder block and an upsampling layer. In each decoder block, the input z is first initially decoded through a convolution operation. Then, through squeezing and excitation operations, the correlation between feature channels is decoded to filter out the attention between channels. Then, through multi-head self-attention operations, the connection relationship between each pixel of the feature map and its horizontal, vertical, and depth directions is decoded. Then, through an aggregation operation, the connection relationship is summed to decode the refined spatiotemporal features that incorporate long-distance information of the video. Finally, the upsampling block uses a convolution operation to increase the dimension of the spatiotemporal features decoded by the decoder to obtain the predicted target frame.

[0069] The scene-general hierarchical variational encoder φ and scene-general hierarchical variational decoder θ can be trained end-to-end, and their loss can be written as:

[0070]

[0071] Using negative log-likelihood To estimate the prediction loss It aims to maintain the consistency between the target frame and the predicted frame. The KL divergence term of the scene-general hierarchical variational autoencoder can be further written as:

[0072] KL(q φ (z|x in )||pθ (z)

[0073]

[0074] Example:

[0075] like Figure 1 As shown, the scene-dependent video anomaly detection and prediction method provided by the embodiment of the present invention includes:

[0076] A preset number of video frames in the video to be processed are obtained, and the preset number of video frames are scaled, cropped, and mean normalized to obtain video data with a dimension of 3×T×224×224. Optionally, T represents a preset number of video frames selected from the video to be processed. In this embodiment, T=8. The video data is sent to a forward and backward frame prediction network framework.

[0077] The forward and backward frame prediction network framework consists of a forward network and a backward network. Both have the same network architecture and are constructed using a score-based scene-dependent autoencoder model to solve the scene-dependent problem of abnormal events. The input video clip is processed by the forward network and the predicted video frame is output, which is regarded as the current frame. By comparing the difference between the predicted value and the true value of the current frame, it is determined whether an abnormal event has occurred, thereby solving the video anomaly detection task.

[0078] Specifically, in the anomaly detection inference process, the video is divided into n video clips with T frames. In each T-frame segment in the video, the fraction of frames with the minimum peak signal-to-noise ratio (PSNR) value is regarded as the score of the video clip. In order to quantify the probability of anomaly occurrence, each PSNR value score(t) is normalized to obtain an anomaly score S(y) in the range of [0,1], which can be written as:

[0079]

[0080] Then the predicted current frame is added to the input video clip, and the first frame of the video clip is deleted to keep the number of input video frames consistent, forming a new input video clip, which is input into the forward network, and the predicted future frames are output. After the forward network predicts the preset number of future frames in this autoregressive manner, these preset number of predicted future frames are added to the original input video clip, and the preset number of frames at the head of the video clip are deleted to keep the number of input video frames consistent, and then they are input into the backward network. The backward network reversely predicts the observed frame based on the frame predicted by the forward network and part of the observed frame. By comparing the difference between the predicted value and the true value of the observed frame, it is determined whether an abnormal event occurs within the preset number of frames in the future, thereby solving the video anomaly prediction task.

[0081] Specifically, during the anomaly prediction inference process, the backward network uses the future frames predicted by the forward network and observation frame f t-1 ,…,f t+i+1-n Take as input the predicted observation frame f t+i-n First, use the predicted frame and the observed true frame f t+i-n The PSNR value between is used as the anomaly score of the i-th future frame (i∈[1,α], α is the prediction time). Then, the minimum value among the α PSNR values ​​is selected as the prediction score score(t+1:t+α). Finally, each expected value is normalized to obtain the anomaly score S(t+1:t+α) in the range [0,1].

[0082] Figure 2 It is a schematic diagram of a score-based scene-dependent autoencoder model provided by an embodiment of the present invention. The score-based scene-dependent autoencoder model includes a scene-general hierarchical variational autoencoder model, a score-based diffusion model and a scene condition extraction module. After the video clip is input into the model, the scene-general hierarchical variational autoencoder first encodes the input into high-quality scene-general features, and the scene condition extraction module extracts the input scene feature information. Then, the scene-general features and the scene feature information are sent to the score-based diffusion model. In the latent space, the scene-general features are perturbed by a diffusion process. Next, sampling is performed through a denoiser, and the scene conditions are embedded in the denoiser to obtain scene-dependent features by conditional sampling from the perturbation distribution. Finally, the generated scene-dependent features are decoded by a scene-general hierarchical variational autoencoder to obtain a predicted target frame.

[0083] The considered models cover different training stages, and a stage flag ρ∈{1,2,3} is set to represent different training steps (i.e., scene condition extraction module, general hierarchical variational autoencoder model, and score-based diffusion model). Formally, the overall loss function can be described as follows:

[0084]

[0085] where λ kfd is the temporal loss based on keyframe differences The formula can also be used to train the reverse network. In this way, the reverse network can use the reverse input and observed information to predict video anomalies.

[0086] Figure 3 is a schematic diagram of a scenario-general hierarchical variational autoencoder model provided by an embodiment of the present invention. Specifically, the scenario-general hierarchical variational autoencoder includes a likelihood function Prior p θ(z) and the approximate posterior q φ (z|x in ). Let φ and θ be the scene-universal hierarchical variational encoder and scene-universal hierarchical variational decoder, respectively, and z is a feature in the latent space. Given an input video clip x consisting of n frames in =f t-n ,…,f t-1 , the scene-general hierarchical variational encoder encodes the input group into latent features z1,z2,…,z N , N represents the number of layers and groups of the hierarchical autoencoder. The latent variables are represented in the form of feature maps of different resolutions. z1 represents the set of variables with the lowest resolution at the top of the network, z N Representing a set of latent variables at the bottom of the network, corresponding to the highest resolution, each latent variable in the prior and approximate posterior can be represented by a multiplicative normal distribution. In this way, the prior and approximate posterior can be expressed as:

[0087]

[0088]

[0089] The scene-general hierarchical variational encoder φ takes an input video sequence x in The encoding process is performed through three layers of encoder units, each of which contains a residual encoder block and a downsampling layer. In each encoder block, the input x in First, the spatiotemporal features are initially encoded through convolution operations. Then, the correlation between feature channels is obtained through squeezing and excitation operations. Then, the connection relationship in the horizontal, vertical and depth directions of the features is obtained through multi-head self-attention operations. Then, the connection relationship is summed through aggregation operations to obtain spatiotemporal features that integrate long-distance information of the video. Finally, the downsampling block reduces the dimension of the spatiotemporal features encoded by the encoder through convolution operations to obtain the encoded spatiotemporal features z, which follow the approximate posterior distribution q φ (z|x in ).

[0090] The scene-general hierarchical variational decoder θ decodes the encoded spatiotemporal features z. The decoding process is performed through three layers of decoder units, each of which contains a residual decoder block and an upsampling layer. In each decoder block, the input z is first initially decoded through a convolution operation. Then, through squeezing and excitation operations, the correlation between feature channels is decoded to filter out the attention between channels. Then, through multi-head self-attention operations, the connection relationship between each pixel of the feature map and its horizontal, vertical, and depth directions is decoded. Then, through an aggregation operation, the connection relationship is summed to decode the refined spatiotemporal features that incorporate long-distance information of the video. Finally, the upsampling block uses a convolution operation to increase the dimension of the spatiotemporal features decoded by the decoder to obtain the predicted target frame.

[0091] The scene-general hierarchical variational encoder φ and scene-general hierarchical variational decoder θ can be trained end-to-end, and their loss can be written as:

[0092]

[0093] Using negative log-likelihood To estimate the prediction loss It aims to maintain the consistency between the target frame and the predicted frame. The KL divergence term of the scene-general hierarchical variational autoencoder can be further written as:

[0094]

Claims

1. A scene-dependent video anomaly detection and prediction method based on latent space prediction model, characterized in that: The steps include: Step 1: Obtain a preset number of video frames in the video to be processed to form an input video clip, and input it into the forward and backward frame prediction network framework; Step 2: The forward and backward frame prediction network framework consists of a forward network and a backward network. Both have the same network architecture and are constructed using a score-based scene-dependent autoencoder model to solve the scene-dependent problem of abnormal events. Step 3: The input video clip is processed by the forward network and the predicted video frame is output, which is regarded as the current frame; Step 4: By comparing the difference between the predicted value and the true value of the current frame, it is determined whether an abnormal event has occurred, thereby solving the video anomaly detection task; Step 5: Add the predicted current frame to the input video segment and delete the first frame of the video segment to keep the number of input video frames consistent, form a new input video segment, input it into the forward network, and output the predicted future frame; Step 6: After the forward network predicts a preset number of future frames in an autoregressive manner, the preset number of predicted future frames are added to the original input video clip, and the preset number of frames at the head of the video clip are deleted to keep the number of input video frames consistent, and then input into the backward network, which reversely predicts the observed frame based on the frame predicted by the forward network and a part of the observed frame; Step 7: By comparing the difference between the predicted value and the true value of the observed frame, it is determined whether an abnormal event will occur within a preset number of frames in the future, thereby solving the video anomaly prediction task.

2. According to claim 1, a scene-dependent video anomaly detection and prediction method based on latent space prediction model is characterized in that: The score-based scene-dependent autoencoder model includes a scene-universal hierarchical variational autoencoder model, a score-based diffusion model, and a scene condition extraction module; after the video clip is input into the model, the scene-universal hierarchical variational autoencoder first encodes the input into scene-universal features, and the scene condition extraction module extracts the input scene feature information; Then, the scene-general features and scene feature information are fed into the score-based diffusion model. Next, sampling is performed through the denoiser, and the scene conditions are embedded in the denoiser to obtain scene-dependent features by conditional sampling from the perturbation distribution. Finally, the generated scene-dependent features are decoded by the scene-general hierarchical variational autoencoder to obtain the predicted target frame.

3. The scene-dependent video anomaly detection and prediction method based on latent space prediction model according to claim 2 is characterized in that: The scenario-general hierarchical variational autoencoder model includes a likelihood function Prior p θ (z) and the approximate posterior q φ (z|x in ); define φ and θ as the scene-general hierarchical variational encoder and scene-general hierarchical variational decoder respectively, z is the feature in the latent space; given an input video clip x consisting of n frames in =f t-n ,…,f t-1 , the scene-general hierarchical variational encoder encodes the input group into latent features z1,z2,…,z N , N represents the number of layers and groups of the hierarchical autoencoder; the latent variables are represented in the form of feature maps of different resolutions; z1 represents the set of variables with the lowest resolution at the top of the network, z N Denote the set of latent variables at the bottom of the network, corresponding to the highest resolution, and each latent variable in the prior and approximate posterior is represented by a multiplicative normal distribution, so that the prior and approximate posterior are expressed as:

4. The scene-dependent video anomaly detection and prediction method based on latent space prediction model according to claim 3 is characterized in that: The scene-general hierarchical variational encoder φ is used for the input video sequence x in Encoding; the encoding process is performed through three layers of encoder units, each encoder unit contains a residual encoder block and a downsampling layer; in each encoder block, the input x in First, the spatiotemporal features are initially encoded through convolution operations; then the correlation between feature channels is obtained through squeezing and excitation operations; then the connection relationship in the horizontal, vertical and depth directions of the features is obtained through multi-head self-attention operations; then the connection relationship is summed through aggregation operations to obtain the spatiotemporal features that integrate the long-distance information of the video; finally, the downsampling block reduces the dimension of the spatiotemporal features encoded by the encoder through convolution operations to obtain the encoded spatiotemporal features z, which follow the approximate posterior distribution q φ (z|x in ); The scene-general hierarchical variational decoder θ decodes the encoded spatiotemporal features z; The decoding process is performed through three layers of decoder units, each of which contains a residual decoder block and an upsampling layer. In each decoder block, the input z is first decoded preliminarily through a convolution operation. Then, through squeezing and excitation operations, the correlation between feature channels is decoded to filter out the attention between channels. Then, through multi-head self-attention operations, the connection relationship between each pixel of the feature map and its horizontal, vertical and depth directions is decoded. Then, through aggregation operations, the connection relationship is summed to decode the refined spatiotemporal features that integrate the long-distance information of the video. Finally, the upsampling block uses convolution operations to increase the dimension of the spatiotemporal features decoded by the decoder to obtain the predicted target frame. The scene-general hierarchical variational encoder φ and scene-general hierarchical variational decoder θ can be trained end-to-end, and its loss is written as: Using negative log-likelihood To estimate the prediction loss Aims to maintain consistency between the target frame and the predicted frame; the KL divergence term of the scene-general hierarchical variational autoencoder can be further written as:

5. The scene-dependent video anomaly detection and prediction method based on latent space prediction model according to claim 4 is characterized in that: The score-based diffusion model transforms the scene-general prior p obtained by the scene-general scene variational autoencoder θ (z) is refined into a scene-dependent prior p ψ (z); The score-based diffusion model includes a forward diffusion process and a reverse denoising process; After training the scene-general hierarchical variational encoder φ and decoder θ, the time-dependent score function is trained using the reverse stochastic differential equation Sampling is performed from z(s); the scene-general hierarchical variational decoder θ then decodes the sampled scene-dependent features into video frames, as described by:

6. The scene-dependent video anomaly detection and prediction method based on latent space prediction model according to claim 5 is characterized in that: The forward diffusion process adds noise to the spatiotemporal features encoded by the scene-general hierarchical variational encoder φ, which is indexed by a continuous time variable s∈[0,1] and is represented as where z(0) is the initial state of feature z and z(s) is the perturbation of feature z at time s; this diffusion process is modeled as the solution of a stochastic differential equation: dz=f(z,s)ds+g(s)d where f(·,s):R→R is a vector-valued function, which is considered as the drift coefficient of z(s); g(·):R→R is a scalar function, which is considered as the diffusion coefficient of z(s); ω represents the standard Wiener process of a time-continuous random process; f(z,s) and g(s) make the state z(1) after the feature is fully perturbed follow a normal distribution at the end of the diffusion process; a reverse denoising model ε is trained by sampling from z(1) and then inverting the stochastic differential equation ψ , expressed as: in is a time-reverse Wiener process, ds is an infinitesimal negative time step, is the score function of the characteristic marginal distribution under the forward diffusion process time s; The reverse denoising model is implemented by a noise conditional score network, which is composed of residual blocks; in the residual block, the samples sampled in the input z(s) are first downsampled by a convolution operation, then intermediately processed by a convolution operation, and finally upsampled by a convolution operation, and the output predicted noise is denoised.

7. The scene-dependent video anomaly detection and prediction method based on latent space prediction model according to claim 6 is characterized in that: The generation process of the sampled scene-dependent features can be equivalently regarded as upsampling the scene-general hierarchical variational autoencoder on the prior of the score-based diffusion model; the scene-general hierarchical variational autoencoder and the score-based diffusion model are jointly trained by minimizing the variational upper bound of the negative log-likelihood: in is the frame prediction loss, which is optimized by decomposing the KL divergence term of the score-based diffusion model into its entropy term and cross entropy term, which can be written as:

8. The scene-dependent video anomaly detection and prediction method based on latent space prediction model according to claim 7 is characterized in that: The scene condition extraction module embeds the condition factor into the denoiser ∈ ψ In order to achieve ∈ ψ (z(s))→∈ ψ (z(s),c), where c is the combination of the time step s and the scene condition extracted by the trainable scene information extractor; the scene condition extraction module includes a scene information extractor E sce and a scene information decoder D sce ; Scene information extractor E sce Pass the condition to the denoiser ∈ ψ All hidden layers of , thus forcing the denoiser ∈ ψ Depends on E sce (x in ) to generate scene dependencies; the scene information decoder is responsible for reconstructing the scene information D to which the input belongs sce (E sce (x in ))→scene(x in ).

9. The scene-dependent video anomaly detection and prediction method based on latent space prediction model according to claim 8, characterized in that: The scene information extractor E sce It consists of three convolutional layers. In each convolutional layer, the input x in The scene features are obtained by processing through convolution operation and then activating through LeakyReLU function; Scene Information Decoder D sce It consists of five convolutional layers. In each convolutional layer, the input scene features are processed by convolution operation and then activated by LeakyReLU function to obtain the reconstructed scene graph. This micro autoencoder is pre-trained by reconstruction loss, which can be written as:

10. The scene-dependent video anomaly detection and prediction method based on latent space prediction model according to claim 9, characterized in that: Train the denoiser ∈ using the classifier-free diffusion-guided strategy ψ , including learning from conditional and unconditional distributions, and implementing the following linear combinations: Where λ>1 is the guidance scale, which can enhance the guidance effect; c is the time step s and the extracted scene information E sce (x in ) combination; after the interactive reverse process of conditional denoising, the target frame can be effectively predicted from the denoised frame; An additional temporal coherence term is used to ensure that the predicted frame maintains temporal consistency with the real video sequence; in the training stage, let X = {f t-n ,…,f t-1 ,f t ,f t+1 ,…,f t+n } is a video clip, where f t is the target frame to be predicted; select f t-n ,f t-1 As f t-n ,…,f t-1 In the key frame, select f t+1 ,f t+n As f t+1 ,…,f t+n The time loss based on the key frame difference is written as follows: A stage flag ρ∈{1,2,3} is set to represent different training steps, namely the scene condition extraction module, the general hierarchical variational autoencoder model, and the score-based diffusion model; formally, the overall loss function is described as follows: where λ kfd is the temporal loss based on keyframe differences The weight of In the anomaly detection reasoning process, the video is divided into n video clips with T frames. In each T-frame segment in the video, the fraction of frames with the minimum peak signal-to-noise ratio (PSNR) value is regarded as the score of the video clip. In order to quantify the probability of anomaly occurrence, each PSNR value score(t) is normalized to obtain an anomaly score S(t) in the range of [0,1], written as: During the anomaly prediction inference process, the backward network uses the future frames predicted by the forward network. and observation frame f t-1 ,…,f t+i+1-n Take as input the predicted observation frame f t+i-n ; First use the predicted frame and the observed true frame f t+i-n The PSNR value between is used as the anomaly score of the i-th future frame, i∈[1,α], α is the prediction time; Then, the minimum of the α PSNR values ​​is selected as the prediction score score(t+1:t+α); finally, each expected value is normalized to obtain an anomaly score S(t+1:t+α) in the range [0,1].

Images