Scene-dependent video anomaly detection and prediction method based on latent space prediction model

Abstract

The present invention discloses a scene-dependent video anomaly detection and prediction method based on a latent space prediction model. First, a preset number of video frames in a video to be processed are obtained to form an input video. The input video clip is processed by a forward network, and a predicted video frame is output. The predicted video frame is regarded as the current frame. The difference between the predicted value and the true value of the current frame is compared to determine whether an abnormal event has occurred, thereby solving the video anomaly detection task. After the predicted current frame is added to the input video clip, the predicted future frame is output. After the forward network predicts a preset number of future frames in an autoregressive manner, it is input into a backward network. The difference between the predicted value and the true value of the observed frame is compared to determine whether an abnormal event has occurred within a preset number of frames in the future, thereby solving the video anomaly prediction task. The method of the present invention can effectively handle scene-dependent anomaly detection and prediction.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a scene-dependent video anomaly detection and prediction method based on a latent space prediction model. Background Art

[0002] Video anomaly detection is a fundamental and critical problem in computer vision. Its purpose is to detect unexpected anomalous events in videos. As a key application technology in intelligent surveillance systems, it plays a crucial role in preventing dangerous incidents and ensuring public safety.

[0003] Because the types of abnormal events are inexhaustible and relatively rare, related technologies typically treat video anomaly detection as a semi-supervised task. The training data consists only of normal events without specific labels. The goal is to train a single classifier that faithfully learns the distribution of normal data and avoids misgeneralization to abnormal events. Specifically, existing methods typically construct deep autoencoders based on reconstructing or predicting video frames to learn the characteristics of normal events in the video. Any event that significantly deviates from the learned normal characteristics is considered an anomaly. However, these methods have limited scene understanding and struggle to detect abnormal events that rely on specific scene conditions, such as a cyclist on the sidewalk. In addition to detecting ongoing abnormal events, predicting future abnormal events is also a practical task worthy of research attention. Providing early warning before an abnormal event occurs could potentially prevent significant loss of life and property. However, this task has yet to attract widespread attention. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the present invention provides a scene-dependent video anomaly detection and prediction method based on a latent space prediction model. First, a preset number of video frames in the video to be processed are obtained to form an input video clip, which is input into a forward and backward frame prediction network framework; a score-based scene-dependent autoencoder model is used to construct the method to solve the scene dependency problem of abnormal events; the input video clip is processed by the forward network, and a predicted video frame is output, which is regarded as the current frame; by comparing the difference between the predicted value and the true value of the current frame, it is determined whether the abnormal event occurs, thereby solving the video anomaly detection task; after the predicted current frame is added to the input video clip, The first frame of the video clip is deleted to keep the number of input video frames consistent, forming a new input video clip, which is input into the forward network and outputs the predicted future frames; after the forward network predicts a preset number of future frames in an autoregressive manner, these preset number of predicted future frames are added to the original input video clip, and a preset number of frames at the head of the video clip are deleted to keep the number of input video frames consistent, and then it is input into the backward network. The backward network reversely predicts the observed frame based on the frame predicted by the forward network and a portion of the observed frame; by comparing the difference between the predicted value and the true value of the observed frame, it is determined whether an abnormal event will occur within the preset number of frames in the future, thereby solving the video anomaly prediction task. The method of the present invention can well handle scene-related anomaly detection and prediction.

[0005] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0006] Step 1: Get a preset number of video frames from the video to be processed to form an input video clip, and input it into the forward and backward frame prediction network framework;

[0007] Step 2: The forward and backward frame prediction network framework consists of a forward network and a backward network. Both networks have the same network architecture and are built using a score-based scene-dependent autoencoder model to address the scene-dependent problem of abnormal events.

[0008] Step 3: The input video clip is processed by the forward network and the predicted video frame is output. The predicted video frame is regarded as the current frame.

[0009] Step 4: By comparing the difference between the predicted value and the true value of the current frame, it is determined whether an abnormal event has occurred, thereby solving the video anomaly detection task;

[0010] Step 5: Add the predicted current frame to the input video segment and delete the first frame of the video segment to keep the number of input video frames consistent, forming a new input video segment, which is fed into the forward network and outputs the predicted future frame;

[0011] Step 6: After the forward network predicts a preset number of future frames in an autoregressive manner, these preset number of predicted future frames are added to the original input video clip, and the preset number of frames at the head of the video clip are deleted to keep the number of input video frames consistent. The video clip is then input into the backward network, which reversely predicts the observed frame based on the frame predicted by the forward network and a portion of the observed frame.

[0012] Step 7: By comparing the difference between the predicted value and the true value of the observation frame, it is determined whether an abnormal event will occur within a preset number of frames in the future, thereby solving the video anomaly prediction task.

[0013] Furthermore, the score-based scene-dependent autoencoder model includes a scene-universal hierarchical variational autoencoder model, a score-based diffusion model, and a scene condition extraction module. After a video clip is input into the model, the scene-universal hierarchical variational autoencoder first encodes the input into scene-universal features, while the scene condition extraction module extracts the input scene feature information. Then, the scene-universal features and scene feature information are fed into the score-based diffusion model. Next, sampling is performed through a denoiser, and the scene condition is embedded in the denoiser to conditionally sample from the perturbation distribution to obtain scene-dependent features. Finally, the generated scene-dependent features are decoded by the scene-universal hierarchical variational autoencoder to obtain a predicted target frame.

[0014] Furthermore, the scenario-general hierarchical variational autoencoder model includes a likelihood function Prior p θ (z) and the approximate posterior q φ (z|x in ); define φ and θ as the scene-universal hierarchical variational encoder and scene-universal hierarchical variational decoder respectively, z is the feature in the latent space; given an input video clip x consisting of n frames in =f t-n ,…,f -1 , the scene-general hierarchical variational encoder encodes the input group into latent features z1,z2,…,z N , N represents the number of layers and groups of the hierarchical autoencoder; the latent variables are represented in the form of feature maps of different resolutions; z1 represents the set of variables with the lowest resolution at the top of the network, z N Denote the set of latent variables at the bottom of the network, corresponding to the highest resolution, where each latent variable in the prior and approximate posterior is represented by a multiplicative normal distribution, such that the prior and approximate posterior are expressed as:

[0015]

[0016] Furthermore, the scene-universal hierarchical variational encoder φ is used to transform the input video sequence x inEncoding; the encoding process is performed through three layers of encoder units, each encoder unit contains a residual encoder block and a downsampling layer; in each encoder block, the input x in First, the spatiotemporal features are preliminarily encoded through convolution operations; then, the correlation between feature channels is obtained through squeezing and excitation operations; then, the connection relationship in the horizontal, vertical and depth directions of the features is obtained through multi-head self-attention operations; then, the connection relationship is summed through aggregation operations to obtain spatiotemporal features that integrate long-distance video information; finally, the downsampling block reduces the dimension of the spatiotemporal features encoded by the encoder through convolution operations to obtain the encoded spatiotemporal features z, which follow the approximate posterior distribution q φ (z|x in ).

[0017] Furthermore, the scene-general hierarchical variational decoder θ decodes the encoded spatiotemporal features z; the decoding process is performed by three layers of decoder units, each of which contains a residual decoder block and an upsampling layer; in each decoder block, the input z is first preliminarily decoded by a convolution operation; then, through squeezing and excitation operations, the correlation between feature channels is decoded to filter out the attention between channels; then, through multi-head self-attention operations, the connection relationship between each pixel point of the feature map and its horizontal, vertical and depth directions is decoded; then, through aggregation operations, the connection relationship is summed to decode the refined spatiotemporal features that integrate the long-distance information of the video; finally, the upsampling block performs a dimensionality-up on the spatiotemporal features decoded by the decoder through a convolution operation to obtain the predicted target frame

[0018] Furthermore, the scene-universal hierarchical variational encoder φ and scene-universal hierarchical variational decoder θ can be trained end-to-end, and their loss is written as:

[0019]

[0020] Using negative log-likelihood To estimate the prediction loss Aims to maintain consistency between the target frame and the predicted frame; the KL divergence term of the scene-universal hierarchical variational autoencoder can be further written as:

[0021]

[0022] Furthermore, the score-based diffusion model transforms the scene-general prior p obtained by the scene-general variational autoencoder into θ (z) is refined into a scene-dependent prior p ψ (z); The score-based diffusion model includes a forward diffusion process and a reverse denoising process; after training the scene-universal hierarchical variational encoder φ and decoder θ, the time-dependent score function is trained using the reverse stochastic differential equation Sampling is performed from z(s); the scene-general level variational decoder θ then decodes the sampled scene-dependent features into video frames, as described by:

[0023]

[0024] Furthermore, the forward diffusion process adds noise to the spatiotemporal features encoded by the scene-general hierarchical variational encoder φ, which is indexed by a continuous time variable s∈[0,1] and represented as where z(0) is the initial state of feature z and z(s) is the perturbation of feature z at time s. This diffusion process is modeled as the solution of the stochastic differential equation:

[0025] dz=f(z,s)ds+g(s)d

[0026] where f(·,s):R→R is a vector-valued function that is considered as the drift coefficient of z(s); g(·):R→R is a scalar function that is considered as the diffusion coefficient of z(s); ω represents the standard Wiener process of a time-continuous random process; f(z,s) and g(s) make the state z(1) after the feature is fully perturbed follow a normal distribution at the end of the diffusion process; by sampling from z(1) and then inverting the stochastic differential equation, an inverse denoising model εψ is trained, which is expressed as:

[0027]

[0028] in is a time-reverse Wiener process, ds is an infinitesimal negative time step, is the score function of the characteristic marginal distribution under the forward diffusion process time s.

[0029] Furthermore, the inverse denoising model is implemented by a noise conditional score network, which is composed of a residual block; in the residual block, the samples sampled in the input z(s) are first downsampled by a convolution operation, then intermediately processed by a convolution operation, and finally upsampled by a convolution operation to output the predicted noise for denoising.

[0030] Furthermore, the generation process of the sampled scene-dependent features can be equivalently viewed as upsampling the scene-universal hierarchical variational autoencoder on the prior of the score-based diffusion model; the scene-universal hierarchical variational autoencoder and the score-based diffusion model are jointly trained by minimizing the variational upper bound of the negative log-likelihood:

[0031]

[0032] in is the frame prediction loss, which is optimized by decomposing the KL divergence term of the score-based diffusion model into its entropy term and cross entropy term, which can be written as:

[0033]

[0034] Furthermore, the scene condition extraction module embeds the condition factor into the denoiser ∈ ψ In order to realize ∈ ψ (z(s))→∈ ψ (z(s),c), where c is a combination of the time step s and the scene condition extracted by the trainable scene information extractor; the scene condition extraction module includes a scene information extractor E sce and a scene information decoder D sce ; Scene information extractor E sce Pass the condition to the denoiser ∈ ψ All hidden layers of , thus forcing the denoiser ∈ ψ Depends on E sce (x in ) to generate scene dependencies; the scene information decoder is responsible for reconstructing the scene information D to which the input belongs sce (E sce (x in ))→scene(x in ).

[0035] Furthermore, the scene information extractor E sce It consists of three convolutional layers. In each convolutional layer, the input x in The scene features are obtained by processing through convolution operation and then activated by LeakyReLU function; the scene information decoder D sce It consists of five convolutional layers. In each convolutional layer, the input scene features are processed by convolution operation and then activated by LeakyReLU function to obtain the reconstructed scene graph. This micro autoencoder is pre-trained by reconstruction loss, which can be written as:

[0036]

[0037] Furthermore, the denoiser ∈ ψ , including learning from conditional and unconditional distributions, and implementing the following linear combination:

[0038]

[0039] Where λ>1 is the guidance scale, which can enhance the guidance effect; c is the time step s and the extracted scene information E sce (x in); after the interactive reverse process of conditional denoising, the target frame can be effectively predicted from the denoised frame.

[0040] Furthermore, an additional temporal coherence term is proposed to ensure that the predicted frames maintain temporal consistency with the real video sequence; in the training phase, let X = {f t-n ,…,f t-1 ,f t ,f t+1 ,…,f t+n} is a video clip, where f t is the target frame to be predicted; select f t-n ,f t-1 As f t-n ,…,f t-1 In the key frame, select f t+1 ,f t+n As f t+1 ,…,f t+n The time loss based on the key frame difference is written as follows:

[0041]

[0042] Furthermore, a stage flag ρ∈{1,2,3} is set to represent different training steps, namely the scene condition extraction module, the general hierarchical variational autoencoder model, and the score-based diffusion model; formally, the overall loss function is described as follows:

[0043]

[0044] where λ kfd is the temporal loss based on keyframe differences The weight of .

[0045] Furthermore, in the anomaly detection inference process, the video is divided into n video segments with T frames. In each T-frame segment in the video, the fraction of frames with the minimum peak signal-to-noise ratio (PSNR) value is regarded as the score of the video clip. In order to quantify the probability of anomaly occurrence, each PSNR value score(t) is normalized to obtain an anomaly score S(t) in the range of [0,1], which is written as:

[0046]

[0047] Furthermore, during the anomaly prediction inference process, the backward network uses the future frames predicted by the forward network and observation frame f t-1 ,…,f t+i+1-n Take as input the predicted observation frame f t+i-n ;First use the predicted frame and the observed true frame ft+i-n The PSNR value between is used as the anomaly score of the i-th future frame, i∈[1,α], α is the prediction time; then, the minimum of the α PSNR values ​​is selected as the prediction score score(t+1:t+α); finally, each expected value is normalized to obtain the anomaly score S(t+1:t+α) in the range [0,1].

[0048] The beneficial effects of the present invention are as follows:

[0049] The present invention provides a scene-dependent video anomaly detection and prediction method, which obtains a preset number of video frames in the video to be processed and inputs them into a forward and backward frame prediction framework for video anomaly detection and prediction. Based on this framework, a new score-based scene-dependent autoencoder is designed, which is a diffusion-based model for detecting and predicting scene-dependent abnormal events. In this model, a more powerful hierarchical variational autoencoder is introduced to extract scene-general features, and a score-based diffusion model in the latent space is introduced to refine these features and generate scene-dependent features. An effective conditional adjustment strategy is applied to the diffusion model to generate scene-dependent samples. In addition, a temporal loss is proposed to constrain the motion consistency of objects in the video sequence. The method of the present invention can well handle scene-dependent anomaly detection and prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic flow diagram of the method of the present invention;

[0051] Figure 2 Schematic diagram of the score-based scene-dependent autoencoder model of the present invention;

[0052] Figure 3 This is a schematic diagram of a universal hierarchical variational autoencoder model for the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described below with reference to the accompanying drawings and examples.

[0054] like Figure 1 As shown, the scene-dependent video anomaly detection and prediction method provided by the embodiment of the present invention includes:

[0055] A preset number of video frames from the video to be processed are obtained, and these frames are scaled, cropped, and mean-normalized to obtain video data with a dimension of 3 × T × 224 × 224. Optionally, T represents a preset number of video frames selected from the video to be processed; in this embodiment, T = 8. This video data is fed into a forward and backward frame prediction network framework.

[0056] The forward and backward frame prediction network framework consists of a forward network and a backward network, both of which share the same network architecture. They are built using a score-based scene-dependent autoencoder model to address the scene-dependent nature of abnormal events. The forward network processes the input video clip and outputs a predicted video frame, which is considered the current frame. By comparing the predicted value of the current frame with the true value, the team determines whether an abnormal event has occurred, thereby solving the video anomaly detection task.

[0057] Specifically, in the anomaly detection inference process, the video is divided into n video segments with T frames. In each T-frame segment in the video, the fraction of frames with the minimum peak signal-to-noise ratio (PSNR) value is regarded as the score of the video clip. In order to quantify the probability of anomaly occurrence, each PSNR value score(t) is normalized to obtain an anomaly score S(t) in the range of [0,1], which can be written as:

[0058]

[0059] The predicted current frame is then appended to the input video clip, and the first frame of the video clip is deleted to maintain the same number of input video frames. This new input video clip is then fed into the forward network, which outputs the predicted future frames. After the forward network has predicted a preset number of future frames using this autoregressive method, these predicted future frames are appended to the original input video clip, and a preset number of frames at the beginning of the video clip are deleted to maintain the same number of input video frames. This is then fed into the backward network, which reversely predicts the observed frames based on the frames predicted by the forward network and a portion of the observed frames. By comparing the difference between the predicted values ​​and the true values ​​of the observed frames, it is determined whether an abnormal event will occur within the preset number of frames in the future, thereby solving the video anomaly prediction task.

[0060] Specifically, during the anomaly prediction inference process, the backward network uses the future frames predicted by the forward network and observation frame f t-1 ,…,f t+i+1-n Take as input the predicted observation frame f t+i-n First, use the predicted frame and the observed true frame f t+i-n The PSNR value between is used as the anomaly score of the i-th future frame (i∈[1,α], α is the prediction time). Then, the minimum value among the α PSNR values ​​is selected as the prediction score score(t+1:t+α). Finally, each expected value is normalized to obtain the anomaly score S(t+1:t+α) in the range [0,1].

[0061] Figure 2: This is a schematic diagram of a score-based scene-dependent autoencoder model provided by an embodiment of the present invention. The score-based scene-dependent autoencoder model includes a scene-general hierarchical variational autoencoder model, a score-based diffusion model and a scene condition extraction module. After the video clip is input into the model, the scene-general hierarchical variational autoencoder first encodes the input into high-quality scene-general features, and at the same time, the scene condition extraction module extracts the input scene feature information. Then, the scene-general features and the scene feature information are fed into the score-based diffusion model. In the latent space, the scene-general features are perturbed through a diffusion process. Next, sampling is performed through a denoiser, and the scene conditions are embedded in the denoiser to obtain scene-dependent features by conditional sampling from the perturbation distribution. Finally, the generated scene-dependent features are decoded by the scene-general hierarchical variational autoencoder to obtain the predicted target frame.

[0062] The considered models cover different training stages, and a stage flag ρ∈{1,2,3} is set to represent different training steps (i.e., scene condition extraction module, general hierarchical variational autoencoder model and score-based diffusion model). Formally, the overall loss function can be described as follows:

[0063]

[0064] where λ kfd is the temporal loss based on keyframe differences The formula can also be used to train the reverse network. In this way, the reverse network can use the reverse input and observed information to predict video anomalies.

[0065] Figure 3 is a schematic diagram of a scenario-general hierarchical variational autoencoder model provided by an embodiment of the present invention. Specifically, the scenario-general hierarchical variational autoencoder includes a likelihood function Prior p θ (z) and the approximate posterior q φ (z|x in ). Let φ and θ be the scene-universal hierarchical variational encoder and scene-universal hierarchical variational decoder, respectively, and z be the feature in the latent space. Given an input video clip x consisting of n frames in =f t-n ,…,f t-1 , the scene-general hierarchical variational encoder encodes the input group into latent features z1,z2,…,z N , N represents the number of layers and groups of the hierarchical autoencoder. The latent variables are represented in the form of feature maps of different resolutions. z1 represents the set of variables with the lowest resolution at the top of the network, z NRepresenting a set of latent variables at the bottom of the network, corresponding to the highest resolution, each latent variable in the prior and approximate posterior can be represented by a multiplicative normal distribution. The prior and approximate posterior can then be expressed as:

[0066]

[0067] The scene-general hierarchical variational encoder φ is trained on the input video sequence x in The encoding process is performed by three layers of encoder units, each of which contains a residual encoder block and a downsampling layer. In each encoder block, the input x in First, the spatiotemporal features are initially encoded through convolution operations. Then, the correlation between feature channels is obtained through squeezing and excitation operations. Then, the connection relationship of the features in the horizontal, vertical and depth directions is obtained through multi-head self-attention operations. Then, the connection relationship is summed through aggregation operations to obtain spatiotemporal features that integrate long-distance information of the video. Finally, the downsampling block reduces the dimension of the spatiotemporal features encoded by the encoder through convolution operations to obtain the encoded spatiotemporal features z, which follow the approximate posterior distribution q. φ (z|x in ).

[0068] The scene-general hierarchical variational decoder θ decodes the encoded spatiotemporal features z. The decoding process is performed by three layers of decoder units, each of which contains a residual decoder block and an upsampling layer. In each decoder block, the input z is first initially decoded by a convolution operation. Then, through squeezing and excitation operations, the correlation between feature channels is decoded, and the attention between channels is filtered out. Then, the connection relationship between each pixel point of the feature map and its horizontal, vertical and depth directions is decoded through a multi-head self-attention operation. The connection relationship is then summed through an aggregation operation to decode the refined spatiotemporal features that incorporate long-distance information of the video. Finally, the upsampling block increases the dimension of the spatiotemporal features decoded by the decoder through a convolution operation to obtain the predicted target frame.

[0069] The scene-general hierarchical variational encoder φ and scene-general hierarchical variational decoder θ can be trained end-to-end, and their loss can be written as:

[0070]

[0071] Using negative log-likelihood To estimate the prediction loss It aims to maintain the consistency between the target frame and the predicted frame. The KL divergence term of the scene-universal hierarchical variational autoencoder can be further written as:

[0072] KL(q φ (z|x in )||pθ (z)

[0073]

[0074] Example:

[0075] like Figure 1 As shown, the scene-dependent video anomaly detection and prediction method provided by the embodiment of the present invention includes:

[0076] A preset number of video frames from the video to be processed are obtained, and these frames are scaled, cropped, and mean-normalized to obtain video data with a dimension of 3 × T × 224 × 224. Optionally, T represents a preset number of video frames selected from the video to be processed; in this embodiment, T = 8. This video data is fed into a forward and backward frame prediction network framework.

[0077] The forward and backward frame prediction network framework consists of a forward network and a backward network, both of which share the same network architecture. They are built using a score-based scene-dependent autoencoder model to address the scene-dependent nature of abnormal events. The forward network processes the input video clip and outputs a predicted video frame, which is considered the current frame. By comparing the predicted value of the current frame with the true value, the team determines whether an abnormal event has occurred, thereby solving the video anomaly detection task.

[0078] Specifically, in the anomaly detection inference process, the video is divided into n video segments with T frames. In each T-frame segment in the video, the fraction of frames with the minimum peak signal-to-noise ratio (PSNR) value is regarded as the score of the video clip. In order to quantify the probability of anomaly occurrence, each PSNR value score(t) is normalized to obtain an anomaly score S(y) in the range of [0,1], which can be written as:

[0079]

[0080] The predicted current frame is then appended to the input video clip, and the first frame of the video clip is deleted to maintain the same number of input video frames. This new input video clip is then fed into the forward network, which outputs the predicted future frames. After the forward network has predicted a preset number of future frames using this autoregressive method, these predicted future frames are appended to the original input video clip, and a preset number of frames at the beginning of the video clip are deleted to maintain the same number of input video frames. This is then fed into the backward network, which reversely predicts the observed frames based on the frames predicted by the forward network and a portion of the observed frames. By comparing the difference between the predicted values ​​and the true values ​​of the observed frames, it is determined whether an abnormal event will occur within the preset number of frames in the future, thereby solving the video anomaly prediction task.

[0081] Specifically, during the anomaly prediction inference process, the backward network uses the future frames predicted by the forward network and observation frame f t-1 ,…,f t+i+1-n Take as input the predicted observation frame f t+i-n First, use the predicted frame and the observed true frame f t+i-n The PSNR value between is used as the anomaly score of the i-th future frame (i∈[1,α], α is the prediction time). Then, the minimum value among the α PSNR values ​​is selected as the prediction score score(t+1:t+α). Finally, each expected value is normalized to obtain the anomaly score S(t+1:t+α) in the range [0,1].

[0082] Figure 2 : This is a schematic diagram of a score-based scene-dependent autoencoder model provided by an embodiment of the present invention. The score-based scene-dependent autoencoder model includes a scene-general hierarchical variational autoencoder model, a score-based diffusion model and a scene condition extraction module. After the video clip is input into the model, the scene-general hierarchical variational autoencoder first encodes the input into high-quality scene-general features, and at the same time, the scene condition extraction module extracts the input scene feature information. Then, the scene-general features and the scene feature information are fed into the score-based diffusion model. In the latent space, the scene-general features are perturbed through a diffusion process. Next, sampling is performed through a denoiser, and the scene conditions are embedded in the denoiser to obtain scene-dependent features by conditional sampling from the perturbation distribution. Finally, the generated scene-dependent features are decoded by the scene-general hierarchical variational autoencoder to obtain the predicted target frame.

[0083] The considered models cover different training stages, and a stage flag ρ∈{1,2,3} is set to represent different training steps (i.e., scene condition extraction module, general hierarchical variational autoencoder model and score-based diffusion model). Formally, the overall loss function can be described as follows:

[0084]

[0085] where λ kfd is the temporal loss based on keyframe differences The formula can also be used to train the reverse network. In this way, the reverse network can use the reverse input and observed information to predict video anomalies.

[0086] Figure 3 is a schematic diagram of a scenario-general hierarchical variational autoencoder model provided by an embodiment of the present invention. Specifically, the scenario-general hierarchical variational autoencoder includes a likelihood function Prior p θ(z) and the approximate posterior q φ (z|x in ). Let φ and θ be the scene-universal hierarchical variational encoder and scene-universal hierarchical variational decoder, respectively, and z be the feature in the latent space. Given an input video clip x consisting of n frames in =f t-n ,…,f t-1 , the scene-general hierarchical variational encoder encodes the input group into latent features z1,z2,…,z N , N represents the number of layers and groups of the hierarchical autoencoder. The latent variables are represented in the form of feature maps of different resolutions. z1 represents the set of variables with the lowest resolution at the top of the network, z N Representing a set of latent variables at the bottom of the network, corresponding to the highest resolution, each latent variable in the prior and approximate posterior can be represented by a multiplicative normal distribution. The prior and approximate posterior can then be expressed as:

[0087]

[0088]

[0089] The scene-general hierarchical variational encoder φ is trained on the input video sequence x in The encoding process is performed by three layers of encoder units, each of which contains a residual encoder block and a downsampling layer. In each encoder block, the input x in First, the spatiotemporal features are initially encoded through convolution operations. Then, the correlation between feature channels is obtained through squeezing and excitation operations. Then, the connection relationship of the features in the horizontal, vertical and depth directions is obtained through multi-head self-attention operations. Then, the connection relationship is summed through aggregation operations to obtain spatiotemporal features that integrate long-distance information of the video. Finally, the downsampling block reduces the dimension of the spatiotemporal features encoded by the encoder through convolution operations to obtain the encoded spatiotemporal features z, which follow the approximate posterior distribution q. φ (z|x in ).

[0090] The scene-general hierarchical variational decoder θ decodes the encoded spatiotemporal features z. The decoding process is performed by three layers of decoder units, each of which contains a residual decoder block and an upsampling layer. In each decoder block, the input z is first initially decoded by a convolution operation. Then, through squeezing and excitation operations, the correlation between feature channels is decoded, and the attention between channels is filtered out. Then, the connection relationship between each pixel point of the feature map and its horizontal, vertical and depth directions is decoded through a multi-head self-attention operation. The connection relationship is then summed through an aggregation operation to decode the refined spatiotemporal features that incorporate long-distance information of the video. Finally, the upsampling block increases the dimension of the spatiotemporal features decoded by the decoder through a convolution operation to obtain the predicted target frame.

[0091] The scene-general hierarchical variational encoder φ and scene-general hierarchical variational decoder θ can be trained end-to-end, and their loss can be written as:

[0092]

[0093] Using negative log-likelihood To estimate the prediction loss It aims to maintain the consistency between the target frame and the predicted frame. The KL divergence term of the scene-universal hierarchical variational autoencoder can be further written as:

[0094]

Claims

1. A scene-dependent video anomaly detection and prediction method based on a latent space prediction model, characterized in that: The steps include: Step 1: Get a preset number of video frames from the video to be processed to form an input video clip, and input it into the forward and backward frame prediction network framework; Step 2: The forward and backward frame prediction network framework consists of a forward network and a backward network. Both networks have the same network architecture and are built using a score-based scene-dependent autoencoder model to address the scene-dependent problem of abnormal events. The score-based scene-dependent autoencoder model includes a scene-universal hierarchical variational autoencoder model, a score-based diffusion model, and a scene condition extraction module; the score-based diffusion model converts the scene-universal prior obtained by the scene-universal scene variational autoencoder into Refinement into scene-dependent prior ; The score-based diffusion model includes a forward diffusion process and a backward denoising process; The forward diffusion process is used to implement the scene-universal hierarchical variational encoder The encoded spatiotemporal features are denoised, and the process consists of a continuous time variable Index, represented as ,in It is a feature The starting state, It is a feature In time Time disturbance; This diffusion process is modeled as the solution of a stochastic differential equation: in, is a vector-valued function, considered as The drift coefficient of is a scalar function, considered as The diffusion coefficient of A standard Wiener process representing a time-continuous random process; and The state after the feature is fully perturbed At the end of the diffusion process it follows a normal distribution; By Sampling, then inverting the stochastic differential equation to train a reverse denoising model , expressed as: in is a time-reverse Wiener process, is an infinitesimal negative time step, is the time of the forward diffusion process The score function of the lower characteristic marginal distribution; The reverse denoising model is implemented by a noise conditional score network, which is composed of residual blocks; in the residual block, the input The mid-sampled samples are first downsampled by the convolution operation, then processed by the convolution operation, and finally upsampled by the convolution operation to output the predicted noise for denoising; The generation process of sampling scene-dependent features can be equivalently viewed as upsampling a scene-universal hierarchical variational autoencoder on the prior of a score-based diffusion model; the scene-universal hierarchical variational autoencoder and the score-based diffusion model are jointly trained by minimizing the variational upper bound of the negative log-likelihood: in is the frame prediction loss, which is optimized by decomposing the KL divergence term of the score-based diffusion model into its entropy term and cross entropy term, which can be written as: ； Step 3: The input video clip is processed by the forward network and the predicted video frame is output. The predicted video frame is regarded as the current frame. Step 4: By comparing the difference between the predicted value and the true value of the current frame, it is determined whether an abnormal event has occurred, thereby solving the video anomaly detection task; Step 5: Add the predicted current frame to the input video segment and delete the first frame of the video segment to keep the number of input video frames consistent, forming a new input video segment, which is fed into the forward network and outputs the predicted future frame; Step 6: After the forward network predicts a preset number of future frames in an autoregressive manner, these preset number of predicted future frames are added to the original input video clip, and the preset number of frames at the head of the video clip are deleted to keep the number of input video frames consistent. The video clip is then input into the backward network, which reversely predicts the observed frame based on the frame predicted by the forward network and a portion of the observed frame. Step 7: By comparing the difference between the predicted value and the true value of the observation frame, it is determined whether an abnormal event will occur within a preset number of frames in the future, thereby solving the video anomaly prediction task.

2. The scene-dependent video anomaly detection and prediction method based on latent space prediction model according to claim 1 is characterized in that: After the video clip is input into the model, the scene-general hierarchical variational autoencoder first encodes the input into scene-general features, while the scene condition extraction module extracts the input scene feature information; Then, the scene-general features and scene feature information are fed into a score-based diffusion model. Next, sampling is performed through a denoiser, and the scene conditions are embedded in the denoiser to conditionally sample scene-dependent features from the perturbation distribution. Finally, the generated scene-dependent features are decoded by a scene-general hierarchical variational autoencoder to obtain the predicted target frame.

3. The scene-dependent video anomaly detection and prediction method based on latent space prediction model according to claim 2 is characterized in that: The scenario-general hierarchical variational autoencoder model includes the likelihood function ,a priori and approximate posterior ;Will and They are defined as scene-general level variational encoder and scene-general level variational decoder respectively, is a feature in the latent space; given a Input video clip of frames , the scene-general hierarchical variational encoder encodes the input group into latent features , The number of layers in the hierarchical autoencoder is also the number of groups; the latent variables are represented in the form of feature maps of different resolutions; represents the set of variables with the lowest resolution at the top of the network, Denote the set of latent variables at the bottom of the network, corresponding to the highest resolution, where each latent variable in the prior and approximate posterior is represented by a multiplicative normal distribution, such that the prior and approximate posterior are expressed as: 。 4. The scene-dependent video anomaly detection and prediction method based on a latent space prediction model according to claim 3 is characterized in that: The scene-general hierarchical variational encoder For the input video sequence Encoding; the encoding process is performed through three layers of encoder units, each encoder unit contains a residual encoder block and a downsampling layer; in each encoder block, the input First, the spatiotemporal features are preliminarily encoded through convolution operations; then, the correlation between feature channels is obtained through squeezing and excitation operations; then, the connection relationship of the features in the horizontal, vertical and depth directions is obtained through multi-head self-attention operations; then, the connection relationship is summed through aggregation operations to obtain spatiotemporal features that integrate long-distance information of the video; finally, the downsampling block reduces the dimension of the spatiotemporal features encoded by the encoder through convolution operations to obtain the encoded spatiotemporal features. , these features follow the approximate posterior distribution ; The scene-general hierarchical variational decoder The spatiotemporal characteristics of the encoding Decode; The decoding process is performed by three layers of decoder units, each of which contains a residual decoder block and an upsampling layer; in each decoder block, the input First, the convolution operation is used for preliminary decoding; then, the correlation between feature channels is decoded through squeezing and excitation operations, and the attention between channels is screened; then, the multi-head self-attention operation is used to decode the connection relationship between each pixel point in the feature map and its horizontal, vertical and depth directions; then, the connection relationship is summed through the aggregation operation to decode the refined spatiotemporal features that integrate the long-distance information of the video; finally, the upsampling block uses the convolution operation to increase the dimension of the spatiotemporal features decoded by the decoder to obtain the predicted target frame ; The scene-general hierarchical variational encoder and scene-general level variational decoder It can be trained end-to-end, and its loss is written as: Using negative log-likelihood To estimate the prediction loss , which aims to maintain the consistency between the target frame and the predicted frame; the KL divergence term of the scene-universal hierarchical variational autoencoder can be further written as: 。 5. The scene-dependent video anomaly detection and prediction method based on latent space prediction model according to claim 4 is characterized in that: After training the scene-general level variation encoder and decoder Then, the time-dependent score function is trained using the inverse stochastic differential equation from Sampling was carried out in Scene-Universal Hierarchical Variational Decoder The sampled scene-dependent features are then decoded into video frames. The process is described as follows: 。 6. The scene-dependent video anomaly detection and prediction method based on latent space prediction model according to claim 5 is characterized in that: The scene condition extraction module embeds the condition factor into the denoiser In order to achieve ,in is the time step extracted by the trainable scene information extractor and scene conditions; the scene condition extraction module includes a scene information extractor and a scene information decoder ; Scene information extractor Passing the conditions to the denoiser All hidden layers of , thus forcing the denoiser Depends on Generate scene dependencies; the scene information decoder is responsible for reconstructing the scene information to which the input belongs .

7. The scene-dependent video anomaly detection and prediction method based on latent space prediction model according to claim 6 is characterized in that: The scene information extractor It consists of three convolutional layers. In each convolutional layer, the input The scene features are obtained by processing through convolution operation and then activating through LeakyReLU function; Scene Information Decoder It consists of five convolutional layers. In each convolutional layer, the input scene features are processed by convolution operation and then activated by LeakyReLU function to obtain the reconstructed scene graph. This micro autoencoder is pre-trained by reconstruction loss, which can be written as: 。 8. The scene-dependent video anomaly detection and prediction method based on latent space prediction model according to claim 7 is characterized in that: Training the denoiser using a classifier-free diffusion-guided strategy , including learning from conditional and unconditional distributions, and implementing the following linear combination: in It is a guidance scale, which can enhance the effectiveness of guidance; is the time step and extracted scene information After the interactive reverse process of conditional denoising, the target frame can be effectively predicted from the denoised frame; An additional temporal coherence term is used to ensure that the predicted frames maintain temporal consistency with the real video sequence; in the training phase, is a video clip, in which is the target frame to be predicted; select As In the keyframe, select As The time loss based on the key frame difference is written as follows: Set a stage marker represents the different training steps, namely the scene condition extraction module, the general hierarchical variational autoencoder model, and the score-based diffusion model; formally, the overall loss function is described as follows: in is the temporal loss based on keyframe differences The weight of During the anomaly detection inference process, the video is divided into Has Frame video clip, each in the video In the frame segment, the fraction of frames with the minimum peak signal-to-noise ratio (PSNR) value is regarded as the fraction of the video clip. In order to quantify the probability of anomaly occurrence, for each PSNR value Normalize to get anomaly scores in the range [0,1] , written as: During the anomaly prediction inference process, the backward network uses the future frames predicted by the forward network and observation frame Take as input the observation frame to predict ;First use the predicted frame and the observed true frame The PSNR value between Anomaly scores for future frames, , is the prediction time; Then, select The minimum of the PSNR values ​​is taken as the prediction score ; Finally, each expected value is normalized to obtain an anomaly score in the range [0,1] .

Images