Video anomaly detection method based on predictive reconstruction integrated with latent space autoregression

Abstract

The present invention discloses a prediction reconstruction video anomaly detection method integrating latent space autoregression, which belongs to the field of surveillance video processing, and solves the video anomaly misdetection and missed detection problems caused by the diversity of video information and the scarcity of abnormal events in traditional methods; including firstly performing data preprocessing on the input video sequence by random cropping, random horizontal rotation and standardization operation; defining prediction network constraints, generating prediction frames that meet the constraints; then constructing a reconstruction network in combination with a UNet network and a latent space autoregressive layer, generating a reconstruction frame that meets the constraints; then constructing a discriminator by stacking multiple convolutional layers to discriminate the abnormal values ​​of the input frame; finally adopting a generative adversarial algorithm, training the prediction generation network, the reconstruction network and the discriminator, and optimizing the network parameters. The recognition accuracy obtained by the present invention is relatively high, and the recognition speed is relatively fast, which can meet the real-time requirements of video anomaly detection in large-scale surveillance scenarios.

Description

Technical Field

[0001] The invention belongs to the field of monitoring video processing, relates to latent space autoregression technology, and specifically is a prediction and reconstruction video anomaly detection method integrating latent space autoregression. Background Art

[0002] With the popularization of video surveillance systems, video surveillance plays an indispensable role in various fields such as public safety, traffic conditions, and urban management. However, the massive amount of video data generated by the rapid growth of video surveillance equipment has made intelligent processing of video data an urgent need to be solved. Therefore, how to analyze, identify, and detect video data efficiently and quickly has attracted much attention.

[0003] Surveillance video abnormal event detection is an important branch of video intelligent analysis in the field of security. Surveillance video abnormal event detection aims to use various machine learning and deep learning methods to automatically detect and locate various unconventional events or behaviors in surveillance videos, such as vehicles breaking into the sidewalk, pedestrians walking on the highway, and other abnormal events.

[0004] Compared with other detection tasks in the field of computer vision, the task of detecting abnormal events in videos faces the following challenges:

[0005] 1) The definition of abnormal events is scenario-dependent. That is, the same event is an abnormal event in one scenario, but is considered a normal event in another scenario. For example, a pedestrian walking on the sidewalk is a normal event, but a pedestrian walking on the highway is an abnormal event.

[0006] 2) The scarcity of abnormal events. In real life, abnormal events are rare events after all, and abnormal samples are often rare and difficult to obtain, which leads to an extremely unbalanced situation of positive and negative abnormal samples in the abnormal event detection task.

[0007] 3) Diversity of video information. Video is different from image and text data. It not only has spatial information but also has strong temporal information.

[0008] With the development of deep learning, in response to the challenges faced by video anomaly detection tasks, the use of deep learning methods can, to a certain extent, weaken the dependence of anomaly detection tasks on scenes and increase the generalization ability of the model. At the same time, unsupervised methods can also solve the problems caused by sample imbalance to a certain extent. Unsupervised video anomaly detection methods are currently a hot topic of research, and there are many research papers on this type, which are mainly divided into three specific methods: clustering-based, reconstruction-based, and prediction-based methods. The above methods need to be further improved in terms of algorithm accuracy, algorithm robustness, and real-time performance, and the role of implicit feature space in semantic feature representation is conducive to improving detection efficiency.

[0009] To this end, the present invention proposes a video anomaly detection method that integrates latent space autoregression and predictive reconstruction. Summary of the invention

[0010] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a method for predictive reconstruction of video anomaly detection by integrating latent space autoregression, which is mainly used to meet the real-time requirements of monitoring video anomaly detection in large-scale application scenarios. The present invention adopts a generative adversarial network framework, adopts a method of first predicting and then reconstructing video frames, combines a latent space autoregression model to estimate the probability distribution of implicit feature vectors, and optimizes the generator by minimizing the cross entropy, prediction loss and reconstruction loss of the implicit feature vector to generate high-quality normal samples. In the training stage, the generative adversarial loss is used to optimize the network parameters; in the testing stage, the test video is used as the network input, and the abnormal score value is output, thereby realizing video anomaly detection. The recognition accuracy obtained by the present invention is relatively high, and the recognition speed is relatively fast, which can meet the real-time requirements of video anomaly detection in large-scale monitoring scenarios.

[0011] To achieve the above object, according to an embodiment of the first aspect of the present invention, a method for detecting anomalies of a video by predicting and reconstructing the video by integrating latent space autoregression is proposed. The method for detecting anomalies of a video by predicting and reconstructing the video by integrating latent space autoregression comprises the following steps:

[0012] Step 1: Use random cropping, random horizontal rotation, and normalization to preprocess the input video sequence;

[0013] Step 2: Construct a prediction network by combining the UNet network and the latent space autoregressive layer, define the prediction network constraints, and generate a prediction frame that satisfies the constraints;

[0014] Step 3: For the generated prediction frame, a reconstruction network is constructed by combining the UNet network and the latent space autoregressive layer, and reconstruction constraints are defined to generate a reconstructed frame that satisfies the constraints.

[0015] Step 4: Use multiple convolutional layers to stack and construct a discriminator, define the discriminator constraints, and identify the outliers of the input frame;

[0016] Step 5: Use the generative adversarial training algorithm to train the prediction generation network, reconstruction network and discriminator, and optimize the network parameters; in the testing phase, use the trained model to discriminate the test video sequence, output the anomaly score, and realize anomaly detection.

[0017] Furthermore, the specific execution process of step one is as follows:

[0018] A monitoring device is set in a specific application scenario, and the monitoring device is set to acquire monitoring videos in real time, and the monitoring videos acquired in real time are stored in a video sequence in the order of acquisition time.

[0019] RandomResizedCrop and RandomHorizontalFlip transformations are used to randomly crop and randomly flip the input video frame sequence to enhance the diversity of training data;

[0020] Use Normalize to normalize the training data and normalize the statistical distribution of the training data;

[0021] Furthermore, the specific execution process of step 2 is as follows:

[0022] The prediction generation network is constructed by using a prediction encoder, a latent space autoregressive layer, and a prediction decoder. The prediction encoder and the prediction decoder both use UNet encoder and UNet decoder. Between the prediction encoder and the prediction decoder, a latent space autoregressive layer is constructed to maximize the probability distribution of implicit prediction features.

[0023] Multiple masked fully connected layers are stacked to form a latent space autoregressive layer. For a given input ,After the full connection mapping, the input element With output The relationship is defined as:

[0024]

[0025] The above definition ensures that each conditional probability is calculated only based on its previous input. According to the autoregressive model, each conditional probability is modeled as a polynomial, then the output of the latent space autoregressive layer is The output of is the conditional probability estimate, where is the order of the polynomial;

[0026] Furthermore, in the generative network, the UNet predictive encoder takes the input sample Mapped to latent feature vector ,Autoregressive Density Estimation Network estimate The probability distribution of the UNet prediction decoder decodes the implicit feature vector Hidden Features The prior distribution of can be decomposed into , therefore, the prior distribution The estimation problem is transformed into multiple conditional probability density estimation problems. Then use autoregressive estimation to output multiple conditional probabilities , through the design of the latent space autoregressive layer, ensure that each Only based on Perform calculations;

[0027] In actual calculations, the autoregressive density network of the latent space models each conditional probability as a polynomial, and the autoregressive density estimation of the latent space is expressed as the cross entropy loss of each conditional probability density and its corresponding classification distribution, which is specifically expressed as follows:

[0028]

[0029] Combined with the prediction generation network structure, the constraints of the prediction generator are defined as follows:

[0030]

[0031] The constraints of the prediction generator are expressed as:

[0032]

[0033] in, To predict the image, and To predict the encoder and decoder parameters in the generator, is the hidden feature vector of the prediction generator.

[0034] Furthermore, the specific execution process of step three is as follows:

[0035] The reconstruction generator network is composed of three parts: reconstruction encoder, latent space autoregressive layer, and reconstruction decoder. The structure of the reconstruction generator network is consistent with the prediction generator network structure in step 2. The calculation method of the conditional density of the latent space autoregressive layer in the reconstruction generator is similar to that in step 2.

[0036] The prediction image generated by the prediction network As the input of the reconstruction network, the constraints of the reconstruction network are defined as follows:

[0037]

[0038] in, To predict the image, To reconstruct the image, and To reconstruct the encoder and decoder parameters in the network.

[0039] Furthermore, the specific execution process of step 4 is as follows:

[0040] The discriminator network is defined as a stack of multiple convolutional layers, each with a convolution kernel size of 5*5, and a ReLU layer as the activation function after the convolutional layer. The objective function of the discriminator is defined as:

[0041]

[0042] Furthermore, the specific execution process of step five is as follows:

[0043] The generative adversarial network is used to train the network so that the generator can generate high-quality images. Combining the constraint representation of the above prediction network and reconstruction network, the generator objective function is defined as:

[0044]

[0045]

[0046] in, is the output of the frame discriminator;

[0047] Adopt Adam stochastic gradient descent algorithm to train prediction network, reconstruction network and discriminator network, and optimize network parameters;

[0048] In the training phase, the prediction network and reconstruction network are used to generate high-quality video frames, and the discriminator network is used to determine whether the current input frame is a real frame;

[0049] During the testing phase, for the input test video sequence, the discriminator directly outputs anomaly scores to determine whether the test frame is abnormal. Frames with scores lower than a given threshold are marked as abnormal frames, thereby achieving anomaly detection.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The present invention performs data enhancement on the input video sequence by random cropping, random horizontal rotation, and standardization operations; combines the UNet network and the latent space autoregressive layer to construct a prediction network, defines the prediction network constraints, and generates prediction frames that meet the constraints; for the generated prediction frames, combines the UNet network and the latent space autoregressive layer to construct a reconstruction network, defines the reconstruction constraints, and generates reconstructed frames that meet the constraints; then uses multiple convolutional layers to stack and construct a discriminator, defines the discriminator constraints, and discriminates the abnormal values ​​of the input frames; finally, uses a generative adversarial algorithm to train the prediction generation network, the reconstruction network, and the discriminator, and optimizes the network parameters. In the test phase, the trained model is used to discriminate the test video sequence, output the abnormal score, and realize abnormal detection; the recognition accuracy obtained by the present invention is high, the recognition speed is fast, and it can meet the real-time requirements of video anomaly detection in large-scale monitoring scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 is a flow chart of the present invention;

[0054] Figure 2 This is the network structure diagram of the discriminator in the present invention. DETAILED DESCRIPTION

[0055] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] like Figure 1 to Figure 2 As shown, the present invention proposes a method for detecting anomalies in a video by predicting and reconstructing the video by integrating latent space autoregression, which comprises the following steps:

[0057] Step 1: Use random cropping, random horizontal rotation, and normalization to preprocess the input video sequence;

[0058] The specific process is:

[0059] A monitoring device is set in a specific application scenario, and the monitoring device is set to acquire monitoring videos in real time, and the monitoring videos acquired in real time are stored in a video sequence in the order of acquisition time.

[0060] In this application, RandomResizedCrop and RandomHorizontalFlip transformations are used to randomly crop and randomly horizontally flip the input video frame sequence to enhance the diversity of training data;

[0061] Use Normalize to normalize the training data and normalize the statistical distribution of the training data;

[0062] Step 2: Use the UNet network and the latent space autoregressive layer to construct a prediction network, define the prediction network constraints, and generate a prediction frame that meets the constraints; the specific process is:

[0063] The prediction generation network is constructed using three parts: prediction encoder, latent space autoregressive layer, and prediction decoder.

[0064] The prediction encoder and the prediction decoder both use UNet encoder and UNet decoder. Between the prediction encoder and the prediction decoder, a latent space autoregressive layer is constructed to maximize the probability distribution of the implicit prediction features.

[0065] Multiple masked fully connected layers are stacked to form a latent space autoregressive layer. For a given input ,After the full connection mapping, the input element With output The relationship is defined as:

[0066]

[0067] The above definition ensures that each conditional probability is calculated only based on its previous input. According to the autoregressive model, each conditional probability is modeled as a polynomial, then the output of the latent space autoregressive layer is The output of is the conditional probability estimate, where is the order of the polynomial;

[0068] In the generative network, the UNet predictive encoder takes as input a sample Mapped to latent feature vector ,Autoregressive Density Estimation Network estimate The probability distribution of the UNet prediction decoder decodes the implicit feature vector ;

[0069] Hidden Features The prior distribution of can be decomposed into , therefore, the prior distribution The estimation problem is transformed into multiple conditional probability density estimation problems;

[0070] Autoregressive Density Estimation Network Then use autoregressive estimation to output multiple conditional probabilities , through the design of the latent space autoregressive layer, ensure that each Only based on Perform calculations;

[0071] In actual calculations, the autoregressive density network of the latent space models each conditional probability as a polynomial, and the autoregressive density estimation of the latent space is expressed as the cross entropy loss of each conditional probability density and its corresponding classification distribution, which is specifically expressed as follows:

[0072]

[0073] Combined with the prediction generation network structure, the constraints of the prediction generator are defined as follows:

[0074]

[0075] The constraints of the prediction generator are expressed as:

[0076]

[0077] in, To predict the image, and To predict the encoder and decoder parameters in the generator, is the hidden feature vector of the prediction generator.

[0078] Step 3: For the generated prediction frame, a reconstruction network is constructed by combining the UNet network and the latent space autoregressive layer, and reconstruction constraints are defined to generate a reconstructed frame that satisfies the constraints. The specific process is as follows:

[0079] The reconstruction generator network is composed of three parts: reconstruction encoder, latent space autoregressive layer, and reconstruction decoder. The structure of the reconstruction generator network is consistent with the prediction generator network structure in step 2. The calculation method of the conditional density of the latent space autoregressive layer in the reconstruction generator is similar to that in step 2.

[0080] The prediction image generated by the prediction network As the input of the reconstruction network, the constraints of the reconstruction network are defined as follows:

[0081]

[0082] in, To predict the image, To reconstruct the image, and To reconstruct the encoder and decoder parameters in the network.

[0083] Step 4: Use multiple convolutional layers to stack and construct a discriminator, define the discriminator constraints, and identify the outliers of the input frame; the specific process is as follows:

[0084] The discriminator network is defined as a stack of multiple convolutional layers, each with a convolution kernel size of 5*5, and a ReLU layer as the activation function after the convolutional layer. The objective function of the discriminator is defined as:

[0085]

[0086] In one embodiment, the network structure of the discriminator is as follows: Figure 2 shown.

[0087] Step 5: Use the generative adversarial training algorithm to train the prediction generation network, reconstruction network and discriminator, and optimize the network parameters. In the testing phase, use the trained model to discriminate the test video sequence, output the anomaly score, and realize anomaly detection. The specific process is:

[0088] The generative adversarial network is used to train the network so that the generator can generate high-quality images. Combining the constraint representation of the prediction network and the reconstruction network, the generator objective function is defined as:

[0089]

[0090] in, is the output of the frame discriminator;

[0091] The Adam stochastic gradient descent algorithm is used to train the prediction network, reconstruction network and discriminator network, and optimize the network parameters. In the training phase, the prediction network and reconstruction network are used to generate high-quality video frames, and the discriminator network is used to determine whether the current input frame is a real frame. In the testing phase, for the input test video sequence, the discriminator directly outputs anomaly scores to determine whether the test frame is abnormal. Frames with scores lower than a given threshold are marked as abnormal frames, thereby achieving anomaly detection.

[0092] It should be noted that the prediction generation network and the reconstruction generation network are used in combination with the latent space autoregressive model to extract the implicit feature representation of the video sequence and generate video frames that meet the prediction constraints and reconstruction constraints; the process of optimizing the network parameters based on the discriminator's discrimination results of the generated frames is the training stage;

[0093] The prediction reconstruction generative network in the training phase generates video frames, and then the discriminator outputs anomaly discrimination values ​​to realize abnormal video detection as the testing phase.

[0094] The above formulas are all calculated by removing dimensions and taking numerical values. The formula is a formula that is closest to the actual situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained by simulating a large amount of data.

[0095] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A method for video anomaly detection based on prediction and reconstruction by integrating latent space autoregression, characterized in that: The following steps are involved: Step 1: Use random cropping, random horizontal rotation, and normalization to preprocess the input video sequence; Step 2: Construct a prediction network by combining the UNet network and the latent space autoregressive layer, define the prediction network constraints, and generate a prediction frame that satisfies the constraints; Step 3: For the generated prediction frame, a reconstruction network is constructed by combining the UNet network and the latent space autoregressive layer, and reconstruction constraints are defined to generate a reconstructed frame that satisfies the constraints. Step 4: Use multiple convolutional layers to stack and construct a discriminator, define the discriminator constraints, and identify the outliers of the input frame; Step 5: Use the generative adversarial training algorithm to train the prediction generation network, reconstruction network and discriminator, and optimize the network parameters; use the trained model to discriminate the test video sequence, output the anomaly score, and realize anomaly detection; The specific implementation process of step 2 includes the following: The prediction generation network is composed of three parts: a prediction encoder, a latent space autoregressive layer, and a prediction decoder. The prediction encoder and the prediction decoder both use UNet encoder and UNet decoder. Between the prediction encoder and the prediction decoder, a latent space autoregressive layer is constructed to maximize the probability distribution of implicit prediction features. Multiple masked fully connected layers are stacked to form a latent space autoregressive layer. For a given input After the full connection mapping, the input element With output The relationship is defined as: The above definition ensures that each conditional probability is calculated only based on its previous input; according to the autoregressive model, each conditional probability is modeled as a polynomial, then the output of the latent space autoregressive layer is The output of is the conditional probability estimate, where B is the order of the polynomial; Combined with the prediction generation network structure, the constraints of the prediction generator are defined as follows: L PRE (x, x′)=L PRE (i f ,the g )+L LLK (i f ,the h ) The autoregressive density network of the latent space models each conditional probability as a polynomial. The autoregressive density estimation of the latent space is expressed as the cross entropy loss of each conditional probability density and its corresponding classification distribution, which is specifically expressed as follows: The constraints of the prediction generator are expressed as: Among them, x′ is the predicted image, θ f and θ h are the encoder and decoder parameters in the prediction generator, and Z is the hidden feature vector of the prediction generator.

2. The method for detecting anomalies in video by predicting and reconstructing latent space autoregression according to claim 1 is characterized in that: In both the prediction network and the reconstruction network, latent space autoregression is used for density estimation to maximize the probability of the latent feature space and reduce the generation error of the prediction frame and the reconstructed frame. The formal expression of autoregressive density estimation applied to the prediction generation network is: The UNet predictive encoder maps the input sample x to an implicit feature vector z = f(x; θ f ), autoregressive density estimation network z = h(z, θ h ) estimates the probability distribution of z, and the UNet prediction decoder decodes the implicit feature vector; The prior distribution of the latent eigenvector z can be decomposed into The estimation problem of the prior distribution p(z) is transformed into multiple conditional probability density estimation problems; the autoregressive density estimation network z = h(z, θ h ) uses autoregressive estimation to output multiple conditional probabilities p(z i |z<i), through the design of latent space autoregressive layer, ensure that each z i Only according to {z1, ..., z i-1 } to perform calculations; The method used for autoregressive density estimation to reconstruct the generative network is similar to that used in predicting the generative network.

3. The method for detecting anomalies in video by predicting and reconstructing latent space autoregression according to claim 1 is characterized in that: The specific implementation process of step three includes the following: The reconstruction generation network is composed of three parts: reconstruction encoder, latent space autoregressive layer and reconstruction decoder. The structure of the reconstruction generation network is consistent with the prediction generation network structure in step 2. The predicted image x′ generated by the prediction network is used as the input of the reconstruction network, and the constraints of the reconstruction network are defined as follows: Among them, x′ is the predicted image, To reconstruct the image, θ f′ and θ h′ To reconstruct the encoder and decoder parameters in the network.

4. The method for detecting anomalies in video by fusion latent space autoregression prediction and reconstruction according to claim 1 is characterized in that: The specific implementation process of step 4 includes the following: The discriminator network is defined as a stack of multiple convolutional layers, each with a convolution kernel size of 5*5, and a ReLU layer as the activation function after the convolutional layer; the objective function of the discriminator is defined as:

5. The method for detecting anomalies in video by predicting and reconstructing latent space autoregression according to claim 1, characterized in that: The specific implementation process of step 5 includes the following: The generative adversarial network is used to train the network so that the generator can generate high-quality images. Combining the constraint representation of the above prediction network and reconstruction network, the generator objective function is defined as: in, is the output of the frame discriminator; Adopt Adam stochastic gradient descent algorithm to train prediction network, reconstruction network and discriminator network, and optimize network parameters; In the training phase, the prediction network and reconstruction network are used to generate high-quality video frames, and the discriminator network is used to determine whether the current input frame is a real frame; During the testing phase, for the input test video sequence, the discriminator directly outputs anomaly scores to determine whether the test frame is abnormal. Frames with scores lower than a given threshold are marked as abnormal frames, thereby achieving anomaly detection.

Images