A video anomaly detection method based on the combination of stream reconstruction and frame prediction

Abstract

The present invention discloses a video anomaly detection method based on the combination of flow reconstruction and frame prediction, comprising the following steps: obtaining training samples, frame data processing, reconstruction network model training, prediction network training, and using the trained model for testing and determining whether the current frame image is an abnormal frame; the video anomaly detection method based on the combination of flow reconstruction and frame prediction of the present invention uses a multi-level memory network improved based on TransUNet and a conditional variational autoencoder for training respectively. First, the high correlation between video frames and optical flows is captured. Second, the reconstructed normal flow usually has high quality, and the prediction module can successfully predict future frames with a smaller prediction error. Finally, the reconstruction error expands the prediction error, making the anomaly easier to detect, thereby obtaining anomaly detection results with higher accuracy and robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of video anomaly detection, and in particular to a video anomaly detection method based on the combination of stream reconstruction and frame prediction. Background Art

[0002] With the increasing popularity of videos, automatically identifying abnormal events in videos is becoming increasingly important. Intelligent video anomaly detection can save human resources and improve efficiency to a certain extent. Video anomaly detection is a computer vision technology that aims to automatically identify and detect abnormal behaviors or events from video streams. In the video anomaly detection task, the system analyzes the input video data and tries to find parts that are inconsistent with normal behaviors or scenes. These parts are usually called anomalies. These anomalies may be unusual behaviors, emergencies, abnormal objects or abnormal activities.

[0003] Video anomaly detection has a wide range of applications in many practical scenarios, such as monitoring systems, security, traffic management, intelligent manufacturing, etc. By automatically detecting and alerting abnormal events, video anomaly detection technology can help improve monitoring efficiency, reduce waste of human resources, and enhance the ability to warn of potential risks.

[0004] The implementation of video anomaly detection involves multiple technologies, including feature extraction, anomaly definition and labeling, and anomaly detection algorithms. Feature extraction is used to convert video data into meaningful feature representations that can be used for analysis. Anomaly definition and labeling are to determine what abnormal behavior is and prepare labeled data for model training. Anomaly detection algorithms identify abnormal behavior in videos based on the extracted features and labeled data.

[0005] At present, video anomaly detection is a challenging task because abnormal events are usually low-frequency events, and the pattern of normal behavior may change in different scenarios, resulting in abnormal behavior being difficult to detect and unable to successfully predict future frames with a smaller prediction error. Therefore, it is necessary to comprehensively consider different technical means, including traditional statistical methods and modern deep learning technologies, to obtain anomaly detection results with higher accuracy and robustness. Therefore, the present invention proposes a video anomaly detection method based on the combination of stream reconstruction and frame prediction to solve the problems existing in the prior art. Summary of the Invention

[0006] In response to the above problems, the purpose of the present invention is to propose a video anomaly detection method based on the combination of stream reconstruction and frame prediction to solve the problem in existing video anomaly detection technology that abnormal events are usually low-frequency events and the pattern of normal behavior may change in different scenarios, resulting in abnormal behavior being difficult to detect and the inability to successfully predict future frames with a smaller prediction error.

[0007] In order to achieve the purpose of the present invention, the present invention is implemented by the following technical solution: a video anomaly detection method based on the combination of stream reconstruction and frame prediction, comprising the following steps:

[0008] Step 1: First, obtain training samples from three internationally recognized public video anomaly detection datasets: UCSD Ped2, CUHK Avenue, and ShanghaiTech. Then, process the video data in the training samples into corresponding frame data.

[0009] Step 2: First extract the optical flow of the training and test video frames and all their foreground objects, where each foreground object is identified by a RoI bounding box. For each RoI bounding box identification, construct a space-time cube;

[0010] Step 3: Use the multi-level memory network based on TransUNet as the optical flow reconstruction network to reconstruct the optical flow extracted in step 2, input the space-time cube, output the reconstructed optical flow image, and calculate the reconstruction error and reconstruction loss;

[0011] Step 4: Use the conditional variational autoencoder (CVAE) as the prediction network model, and input the original video frame data and the reconstructed optical flow image into the prediction network model for training. Output the predicted frame data and calculate the prediction error and prediction loss.

[0012] Step 5: Use the trained model for testing, input the image to be detected, and weight the flow reconstruction error and frame prediction error to obtain the anomaly score. If the anomaly score exceeds the preset threshold, the current frame image is detected as an abnormal frame.

[0013] A further improvement is that in step 2, the constructed space-time cube includes the object in the current frame and the content in the same bounding box of the previous t frames, where t=4, and the width and height of the space-time cube are both adjusted to 32.

[0014] A further improvement is that in step three, the multi-level memory network based on TransUNet is a network structure based on Transformer and U-Net, which is used for image segmentation tasks, and the optical flow reconstruction network is composed of an encoder, a memory module, a decoder, a memory module and a memory module.

[0015] A further improvement is that the encoder first extracts features from the input image through ResNet50. The outputs of the three interface layers are retained and used for subsequent skip connections. The feature maps output by ResNet50 are then serialized and fed into the Transformer network model for sequence prediction. A sequence is output, and the output sequences are then merged and reshaped into a new feature map.

[0016] Each level of the decoder first copies the feature map from the encoder, then concatenates each level of the decoder with the upsampled feature map of the lower level, and then concatenates three layers in sequence, each layer contains two convolution blocks, a storage module and an upsampling layer, and the convolution block contains a convolution layer, a batch normalization layer and a ReLU activation layer;

[0017] The memory module is a matrix M∈R N*C , each row of the matrix is ​​called a slot M i , i = 1, 2, 3, ..., N, the memory module represents the features of its input through the weighted sum of similar memory slots.

[0018] A further improvement is that in step 3, the loss function in the reconstruction training process is:

[0019]

[0020] Where M is the number of memory modules, is the matching probability of the kth slot in the i-th memory module, λ recon and λ ent are weighting parameters respectively.

[0021] A further improvement is that in step 4, the prediction network model consists of two encoders E θ and and a decoder D. The encoder encodes the input data into the distribution parameters of the latent variables, while the decoder decodes the latent variables into generated data. The input of the network is the concatenation of the reconstructed optical flow and the original frame image. θ Optical flow y 1:t Encode to get E θ (y 1:t ), thus obtaining the prior distribution p(z|y 1:t ), Allow x 1:t and y 1:t Concatenation, output features The posterior distribution q(x 1:+ ,y 1:t ).

[0022] A further improvement is that during the training of the prediction network model, z is sampled from the posterior distribution and z is compared with the condition E θ (y 1:t ) are connected and finally sent to the decoder D to generate the future frame

[0023] A further improvement is that in step 5, the calculation formula of the anomaly score is:

[0024]

[0025] Where S is the anomaly score, μ r , σ r , μ p , σ p is the mean and standard deviation of the reconstruction error and prediction error of all training samples, w r and w p is the weight of the two scores, S r and S p These are the L2 norm calculation results before and after reconstruction and prediction, respectively.

[0026] The beneficial effects of the present invention are as follows: the video anomaly detection method based on the present invention combines flow reconstruction and frame prediction, and uses a multi-level memory network and conditional variational autoencoder training based on the improved TransUNet respectively. First, the high correlation between video frames and optical flows is captured. Second, the reconstructed normal flow is usually of high quality, and the prediction module can successfully predict future frames with a smaller prediction error. Finally, the reconstruction error expands the prediction error, making anomalies easier to detect, thereby obtaining anomaly detection results with higher accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 It is a flow chart of the video anomaly detection method of the present invention;

[0029] Figure 2 It is a schematic diagram of the overall network model structure of the video anomaly detection of the present invention. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] See also Figure 1 、 Figure 2 This embodiment provides a video anomaly detection method based on the combination of stream reconstruction and frame prediction, including the following steps:

[0032] S10: First, obtain training samples from three internationally recognized public video anomaly detection datasets: UCSD Ped2, CUHK Avenue, and ShanghaiTech. Then use the ffmpeg tool to process the video data in the training samples into corresponding frame data;

[0033] S20: First extract the optical flow of the training and test video frames and all their foreground objects, where each foreground object is identified by a RoI bounding box. For each RoI bounding box identification, construct a space-time cube (STC). The constructed space-time cube (STC) contains the object in the current frame and the content in the same bounding box of the previous t frames, where t = 4, and the width and height of the space-time cube are both adjusted to 32;

[0034] S30: Use the multi-level memory network based on TransUNet as the optical flow reconstruction network to reconstruct the optical flow extracted in step 2, input the space-time cube, output the reconstructed optical flow image, and calculate the reconstruction error and reconstruction loss. The loss function during the reconstruction training process is:

[0035]

[0036] Where M is the number of memory modules, is the matching probability of the kth slot in the i-th memory module, λ recon and λ ent are weighting parameters respectively;

[0037] like Figure 2 As shown, the overall network model structure of this embodiment consists of two branches, namely the optical flow reconstruction branch and the frame prediction branch. The multi-level memory network based on TransUNet in this embodiment is a network structure based on Transformer and U-Net, which is used for image segmentation tasks. The optical flow reconstruction network consists of an encoder, a memory module, a decoder, a memory module and a memory module. The encoder part first extracts features from the input image through resnet50, and the outputs of the three interface layers (stage) will be retained and used for subsequent skip connections. The feature map output by resnet50 is then serialized and sent to the transformer network model for sequence prediction, outputting a sequence, and then merging and reshaping the output sequence into a new feature map.

[0038] Each level of the decoder first copies the feature map from the encoder, then concatenates each level of the decoder with the upsampled feature map of the lower level, and then concatenates three layers in sequence. Each layer contains two convolution blocks, a storage module, and an upsampling layer. The convolution block contains a convolution layer, a batch normalization layer, and a ReLU activation layer.

[0039] The memory module is a matrix M∈R N*C , each row of the matrix is ​​called a slot M i , i = 1, 2, 3, ..., N, the memory module represents the features of its input through the weighted sum of similar memory slots, so it has the ability to remember normal patterns when trained on normal data;

[0040] S40: Conditional variational autoencoder CVAE is used as the prediction network model, and the original video frame data and the reconstructed optical flow image are input into the prediction network model for training, the predicted frame data is output, and the prediction error and prediction loss are calculated. The prediction network model consists of two encoders E θ and and a decoder D. The encoder encodes the input data into the distribution parameters of the latent variables, while the decoder decodes the latent variables into generated data. The input of the network is the concatenation of the reconstructed optical flow and the original frame image. θ Optical flow y 1:t Encode to get E θ (y 1:t ), thus obtaining the prior distribution p(z|y 1:t ), Allow x 1:+ and y 1:t Concatenation, output features The posterior distribution q(x 1:t ,y 1:t ), during the training of the prediction network model, z is sampled from the posterior distribution and z is compared with the condition E θ (y 1:t ) are connected and finally sent to the decoder D to generate the future frame

[0041] S50: Use the trained model for testing, input the image to be detected, and weight the flow reconstruction error and frame prediction error to obtain the anomaly score. The calculation formula of the anomaly score is:

[0042]

[0043] Where S is the anomaly score, μ r , σ r , μ p , σ pis the mean and standard deviation of the reconstruction error and prediction error of all training samples, w r and w p is the weight of the two scores, S r and S p The L2 normal form calculation results before and after reconstruction and prediction are respectively used to determine whether the anomaly score is greater than the set threshold. If it is greater than the preset threshold, step S60 is executed; if the anomaly score is less than the preset threshold, step S70 is executed;

[0044] S60: The image frame is an abnormal frame;

[0045] S70: This frame image is a normal frame.

[0046] The present invention uses flow reconstruction and frame prediction error as the final anomaly detection calculation factors. First, the multi-layer memory module adopts a skip-connected encoder-decoder structure to ensure that normal patterns are well memorized, thereby sensitively identifying abnormal events or activities. Second, a method combining flow reconstruction and frame prediction is used to predict future frames from previous video frames and corresponding optical flows. However, the optical flow is reconstructed in advance, and the reconstruction error amplifies the prediction error, making anomalies easier to detect.

[0047] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A video anomaly detection method based on the combination of stream reconstruction and frame prediction, characterized in that: The following steps are involved: Step 1: First, obtain training samples from three internationally recognized public video anomaly detection datasets: UCSD Ped2, CUHK Avenue, and ShanghaiTech. Then, process the video data in the training samples into corresponding frame data. Step 2: First extract the optical flow of the training and test video frames and all their foreground objects, where each foreground object is identified by a RoI bounding box. For each RoI bounding box identification, construct a space-time cube; Step 3: Use the multi-level memory network based on TransUNet as the optical flow reconstruction network to reconstruct the optical flow extracted in step 2, input the space-time cube, output the reconstructed optical flow image, and calculate the reconstruction error and reconstruction loss. The multi-level memory network based on TransUNet is a network structure based on Transformer and U-Net, which is used for image segmentation tasks. The optical flow reconstruction network consists of an encoder, a memory module, a decoder, a memory module and a memory module. The encoder part first extracts features from the input image through resnet50. The outputs of the three interface layers will be retained and used for subsequent jump connections. The feature map output by resnet50 is then serialized and sent to the transformer network model for sequence prediction. A sequence is output, and then the output sequence is merged and reshaped into a new feature map. Each level of the decoder first copies the feature map from the encoder, then concatenates each level of the decoder with the upsampled feature map of the lower level, and then concatenates three layers in sequence, each layer contains two convolution blocks, a storage module and an upsampling layer, and the convolution block contains a convolution layer, a batch normalization layer and a ReLU activation layer; The memory module is a matrix M∈R N*C , each row of the matrix is ​​called a slot M i , i = 1, 2, 3, ..., N, the memory module represents the features of its input through the weighted sum of similar memory slots; Step 4: Use the conditional variational autoencoder (CVAE) as the prediction network model, and input the original video frame data and the reconstructed optical flow image into the prediction network model for training. Output the predicted frame data and calculate the prediction error and prediction loss. Step 5: Use the trained model to test, input the image to be detected, and weight the flow reconstruction error and frame prediction error to obtain an anomaly score. If the anomaly score exceeds the preset threshold, the current frame image is detected as an anomaly frame. The calculation formula of the anomaly score is: Where S is the anomaly score, μ r , σ r , μ p , σ p is the mean and standard deviation of the reconstruction error and prediction error of all training samples, w r and w p is the weight of the two scores, S r and S p These are the L2 norm calculation results before and after reconstruction and prediction, respectively.

2. The video anomaly detection method based on combining stream reconstruction and frame prediction according to claim 1, characterized in that: In step 2, the constructed space-time cube includes the object in the current frame and the content in the same bounding box of the previous t frames, where t=4, and the width and height of the space-time cube are both adjusted to 32.

3. The video anomaly detection method based on combining stream reconstruction and frame prediction according to claim 1, characterized in that: In step 3, the loss function in the reconstruction training process is: Where M is the number of memory modules, is the matching probability of the kth slot in the i-th memory module, λ recon and λ ent are weighting parameters respectively.

4. The video anomaly detection method based on combining stream reconstruction and frame prediction according to claim 1, characterized in that: In step 4, the prediction network model consists of two encoders E θ and and a decoder D. The encoder encodes the input data into the distribution parameters of the latent variables, while the decoder decodes the latent variables into generated data. The input of the network is the concatenation of the reconstructed optical flow and the original frame image. θ Optical flow y 1:t Encode to get E θ (y 1:t ), thus obtaining the prior distribution p(z|y 1:t ), Allow x 1:t and y 1:t Concatenation, output features The posterior distribution q(x 1:t ,y 1:t ).

5. The video anomaly detection method based on combining stream reconstruction and frame prediction according to claim 4 is characterized in that: During the training of the prediction network model, z is sampled from the posterior distribution and compared with the condition E θ (y 1:t ) are connected and finally sent to the decoder D to generate the future frame

Images