A video anomaly event detection method based on a cascaded attention U-Net

Abstract

The application discloses a video abnormal event detection method based on a cascaded attention U-Net, and belongs to the technical field of video processing. Since a deep neural network has strong feature extraction and representation capabilities, a video abnormal event detection method based on deep learning is widely applied. The deep neural network is trained through a large amount of experimental data, does not need to manually design features, integrates feature extraction and model construction together, and realizes an end-to-end abnormal event detection method. The application proposes a video abnormal event detection method based on a cascaded attention U-Net. In a first attention U-Net block, a time migration module is combined as a frame prediction model, second and third attention U-Net cascaded modules are combined as frame reconstruction modules, finally, an abnormal score is calculated by combining a predicted frame and a reconstructed frame to complete abnormal event recognition, and the detection performance is further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a video abnormal event detection method based on cascaded attention U-Net. BACKGROUND

[0002] In order to ensure social public safety and improve the management ability of smart city, video monitoring technology is widely used in crowded public places such as campus, large shopping malls, airports and railway stations. Video abnormal event detection technology mainly uses image processing technology to automatically analyze and identify abnormal behaviors in video, and timely detects a small number of abnormal events inconsistent with most normal events from massive monitoring video data. Under normal circumstances, manual monitoring of massive video will consume a lot of manpower and material resources. Moreover, long-term human monitoring will cause visual fatigue of the monitoring personnel, and further cause missed detection of abnormal events in the video. Therefore, it is of great significance to study intelligent processing of massive video and automatically detect and identify abnormal events in video.

[0003] Video abnormal event detection methods can be divided into: machine learning-based abnormal event detection methods and deep learning-based abnormal event detection methods. The machine learning-based abnormal event detection method first extracts the contour features, RGB texture information, gradient and other features of the moving target from the original high-dimensional video sequence data, and then uses probability models such as hidden Markov model, Markov random field and Gaussian mixture model to construct the model, and then completes the identification of abnormal events. However, in the feature extraction process, it is easy to be affected by image noise, illumination change and occlusion, which will further affect the judgment of the probability model on the abnormal event.

[0004] Due to the strong feature extraction and representation ability of deep neural network, deep learning-based video abnormal event detection method is widely used. Deep neural network is trained by a large amount of experimental data, without manually designing features, integrating feature extraction and model construction together, and realizing end-to-end abnormal event detection method. The present application proposes a video abnormal event detection method based on cascaded attention U-Net, which combines a time migration module as a frame prediction model in the first attention U-Net block, and the second and third attention U-Net cascade modules as frame reconstruction modules. Finally, the abnormal score is calculated by combining the predicted frame and the reconstructed frame to complete the abnormal event identification, and the detection performance is further improved. SUMMARY

[0005] The present application aims to solve the above problems, and provides a video abnormal event detection method based on cascaded attention U-Net.

[0006] To achieve the above object, the technical scheme adopted by the present application is as follows: a video abnormal event detection method based on a cascaded attention U-Net, comprising the following steps:

[0007] S1: constructing a U-Net down-sampling module.

[0008] S2: constructing a time migration module;

[0009] S3: constructing an U-Net up-sampling module based on attention gates, using a continuous up-sampling module for feature perception;

[0010] S4: constructing a video frame prediction module, which is composed of a time migration module and an attention U-Net network;

[0011] S5: constructing a cascaded video frame reconstruction module, which is composed of two cascaded attention U-Net networks;

[0012] S6: taking the output of the first attention U-Net network in the reconstruction module as the input of the second attention U-Net network in the reconstruction module, and sequentially performing steps (1-1) and (1-3) to obtain a reconstructed video frame.

[0013] S7: constructing a multi-element loss function to train the cascaded attention U-Net network;

[0014] S8: combining the predicted frame and the reconstructed frame to calculate an abnormality detection score, thereby realizing video abnormal event detection

[0015] Further, the U-Net down-sampling module specifically includes perception by a continuous down-sampling module, and the output z i of each layer of the down-sampling module can be represented as:

[0016] z i =maxpooling(σ(W i 2 *σ(W i 1 *x i ))) (1)

[0017] wherein i represents the i-th down-sampling module, x i represents the input of the i-th down-sampling module, W i 1 , W i 2 represent weight matrices learned by two consecutive convolution layers in the i-th down-sampling module, * represents a convolution operation, σ represents a ReLU nonlinear activation function, maxpooling represents a pooling layer, and abstract high-dimensional features are gradually extracted through the down-sampling module operation, and the output feature zi the size of H i x W i x C i , where C i represents the number of channels of the output of the i-th down-sampling block.

[0018] Further, the temporal migration module specifically includes taking the output of the last layer frame prediction down-sampling module as the input of the temporal migration module, first using two consecutive convolution layers to increase the number of channels, and then using the temporal migration module to migrate the channel information of the extracted features, realize the information interaction of different time channels, and focus on the change of abnormal behavior in different time dimensions. The output feature can be represented as:

[0019]

[0020] where z shift represents the output of the temporal migration module, z last represents the output of the last frame prediction down-sampling module, is the weight matrix learned by the two consecutive convolution layers in the temporal migration module, and σ represents the ReLU nonlinear activation function. The size of the output of the temporal migration module is H shift x W shift x C shift .

[0021] Further, the attention gate-based U-Net up-sampling module specifically includes that the attention gate can gradually suppress the feature response of irrelevant background regions and enhance the salient features of the regions of interest, aiming to focus on regions of different shapes and sizes, which can improve the sensitivity and prediction accuracy of the model, thereby realizing more accurate and robust abnormal event recognition performance. The attention gate can be represented as:

[0022]

[0023]

[0024]

[0025]

[0026] where l represents the l-th up-sampling module, (i,j) represents the index value, x l represents the feature that needs to be up-sampled, ψ l is the feature that completes the convolution operation on x l , g l represents the feature corresponding to x l in the down-sampling module, is the feature that completes the convolution operation on g l , Ml attention coefficient obtained by formula (5), W1 l 、 W l , b l is a weight matrix learned in the convolution operation, and σ represents a ReLU nonlinear activation function, and δ represents a Sigmoid nonlinear activation function, is the output obtained by the attention gate.

[0027] Finally, the output of the upsampling module is calculated using the features obtained by the attention gate:

[0028]

[0029] where concentrate represents feature aggregation, is a parameter matrix learned in the convolution operation, and σ represents a ReLU nonlinear activation function, and upsampling represents an upsampling operation, and y l represents the output feature of the upsampling module, and the size can be represented as H l ×W l ×C l .

[0030] Further, the video frame prediction module specifically includes the RGB images of consecutive frames as the input of the prediction frame module, first uses S1 to obtain down-sampled features of different scales; then performs step S2 on the features of the last layer of the down-sampling module to complete channel information migration on the extracted features and obtain time interaction information; finally, the features obtained by steps S1 and S2 are combined with step S3 to complete video frame prediction, and the next future video frame is predicted for the consecutive multiple video frames.

[0031] Further, the predicted video frame obtained by step S4 is used as the input of the cascaded attention U-Net network, the first attention U-Net network in the reconstruction module sequentially performs steps S1 and S3, and then the output of the first attention U-Net network in the reconstruction module is used as the input of the second attention U-Net network in the reconstruction module, and steps S1 and S3 are sequentially performed to obtain the reconstructed video frame.

[0032] Further, steps S1 to S5 complete the construction of the network model for cascaded frame prediction and frame reconstruction, in order to minimize the gap between the predicted reconstructed frame and the real frame, fully considering the similarity between the predicted reconstructed frame and the real frame in appearance and motion, the multi-element loss function is composed of intensity loss, gradient loss and structural similarity loss.

[0033] where the intensity loss can be represented as:

[0034]

[0035] where I denotes the real video frame, denotes the predicted reconstruction frame obtained by the network model of cascaded frame prediction and frame reconstruction, and (i, j) denotes an index value.

[0036] The gradients in the horizontal and vertical directions can be calculated as:

[0037]

[0038]

[0039] In combination with equations (9) and (10), the gradient loss can be represented as:

[0040]

[0041] In combination with equations (8) and (11), the loss function can be represented as:

[0042]

[0043] where SSIM denotes the structural similarity loss, and a, b and g denote the weights of the loss function.

[0044] Further, the specific operation in S7 includes performing steps S1 to S6, and an optimal cascaded frame prediction and frame reconstruction network model can be obtained, and abnormality detection is performed according to the difference between the predicted reconstruction frame and the corresponding real frame;

[0045] First, the peak signal-to-noise ratio (PSNR) is calculated to evaluate the image quality, which can be represented as:

[0046]

[0047] where I denotes the real video frame, and N denotes the number of video frames. The higher the value of PSNR, the better the quality of the output video frame obtained by the model. In order to further improve the detection performance of the model, the abnormality detection score combining the predicted video frame and the reconstructed video frame is calculated based on the PSNR value; The abnormality score of the predicted video frame obtained by the first attention U-Net model can be represented as:

[0048]

[0049]

[0050] where I denotes the real video frame, and N denotes the number of video frames. The higher the value of PSNR, the better the quality of the output video frame obtained by the model. In order to further improve the detection performance of the model, the abnormality detection score combining the predicted video frame and the reconstructed video frame is calculated based on the PSNR value;

[0051] The abnormality score of the reconstructed video frame obtained by the two cascaded attention U-Net models can be represented as:​​​

[0052]

[0053] wherein, represents a predicted video frame, I represents a real video frame;

[0054] The final anomaly score can be represented as:

[0055]

[0056] wherein, alpha represents the weight of the anomaly score, and according to the anomaly score, it can be calculated whether an abnormal event occurs in the current video frame.

[0057] Compared with the prior art, the present application has the following beneficial effects:

[0058] The present application learns from the overall characteristics of human visual perception, uses a cascaded video frame prediction network and a video frame reconstruction network, effectively amplifies the reconstruction error of abnormal events, reduces the reconstruction error of normal events, and improves the performance of video-based abnormal event detection. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is the overall flowchart of the embodiments of the present application;

[0060] Figure 2 is the time migration module involved in the present application;

[0061] Figure 3 is the prediction network model diagram based on the time migration module and attention U-Net involved in the present application;

[0062] Figure 4 is the reconstruction network model diagram based on the cascaded attention U-Net involved in the present application;

[0063] Figure 5 is the test effect diagram of the method designed in the present application on the Avenue public data set;

[0064] Figure 6 is the test effect diagram of the method designed in the present application on the Ped2 public data set. DETAILED DESCRIPTION

[0065] In order to make the technical means, creative features, purposes and effects achieved by the present application easy to understand, the present application will be further described below in conjunction with specific embodiments.

[0066] In order to establish an intelligent video monitoring system for automatically identifying abnormal events, the application proposes a video abnormal event detection method based on a cascaded attention U-Net, aiming to cascade a frame prediction model and a reconstruction frame model to further improve the performance of abnormal event detection. To achieve the above purpose, the technical solution adopted by the application is as follows:

[0067] (1-1) Construct the U-Net down-sampling module. Perception is performed by consecutive down-sampling modules, and the output z i of each down-sampling module can be represented as:

[0068] z i =maxpooling(σ(W i 2 *σ(W i 1 *x i ))) (1)

[0069] Where i represents the i-th down-sampling module, x i represents the input of the i-th down-sampling module, W i 1 , W i 2 represents the weight matrix learned in the i-th down-sampling module, * represents convolution operation, sigma represents ReLU nonlinear activation function, maxpooling represents pooling layer, through the operation of the down-sampling module, abstract high-dimensional features are gradually extracted, and the size of the output feature z i of the i-th down-sampling module is H i x W i x C i , wherein C i represents the number of channels output by the i-th down-sampling block.

[0070] (1-2) Construct the time migration module. The output of the last frame prediction down-sampling module is taken as the input of the time migration module, first two consecutive convolution layers are used to increase the number of channels, and then the time migration module is used to migrate the channel information of the extracted features, realize the information interaction of different time channels, and pay attention to the change of abnormal behavior in different time dimensions, and the output feature can be represented as:

[0071]

[0072] Where z shift represents the output of the time migration module, z last represents the output of the last frame prediction down-sampling module, is the weight matrix learned in the time migration module, and sigma represents the ReLU nonlinear activation function. The size of the output of the time migration module is Hshift xW shift x C shift .

[0073] (1-3) Construct an attention gate-based U-Net upsampling module, which uses a continuous upsampling module for feature perception. The attention gate can gradually suppress the feature response of irrelevant background regions and enhance the salient features of the region of interest, aiming to focus on regions of different shapes and sizes, which can improve the model sensitivity and prediction accuracy, thereby achieving more accurate and robust anomaly event recognition performance. The attention gate can be expressed as:

[0074]

[0075]

[0076]

[0077]

[0078] where l represents the l-th upsampling module, (i,j) represents the index value, x l represents the feature that needs to be upsampled, ψ l is the feature that completes the convolution operation on x l , g l represents the feature corresponding to x l in the down-sampling module, is the feature that completes the convolution operation on g l , M l represents the attention coefficient obtained from formula (5), W1 l , W l , b l are the weight matrices learned in the convolution operation, σ represents the ReLU nonlinear activation function, δ represents the Sigmoid nonlinear activation function, is the output obtained by the attention gate. Finally, the output of the upsampling module is calculated using the features obtained by the attention gate:

[0079]

[0080] where concentrate represents feature aggregation, is the parameter matrix learned in the convolution operation, σ represents the ReLU nonlinear activation function, upsampling represents the up-sampling operation, y l represents the output feature of the upsampling module, and the size can be represented as H l x W l x C l .

[0081] (1-4) Construct a video frame prediction module, which is composed of a temporal migration module and an attention U-Net network. At time t, HxW RGB images of consecutive T frames are taken as the input of the prediction frame module. First, the down-sampling features of different scales are obtained by using step (1-1); then, the channel information migration is completed for the extracted features by using step (1-2), and the temporal interaction information is obtained; finally, the video frame prediction is completed by using the features obtained by using steps (1-1) and (1-2) and combining step (1-3), and the next future video frame of the consecutive multiple video frames is predicted. The video frame prediction module can expand the reconstruction difference of abnormal samples, which is helpful for the model to identify abnormal events.

[0082] (1-5) Construct a cascaded video frame reconstruction module, which is composed of two cascaded attention U-Net networks. Specifically, the predicted video frame obtained by using step (1-4) is taken as the input of the cascaded attention U-Net network, the first attention U-Net network in the reconstruction module sequentially performs steps (1-1) and (1-3), and then the output of the first attention U-Net network in the reconstruction module is taken as the input of the second attention U-Net network in the reconstruction module, and steps (1-1) and (1-3) are sequentially performed to obtain the reconstructed video frame The reconstructed video frame can reduce the reconstruction error of the normal video frame, and further expand the gap with the reconstruction error of the abnormal behavior, which is helpful for the model to identify abnormal events.

[0083] (1-6) Construct a multi-element loss function to train the cascaded attention U-Net network. Steps (1-1) to (1-5) complete the construction of the network model of cascaded frame prediction and frame reconstruction. In order to minimize the gap between the predicted reconstruction frame and the real frame, the similarity of the predicted reconstruction frame and the real frame in appearance and motion is fully considered, and the multi-element loss function is composed of intensity loss, gradient loss and structural similarity loss. The intensity loss can be expressed as:

[0084]

[0085] Where I represents the real video frame, represents the predicted reconstruction frame obtained by the network model of cascaded frame prediction and frame reconstruction, and (i,j) represents the index value.

[0086] The gradients in the horizontal and vertical directions can be calculated as:

[0087]

[0088] ​​

[0089] Combining equations (9), (10), the gradient loss can be expressed as:

[0090]

[0091] Combining equations (8), (11), the loss function can be expressed as:

[0092]

[0093] wherein, represents the structural similarity loss, and a, b and g represent the weights of the loss function. In the training process, the network parameters are adjusted by minimizing the loss function and back propagation. In order to obtain the optimal abnormal event detection model, only normal data is used for training. In this case, the historical information in the model is beneficial to the normal behavior frame, which will result in smaller error between the predicted reconstruction frame and the real frame. In contrast, the abnormal video frame cannot be accurately predicted, and the error between the predicted reconstruction frame of abnormal behavior and the real frame will be larger.

[0094] (1-7) Calculate the anomaly detection score combining the predicted frame and the reconstructed frame to realize video anomaly event detection. By performing steps (1-1) to (1-6), the optimal cascaded frame prediction and frame reconstruction network model can be obtained, and the anomaly detection is performed according to the difference between the predicted reconstruction frame and its corresponding real frame. First, the peak signal-to-noise ratio (PSNR) is calculated to evaluate the image quality, which can be expressed as:

[0095]

[0096] wherein, represents the predicted video frame or the reconstructed video frame, I represents the real video frame, and N represents the number of video frames. The higher the value of PSNR, the better the quality of the output video frame obtained by the model. In order to further improve the detection performance of the model, the anomaly detection score combining the predicted video frame and the reconstructed video frame is calculated based on the PSNR value. The anomaly score of the predicted video frame obtained by the first attention U-Net model can be expressed as:

[0097]

[0098] wherein, represents the predicted video frame, and I represents the real video frame. The anomaly score of the reconstructed video frame obtained by the cascaded two attention U-Net models can be expressed as:

[0099]

[0100] wherein, represents the predicted video frame, and I represents the real video frame. The final anomaly score can be expressed as:

[0101]

[0102] wherein a represents the weight of the abnormal score, according to which the abnormal score can be calculated whether an abnormal event occurs in the current video frame.

[0103] It is apparent for a person skilled in the art that the present application is not limited to the details of the above-described exemplary embodiments, but can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and not restrictive, the scope of the present application being defined by the appended claims rather than the above description, and it is intended to embrace all changes and modifications that fall within the meaning and scope of the equivalent elements of the claims. Any reference signs in the claims should not be construed as limiting the claims to the figures in which the reference signs are used.

[0104] Furthermore, it should be understood that although the present specification describes exemplary embodiments, not every exemplary embodiment contains only one independent technical solution, and the present specification is described in this way only for the sake of clarity, and a person skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that a person skilled in the art can understand.

Claims

1. A video abnormal event detection method based on cascaded attention U-Net, characterized in that: The specific steps are: S1: Build U-Net downsampling module; S2: build time migration module; S3: Construct a U-Net upsampling module based on attention gate and use continuous upsampling modules for feature perception; S4: Construct a video frame prediction module, which consists of a time migration module and an attention U-Net network; S5: Construct a cascaded video frame reconstruction module, which consists of two cascaded attention U-Net networks; S6: The output of the first attention U-Net network in the reconstruction module is used as the input of the second attention U-Net network in the reconstruction module, and steps S1 and S3 are performed in sequence to obtain a reconstructed video frame; S7: Construct a multivariate loss function and train the cascaded attention U-Net network; The specific operation in S7 includes performing steps S1 to S6 to obtain an optimal cascade frame prediction and frame reconstruction network model, and performing anomaly detection based on the difference between the predicted and reconstructed frame and its corresponding real frame; First, calculate the peak signal-to-noise ratio (PSNR) to evaluate the image quality, which can be expressed as: in, Represents the predicted video frame or the reconstructed video frame, represents a real video frame, Indicates the number of video frames. The higher the PSNR value, the better the quality of the output video frames obtained by the model. To further improve the detection performance of the model, the anomaly detection score of the combined predicted and reconstructed video frames is calculated based on the PSNR value. The anomaly score of the predicted video frame obtained by the first attention U-Net model is expressed as: in, represents the predicted video frame, represents a real video frame; The anomaly score of the reconstructed video frame obtained by the cascaded two attention U-Net models is expressed as: in, represents the predicted video frame, represents a real video frame; The final anomaly score is expressed as: in, The weight of the anomaly score is used to calculate whether an abnormal event has occurred in the current video frame; S8: Calculate the anomaly detection score by combining the predicted frame and the reconstructed frame to realize video abnormal event detection.

2. The method for detecting abnormal events in videos based on cascaded attention U-Net according to claim 1, characterized in that: The U-Net downsampling module specifically includes perception by continuous downsampling modules, and the output z of the downsampling module of each layer is i Expressed as: With i =maxpooling(σ(W i 2 *σ(W i 1 *x i ))) (1) in, Indicates the Downsampling modules, Indicates the The input of the downsampling module, 、 Indicates the The weight matrix learned by two consecutive convolutional layers in a downsampling module, * represents the convolution operation, ReLU nonlinear activation function, maxpooling represents the pooling layer, through the downsampling module operation, gradually extracting abstract high-dimensional features, The output feature z of the downsampling module i The size of ,in Indicates the The number of channels that the block outputs.

3. The video abnormal event detection method based on cascaded attention U-Net according to claim 1 is characterized in that The time migration module specifically uses the output of the last layer of frame prediction downsampling module as the input of the time migration module. First, two consecutive convolutional layers are used to increase the number of channels. Then, the time migration module is used to migrate the channel information of the extracted features, realizing information interaction between different time channels and focusing on changes in abnormal behavior in different time dimensions. Its output features can be expressed as: Among them, z shift represents the output of the time migration module, represents the output of the last frame prediction downsampling module, 、 is the weight matrix learned by two consecutive convolutional layers in the time migration module, Represents the ReLU nonlinear activation function, and the output size of the time migration module is .

4. The video abnormal event detection method based on cascaded attention U-Net according to claim 1 is characterized in that The U-Net upsampling module based on the attention gate specifically includes an attention gate that gradually suppresses the feature responses of irrelevant background areas and enhances the salient features of the region of interest. It aims to focus on regions of different shapes and sizes, improve model sensitivity and prediction accuracy, and thus achieve more accurate and robust abnormal event recognition performance; Attention gate is expressed as: in, Indicates the Upsampling modules, Represents the index value, Indicates the features that need to be upsampled, For Complete the features of the convolution operation, Indicates that the module is used The corresponding features, For Complete the features of the convolution operation, represents the attention coefficient obtained by formula (5), 、 、 、 、 、 is the weight matrix learned in the convolution operation, represents the ReLU nonlinear activation function, express Sigmoid Non-linear activation function, is the output of the attention gate; Finally, the features obtained by the attention gate are used to calculate the output of the upsampling module: in, represents feature aggregation, 、 is the parameter matrix learned in the convolution operation, represents the ReLU nonlinear activation function, represents the upsampling operation, Represents the output features of the upsampling module, and its size can be expressed as .

5. The method for detecting abnormal events in videos based on cascaded attention U-Net according to claim 1, characterized in that: The video frame prediction module specifically includes RGB images of consecutive frames as the input of the prediction frame module. First, S1 is used to obtain down-sampled features of different scales; then step S2 is performed on the features of the last layer of down-sampling module to complete channel information migration of the extracted features and obtain temporal interaction information; finally, the features obtained in steps S1 and S2 are combined with step S3 to complete video frame prediction and predict the next future video frame for multiple consecutive video frames.

6. The method for detecting abnormal events in videos based on cascaded attention U-Net according to claim 1, characterized in that: The predicted video frame obtained in step S4 is used as the input of the cascaded attention U-Net network, and the first attention U-Net network in the reconstruction module performs steps S1 and S3 in sequence. Then, the output of the first attention U-Net network in the reconstruction module is used as the input of the second attention U-Net network in the reconstruction module, and steps S1 and S3 are performed in sequence to obtain the reconstructed video frame.

7. The method for detecting abnormal events in videos based on cascaded attention U-Net according to claim 1, characterized in that: Steps S1 to S5 complete the construction of the network model for cascaded frame prediction and frame reconstruction. In order to minimize the gap between the predicted and reconstructed frames and the real frames, and fully consider the similarity in appearance and motion between the predicted and reconstructed frames and the real frames, the multivariate loss function consists of intensity loss, gradient loss, and structural similarity loss. The strength loss is expressed as: in, represents a real video frame, Represents the predicted and reconstructed frame obtained by the cascaded frame prediction and frame reconstruction network model, Represents the index value; The gradients in the horizontal and vertical directions are calculated as: Combining formulas (9) and (10), the gradient loss can be expressed as: Combining formulas (8) and (11), the loss function can be expressed as: in, represents the structural similarity loss, 、 and Represents the weight of the loss function.

Images