A video abnormal event detection method based on a double-flow space-time autoencoder

Abstract

The application belongs to the field of video monitoring, and discloses a video abnormal event detection method based on a double-flow space-time autoencoder, which specifically comprises the following steps: video preprocessing; setting a deep space-time autoencoder; constructing a pseudo-abnormal dataset and performing adversarial training; training a binary classifier using reconstruction error; and abnormal event detection. The application can effectively model normal events by effectively combining appearance and motion information, improve the performance of the reconstruction framework by ignoring redundant environmental information to detect abnormal events, and further improve the accuracy of abnormal event detection by using a pseudo-abnormal dataset in the adversarial branch of each of the two branches, which can enhance the ability to identify abnormal events. Experiments on three datasets prove that the method is superior to a series of state-of-the-art methods at the frame level and the pixel level.

Description

Technical Field

[0001] This invention belongs to the field of video surveillance in intelligent security, and particularly relates to a video anomaly event detection method based on a dual-stream spatiotemporal autoencoder. Background Technology

[0002] With the rapid development of internet technology, intelligent surveillance is playing an increasingly important role in maintaining order in public places, and anomaly detection technology is a core technology in this field. However, most video content supervision and analysis currently relies on manual labor. This traditional manual anomaly detection not only wastes a significant amount of human resources but also fails to provide timely warnings of anomalies. Due to the low frequency of anomalies; the lack of clear definitions for positive and anomalous events in different scenarios; and the high computational cost and slow speed of anomaly detection methods, which cannot meet real-time requirements, anomaly detection remains a highly challenging task.

[0003] Because anomalous events are rare and anomalous data is scarce, most methods employ unsupervised training of anomaly detection models containing only normal samples. Autoencoders have shown good performance in video anomaly detection, adhering to the reconstruction-based assumption that the reconstruction error of anomalous events is much larger than that of normal events due to their irregularity. Chong et al. and Zhao et al. added 3D convolutional and convolutional long short-term memory networks to the original autoencoder to extract temporal information. Wang et al. proposed a hybrid autoencoder to learn appearance and motion features from long-term videos. However, previous methods often simply integrate appearance and motion information and fail to accurately simulate normal events.

[0004] Recently, two-stream networks have been used for video anomalous event detection. The basic idea is to use a single-frame RGB image as input to process spatial information and a multi-frame optical flow map as input to process temporal information. Anomalous events correspond to unexpected appearances and motions compared to normal events. Wei et al. used a two-stream fully convolutional neural network (FCN) to extract complementary information. Furthermore, Li et al. combined a two-stream framework with an autoencoder to explore appearance and motion cues for anomaly detection. In these methods, due to the powerful representation of deep neural networks, the combination of an autoencoder and two-stream network cannot guarantee large reconstruction errors for anomalous events. Moreover, two-stream networks require training on a large number of labeled datasets, but the current lack of anomalous samples makes them prone to overfitting.

[0005] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0006] Due to the powerful representation capabilities of deep neural networks, the combination of autoencoders and two-stream networks cannot guarantee a large reconstruction error for anomalous events. Furthermore, training two-stream networks requires a large labeled dataset, but the number of anomalous event samples is far smaller than that of normal events, making them prone to overfitting. Summary of the Invention

[0007] To address the problems of existing technologies, this invention provides a video anomaly event detection method based on a dual-stream spatiotemporal autoencoder. This method fully utilizes both spatial and temporal information. The spatial stream uses detected objects as input to reconstruct the appearance model of normal events, while the temporal stream uses optical flow maps as input to reconstruct the temporal model of normal events. The dual-stream approach effectively leverages both appearance and motion information. Furthermore, inspired by pseudo-labels, an adversarial branch based on a pseudo-dataset is proposed. On one hand, it can compensate for the lack of anomaly data required by classical supervised learning and solve the overfitting problem caused by training the autoencoder only on normal events. On the other hand, it can ensure a larger anomaly reconstruction error for anomaly events, thus improving detection accuracy.

[0008] The present invention is achieved through the following technical solution.

[0009] This invention provides a video anomaly event detection method based on a dual-stream spatiotemporal autoencoder, the method specifically including:

[0010] Step 1: Video preprocessing;

[0011] Step 2: Set up a dual-stream spatiotemporal autoencoder;

[0012] Step 3: Construct a pseudo-anomaly dataset and perform adversarial training;

[0013] Step 4: Train a binary classifier using the reconstruction error;

[0014] Step 5: Abnormal event detection.

[0015] Furthermore, in step one, the video is first processed into frames. The YOLOv3 object detection method and the Selflow optical flow estimation method are used to process the UCSD Ped1, UCSD Ped2 and Avenue datasets to extract the foreground objects in a single video frame and generate optical flow maps of consecutive video frames.

[0016] Furthermore, the dual-stream spatiotemporal autoencoder in step two consists of a spatial autoencoder and a temporal autoencoder, following the U-Net architecture. The spatial stream learns the appearance features of the target object, while the temporal stream learns higher-level motion features from the optical flow map; each encoder e * It consists of three blocks, including convolutional layers and max pooling layers; for spatial flow, the loss function for reconstructing the input object x is as follows:

[0017]

[0018] in, This is the output of the standard branch, with an input size of h = w = 64.

[0019] For the temporal flow, the loss function for reconstructing the input optical flow x′ is as follows:

[0020]

[0021] in, This is the output of the standard branch, with an input size of h = w = 64 and c = 2.

[0022] Furthermore, in step three, a pseudo-anomaly dataset is first designed and constructed. This dataset can be applied to different datasets and does not contain objects that might occur in real-world anomaly detection scenarios, such as people, cars, or skaters. Two images are randomly selected from the image dataset and the training dataset. i and I j A pseudo-anomaly image is generated, as shown below:

[0023]

[0024] Where i ≠ j.

[0025] Furthermore, the pseudo-abnormal samples are input into the adversarial branch, and the autoencoder must process these cases that exceed normal events, enabling it to handle situations beyond normal events.

[0026] Furthermore, in step four, a binary classifier is used, employing a classic supervised strategy to distinguish between normal and pseudo-abnormal examples, specifically including:

[0027] After training the spatial flow and temporal flow to their optimal values ​​respectively, two binary classifiers are proposed, using a similar classical supervision strategy to distinguish between normal and pseudo-abnormal examples. The input of the binary classifier is the reconstruction error between the input and output of the normal reconstruction branch and the adversarial reconstruction branch.

[0028]

[0029] Where x is the input of the spatial stream or the temporal stream,

[0030]

[0031] in, The input batch is of size N and number i th hypercube

[0032] It is a prediction of normal and adversarial reconstruction of spatial or temporal flows.

[0033] Furthermore, the anomaly detection performed in step five specifically includes:

[0034] (1) Use the normal branch of the two-stream network without using the adversarial branch to obtain the appearance and motion representation;

[0035] (2) Calculate the spatial and temporal reconstruction errors and input them into the classifier;

[0036] (3) The two binary classifiers output the normal score for each object;

[0037] (4) Pixel-level anomaly detection can be obtained by recombining the detected anomaly scores.

[0038] Furthermore, the normal score is normalized between 0 and 1, and the abnormal score of any pixel in each object is calculated by subtracting the average of the two normal scores from 1. An abnormal event can be identified by using an appropriate threshold between 0 and 1.

[0039] Furthermore, the normal score is a score for each pixel, while the frame-level abnormal score is the maximum value among the pixel scores of the corresponding frame.

[0040] Based on the above technical solutions and the technical problems solved, please analyze the advantages and positive effects of the technical solution to be protected by this invention from the following aspects:

[0041] First, addressing the technical problems existing in the prior art and the difficulty in solving them, this paper closely analyzes, in conjunction with the technical solution to be protected by this invention and the results and data obtained during the research and development process, how the technical solution of this invention solves the technical problems, and the inventive technical effects brought about by solving these problems. The specific description is as follows:

[0042] This invention proposes a novel two-stream spatiotemporal autoencoder method to determine anomalous events in videos. The new method makes full use of the spatial and temporal information of video frames extracted by the two-stream network.

[0043] This invention combines adversarial training methods, using pseudo-abnormal samples as input to the adversarial branch, to prevent the autoencoder from reconstructing abnormal frames well and further enhancing its ability to distinguish between normal and abnormal events.

[0044] Experimental results on three common datasets show that the proposed method outperforms most existing methods in frame-level and pixel-level standards.

[0045] Second, considering the technical solution as a whole or from the perspective of the product, the technical effects and advantages of the technical solution to be protected by this invention are specifically described as follows:

[0046] This invention designs a novel adversarially trained dual-stream spatiotemporal autoencoder network for video anomaly detection. This network fully utilizes appearance features extracted from the spatial stream and motion patterns learned from the temporal stream, effectively modeling normal events. The method uses object detection to extract foreground objects, ignoring redundant environmental information to reduce model parameters and improve detection time. Furthermore, the adversarial branch further enhances anomaly detection accuracy by using a pseudo-anomaly dataset as input; these pseudo-anomaly samples enhance the model's ability to identify anomalous events. On three basic datasets, the method outperforms a range of state-of-the-art methods at both the frame and pixel levels.

[0047] Third, as supplementary evidence of the inventive step of the claims of this invention, it is also reflected in the following important aspects:

[0048] The technical solution of this invention solves a technical problem that people have long desired to solve but have been unable to achieve:

[0049] Traditional manual anomaly detection is time-consuming and labor-intensive, and current video anomaly detection methods also suffer from numerous problems, such as a lack of anomaly event samples, the inability of autoencoders to guarantee significant reconstruction errors for anomalies, and the high computational cost, slow speed, and inability to meet real-time requirements. The solution presented in this invention successfully addresses these technical problems using a dual-stream spatiotemporal autoencoder with adversarial training, demonstrating inventiveness. Attached Figure Description

[0050] Figure 1 This is a flowchart of a method for a dual-stream spatiotemporal autoencoder with adversarial training for video anomaly detection provided in an embodiment of the present invention.

[0051] Figure 2 This is a flowchart illustrating the basic steps of video anomaly detection provided in this embodiment of the invention;

[0052] Figure 3 This is an overall structural diagram of the model provided in the embodiment of the present invention;

[0053] Figure 4 This is a structural diagram of the self-encoder provided in an embodiment of the present invention;

[0054] Figure 5 These are frame-level and pixel-level ROC curves on the UCSD dataset provided in this embodiment of the invention;

[0055] Figure 6These are some comparisons of reconstruction error maps on the UCSD ped1 and ped2 datasets provided in this embodiment of the invention;

[0056] Figure 7 This is a diagram showing the spatial, temporal, and dual-stream results of adversarial training provided in this embodiment of the invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0058] I. Explanatory and Illustrative Embodiments. To enable those skilled in the art to fully understand how the present invention is specifically implemented, this section provides an explanatory and illustrative description of the embodiments described in the claims.

[0059] This invention provides a video anomaly event detection method based on a dual-stream spatiotemporal autoencoder, the method specifically including:

[0060] S1: Video preprocessing;

[0061] S2: Set the dual-stream spatiotemporal autoencoder;

[0062] S3: Construct a pseudo-anomaly dataset and perform adversarial training;

[0063] S4: Train a binary classifier using reconstruction error;

[0064] S5: Abnormal event detection.

[0065] S1 first processes the video into frames, and uses the YOLOv3 object detection method and the Selflow optical flow estimation method to process the UCSD Ped1, UCSD Ped2 and Avenue datasets, extracting foreground objects in a single video frame and generating optical flow maps of consecutive video frames.

[0066] The dual-stream spatiotemporal autoencoder in S2 consists of a spatial autoencoder and a temporal autoencoder, following the U-Net architecture. The spatial stream learns the appearance features of the target object, while the temporal stream learns higher-level motion features from the optical flow map; each encoder e * It consists of three blocks, including convolutional layers and max pooling layers; for spatial flow, the loss function for reconstructing the input object x is as follows:

[0067]

[0068] in, This is the output of the standard branch, with an input size of h = w = 64.

[0069] For the temporal flow, the loss function for reconstructing the input optical flow x′ is as follows:

[0070]

[0071] in, This is the output of the standard branch, with an input size of h = w = 64 and c = 2.

[0072] In step S3, a pseudo-anomaly dataset is first designed and constructed. This dataset can be applied to different datasets and does not contain objects that might occur in real-world anomaly detection scenarios, such as people, cars, or skaters. Two images, I, are randomly selected from the image dataset and the training dataset. i and I j A pseudo-anomaly image is generated, as shown below:

[0073]

[0074] Where i ≠ j.

[0075] The pseudo-abnormal samples are input into the adversarial branch, and the autoencoder must process these cases that exceed normal events to enable it to handle situations beyond normal events.

[0076] The S4 section utilizes a binary classifier and a classic supervised strategy to distinguish between normal and pseudo-abnormal examples, specifically including:

[0077] After training the spatial flow and temporal flow to their optimal values ​​respectively, two binary classifiers are proposed, using a similar classical supervision strategy to distinguish between normal and pseudo-abnormal examples. The input of the binary classifier is the reconstruction error between the input and output of the normal reconstruction branch and the adversarial reconstruction branch.

[0078]

[0079] Where x is the input of the spatial stream or the temporal stream,

[0080]

[0081] in, The input batch is of size N and number i th hypercube

[0082] It is a prediction of normal and adversarial reconstruction of spatial or temporal flows.

[0083] Anomaly detection is performed in S5, specifically including:

[0084] (1) Use the normal branch of the two-stream network without using the adversarial branch to obtain the appearance and motion representation;

[0085] (2) Calculate the spatial and temporal reconstruction errors and input them into the classifier;

[0086] (3) The two binary classifiers output the normal score for each object;

[0087] (4) Pixel-level anomaly detection can be obtained by recombining the detected anomaly scores.

[0088] The normal score is normalized between 0 and 1. The abnormal score of any pixel in each object is calculated by subtracting the average of the two normal scores from 1. An abnormal event can be identified by using an appropriate threshold between 0 and 1. The normal score is the score for each pixel, while the frame-level abnormal score is the maximum value of the pixel detection in the corresponding frame.

[0089] II. Application Examples. To demonstrate the inventiveness and technical value of the technical solution of this invention, this section provides application examples of the technical solution of the claims on specific products or related technologies.

[0090] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0091] III. Evidence of the Relevant Effects of the Embodiments. The embodiments of the present invention have achieved some positive effects during research and development or use, and indeed possess significant advantages compared to existing technologies. The following description, in conjunction with data, charts, and other materials from the experimental process, illustrates these advantages.

[0092] In this embodiment, the proposed method was evaluated on three benchmark anomaly detection datasets: the Avenue dataset, the UCSD pedestrian datasets Ped1 and Ped2, using both frame-level and pixel-level evaluation criteria. The training dataset contains only video clips under normal conditions, while the test dataset includes both anomalous and normal events.

[0093] Specifically, Figure 1 This is a flowchart of a video anomaly event detection method based on a dual-stream spatiotemporal autoencoder provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the basic steps of video anomaly detection provided in this embodiment of the invention; the network structure of this invention is based on a dual-stream spatiotemporal autoencoder network architecture as follows: Figure 3 As shown, the specific autoencoder network structure is as follows: Figure 4 As shown, it consists of an encoder and a decoder, with both the input and output sizes being 64*64*32.

[0094] Specifically, based on the previous evaluation methods, receiver operating characteristic (ROC) curves and average error rate (EER) are used to measure frame-level and pixel-level accuracy. The results for the UCSD ped1 and ped2 datasets are shown in Tables 1 and 2. Figure 5 (a) and (b) show the frame-level and pixel-level ROC curves on the UCSD Ped1 dataset, and (c) and (d) show the frame-level and pixel-level ROC curves on the UCSD Ped2 dataset. A larger area under the curve indicates better performance. Quantitative analysis shows that the method of this invention outperforms existing methods in both frame-level and pixel-level evaluation, achieving a frame-level accuracy of 91.5% on Ped1 and 97.9% on Ped2. Furthermore, the method of this invention achieves 82.7% accuracy on the pixel-level standard of Ped1 and 95.1% on Ped3. As shown in Table 3, the AUC and EER of the method of this invention are superior to other methods on the Avenue dataset. The method of this invention exhibits high anomaly detection accuracy at the frame level, achieving an AUC of 86.7% and an EER of 19.5% on the Avenue dataset.

[0095] Table 1

[0096]

[0097] Table 2

[0098]

[0099] Table 3

[0100]

[0101] Specifically, in order to better evaluate the performance of the anomaly detection method of the present invention, anomaly score curves and reconstruction error maps are plotted for each video to qualitatively analyze the model of the present invention. Figure 6 This shows a comparison of anomaly scoring curves across different test videos from UCSD ped2 and Avenue. The red area represents frame-level ground truth of the anomaly event; when the anomaly event occurs, the predicted anomaly score increases significantly. Figure 6 As shown, the test results perfectly match the actual ground conditions, indicating that the proposed method achieves better performance than existing methods. Figure 7 This paper presents a comparison of reconstruction error maps of the method of the present invention on the UCSD ped1 and ped2 datasets. The left column represents samples from the anomalous dataset. The middle two columns show the reconstruction error maps of the spatial and temporal flows. The right column shows the pixel-level reconstruction error map of the method of the present invention. It can be observed that the method of the present invention can successfully locate anomalous events.

[0102] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A video anomaly event detection method based on a dual-stream spatiotemporal autoencoder, characterized in that: The video abnormal event detection method based on dual-stream spatiotemporal autoencoder utilizes spatial and temporal information. The spatial stream uses the detected objects as input to reconstruct the appearance model of normal events, while the temporal stream uses the optical flow map as input to reconstruct the temporal model of normal events. The dual stream can make full use of appearance and motion information. The video anomaly event detection method based on a dual-stream spatiotemporal autoencoder specifically includes: Step 1: Video preprocessing; Step 2: Set up a dual-stream spatiotemporal autoencoder; Step 3: Construct a pseudo-anomaly dataset and perform adversarial training; Step 4: Train a binary classifier using the reconstruction error; Step 5: Abnormal event detection; In step three, a pseudo-anomaly dataset is first designed and constructed, which does not contain objects that may occur in real anomaly detection scenarios, including people, cars, and skaters; then, two images are randomly selected from the image dataset and the training dataset. and The pseudo-anomaly sample is generated as follows: ； in, ; The pseudo-abnormal samples are input into the adversarial branch, and the autoencoder must process these cases that are beyond the normal events to enable it to handle cases that are beyond the normal events. Step four, which utilizes a binary classifier and a classic supervised strategy to distinguish between normal and pseudo-abnormal examples, specifically includes: After training the spatial flow and temporal flow to their optimal values ​​respectively, two binary classifiers are proposed, using a similar classical supervision strategy to distinguish between normal and pseudo-abnormal examples. The input of the binary classifier is the reconstruction error between the input and output of the normal reconstruction branch and the adversarial reconstruction branch. ； in, It is the input of spatial or temporal streams. ; ； in, It is the size of The input batch is hypercube It is a prediction of normal and adversarial reconstruction of spatial or temporal flows. ; Step five involves anomaly detection, specifically including: (1) Using the normal branches of the two-stream network, without using adversarial branches, to obtain representations of appearance and motion; (2) Calculate the spatial and temporal reconstruction errors and input them into the classifier; (3) The two binary classifiers output the normal score for each object; (4) Pixel-level abnormal event detection is obtained by recombining the detected abnormal scores.

2. The video anomaly event detection method based on a dual-stream spatiotemporal autoencoder as described in claim 1, characterized in that, The first step involves processing the video into frames. The YOLOv3 object detection method and the Selflow optical flow estimation method are used to process the UCSD Ped1, UCSD Ped2 and Avenue datasets, extracting foreground objects from individual video frames and generating optical flow maps of consecutive video frames. The dual-stream spatiotemporal autoencoder in step two consists of a spatial autoencoder and a temporal autoencoder, following the U-Net architecture. The spatial stream learns the appearance features of the target object, while the temporal stream learns higher-level motion features from the optical flow map; each encoder... It consists of three blocks, including convolutional layers and max pooling layers; for spatial flow, it reconstructs the input object. The loss function is shown below: ； in, This is the output of the standard branch, with an input size of [value missing]. ; For temporal streams, reconstruct the input optical flow. The loss function is shown below: ； in, This is the output of the standard branch, with an input size of [value missing]. .

3. The video anomaly event detection method based on a dual-stream spatiotemporal autoencoder as described in claim 1, characterized in that, The normal score is normalized between 0 and 1. The abnormal score of any pixel in each object is calculated by subtracting the average of the two normal scores from 1, and then using an appropriate threshold between 0 and 1 to determine the abnormal event.

4. The video anomaly event detection method based on a dual-stream spatiotemporal autoencoder as described in claim 3, characterized in that, The normal score is the score for each pixel, while the frame-level anomaly score is the maximum value detected in the pixels of the corresponding frame.

Images