Video anomaly detection-based thrown object detection method and device, and storage medium

Abstract

According to the thrown object detection method and device based on video anomaly detection and the storage medium, normal continuous video frames without thrown objects are used as input, then an improved auto-encoder model is trained to achieve anomaly detection of the thrown objects, multi-frame smoothing is carried out on an error graph obtained after detection, interference of a running vehicle is reduced, a brightness threshold value is set, and the thrown object detection accuracy is improved. If highlight areas exist in the smoothed error graph, the areas are contained in a minimum bounding rectangular frame, then a result is displayed and stored, it is judged that the thrown objects exist in the current time period, and meanwhile early warning is given to a system. According to the method, the detection precision of the throwing object abnormal event can be improved, a throwing object data set does not need to be marked, the problem that throwing object types cannot be exhaustive is solved, and intelligentization of road surface throwing object detection is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video image processing, and is a method, device and storage medium for detecting spilled objects based on video anomaly detection. Background Art

[0002] Spilled objects on the road, including goods, beverage bottles, cartons and waste tires, are common causes of road traffic accidents. Especially in tunnels, where the light is dim, spilled objects on the road are easily overlooked, posing a serious threat to driving safety. If a vehicle encounters spilled objects while driving at high speed, the driver may take emergency evasive measures due to the short reaction time, which may easily lead to rear-end collisions, collisions with guardrails or rollovers, and even cause chain reactions. Timely and accurate detection of spilled objects on the road is a key link in video traffic monitoring, and is even more crucial to traffic management systems.

[0003] Computer vision technology can continuously monitor road conditions, which is more efficient than manual supervision. It can process and analyze a large amount of video data in real time, quickly identify spilled objects and issue warnings, which is crucial to enhancing road safety and improving the level of road informatization. With the promotion and popularization of deep learning, deep learning networks such as the yolo series, RetinaNet and CenterMask are used for spilled object detection, directly outputting the category and location of the spilled objects. The above deep learning networks can generally only output fixed categories of spilled objects and require sufficient training samples and annotations, but there is a problem of not being able to detect spilled objects outside their category range. For the problem that there are many types of detection objects and it is impossible to exhaust them, some detection models propose to compare the object to be tested with the normal sample. If there is inconsistency, it indicates an abnormal detection idea, such as CN110827265A "Image Anomaly Detection Method Based on Deep Learning" and CN110211114A "A Visual Detection Method for Missing Strips in Packing Based on Deep Learning", but the detection objects of these methods are limited to relatively static and simple detection objects and detection environments, such as workpieces on assembly lines, and their normal samples are relatively stable and single. In the detection of spilled objects on the road, there are various vehicles on the road in addition to spilled objects. Video images of roads without vehicles, roads with various vehicles or only some vehicles are all normal samples. Therefore, there is uncertainty in normal samples in the detection of spilled objects on the road. For example, in the solution of CN110211114A, due to the complex scene conditions of the road, it is difficult for the encoder in the existing autoencoder model to extract effective representation features of normal road conditions, which makes it difficult for the decoder to reconstruct the video frames to output correct normal samples, so that the reconstruction error of the normal video frames without spilled objects is very large, resulting in false detection of abnormal events of spilled objects.

[0004] In summary, road spill detection faces many challenges: first, the attributes of spilled objects such as size, color and shape are highly uncertain, and it is impossible to build a unified model based on deep learning target detection algorithm to detect all spilled objects; second, although some anomaly detection models can solve the problem of inexhaustible types of training samples by comparing the differences between normal and abnormal types, due to the complexity of road scenes, in addition to spilled objects, there are many types of vehicles and different speeds of different vehicles, which will make the reconstruction error of the model very large; finally, in addition to vehicles with changing motion, external changes in the road environment such as light and weather will also affect the construction of normal samples, and thus affect the detection accuracy. Summary of the invention

[0005] The problem to be solved by the present invention is that the existing deep learning network can generally only output fixed categories of spilled objects in road spilled object detection and requires sufficient training samples and annotations, and there is a problem of missed detection for spilled objects outside its category range; although the anomaly detection model based on the positive example comparison idea can solve the problems of insufficient training samples and a large number of annotations, it cannot cope with dynamically changing detection scenarios and lacks stable positive example comparison objects. The existing autoencoder is difficult to extract and reconstruct complex road conditions, which easily leads to false detection of spilled objects.

[0006] The technical solution of the present invention is: a method for detecting spilled objects based on video anomaly detection, obtaining a surveillance video taken by a surveillance camera, performing frame extraction on the video, outputting each frame as a picture and converting it into a grayscale image, screening out time periods without spilled objects to construct a normal continuous video frame sequence, training an improved autoencoder model as an anomaly detection model, and then using the trained improved autoencoder model to detect real-time surveillance video, calculating the error between the output of the improved autoencoder model and the real image to obtain a grayscale error map, performing multi-frame smoothing on the obtained grayscale error map to eliminate the interference of vehicle errors, setting a brightness threshold, and judging whether there is spilled objects in the current time period by judging whether there is an area exceeding the brightness threshold;

[0007] The improved autoencoder model first adds a coordinate gating module after the first convolution block of the encoder, and then introduces a feature fusion module between the encoder and the decoder, and first downsamples and fuses the four feature layers output by the encoder and then upsamples and fuses them. The sizes of the four feature layers after fusion remain unchanged and serve as the input of the decoder;

[0008] The multi-frame smoothing of the error map is as follows: N consecutive frames of grayscale error maps are selected for alignment, and the average value is calculated by superposition to smooth the image, and multi-frame smoothing is performed once for every N frames of the real-time monitoring video.

[0009] Further, the present invention comprises the following steps:

[0010] Step 1: Collect surveillance video of the road section to be detected;

[0011] Step 2: Cut the video collected in Step 1 into frames, output each frame as an image, then cut out the time information in each image and convert it into a grayscale image, filter out the time period without spilled objects, and construct a normal continuous video frame sequence as the training sample input of the anomaly detection model;

[0012] Step 3: The improved autoencoder model is trained with the normal continuous video frame sequence constructed in Step 2, and then the real-time continuous video frames are input into the model for spilled object detection, and the detected video frame sequence is output;

[0013] Step 4: Subtract the detected video frame sequence from the original video frame sequence to obtain an error map;

[0014] Step 5: Smooth the error map over multiple frames. The number of smoothed frames is set according to the length of the video to highlight the static features of the spilled objects and reduce the interference of moving vehicles. The first frame is used as a reference for multi-frame alignment and then smoothing. Set the brightness threshold. If there are areas above the brightness threshold in the smoothed image, these areas are included in a minimum bounding rectangle. Then the results are displayed and saved to determine whether there are spilled objects in the current period and to warn the system.

[0015] Furthermore, the surveillance video collected for the road section to be detected is of a fixed viewing angle and a fixed scene, and the frame rate is 30FPS.

[0016] Furthermore, the improved autoencoder model first adds a coordinate gating module after the first convolution block of the encoder, and generates a static gating mask related to the pixel coordinates of the image and multiplies the convolution result element by element to enhance or suppress the features of different positions, thereby enhancing the model's perception of spatial changes. Then, a feature fusion module is introduced between the encoder and the decoder to fuse the four feature layers T1, T2, T3, and T4 output by the encoder, including downsampling fusion and upsampling fusion. First, (T1, T2), (T2, T3), and (T3, T4) are fused. 4) Perform downward feature fusion in pairs. When T3 and T4 are fused, T3 contains the information of T2, and T2 at this time integrates the information of T1, so that the information of T1 is indirectly transmitted to T4, so that the information of the upper and lower layers is interacted. Then perform bottom-up feature fusion and pass the information back to the shallow layer. Perform upward feature fusion in pairs on (T4, T3), (T3, T2) and (T2, T1), respectively, so that the information of T4 is passed to T1 again, thereby realizing two-way transmission of information. Finally, the obtained four feature layers are input into the decoder and the reconstructed image is output.

[0017] Furthermore, the mean square error (MSE) is used to calculate the error between the output of the improved autoencoder model and the real image to obtain a grayscale error map.

[0018] Furthermore, a brightness threshold is set according to the characteristics of the scene, preferably set to 50.

[0019] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the above-mentioned method for detecting spilled objects based on video anomaly detection, wherein the input video is detected by an anomaly detection model to obtain an error map, and determine whether there are spilled objects on the road surface in the current period.

[0020] The present invention also provides a computer-readable storage medium, in which at least one instruction or at least one program is stored. When the at least one instruction or at least one program is executed, the above-mentioned method for detecting spilled objects based on video anomaly detection is implemented.

[0021] The present invention provides an improved autoencoder model, which enhances the encoder's feature extraction of normal video frames without spilled objects by adding a coordinate gating module and a feature fusion module, learns the characterization features of various normal road surfaces, reduces the reconstruction error of normal video frames, reduces false detections, and realizes accurate detection of abnormal events of spilled objects.

[0022] In view of the problems that the current deep learning target detection model cannot exhaustively classify the categories of spilled objects and the existing abnormal detection model has poor effect on detecting spilled objects, the present invention uses an improved autoencoder model to detect spilled objects. First, normal continuous video frames of the road surface without spilled objects are constructed as the training set of the model, and there is no need to label the spilled object data set, which solves the problem that the categories of spilled objects cannot be exhausted. Then, the improved autoencoder model is used to enhance the encoder's feature extraction of normal video frames without spilled objects by adding a coordinate gating module and a feature fusion module, improve the ability to learn the characterization features of complex road conditions, reduce the reconstruction error of normal video frames, reduce false detections, and achieve accurate detection of abnormal events of spilled objects.

[0023] In order to solve the problem that the detection accuracy is affected by the changes in ambient light and the changes in time information in video monitoring, the present invention first crops and grayscales the video frame sequence and then trains and detects the model.

[0024] Aiming at the interference problem caused by traveling vehicles in the detection error map, the present invention performs multi-frame smoothing on the detected error map to reduce interference and sets a brightness threshold. If there are areas above the threshold in the smoothed image, these areas are included in a minimum bounding rectangular frame, and the results are displayed and saved. It is determined that there are spilled objects in the current period, and an early warning is issued to the system, thereby realizing intelligent detection of spilled objects on the road surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The present invention is a flow chart for realizing the detection of spilled objects on the road surface.

[0026] Figure 2 This is a structural block diagram of the improved autoencoder model of the present invention.

[0027] Figure 3 This is a detection error diagram of the present invention applied in a real tunnel scenario.

[0028] Figure 4 This is a diagram of the spilled object detection results when the present invention is applied in a real tunnel scenario. DETAILED DESCRIPTION

[0029] The present invention obtains the video taken by the road monitoring camera, cuts the video into frames, outputs each frame as a picture, then cuts out the time information in each picture, converts it into a grayscale picture, filters out the time period without spilled objects, and constructs a normal continuous video frame sequence, then uses the improved autoencoder model for training, saves the trained weight file, and then detects the real-time continuous video frame sequence, performs multi-frame smoothing on the error map obtained by the detection, sets the brightness threshold, and determines whether there is spilled objects in the current time period. The flow diagram of the present invention is as follows: Figure 1 shown.

[0030] The implementation of the present invention is described below through an example of tunnel spilled object detection, and the implementation steps are as follows.

[0031] Step 1: Collect surveillance video of the tunnel section to be inspected. The surveillance video needs to be fixed angle, fixed scene, and have a frame rate of 30FPS.

[0032] Step 2: Cut the video collected in Step 1 into frames, output each frame as an image, then crop the time information in each image and convert it into a grayscale image, filter out the time period without spilled objects, and construct a normal continuous video frame sequence as the training sample input of the anomaly detection model.

[0033] Step 3: Improved autoencoder model such as Figure 2As shown, a coordinate gating module is first added after the first convolution block of the encoder. The coordinate gating module is a CoordGate module. CoordGate creates a gating mask by inputting a static coordinate map into a pixel-by-pixel fully connected encoding network. In the present invention, the coordinate gating module is located after the first convolution block of the encoder, and enhances or suppresses features at different positions by generating a static gating mask related to the pixel coordinates of the image and multiplying the convolution result of the first convolution block element by element, thereby enhancing the model's perception of spatial changes. Then a feature fusion module is introduced between the encoder and the decoder to perform feature fusion on the four feature layers (T1, T2, T3, T4) output by the encoder, including downsampling fusion and upsampling fusion. First, downward feature fusion is performed on (T1, T2), (T2, T3) and (T3, T4) in pairs. When T3 and T4 are fused, T3 already contains the information of T2, and T2 at this time has also fused the information of T1. Therefore, the information of T1 is indirectly passed to T4, so that the information of the upper and lower layers is interacted. Then, bottom-up feature fusion is performed to transmit the information back to the shallow layer. The upward feature fusion is performed on (T4, T3), (T3, T2) and (T2, T1) in pairs, so that the information of T4 can also be transmitted to T1, thereby realizing two-way transmission of information and improving accuracy. Finally, the obtained four feature layers are input into the decoder to train the normal continuous video frame sequence constructed in Step 2. After the model training is completed, the real-time continuous video frames are input into the model for detection, and the detected video frame sequence is output.

[0034] Step 4: Use the mean square error (MSE) to calculate the error between the output of the autoencoder model and the real image, and get the grayscale error graph as shown below: Figure 3 shown.

[0035] Step 5: Multi-frame smoothing of the error map, that is, first use the image alignment technology to align the first frame as the reference, and then calculate the average value of all aligned images to generate a smoothed result map. The number of smoothed frames is set according to the video length. In this embodiment, 150 frames are set for multi-frame smoothing to highlight the static characteristics of the scattered objects and reduce the error interference of the moving vehicles. The brightness threshold is set. If there are highlighted areas in the smoothed result map, there are scattered objects in the current period. These areas are included in a minimum bounding rectangular frame, such as Figure 4 As shown, the result is then displayed and saved, and the system is warned at the same time. The brightness threshold is set according to the characteristics of the scene, and preferably the brightness threshold is set to 50.

[0036] The embodiment of the present invention also provides an electronic device, including a processor and a memory, wherein at least one instruction or at least one program is stored in the memory, and the at least one instruction or at least one program is loaded and executed by the processor to implement the method for detecting spilled objects based on video anomaly detection described in the above embodiment, detect the input video through the anomaly detection model, output an error map, and judge whether there is a spilled object in the current period. The electronic device of the present invention is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, smart phones, wearable devices, and other similar data computing devices. The processor can be various general and / or special processing components with processing and computing capabilities. Some examples of processors include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc.

[0037] The embodiment of the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program is used to make the computer perform the method according to the embodiment of the present invention when executed by the processor of the computer. The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or completely on a remote machine or server.

[0038] The current deep learning target detection technology mainly relies on model training through data containing spilled objects to distinguish between spilled objects and other objects, but when new object categories appear, the model may not be able to accurately identify and detect; the video-based anomaly detection model can solve the problem of insufficient training samples and a large number of annotations, but due to the large number of road vehicle types and different vehicle speeds, the existing autoencoder is poor in extracting and reconstructing features, resulting in low detection accuracy for spilled objects. The present invention uses an anomaly detection model to detect spilled objects without annotating the spilled object data set, which solves the problem of not being able to fully characterize all categories of spilled objects due to insufficient sample size. At the same time, by adding a coordinate gating module and a feature fusion module, the encoder's feature extraction of normal video frames without spilled objects is enhanced, the reconstruction error of normal video frames is reduced, and false detection is reduced. The improved autoencoder model is trained by normal continuous video frames without spilled objects, and the detection accuracy of abnormal events of spilled objects on the road surface is improved, realizing the intelligent detection of road spilled objects.

Claims

1. A method for detecting spilled objects based on video anomaly detection, characterized in that Obtain surveillance video taken by a surveillance camera, extract frames from the video, output each frame as a picture and convert it into a grayscale image, filter out time periods without spilled objects to construct a normal continuous video frame sequence, train an improved autoencoder model as an anomaly detection model, and then use the trained improved autoencoder model to detect real-time surveillance video, calculate the error between the output of the improved autoencoder model and the real image, and obtain a grayscale error map. Perform multi-frame smoothing on the obtained grayscale error map to eliminate the interference of vehicle errors, set a brightness threshold, and determine whether there is spilled objects in the current period by judging whether there is an area exceeding the brightness threshold; The improved autoencoder model first adds a coordinate gating module after the first convolution block of the encoder, and then introduces a feature fusion module between the encoder and the decoder, and first downsamples and fuses the four feature layers output by the encoder and then upsamples and fuses them. The sizes of the four feature layers after fusion remain unchanged and serve as the input of the decoder; The multi-frame smoothing of the error map is as follows: N consecutive frames of grayscale error maps are selected for alignment, and the average value is calculated by superposition to smooth the image, and multi-frame smoothing is performed once for every N frames of the real-time monitoring video.

2. A method for detecting spilled objects based on video anomaly detection according to claim 1, characterized in that The following steps are involved: Step 1: Collect surveillance video of the road section to be detected; Step 2: Cut the video collected in Step 1 into frames, output each frame as an image, then cut out the time information in each image and convert it into a grayscale image, filter out the time period without spilled objects, and construct a normal continuous video frame sequence as the training sample input of the anomaly detection model; Step 3: The improved autoencoder model is trained with the normal continuous video frame sequence constructed in Step 2, and then the real-time continuous video frames are input into the model for spilled object detection, and the detected video frame sequence is output; Step 4: Subtract the detected video frame sequence from the original real-time video frame sequence to obtain an error map; Step 5: Smooth the error map over multiple frames. The number of smoothed frames is set according to the length of the video to highlight the static features of the spilled objects and reduce the interference of moving vehicles. The first frame is used as a reference for multi-frame alignment and then smoothing. Set the brightness threshold. If there are areas above the brightness threshold in the smoothed image, these areas are included in a minimum bounding rectangular frame. The results are then displayed and saved to determine that there are spilled objects in the current period and to warn the system.

3. A method for detecting spilled objects based on video anomaly detection according to claim 1 or 2, characterized in that The surveillance video collected for the road section to be detected has a fixed perspective and fixed scene, and the frame rate is 30FPS.

4. A method for detecting spilled objects based on video anomaly detection according to claim 1 or 2, characterized in that The improved autoencoder model first adds a coordinate gating module after the first convolution block of the encoder. The coordinate gating module generates a static gating mask related to the pixel coordinates of the image and multiplies the convolution result element by element to enhance or suppress the features at different positions, so as to enhance the model's perception of spatial changes. Then, a feature fusion module is introduced between the encoder and the decoder to fuse the four feature layers T1, T2, T3, and T4 output by the encoder, including downsampling fusion and upsampling fusion. First, (T1, T2), (T2, T3), and (T3, When T3 and T4 are fused, T3 contains the information of T2, and T2 at this time fuses the information of T1, so that the information of T1 is indirectly transmitted to T4, so that the information of the upper and lower layers interact with each other. Then, feature fusion is performed from bottom to top, and the information is transmitted back to the shallow layer. The upward feature fusion is performed on (T4, T3), (T3, T2) and (T2, T1) respectively, so that the information of T4 is transmitted to T1 again, thereby realizing the two-way transmission of information. Finally, the obtained four feature layers are input into the decoder to output the reconstructed image.

5. A method for detecting spilled objects based on video anomaly detection according to claim 1 or 2, characterized in that The mean square error (MSE) is used to calculate the error between the output of the improved autoencoder model and the real image to obtain a grayscale error map.

6. A method for detecting spilled objects based on video anomaly detection according to claim 1 or 2, characterized in that Set the brightness threshold according to the scene characteristics and set it to 50.

7. An electronic device, characterized in that The electronic device includes a processor and a memory, wherein at least one instruction or at least one program is stored in the memory, and the at least one instruction or at least one program is loaded and executed by the processor to implement the method for detecting spilled objects based on video anomaly detection as described in any one of claims 1 to 6, wherein the input video is detected by an anomaly detection model to obtain an error map, and it is determined whether there are spilled objects on the road surface in the current period.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction or at least one program. When the at least one instruction or the program is executed, the method for detecting spilled objects based on video anomaly detection as described in any one of claims 1 to 6 is implemented.

Images