A method for detecting projectiles by fusing differential video sequences and convolutional neural networks

Abstract

A method for detecting spilled objects by integrating differential video sequences and convolutional neural networks is proposed. First, a static background model of the scene is constructed. Then, a differential video sequence is built using a Gaussian-weighted background subtraction method to extract the foreground. Next, stationary targets within the foreground are extracted and combined with a convolutional neural network model to analyze and exclude non-sprayed objects from the stationary targets in the foreground, thus detecting the spilled object in the current video frame. This invention fully utilizes existing tunnel monitoring facilities, minimizing hardware investment. It can automatically and in real-time monitor the occurrence of spilled object events in tunnels, achieving intelligent road detection of spilled object events in tunnel traffic.

Description

Technical Field

[0001] This invention belongs to the field of video image processing technology and relates to tunnel video monitoring. It is a method for detecting spilled objects by fusing differential video sequences and convolutional neural networks. Background Technology

[0002] In recent years, with the rapid development of my country's economy, the volume of urban roads has been insufficient to meet the ever-increasing traffic volume. Tunnels, as a supplement to urban transportation, play a vital role. However, tunnel surfaces are not as wide as urban roads, and vehicles inside typically cannot change lanes. If debris is left on the road surface, it can easily cause traffic accidents. Utilizing computer vision technology to detect debris on tunnel surfaces, replacing manual supervision with automated monitoring, and quickly locating the location of debris from massive amounts of video data to issue warnings, is of great significance for improving tunnel surface safety and the level of road information technology.

[0003] In real-world tunnel monitoring environments, interference from artificial lighting and vehicle headlights causes inconsistencies in the outlines and shadows of targets within the tunnel. Furthermore, the variety of debris samples makes it impossible for deep learning methods, which rely on prior models, to construct a unified model to detect all debris. Therefore, debris detection in the specific environment of a tunnel presents a significant challenge. Summary of the Invention

[0004] The problem this invention aims to solve is that with the large-scale construction of tunnels, the number of video surveillance cameras inside tunnels is enormous, and relying solely on manual labor is insufficient to promptly detect and clear road debris. Currently, some methods detect debris by learning features from debris models, while others use inter-frame difference methods. However, on the one hand, the types of debris appearing on tunnel surfaces are diverse, and due to artificial lighting and the illumination of passing vehicle lights, even pixels at the same location will exhibit different feature information at different times, making it difficult to encompass all debris through training alone. On the other hand, if traditional inter-frame difference methods cannot accurately detect debris in a short time, it will quickly blend into the background, making it impossible to detect debris that has already blended into the background. Furthermore, traditional inter-frame difference methods can only detect foreground objects; for objects in the foreground, thresholding or model training is still required for judgment, and without massive amounts of debris data, it is impossible to represent all types of debris.

[0005] The technical solution of the present invention is as follows: a method for detecting projectiles by integrating differential video sequences and convolutional neural networks. The method constructs a static background model of the scene, then constructs a differential video sequence based on Gaussian weighted background subtraction, extracts the foreground, extracts the stationary targets in the foreground, and combines it with a convolutional neural network model to analyze and exclude non-projectiles in the stationary targets in the foreground, thereby detecting projectiles in the current video frame.

[0006] Furthermore, the present invention includes the following steps:

[0007] Step 1: Static background model construction. Collect surveillance videos during periods of low traffic density and learn to obtain a static background model.

[0008] Step 2: Construction of differential video sequence. Obtain video frames of the monitoring video, perform Gaussian weighted background difference with the static background model frame by frame, extract the foreground of the difference, and obtain a differential video sequence with a continuous frame size of N.

[0009] Step 3: Extraction of stationary objects in the foreground. For the obtained differential video sequence, if the probability of the same pixel position in the differential video sequence being in the foreground exceeds the set threshold, it is determined to be a stationary target in the foreground; otherwise, it is a moving target. Then, morphological operations are performed on the foreground, and the edges of the stationary targets in the foreground are obtained through edge detection. The obtained edges are then subjected to dilation and erosion, and adjacent edges are merged. Finally, contour detection is used to obtain the coordinate information of the stationary target in the corresponding video frame of the monitoring video.

[0010] Step 4: Construct a convolutional neural network for object detection. Pre-train the network on the set non-spraying objects to obtain a convolutional neural network model for detecting and recognizing non-spraying objects. Input the current frame of the surveillance video into the convolutional neural network model to identify the non-spraying objects in the current frame.

[0011] Step 5: Debris Analysis and Early Warning. A debris warning signal sequence is set for the corresponding video frames of the monitoring video. The initial value of the sequence is 0. The candidate boxes of stationary targets extracted from the foreground and the candidate boxes of non-debris objects identified by the convolutional neural network are filtered by the intersection-overall ratio (IoU) to eliminate non-debris objects among the stationary targets. The remaining ones are judged as debris. If debris is detected in the current frame, the value of the debris warning signal sequence corresponding to the current frame is updated to 1, indicating that debris has appeared in the current frame. After completing the debris detection of a differential video sequence, the frequency of debris appearing in the debris warning signal sequence is counted. If it exceeds the set threshold, the debris is marked in the current frame and an early warning is reported.

[0012] Existing background subtraction algorithms process data based on absolute or average thresholds. However, tunnel scenes often have localized areas that are too bright or too dark, making existing subtraction methods unsuitable for effectively extracting the foreground. This invention addresses the impact of vehicle and lighting lights on tunnel areas by increasing the weight of local color information features. It employs a Gaussian-weighted background subtraction method to eliminate interference from shadows and flickering vehicle lights before performing background subtraction, thus extracting a clearly defined foreground.

[0013] The narrow road surface inside tunnels and the short time from the appearance of spilled materials to their landing make them undetectable by conventional frame difference methods. Furthermore, the constant movement or flashing of vehicle lights inside tunnels interferes with the color and other feature information of the spilled materials. This invention constructs a differential video sequence by learning a static background model. If the probability of the foreground appearing at the same pixel location exceeds a certain threshold in the differential video sequence, it is detected as a stationary target in the foreground, thus suppressing the movement or flashing of vehicle lights.

[0014] This invention addresses the problem that current deep learning methods struggle to encompass all features of spilled objects. It first effectively extracts static foregrounds based on the foreground frequency of differential video sequences, then uses a convolutional neural network to identify non-sprayed objects such as people, vehicles, and traffic cones. The overlap rate between these objects and the static targets in the foreground is calculated. By setting a spilled object warning signal sequence, the accuracy of detection is improved. If the frequency of spilled objects in the warning sequence exceeds a certain threshold, the spilled object is bounded in the current video frame, and the warning is simultaneously reported to staff for timely obstacle removal.

[0015] This invention makes full use of existing tunnel monitoring facilities, which can save hardware investment to the greatest extent. It can automatically monitor the occurrence of debris throwing events in the tunnel in real time, realizing intelligent road detection of debris throwing events in tunnel traffic. Attached Figure Description

[0016] Figure 1 This is a flowchart of the projectile detection process that integrates differential video sequences and convolutional neural networks according to the present invention.

[0017] Figure 2 This is the first embodiment of the invention applied to the detection of spilled material in a real tunnel scenario.

[0018] Figure 3 This is the second embodiment of the invention applied to the detection of spilled material in a real tunnel scenario. Detailed Implementation

[0019] This invention combines traditional frame difference methods with deep learning methods to significantly improve the accuracy of identifying debris events in tunnels. Considering the interference of artificial and vehicle lighting on the scene within tunnels, and the fact that deep learning methods using prior models to construct training sets are not suitable for detecting debris with diverse features, this invention learns a static background model and uses a Gaussian weighted background difference method based on local regions to extract the foreground. It constructs a difference video sequence, analyzes the frequency of foreground pixel occurrences to identify stationary objects in the foreground, and uses deep learning to identify non-debris objects such as people, vehicles, and cone-shaped structures. The overlap rate between these objects and stationary targets in the foreground is calculated using parameters such as the intersection-union ratio (IUU). Then, debris warning analysis is performed in a designated road surface area to identify debris events on the tunnel surface and upload debris warning signals.

[0020] The flowchart of this invention is shown below. Figure 1As shown.

[0021] The specific implementation steps for detecting tunnel debris are as follows:

[0022] Step 1: Static background model construction. This involves acquiring surveillance video during periods of low traffic density. Low traffic density is generally considered to be periods where the total number of vehicles displayed on each frame of the monitored road does not exceed 10 within a consecutive 10-second period. This invention does not directly use video with no vehicles to further improve the robustness of the background model construction. One frame is taken every 12 frames of the acquired surveillance video, for a total of N0 frames. The pixel values ​​at the same location in these frames are then averaged to obtain the pixel value for the corresponding location of the static background. The calculation formula is as follows:

[0023]

[0024] In the formula, N0 is assumed to be 300, and f i (x0, y0) represents the pixel value at position (x0, y0) in the i-th frame, and Background0(x0, y0) represents the pixel value at position (x0, y0) in the generated background model frame.

[0025] Step 2: Construction of the Differential Video Sequence. This invention constructs a differential video sequence with a continuous frame size of N, where N is set to 30, representing the frame rate of the high-definition video surveillance within one second, or one detection cycle. The static background model and the current frame of the acquired surveillance video are simultaneously subjected to Gaussian weighted binarization based on local color features, followed by differencing to reduce interference from artificial lighting and vehicle lights, thus obtaining the foreground. The Gaussian weighted binarization is as follows: Any pixel point (x0, y0) in the image is taken and extended outwards into a 3x3 pixel matrix H. This pixel point is set as the center point (0, 0) of the matrix, and the coordinates of the eight outwardly extended points are (-1, 1), (0, 1), (1, 1), (-1, 0), (1, 0), (-1, -1), (0, -1), (1, -1). Based on the principle that the closer to the center of the image patch, the greater its weight; and the farther away, the smaller its weight, this is applied to the Gaussian distribution formula:

[0026]

[0027] In the formula, σ represents the degree of dispersion of the normal distribution, which is taken as 1.5 here.

[0028] Based on the above calculations, a 3x3 weight matrix is ​​obtained. The normalized weight matrix is ​​denoted as M. The normalized weight matrix M is multiplied by the original pixel matrix H according to their corresponding positions to obtain the color threshold λ for this region. The entire grayscale image is traversed. If the value of the pixel at the current position is greater than the color threshold λ of the neighboring region, it is set to (255, 255, 255); otherwise, it is set to (0, 0, 0).

[0029] During the initialization of the constructed differential video sequence, 30 identical binarized images of the static background model, weighted by Gaussian based on local color features, are used. Then, each time a video frame is received, the static background model and the current video frame are simultaneously weighted by Gaussian based on local color features, and the resulting differential video sequence is updated cyclically from left to right.

[0030] f i，pro =Background bin -f i，bin

[0031] Background in the formula bin For a static background model, f is the image after Gaussian weighting of local color features. i，bin To monitor the image of the current i-th frame of the video, which is Gaussian weighted based on local color features, f i，pro This is used to monitor the foreground image after Gaussian weighted difference based on local color features in the current i-th frame of the video and the static background model, and is also used to cyclically update the foreground image in the corresponding difference video sequence from left to right.

[0032] Step 3: Extraction of stationary foreground objects. For the differential video sequence corresponding to the current frame after the update, if the probability of the foreground appearing at the same pixel position exceeds a certain threshold rate in the differential video sequence at this time, it is retained as the foreground; otherwise, it is the background. The reason for setting the rate is to reduce the interference of vehicle lights.

[0033] First, the differential video sequences are normalized one by one and accumulated to obtain the foreground image to be processed. If the value of a pixel at a certain position in the foreground image is greater than the sequence size multiplied by rat, it is updated as the foreground; otherwise, it is updated as the background. At this point, the stationary target in the foreground is obtained.

[0034]

[0035] In the formula, frame i，j To monitor the pixel value at position (i, j) of the current frame of the video, N is the size of the differential video sequence, and rate is the threshold for the probability of foreground appearance.

[0036] Then, the foreground is first filtered, eroded, and dilated to filter out discrete small pixels. Then, the edges of the stationary targets in the foreground are obtained through edge detection. These contours are dilated and eroded to merge adjacent edges to avoid the same stationary target pixel from being discrete due to insufficient lighting. Finally, the coordinate information of the stationary targets in the foreground of the real-time frame is obtained through contour detection.

[0037] Step 4: Convolutional Neural Network Object Detection. The dataset includes categories such as pedestrians, cars, and traffic cones—non-scattered objects temporarily appearing in the tunnel. Due to the extreme imbalance of the samples in the dataset (for example, the number of traffic cones is significantly less than that of cars), the gradient is dominated by the larger category during network training, resulting in poor learning of traffic cone features. Therefore, data augmentation is performed on the smaller categories using methods such as flipping, cropping, and color transformation, while simultaneously improving the class balance loss function.

[0038]

[0039] In the formula, Loss cls The loss is the class balance loss, where j is the sample index and N is the number of samples. m N represents the number of class m in the sample set. pos N represents the number of positive samples among all samples. neg G(N) represents the number of negative samples among all samples. m ) represents the weight of class m in the loss function. Let m be the class loss for the j-th sample corresponding to class m.

[0040] Next, the real-time frames of the tunnel monitoring video are input into a pre-trained convolutional neural network to identify non-spraying objects such as people, vehicles, and traffic cones, and to obtain the coordinate information of the non-spraying objects.

[0041] Step 5: Debris Analysis and Early Warning. The tunnel surface area is manually delineated, and a debris warning sequence of size N is set to reduce interference from headlight reflections on the tunnel walls. The candidate bounding boxes of stationary targets extracted from the foreground are overlapped with candidate bounding boxes of non-debris objects identified by the convolutional neural network, based on information such as the intersection-union ratio (IU). If the overlap between the two exceeds 50% of the foreground stationary contour candidate boxes or the trained non-debris object candidate boxes, or if the IU of the two exceeds 50%, then the stationary object in the foreground is identified as non-debris. Otherwise, the current frame is considered to have detected debris, and the value in the corresponding debris warning sequence for the current frame is set to 1.

[0042] Finally, the remaining stationary targets in the foreground are bounded and marked as spilled objects. The frequency of spilled objects appearing in the spilled object warning sequence is counted. If the frequency of spilled objects in the spilled object warning sequence corresponding to the current frame exceeds 80%, a warning is issued and reported to the staff. Since pedestrians or vehicles may stop briefly before moving on, this invention can exclude such targets that combine static and dynamic states by counting the frequency of spilled objects in the spilled object warning sequence, thus preventing false alarms.

[0043] Current methods for detecting spilled objects primarily rely on training models that include spilled objects to distinguish them from other objects. This approach is inaccurate, failing to accurately represent and detect new object categories. This invention eliminates the need to add spilled objects to the training model. Including spilled objects reduces the detection accuracy of existing categories and, without sufficient data, prevents the model from representing all spilled object categories. Furthermore, this invention avoids background updates. Since tunnel surveillance cameras are fixed, learning to construct a static background reduces computational load, achieving real-time detection performance. This invention utilizes a Gaussian weighted background difference method based on local features to extract better foreground targets, exhibiting robustness against interference from vehicle lights and shadows. Frequent flashing of vehicle lights within tunnels can cause inconsistencies in pixel representation at uniform locations; this invention addresses this through frequency analysis to enhance foreground representation accuracy. Finally, the addition of a spilled object warning sequence in the final spilled object discrimination enhances anti-interference capabilities, maintaining an alarm accuracy rate of over 98% even during peak tunnel periods. In addition, this invention also augments the imbalanced data and improves the loss function when training on non-projectile targets.

[0044] Figure 2 and Figure 3 This embodiment of the invention is applied to the detection of debris in a real tunnel scenario, which can eliminate interference from foreground elements such as vehicles that are not debris. In particular... Figure 3 It is evident that the method of the present invention, through the screening and probability discrimination of non-sprayed objects using a convolutional neural network, can exclude static traffic cones, pedestrians with a combination of static and dynamic elements, and moving vehicles in the video, thus avoiding false alarms.

Claims

1. A method for detecting projectiles that integrates differential video sequences and convolutional neural networks, characterized in that: A static background model of the scene is constructed, and then a difference video sequence is constructed using a Gaussian weighted background subtraction method to extract the foreground. Then, stationary targets in the foreground are extracted and combined with a convolutional neural network model to analyze and exclude non-projectile objects among the stationary targets in the foreground, and to detect projectile objects in the current video frame. This process includes the following steps: Step 1: Static background model construction: Collect surveillance videos of the road section to be detected and learn to obtain a static background model; Step 2: Construction of the differential video sequence. N video frames are acquired from the surveillance video. Each video frame and the static background model are subjected to Gaussian weighted binarization and then differentially analyzed to extract the foreground, resulting in a differential video sequence with N consecutive frames. Specifically, the static background model and the acquired current frame of the surveillance video are simultaneously subjected to Gaussian weighted binarization based on local color features. The Gaussian weighted binarization is as follows: The image pixel position (x0, y0) is taken and extended outwards into a 3x3 pixel matrix H. This pixel position is set as the center point (0, 0) of the matrix. The coordinates of the eight points extending outwards are (-1, 1), (0, 1), (1, 1), (-1, 0), (1, 0), (-1, -1), (0, -1), (1, -1). The closer to the center of the image patch, the greater its weight; the farther away, the smaller its weight. Substituting this into the Gaussian distribution formula... In the formula, σ represents the degree of dispersion of the normal distribution. Based on the above calculations, a 3x3 weight matrix is ​​obtained. The normalized weight matrix is ​​set as M. The normalized weight matrix M is multiplied with the pixel matrix H according to the corresponding positions to obtain the color threshold λ of the region. The entire image after grayscale is traversed. If the value of the pixel at the current position is greater than the color threshold λ of the neighboring region, it is set to (255,255,255); otherwise, it is set to (0,0,0). Step 3: Extraction of stationary objects in the foreground. For the obtained differential video sequence, if the probability of the same pixel position in the differential video sequence being in the foreground exceeds the set threshold, it is determined to be a stationary target in the foreground; otherwise, it is a moving target. Then, morphological operations are performed on the foreground, and the edges of the stationary targets in the foreground are obtained through edge detection. The obtained edges are then subjected to dilation and erosion, and adjacent edges are merged. Finally, contour detection is used to obtain the coordinate information of the stationary target in the corresponding video frame of the monitoring video. Step 4: Construct a convolutional neural network for object detection. Pre-train the network on the set non-spraying objects to obtain a convolutional neural network model for detecting and recognizing non-spraying objects. Input the current frame of the surveillance video into the convolutional neural network model to identify the non-spraying objects in the current frame. Step 5: Debris Analysis and Early Warning. A debris warning signal sequence is set for the corresponding video frames of the monitoring video. The initial value of the sequence is 0. The candidate boxes of stationary targets extracted from the foreground and the candidate boxes of non-debris objects identified by the convolutional neural network are filtered by the intersection-overall ratio (IoU) to eliminate non-debris objects among the stationary targets. The remaining ones are judged as debris. If debris is detected in the current frame, the value of the debris warning signal sequence corresponding to the current frame is updated to 1, indicating that debris has appeared in the current frame. After completing the debris detection of a differential video sequence, the frequency of debris appearing in the debris warning signal sequence is counted. If it exceeds the set threshold, the debris is marked in the current frame and an early warning is reported.

2. The method for detecting projectiles by fusing differential video sequences and convolutional neural networks according to claim 1, characterized in that: In Step 1, surveillance video with low traffic density is collected for the road section to be detected. The total number of vehicles displayed on each surveillance frame within 10 consecutive seconds does not exceed 10. One frame is taken every n frames of the collected surveillance video, for a total of N0 frames. Then, the pixel values ​​at the same position in these frames are accumulated and averaged according to their corresponding pixel positions, which is used as the pixel value of the corresponding position of the static background.

3. The method for detecting projectiles by fusing differential video sequences and convolutional neural networks according to claim 1, characterized in that: In Step 3, the differential video sequence is first normalized and accumulated to obtain the differential foreground image to be processed. If the value of a pixel at a certain position in the differential foreground image to be processed is greater than the sequence size multiplied by a set threshold rate, it is updated to a stationary target in the foreground; otherwise, it is updated to a moving target, thus obtaining the stationary target in the foreground.

4. The method for detecting projectiles by fusing differential video sequences and convolutional neural networks according to claim 1, characterized in that: Step 4: For pre-training on non-sprayed samples, perform data augmentation on the extremely imbalanced classes in the non-sprayed sample dataset, using the class-balanced loss function: In the formula, Loss cls The loss is the class balance loss, where j is the sample index and N is the number of samples. m N represents the number of class m in the sample set. pos N represents the number of positive samples among all samples. neg G(N) represents the number of negative samples among all samples. m ) represents the weight of class m in the loss function. Let m be the class loss for the j-th sample corresponding to class m.

5. A method for detecting projectiles by fusing differential video sequences and convolutional neural networks according to any one of claims 1-4, characterized in that: The differential video sequence is set to a size N of 30, which represents the frame rate of the video surveillance within one second.

Images