A road litter identification method based on frame difference method and semantic segmentation combination

Abstract

The application discloses a road litter identification method based on frame difference method and semantic segmentation combination, comprising: acquiring target video stream data of a monitoring position; setting a background frame window, setting a pointer to read a video stream based on the set background frame window size; calculating the background of the current moment after reading the image data of one background frame window; calculating the background difference between the background of the next moment and the background of the current moment based on the background of the adjacent moment; then obtaining the background frame difference graph; calculating the proportion of black pixel points in the obtained background frame difference graph to determine whether it is a suspected litter; and performing semantic segmentation on the background frame difference graph with suspected litter to determine whether it is litter falling on the target road. The application realizes the identification of road litter by combining frame difference method and semantic segmentation, reduces the influence of environmental factors, and improves the accuracy of identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of road spillage identification, and particularly relates to a road spillage identification method based on frame difference method and semantic segmentation. BACKGROUND

[0002] With the development of economy and highway construction, vehicles have become one of the main means of commercial transportation and people's travel. Once a traffic accident occurs on the road, it will cause significant casualties and property losses. The presence of spillage is currently one of the causes of road traffic accidents, and it is highly dangerous, especially on tunnels and highways, where there is a large amount of traffic and fast speed, and the influence of spillage is more obvious. Therefore, road safety has become the focus of attention. Spillage detection is an important part of target detection. Spillage detection through road monitoring images can reduce road safety hazards as much as possible and improve the safety of highway travel.

[0003] At present, in the field of highway spillage identification, most detection algorithms still use deep learning neural networks to perform deep learning on the scene, and then use the learning results to identify spillage in the actual scene. The idea of this solution is to convert the spillage identification task into target identification, and after a large amount of training of the deep learning network or algorithm, the experience and a series of parameters obtained in the learning are used to identify unfamiliar scenes. Common deep learning networks include resnet, and common target identification algorithms include yolo, R-CNN series, etc. The current technology has the following problems.

[0004] 1. There are many types of spillage, and it is difficult to effectively identify spillage using existing identification schemes of traditional target identification algorithms (such as yolo).

[0005] 2. Traditional deep learning networks are difficult to effectively identify spillage in different backgrounds and environments, such as tunnels, night and bad weather environments.

[0006] 3. Traditional target identification algorithms are easily affected by video noise. In actual application scenarios, the monitoring camera will not always be in the best working state, so there will be noise from time to time. This influence will cause great trouble to the traditional solution.

[0007] 4. When the learning results and system parameters of familiar environments are transferred to a variety of unfamiliar environments in the traditional algorithm, the correctness of the learning results will be discounted.

[0008] 5. Traffic accidents are sporadic events. When a traffic accident occurs, due to various objective limitations, the number of clear videos or pictures that can be collected is limited, and the clarity is difficult to guarantee. Existing recognition methods using deep learning often rely heavily on the diversity and reliability of data. When the dataset is insufficient or the data diversity is insufficient, it is often difficult to meet the indicator requirements. Summary of the Invention

[0009] To address the aforementioned issues, this invention provides a road debris identification method based on a combination of frame difference and semantic segmentation. This method employs a combined approach to identify debris on highways. First, video stream data is read frame by frame, and the background difference between adjacent frames is used for initial debris identification. Then, semantic segmentation is performed on the original frame image of suspected debris to achieve final classification and identification. The use of frame difference for initial debris identification overcomes the influence of environmental factors on target detection, while semantic segmentation of the original frame image of suspected targets ensures the accuracy of the final identification. Furthermore, the combination of these two methods enhances the algorithm's transferability.

[0010] This invention provides a method for identifying road debris based on a combination of frame difference and semantic segmentation. The specific technical solution is as follows:

[0011] Acquire target video stream data at the monitored location;

[0012] Set a background frame window, and set a pointer to read the video stream based on the set background frame window size;

[0013] For each background frame window of image data read, the image images of the previous x-1 frames and the current frame are taken as a window, and the background at the current moment is calculated using the median method.

[0014] Based on the backgrounds of adjacent time points, calculate the background difference between the background of the next time point and the background of the current time point;

[0015] Perform image processing operations on the background difference image to obtain the background frame difference map;

[0016] The proportion of black pixels in the calculated background frame difference image is used to determine that there is suspected litter residue on the road surface being monitored and detected when the proportion exceeds a set threshold.

[0017] Semantic segmentation is performed on the background frame difference map showing suspected spilled material to determine whether it is spilled material that has fallen on the target road.

[0018] Furthermore, the background at the current moment is calculated as follows:

[0019] During the background frame window reading process, the duration of color appearance in each region of each frame is recorded from the read background frame window;

[0020] The color with a duration exceeding a set time threshold is determined as the background color of the small area.

[0021] Further, the image processing is sequentially carried out on the background difference image, including grayscale processing, binaryzation processing and noise reduction processing.

[0022] Further, the semantic segmentation adopts the original frame picture of the current frame in which the suspected litter is detected.

[0023] Further, the semantic segmentation adopts the mmsegmentation model.

[0024] Further, the mmsegmentation model is trained by using the Highway Driving Dataset data set.

[0025] The beneficial effects of the present application are as follows:

[0026] Based on the set background frame window, the video stream data is read by a pointer, the video stream of the monitoring camera can be processed in real time, the background difference of adjacent frames is calculated, the background frame difference image is obtained through image processing, the pixel point proportion is calculated to realize the initial identification of the litter, the semantic segmentation is further used for identification, the object type is judged, and the situation that the object stays on the road surface is alarmed in real time, the frame difference method and the semantic segmentation are combined to realize the accurate identification of the road litter, and the influence of the identification difficulty caused by the environment is overcome. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 It is a method flowchart of the present application. DETAILED DESCRIPTION

[0028] In the following description, the technical solutions in the embodiments of the present application are clearly and completely described, obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the present application.

[0029] In the description of the embodiments of the present application, it should be noted that the indicated position or positional relationship is based on the position or positional relationship shown in the drawings, or the position or positional relationship commonly used when the product of the present application is used, or the position or positional relationship commonly understood by those skilled in the art, or the position or positional relationship commonly used when the product of the present application is used, only for the convenience of describing the present application and simplifying the description, and is not intended to indicate or imply that the indicated device or element must have a particular position, be constructed and operated in a particular position, and therefore cannot be understood as a limitation of the present application. In addition, the terms "first", "second" are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0030] In the description of the embodiments of the present application, it should be noted that, unless otherwise explicitly specified and limited, the terms "set", "connected" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be directly connected, or indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0031] Embodiment 1

[0032] Embodiment 1 of the present application discloses a road litter identification method based on frame difference method and semantic segmentation combination, as shown in Figure 1 First, a first identification of litter is performed by frame difference method;

[0033] Frame difference method is a method of obtaining the contour of moving target by difference operation on adjacent two frames in video image sequence, which can be well applied to the case of multiple moving targets and camera movement.

[0034] A video is composed of several frames, a frame is essentially a still picture, a series of still pictures played quickly can constitute the entire video, the principle is to use the current frame and the past several frames for comparison, when an abnormal object movement appears in the monitoring scene, there will be a more obvious difference between frames, two frames are subtracted, the absolute value of the brightness difference of the two frame images is obtained, it is judged whether it is greater than the threshold to analyze the motion characteristics of the video or image sequence, to determine whether there is object movement in the image sequence, for those areas whose color and texture characteristics have no change or little change in the past period of time, it can be determined that they are static background, and those areas with dramatic changes can be determined as moving objects. Since the monitoring camera on the highway is stationary, the background in the video is also stationary, so frame difference method can be used to identify moving objects; the litter will eventually fall on the highway in a stationary state, in the process of falling, the litter is a moving object, and after the litter stops moving, the frame difference method cannot detect this moving object, when a moving object disappears in the video and "integrates into the background" to escape the detection of the frame difference method, it can be considered that the object has stayed on the highway, at this time the use of the frame difference method has ended, and the next step of detection is entered.

[0035] In combination Figure 1 As shown in the figure, the specific identification process is as follows:

[0036] Obtain target video stream data of the monitoring position;

[0037] Specifically, video stream data of the highway camera at the position to be monitored and identified can be obtained.

[0038] Set a background frame window, and set a pointer to read the video stream based on the set background frame window size x;

[0039] In this embodiment, 60 frames (i.e., x is 60) are set as a background frame window to calculate the background, and a pointer R is set to read the obtained video stream data.

[0040] After reading the image data of one background frame window, the image pictures of the previous x-1 frames and the current frame are taken as a window, and the background at the current time is calculated by using the median method.

[0041] Based on the background of the adjacent time, the background difference between the next time background and the current time background is calculated.

[0042] In this embodiment, the specific process is as follows:

[0043] R starts to read the frames in the video stream and moves forward, and R takes the image pictures of the previous x-1 frames and the current frame as a window every time x frames are read, and uses the median method to calculate the background B_t at the current time t.

[0044] In the next moment R moves forward a frame, and the current frame and the previous x-1 frames are calculated using the median method to obtain the background B_(t+1) at the current t+1 moment;

[0045] In this embodiment, the background at the current moment is calculated as follows:

[0046] In the background frame window reading process, the duration of each region appearing color in each frame is recorded from the read background frame window;

[0047] The color with a duration exceeding a set time threshold is determined as the background color of the small region.

[0048] After obtaining the backgrounds at the adjacent two moments, the background difference, i.e., the frame difference, at the t and t+1 moments is calculated.

[0049] Based on the pointer, the frame images of the video stream are continuously read to realize real-time target detection of the monitoring video.

[0050] The background difference at the adjacent moments is calculated to obtain the background difference image, i.e., the frame difference image, and image processing is performed on the background difference image to obtain the background frame difference image at the current moment.

[0051] In this embodiment, the image processing includes grayscale processing, binarization processing, and noise reduction processing performed on the background difference image in sequence.

[0052] The proportion of black pixel points in the calculated background frame difference image is calculated, and when the proportion exceeds a set threshold, it is determined that the background images at the current frame and the previous frame have changed, i.e., a suspected spilling object is retained on the road surface and integrated into the background. If the threshold is not reached, it means that the road surface is normal and no abnormal event occurs.

[0053] The result of one-time recognition is subjected to semantic segmentation using a third-party semantic segmentation model and partial optimization. The semantic segmentation model can divide a picture of an expressway into multiple categories such as vehicles, pedestrians, road surfaces, backgrounds, and traffic signs, and can accurately classify each pixel point in the picture into a certain category. After it is found that an object is retained on the expressway in the previous step, semantic segmentation is performed on the frame to determine the specific object and whether the specific position is retained on the expressway road surface. When the spilling position is not on the expressway road surface (e.g., on the roadside), the algorithm can choose not to alarm; when the spilling position is on the road surface, the algorithm alarms and notifies the relevant departments to handle.

[0054] In combination with Figure 1 as shown in the specific implementation is as follows:

[0055] The background frame difference image with a suspected spilling object is subjected to semantic segmentation to determine whether it is a spilling object on the target road.

[0056] When a litter event occurs, the current time and the road section where the litter event occurs are reported to the control terminal, so that the road surface is cleaned.

[0057] In this embodiment, the semantic segmentation uses the original frame picture of the current frame in which the suspected litter retention is detected.

[0058] In this embodiment, the semantic segmentation uses the mmsegmentation model as a basic model for training and optimization.

[0059] Specifically, in the training of the mmsegmentation model, The Highway Driving Dataset dataset is used for training, which contains a large amount of highway related data.

[0060] The present application is not limited to the foregoing specific embodiments. The present application extends to any novel one, or any new combination, of the characteristics disclosed in this specification, as well as to any novel method or process steps disclosed, or any new combination thereof.

Claims

1. A road litter identification method based on a combination of frame difference method and semantic segmentation, characterized in that, The method comprises the following steps: acquiring target video stream data of a monitoring position; setting a background frame window, and setting a pointer to read the video stream based on the set background frame window size; after reading image data of one background frame window, taking the image pictures of the previous x-1 frames and the current frame as a window, calculating the background at the current time through the median method, and the calculation process is as follows: during the reading process based on the background frame window, recording the duration of the color appearing in each region in each frame from the read background frame window; judging the color with a duration exceeding a set time threshold as the background color of the small region; calculating the background difference between the background at the next time and the background at the current time based on the background at the adjacent time; performing image processing operation on the background difference image to obtain a background frame difference image; calculating the proportion of black pixel points in the obtained background frame difference image, and judging that there is a suspected spilling object retained on the road surface in the current monitoring detection when the proportion exceeds a set threshold; performing semantic segmentation on the background frame difference image where the suspected spilling object appears to determine whether it is a spilling object falling on the target road; the semantic segmentation adopts the original frame picture of the current frame where the suspected spilling object is detected; the semantic segmentation adopts an mmsegmentation model; and the mmsegmentation model is trained by using a The Highway Driving Dataset data set.

2. The road throw identification method according to claim 1, wherein the background at the current time, and the image processing is sequentially performing grayscale processing, binarization processing and noise reduction processing on the background difference image.

Images