Road thrown object detection method based on foreground and background comparison and related equipment

Abstract

The invention belongs to the technical field of image processing, and discloses a road spilled object detection method based on foreground and background comparison and related equipment, and the method directly extracts a road background from a video stream of a road, avoids the problems of multiple modeling and high calculation amount caused by the use of complex background modeling methods such as a Gaussian mixture model, and improves the detection accuracy. And the detection efficiency is obviously improved. Meanwhile, the video stream frame and the background are differentiated and then converted into the binary image, and the vehicle area shielding processing is performed on the image by using the vehicle detection result, so that the false detection is effectively reduced, and the detection accuracy is improved. In addition, the throwing object is marked by calculating the center-of-mass coordinates of the connected domain, accurate positioning of the throwing object is achieved, and the detection precision and practicability are further enhanced. By adopting the detection method, not only is the detection efficiency improved, but also the detection precision is optimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a method for detecting road spillage based on foreground and background comparison and related equipment. Background Art

[0002] With the development of the economy, the passenger and freight transportation on highways has grown rapidly, and the attention to highway driving safety has also increased day by day. However, spillage has become an important cause of highway accidents. At present, the detection of spillage mainly relies on manual visual observation, with low efficiency and unsustainable. In order to improve the detection efficiency, video detection by highway cameras has been adopted in the prior art, achieving efficient all-weather detection. At present, relatively few studies on spillage detection have been conducted at home and abroad. However, the use of deep learning and machine learning methods is one of the more commonly used methods at present. The deep learning method uses a large number of pre-planned datasets for training and can achieve excellent detection results in the corresponding test sets.

[0003] For example, the patent document with the Chinese patent application number CN202110100793.4 discloses "A Method for Detecting Road Surface Spillage on Highways Based on Surveillance Video" which proposes a machine learning solution: first, different pictures of highways collected are processed to make a dataset; then, a segmentation model is used to segment the highway dataset images to obtain highway area images; a feature image is obtained through the ORB corner detection method and a classification model is obtained; finally, the model is trained, and the trained model is used to detect highway videos to identify spillage. Although this method can detect spillage on the road surface, if some spillage on the road surface is not included in the training set, this spillage cannot be identified by this model.

[0004] In view of the above problems, scholars have proposed a method for detecting road litter based on foreground and background contrast to achieve the detection of litter on the road surface. For another example, the patent document with the application number CN202210251759.1 discloses a "Method for Detecting Road Litter in Highway Surveillance Videos Based on a Hybrid Background Model". Although this method can detect road litter in highway surveillance videos, the hybrid Gaussian model is used for background modeling. This model has a large amount of computation during background modeling. If there are always moving objects in the video, continuous computation is required, which is time-consuming. The patent document with the application number CN202210262727.1 discloses a "Method for Detecting Road Litter Based on Dual Background". First, it performs background modeling on the highway surveillance video to obtain a background frame video. Then, it subtracts the historical background frame from the current background frame in the background frame video to obtain a difference image between the current background frame and the historical background frame. Next, it performs mathematical morphological operations on the difference image to obtain moving objects in the background frame video. Finally, it removes moving objects with too small or too large areas, too far from the camera, and not on the road surface from the obtained moving objects, determines the remaining moving objects as litter, and marks them in the video. This method also uses the hybrid Gaussian model for background modeling and has the above-mentioned drawbacks. Moreover, when judging litter, based on the fact that the image background changes before and after the litter falls, and at the same time, due to the operation of subtracting the historical background frame from the current background frame in the background frame video it uses to obtain the suspected litter area, two background modelings using the hybrid Gaussian model are required, so it is more time-consuming. And after obtaining the differential binary image, it cannot determine it as litter only based on the white pixel values, resulting in a decrease in accuracy.

[0005] It can be seen that the existing litter detection methods require multiple background modelings, have a large amount of computation, affect the detection efficiency and have poor detection accuracy. Summary of the Invention

[0006] The present invention provides a method and related device for detecting road litter based on foreground and background contrast to solve the technical problems of existing litter detection methods that require multiple background modelings, have a large amount of computation, affect the detection efficiency and have poor detection accuracy.

[0007] To achieve the above object, the present invention adopts the following technical solutions: A method for detecting road litter based on foreground and background contrast, including: Extracting and obtaining a road background based on the video stream of the road; Converting the difference image between each frame image of the video stream and the extracted road background into a binary image to obtain a video image set; Performing vehicle area masking processing on the video image set based on the vehicle detection results of each frame image of the video stream to obtain a masked image; Based on the connected components obtained from the recognized occluded image, calculate the centroid coordinates of the connected components in each frame of the image; Based on the centroid coordinates of the connected components, mark the road spillage.

[0008] Further, the extraction of the road background based on the video stream of the road includes: Starting from a certain frame S0 of the video stream, take the n frames after frame S0 as background extraction candidate frames, denoted as [S0, S1, …, Si, …, Sn] in sequence, where 0 ≤ i ≤ n; Mark the road area where the spillage to be detected in the image as a polygon, set all pixels outside the polygon area to 0, retain the original values of the pixels inside the area, and apply the marked polygon detection area to all frame images; Use the yolov5 model to detect the vehicles in the detection area of the background candidate frames, mark the detected vehicle targets with bounding boxes, and record the frame with the fewest bounding boxes as the S* frame; If the S* frame contains 0 target boxes, extract the image corresponding to the S* frame as the road background; If the S* frame contains N target boxes, save the coordinate information of these N target boxes to a preset first list A; Compare the image S of every preset number of frames 1 with the image corresponding to the S* frame. If the image S 1 has no vehicle information in the corresponding area of the first list A, assign the pixel values of this area to the S* frame image. When all areas in the list A have been assigned values, extract the assigned background to obtain the road background.

[0009] Further, the conversion of the difference image between each frame image of the video stream and the extracted road background into a binary image to obtain a video image set includes: Starting from the S* frame moment, grayscale each frame image S in the video stream after the S* frame and the extracted road background respectively to obtain the grayscale image of each frame in the video stream and the background grayscale image; Subtract each frame grayscale image in the video stream from the background grayscale image pixel by pixel to obtain the difference image of the foreground and background; perform binary processing on the difference image to obtain the video image set.

[0010] Further, the vehicle area occlusion processing of the video image set based on the vehicle detection results of each frame image of the video stream to obtain the occluded image includes: Use the yolov5 model to detect the vehicles in each frame image after the S* frame, and save the vehicle detection results to a preset third list C; According to the target box area information recorded in the third list C, all the pixel values of the corresponding areas in the video image set are assigned 0, and the masked image after the black pixel assignment process is obtained.

[0011] Further, calculating the centroid coordinates of the connected components in each frame of image based on the connected components recognized in the masked image includes: Performing an opening operation on the masked image to achieve denoising processing, thereby eliminating random noise interference and obtaining a denoised image; Using the 8-neighbor connection method to find the connected components of the white pixels in the denoised image, and calculating the centroid coordinates of each connected component. The specific calculation formula for the centroid coordinates of the connected component is as follows:

[0012]

[0013] In the formula, represents the abscissa of the white pixel points in the connected component, represents the ordinate of the white pixel points in the connected component, , n represents the total number of white pixel points in the connected component; represents the abscissa of the centroid of the connected component; represents the ordinate of the centroid of the connected component.

[0014] Further, marking the road spillage based on the centroid coordinates of the connected components includes: Judging based on the centroid coordinates of the connected components in each frame of image; If the distance between the centroid coordinates of the connected components in different frames of images is less than the preset number of pixels, it means that the same connected component appears repeatedly in these frames of images. When the number of repeated appearances exceeds the preset number of times, the corresponding connected component is marked as a road spillage.

[0015] Further, after marking the road spillage based on the centroid coordinates of the connected components, it further includes: Using a bounding rectangle to mark the connected component corresponding to the road spillage, and the bounding rectangle is displayed in the image or video and feedbacks road prompt information.

[0016] A road spillage detection system based on foreground and background contrast includes: A road background extraction module for extracting the road background based on the video stream of the road; A differential conversion module for converting the differential image of each frame of the video stream and the extracted road background into a binary image to obtain a video image set; A shielding processing module, configured to perform vehicle area shielding processing on a video image set based on the vehicle detection results of each frame of the video stream to obtain a shielded image; A coordinate calculation module, configured to calculate the centroid coordinates of the connected regions in each frame of the image based on the connected regions identified from the shielded image; A road spillage marking module, configured to mark road spillages based on the centroid coordinates of the connected regions.

[0017] A device, comprising: A memory, configured to store a computer program; A processor, configured to implement the steps of the above-mentioned road spillage detection method based on foreground and background contrast when executing the computer program.

[0018] A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, is used to implement the steps of the above-mentioned road spillage detection method based on foreground and background contrast.

[0019] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a road spillage detection method based on foreground and background contrast. This method directly extracts the road background from the road video stream, avoiding the problems of multiple modeling and high computational complexity caused by complex background modeling methods such as the Gaussian mixture model, and significantly improving the detection efficiency. At the same time, after differentiating the video stream frames from the background and converting them into binary images, the vehicle area shielding processing is performed on the images using the vehicle detection results, effectively reducing false detections and improving the detection accuracy. In addition, by calculating the centroid coordinates of the connected regions to mark the spillages, the precise positioning of the spillages is achieved, further enhancing the detection accuracy and practicality. The adoption of this detection method not only improves the detection efficiency but also optimizes the detection accuracy.

[0020] In the present invention, preferably, extracting the road background from the road video stream includes steps such as selecting background extraction candidate frames, marking polygon detection regions, detecting vehicles using the yolov5 model and determining the S* frames, and comparing and assigning values, which can accurately extract the road background without vehicles, providing a reliable basis for subsequent differential processing.

[0021] In the present invention, preferably, the process of differentiating each frame of the image in the video stream from the extracted road background and converting it into a binary image is clarified. Through grayscale conversion, differentiation, and binary processing, the foreground objects (such as spillages) can be highlighted, providing clear image data for subsequent vehicle area shielding and connected region identification.

[0022] In the present invention, preferably, the shielding processing method detects vehicles by using the yolov5 model and sets the pixel values of the detected result area to 0, effectively reducing false detections caused by vehicle movement and improving the accuracy of spill detection.

[0023] In the present invention, preferably, by performing denoising processing, finding connected regions using the 8-neighbor method, and calculating the centroid coordinates, the precise positioning of the spill can be achieved, providing a basis for subsequent marking of the spill.

[0024] In the present invention, preferably, by judging the distance and the number of repeated occurrences of the centroid coordinates of the connected regions in different frame images, the spill can be accurately identified, avoiding misjudgment and missed judgment and improving the detection accuracy.

[0025] In the present invention, preferably, after marking the road spill, an external rectangle frame is used for marking and displayed in the image or video, and at the same time, road prompt information is fed back. This step not only makes the detection result more intuitive and understandable, but also can timely remind relevant personnel to pay attention to and handle the spill, improving the safety and traffic efficiency of the road. Description of the Drawings

[0026] Figure 1 is a flowchart of a method for detecting road spills based on foreground and background contrast provided by an embodiment of the present invention; Figure 2 is the first simulation experiment result provided by an embodiment of the present invention; among them, (a) is the calibrated background extraction area and the recognition result of vehicles by the yolov5 model within the area; (b) is the road background result extracted according to (a) Figure 3 is the second simulation experiment result provided by an embodiment of the present invention; (a) is the road background extracted during the test; (b) is the detected spill marking frame.

[0027] Figure 4 is a flowchart of a method for detecting road spills based on foreground and background contrast provided by the present invention; Figure 5 is a schematic structural diagram of a system for detecting road spills based on foreground and background contrast provided by the present invention. Detailed Embodiments

[0028] To further understand the content of the present invention, the following describes the present invention in detail with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are only for explaining the present invention and not for limiting it.

[0029] Embodiment 1 This embodiment provides a method for detecting road spills based on foreground and background contrast, including: Extract the road background based on the road-based video stream extraction; Convert the difference image between each frame image of the video stream and the extracted road background into a binary image to obtain a video image set; Perform vehicle area masking processing on the video image set based on the vehicle detection results of each frame image of the video stream to obtain a masked image; Calculate the centroid coordinates of the connected components in each frame image based on the connected components identified in the masked image; Mark the road spillage based on the centroid coordinates of the connected components.

[0030] The following is a further explanatory description of this embodiment with reference to the accompanying drawings: As Figure 1 shown, this embodiment provides a method for detecting road spillage based on foreground and background contrast, and the specific steps are as follows: Step 1: Shoot the highway surface through a camera to obtain a video or a video stream; Step 2: Use the captured video stream to extract the road background: (2a) Starting from a certain frame S0 of the video stream, take the n frames after it as background extraction candidate frames, and record them in sequence as [S0, S1,…, Si,…,Sn], 0≤i≤n; (2b) Mark the road area where spillage needs to be detected in the image as a polygon, set all pixels outside the polygon area to 0, retain the original values of the pixels inside the area, and apply the marked polygon detection area to all frame images; (2c) Use the yolov5 model to detect vehicles in the detection area of the background candidate frames [S0, S1, S2,…, Sn], mark the detected vehicle targets with bounding rectangles, and record the frame with the fewest marked boxes as the S* frame, and then make a judgment on it: If S* contains 0 target boxes, then the S* frame is the extracted background frame, If the S* frame contains N target boxes, then save the coordinate information of these N target boxes to a preset first list A, denoted as A={Box1, Box2,…, Boxi…, BoxN}, 1≤i≤N, and execute step (2d); (2d) Compare the image S 1 with the S* frame image every 7 frames: If the image S 1 has no vehicle information in the area corresponding to Boxi, then assign the pixel values of this area to the S* frame image. When all areas in the list A are assigned, the background extraction is completed, denoted as the image S^; Otherwise, do not process; Step 3: Starting from the S* frame moment of the proposed background, grayscale each frame image S in the video stream after the S* frame and the extracted road background S^ respectively, subtract the grayscale image of each frame in the video stream from the background grayscale image pixel by pixel to obtain the differential image of the foreground and background, and then binarize these differential images to obtain the video image set DS; Step 4: Perform preprocessing of blackening, denoising, and region finding on the video image set DS: (4a) Use the existing yolov5 model to detect vehicles for each frame image S after the S* frame, save the detection results to the preset third list C, and assign all pixel values of the corresponding regions in the video image set DS to 0, that is, black pixels, according to the target box region information recorded in the third list C, to obtain the image after the black pixel assignment process; (4b) Perform denoising processing on the image after the blackening process by performing an opening operation to eliminate random noise interference and obtain the denoised image; (4c) Use the 8-neighbor method to find the connected regions of the white pixels in the denoised image, calculate the centroid coordinates of each connected region, and save the connected region coordinate information to the preset fourth list D; Step 5: Perform the same processing as in steps (3) to (4) on each frame image after the S* frame to obtain the connected region information in each frame image, and then judge the centroid coordinates of the connected regions in each frame image: If the distance between the centroid coordinates of the connected regions in different frame images is less than 5 pixels, the same connected region appears repeatedly in these frame images. When the number of repeated appearances exceeds 6, mark this connected region as a road spill (road debris); Otherwise, do not process the connected region; Step 6: Draw the circumscribed rectangle of the connected region marked as a road spill, display it in the image or video, and give a spill prompt message.

[0031] As another preferred method of this embodiment, the more specific steps are as follows: S1. Obtain the video stream, calibrate the region, and extract the road background.

[0032] S1.1. Shoot the highway pavement through a camera to obtain a video or video stream.

[0033] S1.2. Starting from a certain frame S0 of the video stream, take the n frames after it as the background extraction candidate frames, denoted as [S0, S1,…, Si,…,Sn] in sequence, where 0≤i≤n; S1.3. Mark the road area where debris needs to be detected in the image as a polygon, set all pixels outside the polygon area to 0, retain the original values of the pixels inside the area, and apply the marked polygon detection area to all frame images; S1.4. Detect the vehicles within the detection area of the background candidate frames [S0, S1, S2, …, Sn] using the YOLOV5 model, mark the detected vehicle targets with bounding rectangles, record the frame with the fewest marked boxes as the S* frame, and then make a judgment on it: If the S* frame contains 0 target boxes, then the S* frame is the extracted background frame.

[0034] If the S* frame contains N target boxes, save the coordinate information of these N target boxes to the preset first list A, denoted as A = {Box1, Box2, …, Boxi…, BoxN}, where Boxi represents the coordinates of the i-th target box, 1 ≤ i ≤ N, and execute S1.5; S1.5. Extract the road background: S1.5.1. Record the vehicle information in every 7th image S 1 into the preset second list B, denoted as B = {box1, box2, …, boxj…, boxm}, where boxj represents the coordinates of the j-th target box, 1 ≤ j ≤ m; S1.5.2. Compare boxj in every 7th image S 1 with Boxi in the S* frame image: Take Boxi in the preset first list A and boxj in the second list B. Denote the minimum and maximum abscissas in the Boxi target box as [Boxi_xmin, Boxi_xmax], and the minimum and maximum abscissas in the boxj target box as [boxj_xmin, boxj_xmax]: If Boxi_xmin ≥ boxj_xmax and Boxi_xmax ≤ boxj_xmin, then the two rectangles do not overlap, indicating that there is no vehicle information in the area corresponding to Boxi in image S 1 and execute step S1.5.4; Otherwise, execute step S1.5.3; S1.5.3. Judge the ordinates in the target boxes: Take Boxi in the first list A and boxj in B. Denote the minimum and maximum ordinates in the Box i target box as [Boxi_ymin, Boxi_ymax], and the minimum and maximum ordinates in the boxj target box as [boxj_ymin, boxj_ymax]: If Boxi_ymin ≥ boxj_ymax and Boxi_ymax ≤ boxj_ymin, then the two rectangles do not overlap, indicating that image S 1When there is no vehicle information in the corresponding area of Boxi, step S1.5.4 is executed; S1.5.4. Assign the pixel values of the area without vehicles to the S* frame image. When all areas in list A have been assigned, the background extraction is completed and denoted as image S^.

[0035] S2. Perform differential and binarization operations on the background S^ and each frame image of the video.

[0036] S2.1. Starting from the moment of the S* frame for background extraction, grayscale each frame image S in the video stream after the S* frame and the extracted road background S^ respectively; S2.2. Subtract the two grayscale images pixel by pixel to obtain the differential image of the foreground and background; S2.3. Select the threshold t as 127 and perform binarization processing on the differential image to obtain the video image set DS. Judge the magnitude relationship between the absolute value of each pixel in the differential image and the threshold t: If the absolute value of the pixel is greater than the threshold t, assign the pixel value 255, that is, white; Otherwise, assign the pixel value 0, that is, black.

[0037] S3. Eliminate the interference of vehicles in the area to be detected.

[0038] S3.1. Use the existing yolov5 model to detect vehicles for each frame image S after the S* frame and save the detection results to the preset third list C; S3.2. According to the target box area information recorded in list C, assign all pixel values in the corresponding area of the video image set DS to 0, that is, black pixels, to obtain the image after the black pixel assignment process.

[0039] S4. Denoise the video image set DS.

[0040] S4.1. Perform erosion operation on the binarized image, that is, select a rectangular structuring element with a size of 5*5, remove the noise points smaller than the structuring element in the image, and make the object edges in the image shrink inward; S4.2. Apply dilation operation to the eroded image to restore the size of the object.

[0041] S5. Mark the white pixel connected regions in the denoised image and calculate the centroid coordinates.

[0042] S5.1. Search for connected components of white pixels in the differential binary image DS processed in S4 according to the 8-neighbor method, calculate the centroid coordinates of each connected component, and save the connected component information to the list D. If there are white pixels in one or more of the eight directions above, below, to the left, to the right, and the four diagonals of a white pixel point, mark these white pixel points as the same connected component and give the number of this connected component; S5.2. Read the coordinates of each white pixel point within the connected component according to the number of the connected component , , ; S5.3. Calculate the abscissa and ordinate of the centroid of the connected component:

[0043]

[0044] where represents the abscissa of the white pixel points within the connected component, represents the ordinate of the white pixel points within the connected component, , and n represents the total number of white pixel points within this connected component.

[0045] S6. Compare the centroid coordinates of the connected components in different frame images.

[0046] S6.1. Perform the same processing as S2 - S5 on each frame image after the S* frame to obtain the connected component information in each frame image; S6.2. Judge the centroid coordinates of the connected components in each frame image to determine the centroid coordinates: If the distance between the centroid coordinates of the connected components in different frame images is less than 5 pixels, the same connected component appears repeatedly in these frame images. When the number of repeated appearances exceeds 6, the coordinates of this connected component are the centroid, and execute S7; Otherwise, do not process the connected component.

[0047] S7. Mark the spill area.

[0048] S7.1. Take the centroid coordinates obtained in S6 as the position of the spill in the image; S7.2. Draw the circumscribed rectangle of the connected component marked as the road spill according to the centroid coordinates to display in the image or video, and give the spill prompt information.

[0049] It can be seen that this embodiment provides a method for detecting road spillage based on foreground and background contrast. Compared with the existing detection methods, it has the following advantages: First, since this embodiment detects spillage according to the method of foreground and background contrast of the road surface, it overcomes the problem of time-consuming background modeling in the prior art, making this method not only speed up the extraction speed of the road background, reduce the number of modeling times, but also be more simple and convenient to use.

[0050] Second, this embodiment determines the spillage by whether the centroid coordinates of the white pixel connected domain in the binary image move, improving the detection accuracy.

[0051] For the method for detecting road spillage based on foreground and background contrast provided by this embodiment, a simulation implementation is carried out as follows: In the simulation experiment, the software used is pycharm software, which runs the simulation in the environment of Python = 3.8. The video images are high-speed road videos downloaded from public websites and videos taken on community roads.

[0052] As Figure 2 shown, specifically as Figure 2 in (a) and (b), it can be seen that the road background extracted by this method is relatively clean.

[0053] As Figure 3 shown, specifically as Figure 3 in (a) and (b), it can be seen that the road spillage can be correctly identified.

[0054] Embodiment 2 As Figure 4 shown, this embodiment provides a method for detecting road spillage based on foreground and background contrast. This method is used for the management of streets, roads and highways. The situations to avoid endangering the normal operation of vehicles include the following steps: Extract the road background based on the video stream of the road; Convert the difference image between each frame image of the video stream and the extracted road background into a binary image to obtain a video image set; Perform vehicle area masking processing on the video image set based on the vehicle detection results of each frame image of the video stream to obtain a masked image; Based on the connected domains identified from the masked image, calculate the centroid coordinates of the connected domains in each frame image; Mark the road spillage based on the centroid coordinates of the connected domains.

[0055] In this embodiment, the extraction of the road background based on the video stream of the road includes: Starting from a certain frame S0 of the video stream, the n frames after frame S0 are used as background extraction candidate frames, denoted as [S0, S1, …, Si, …, Sn] in sequence, where 0 ≤ i ≤ n; Mark the road area where the spill to be detected in the image as a polygon, set all pixels outside the polygon area to 0, retain the original values of the pixels inside the area, and apply the marked polygon detection area to all frame images; Use the yolov5 model to detect vehicles in the detection area of the background candidate frames, mark the detected vehicle targets with bounding boxes, and record the frame with the fewest bounding boxes as the S* frame; If the S* frame contains 0 target boxes, extract the image corresponding to the S* frame as the road background; If the S* frame contains N target boxes, save the coordinate information of these N target boxes to a preset first list A; For the image S at every preset number of frames 1 Compare it with the image corresponding to the S* frame. If the image S 1 Has no vehicle information in the corresponding area of the first list A, assign the pixel values of this area to the S* frame image. After all areas in list A are assigned, extract the background after assignment to obtain the road background.

[0056] Specifically, converting the difference image between each frame image of the video stream and the extracted road background into a binary image to obtain a video image set includes: Starting from the S* frame moment, grayscale each frame image S in the video stream after the S* frame and the extracted road background respectively to obtain the grayscale image of each frame in the video stream and the background grayscale image; Subtract the grayscale image of each frame in the video stream from the background grayscale image pixel by pixel to obtain the difference image between the foreground and background; perform binary processing on the difference image to obtain a video image set.

[0057] Here, performing vehicle area occlusion processing on the video image set based on the vehicle detection results of each frame image of the video stream to obtain an occluded image includes: Use the yolov5 model to detect vehicles in each frame image after the S* frame, and save the vehicle detection results to a preset third list C; According to the target box area information recorded in the third list C, assign all pixel values of the corresponding area in the video image set to 0 to obtain an occluded image after processing with black pixels assigned.

[0058] Here, calculating the centroid coordinates of the connected regions in each frame image based on the recognized occluded image includes: Perform opening operation on the occluded image to achieve denoising processing, thereby eliminating random noise interference to obtain a denoised image; Use the 8-neighbor connection method to find the connected components of the white pixels in the denoised image, and calculate the centroid coordinates of each connected component. The specific formula for calculating the centroid coordinates of the connected component is as follows:

[0059]

[0060] In the formula, represents the abscissa of the white pixel points in the connected component, represents the ordinate of the white pixel points in the connected component, , n represents the total number of white pixel points in the connected component; represents the abscissa of the centroid of the connected component; represents the ordinate of the centroid of the connected component.

[0061] In this embodiment, marking the road spillage based on the centroid coordinates of the connected component includes: Judging based on the centroid coordinates of the connected components in each frame of the image; If the distance between the centroid coordinates of the connected components between different frames of images is less than a preset number of pixels, it means that the same connected component appears repeatedly in these frames of images. When the number of repeated appearances exceeds the preset number of times, mark the corresponding connected component as a road spillage.

[0062] This detection method further includes: using an external rectangular frame to mark the connected component corresponding to the road spillage, and the external rectangular frame is displayed in the image or video and feedbacks road prompt information.

[0063] As Figure 5 shown, this embodiment also provides a road spillage detection system based on foreground and background contrast, including: a road background extraction module for extracting the road background based on the video stream of the road; a differential conversion module for converting the differential image between each frame of the video stream and the extracted road background into a binary image to obtain a video image set; a masking processing module for performing vehicle area masking processing on the video image set based on the vehicle detection results of each frame of the video stream to obtain a masked image; a coordinate calculation module for calculating the centroid coordinates of the connected components in each frame of the image based on the connected components identified in the masked image; a spillage marking module for marking the road spillage based on the centroid coordinates of the connected components.

[0064] The present invention also provides a device, including: a memory for storing a computer program; a processor for implementing the steps of the road spillage detection method based on foreground and background contrast when executing the computer program.

[0065] When the processor executes the computer program, it implements the steps of the above-mentioned road spill detection based on foreground and background comparison, for example: extracting the road background based on the video stream of the road; converting the difference image between each frame image of the video stream and the extracted road background into a binary image to obtain a video image set; performing vehicle area masking processing on the video image set based on the vehicle detection result of each frame image of the video stream to obtain a masked image; calculating the centroid coordinates of the connected components in each frame image based on the connected components identified in the masked image; and marking the road spills based on the centroid coordinates of the connected components.

[0066] Alternatively, when the processor executes the computer program, it implements the functions of each module in the above system, for example: a road background extraction module for extracting the road background based on the video stream of the road; a difference conversion module for converting the difference image between each frame image of the video stream and the extracted road background into a binary image to obtain a video image set; a masking processing module for performing vehicle area masking processing on the video image set based on the vehicle detection result of each frame image of the video stream to obtain a masked image; a coordinate calculation module for calculating the centroid coordinates of the connected components in each frame image based on the connected components identified in the masked image; and a spill marking module for marking the road spills based on the centroid coordinates of the connected components.

[0067] Exemplarily, the computer program can be divided into one or more modules / units. The one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of completing preset functions, and the instruction segments are used to describe the execution process of the computer program in the road spill detection device based on foreground and background comparison. For example, the computer program can be divided into a road background extraction module, a difference conversion module, a masking processing module, a coordinate calculation module, and a spill marking module; the specific functions of each module are as follows: the road background extraction module for extracting the road background based on the video stream of the road; the difference conversion module for converting the difference image between each frame image of the video stream and the extracted road background into a binary image to obtain a video image set; the masking processing module for performing vehicle area masking processing on the video image set based on the vehicle detection result of each frame image of the video stream to obtain a masked image; the coordinate calculation module for calculating the centroid coordinates of the connected components in each frame image based on the connected components identified in the masked image; and the spill marking module for marking the road spills based on the centroid coordinates of the connected components.

[0068] The road spillage detection device based on foreground and background contrast can be computing devices such as desktop computers, notebooks, palmtop computers, and cloud servers. The road spillage detection device based on foreground and background contrast may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the above are examples of the road spillage detection device based on foreground and background contrast, which do not constitute a limitation on the road spillage detection device based on foreground and background contrast. It may include more components than the above, or combine certain components, or different components. For example, the road spillage detection device based on foreground and background contrast may also include input / output devices, network access devices, buses, etc.

[0069] The so-called processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor is the control center of the road spillage detection based on foreground and background contrast, and connects various parts of the entire road spillage detection device based on foreground and background contrast through various interfaces and lines.

[0070] The memory can be used to store the computer programs and / or modules. The processor realizes various functions of the road spillage detection device based on foreground and background contrast by running or executing the computer programs and / or modules stored in the memory, and by calling the data stored in the memory.

[0071] The memory may mainly include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, phone book, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disks, memory, plug-in hard disks, smart media cards (SMCs), secure digital (SD) cards, flash cards, at least one magnetic disk storage device, flash device, or other volatile solid-state storage devices.

[0072] The present invention also provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the method for detecting road spillage based on foreground and background contrast as described above.

[0073] If the modules / units integrated in the road spillage detection system based on foreground and background contrast are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0074] Based on such understanding, all or part of the processes in the method for detecting road spillage based on foreground and background contrast of the present invention can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the method for detecting road spillage based on foreground and background contrast as described above can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or preset intermediate form, etc.

[0075] The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0076] It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0077] The above embodiments are only one of the implementation manners capable of implementing the technical solution of the present invention. The scope of protection required by the present invention is not limited only by this embodiment, but also includes any changes, substitutions and other implementation manners that are easily conceivable by those skilled in the art within the technical scope disclosed by the present invention. In summary, the present invention provides a method for detecting road spillage based on foreground and background contrast and related devices, which has the following advantages compared with the existing spillage detection measures: It mainly solves the problems of the prior art, such as the large number of background modeling times, long time consumption, and low detection accuracy. This method obtains a video or video stream by shooting a highway surface; extracts the road background from the video stream, and performs differential processing on each frame image of the video stream and the extracted background; then converts the differential image into a binary image; uses the existing yolov5 model to detect the video image, assigns black pixels to the detected vehicle target boxes in the binary image, and then performs denoising processing to eliminate random noise interference; marks the connected regions of white pixels in the binary image by the 8-neighbor method, calculates the centroid coordinates of the connected region, determines the coordinates that appear more than the preset number of times as spillage, and frames them in the video image. Using this detection method reduces the number of background modeling times, reduces the background modeling time, improves the detection accuracy, and can be used for road management.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: still can modify the specific implementation manners of the present invention or make equivalent replacements, and any modification or equivalent replacement without departing from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. A method for detecting road spilled objects based on foreground and background comparison, characterized in that: include: Obtain road background based on road video stream extraction; The difference image between each frame image of the video stream and the extracted road background is converted into a binary image to obtain a video image set; Based on the vehicle detection result of each frame of the video stream, the video image set is subjected to vehicle region masking processing to obtain a masked image; Based on the connected domain obtained by identifying the masked image, the centroid coordinates of the connected domain in each frame of the image are calculated; Road spills are marked based on the centroid coordinates of the connected domain.

2. The method for detecting road spilled objects based on foreground and background comparison according to claim 1, characterized in that: The road background is obtained by extracting the road-based video stream, including: Starting from a certain frame S0 of the video stream, the n frames after frame S0 are taken as candidate frames for background extraction, which are recorded as [S0, S1, …, Si, …, Sn], 0≤i≤n; The road area where the spilled objects are to be detected in the image is marked as a polygon, all pixels outside the polygon area are set to 0, and the pixels inside the area retain their original values, and the marked polygon detection area is applied to all frame images; The yolov5 model is used to detect vehicles in the detection area of ​​the background candidate frame, and the detected vehicle targets are marked with a marking frame, and the frame with the least marking frame is recorded as the S* frame; If the S* frame contains 0 target frames, the image corresponding to the S* frame is extracted as the road background; If the S* frame contains N target frames, the coordinate information of the N target frames is saved in a preset first list A; The image S is set every preset number of frames. 1 Compare with the image corresponding to the S* frame. If the image S 1 If there is no vehicle information in the corresponding area of ​​the first list A, the pixel value of the area is assigned to the S* frame image. When all areas in list A are assigned, the assigned background is extracted to obtain the road background.

3. The method for detecting road spilled objects based on foreground and background comparison according to claim 1, characterized in that: The step of converting the difference image between each frame of the video stream and the extracted road background into a binary image to obtain a video image set includes: Starting from the S* frame time, each frame image S in the video stream after the S* frame and the extracted road background are grayed out to obtain a gray image of each frame in the video stream and a background gray image; Subtract the grayscale image of each frame in the video stream from the background grayscale image pixel by pixel to obtain the difference image between the foreground and the background; binarize the difference image to obtain a video image set.

4. The method for detecting road spilled objects based on foreground and background comparison according to claim 1, characterized in that: The vehicle region masking process is performed on the video image set based on the vehicle detection result of each frame of the video stream to obtain the masked image, including: Use the yolov5 model to detect vehicles in each frame after the S* frame, and save the vehicle detection results into the preset third table C; According to the target frame area information recorded in the third table C, all pixel values ​​of the corresponding area in the video image set are assigned to 0, so as to obtain a masked image after being assigned black pixels.

5. The method for detecting road spilled objects based on foreground and background comparison according to claim 1, characterized in that: The method of calculating the centroid coordinates of the connected domain in each frame of the image based on identifying the connected domain obtained from the masked image includes: Perform an opening operation on the masked image to achieve denoising, thereby eliminating random noise interference and obtaining a denoised image; The 8-neighbor method is used to find the connected domains for the white pixels in the denoised image, and the centroid coordinates of each connected domain are calculated. The specific calculation formula for the centroid coordinates of the connected domain is as follows: In the formula, Represents the horizontal coordinate of the white pixel in the connected domain, Represents the ordinate of the white pixel in the connected domain, , n represents the total number of white pixels in the connected domain; Represents the horizontal coordinate of the centroid of the connected domain; Represents the centroid ordinate of the connected domain.

6. The method for detecting road spilled objects based on foreground and background comparison according to claim 1, characterized in that: The method of marking the road spilled objects based on the centroid coordinates of the connected domain includes: The judgment is made based on the coordinates of the centroid of the connected domain in each frame of the image; If the distance between the centroid coordinates of the connected domains between different frame images is less than a preset number of pixels, it means that the same connected domain appears repeatedly in these frame images. When the number of repetitions exceeds the preset number, the corresponding connected domain is marked as road spills.

7. The method for detecting road spilled objects based on foreground and background comparison according to claim 1, characterized in that: After marking the road spilled objects based on the centroid coordinates of the connected domain, the method further includes: The connected domain corresponding to the road spilled objects is marked with an external rectangular frame, which is displayed in the image or video and feeds back road prompt information.

8. A road spill detection system based on foreground and background comparison, characterized in that: include: A road background extraction module is used to obtain the road background based on the road video stream extraction; A differential conversion module is used to convert the differential image between each frame of the video stream and the extracted road background into a binary image to obtain a video image set; The mask processing module is used to perform vehicle area masking processing on the video image set based on the vehicle detection result of each frame image of the video stream to obtain a masked image; A coordinate calculation module, used to calculate the centroid coordinates of the connected domain in each frame of the image based on the connected domain obtained by identifying the masked image; The spilled object marking module is used to mark road spilled objects based on the centroid coordinates of the connected domain.

9. A device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the road spilled object detection method based on foreground and background comparison as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the steps of the road spilled object detection method based on foreground and background comparison according to any one of claims 1 to 7.

Images