A method for detecting road surface litter based on unknown class perception
By using a road debris detection method based on unknown category perception, an unknown category distribution is generated by using a target detection model and a semantic segmentation model. Combined with a probability density function and suppression operation, the debris is identified and filtered out, which solves the problem of limited detection types in existing technologies and achieves efficient multi-category debris detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 浙江省机电设计研究院有限公司
- Filing Date
- 2023-05-19
- Publication Date
- 2026-05-15
AI Technical Summary
Existing deep learning-based methods for detecting road debris cannot effectively handle the diverse types of debris with indistinct image features in real-world scenarios, resulting in low detection efficiency and limited practical value.
The target detection model and semantic segmentation model are trained using a traffic participant dataset to generate unknown category distributions and known category distributions. The perception parameters are fitted by a probability density function, and combined with the road surface semantic segmentation results, candidate boxes for suspected litter are selected and suppressed to identify unknown category targets as litter.
Without any training data on road debris, it can detect a large number of different types of road debris, meeting the detection needs of real-world scenarios and improving the accuracy and efficiency of detection.
Smart Images
Figure CN116597357B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of computer vision technology, and in particular relates to a method for detecting road debris based on unknown category perception. Background Technology
[0002] In recent years, with the continuous improvement of the highway network, highways have become an important support for the national economic cycle. Many road sections have a high volume of truck traffic, which can easily cause cargo to fall off during transit, resulting in road debris. Road debris can easily lead to traffic accidents and is one of the major hidden dangers threatening driving safety. Related operating companies attach great importance to this issue and invest a lot of manpower in patrols to promptly eliminate hazards; however, there are problems with inefficiency and unsatisfactory results, and the use of technological means is still needed.
[0003] Existing deep learning-based methods for detecting road debris suffer from limitations in the types of debris that can be detected. They cannot effectively handle the diverse types of debris with indistinct image features in real-world scenarios, which greatly limits the practical value of such methods. Summary of the Invention
[0004] The purpose of this application is to propose a method for detecting road debris based on unknown category perception, so as to overcome the technical problems mentioned in the background art.
[0005] To achieve the above objectives, the technical solution of this application is as follows:
[0006] A method for detecting road debris based on unknown category perception includes:
[0007] A target detection model and a semantic segmentation model are trained using a traffic participant dataset. The target detection model is used to detect foreground targets, and the semantic segmentation model is used to segment the road surface area.
[0008] The trained object detection model is used to detect candidate boxes for the unknown category dataset and the known category dataset respectively. Candidate boxes with an intersection-union ratio greater than the first threshold are retained. The perceptual parameters corresponding to the retained candidate boxes are calculated. The perceptual parameter set is fitted with a probability density function to generate the unknown category distribution and the known category distribution.
[0009] The image to be identified is input into the trained object detection model and semantic segmentation model to obtain a set of foreground candidate boxes and a road surface semantic segmentation result matrix, and the set of target candidate boxes is selected from the set of foreground candidate boxes.
[0010] By utilizing known and unknown category distributions, unknown category targets are perceived from the target candidate box set, resulting in a suspected debris candidate box set.
[0011] Furthermore, the road spill detection method based on unknown category perception also includes:
[0012] Suppression operations are performed on the suspected debris candidate box set to suppress non-debris targets, and candidate boxes that have not been suppressed are selected to obtain the final debris candidate box set.
[0013] Furthermore, the calculation retains the perceptual parameters corresponding to the candidate boxes, and the calculation formula is as follows:
[0014] p uccp (f)=ζ di ·ρ ue
[0015] Where, candidate box f belongs to the set of retained candidate boxes, ζ di ρ is the logical value corresponding to the candidate box f. ve Let f be the maximum feature value of the candidate box.
[0016] Furthermore, the step of filtering the target candidate box set from the foreground candidate box set includes:
[0017] For any foreground candidate box, expand it according to the expansion coefficient with the center of the foreground candidate box as the origin to generate a new candidate box;
[0018] Based on the road surface semantic segmentation result matrix, the confidence score of the new candidate box located in the roadside area is calculated. When the confidence score is greater than the second threshold, it is determined to be a target candidate box. All target candidate boxes are combined into a target candidate box set.
[0019] Furthermore, the step of perceiving unknown category targets from the target candidate box set using known and unknown category distributions includes:
[0020] There are target candidate boxes Determine whether it belongs to the "unknown category" based on the following conditions:
[0021]
[0022] Where κ=p ucpp (f) represents the perceptual parameter magnitude of the target candidate box f, where γ is a minimum value. When the above condition is met, the target candidate box f is determined to belong to the "unknown category"; when the above condition is not met, the target candidate box f is determined not to belong to the "unknown category", θ kno Given the class distribution, θ unk The distribution is for an unknown category.
[0023] Furthermore, the suppression operation on the suspected projectile candidate box set to suppress non-projectile targets includes:
[0024] For a candidate bounding box f that is suspected to contain debris, it will be suppressed if the following condition is met:
[0025]
[0026] in, This is the segmentation result of road surface semantic segmentation. y is the width of the image to be recognized. c η is the height of the center coordinates of the candidate box f. ssio ω is the scale parameter; h is the width of the candidate box f and h is the height of the candidate box f.
[0027] Then, for any candidate box of suspected spilled material It will be suppressed if the following conditions are met:
[0028]
[0029] Among them, y ty η is the y-axis value of the top-left corner of the candidate box f. ssro The size is defined as:
[0030]
[0031] in, For matrix 5η ssro The sum of all values for the following height, and For matrix The sum of all values in the expression.
[0032] This application proposes a method for detecting road debris based on unknown category perception. It studies relevant techniques for target detection based on unknown category perception, identifies unknown category targets located on the road surface as debris, and proposes a method for detecting road debris based on unknown category perception. Without any training data on road debris, this application can detect a large number of different categories of road debris, better meeting the detection needs of real-world scenarios. Attached Figure Description
[0033] Figure 1 This is a flowchart of the spill detection method based on unknown category perception in this application. Detailed Implementation
[0034] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0035] In one embodiment, such as Figure 1 As shown, a method for detecting spilled material based on unknown category perception is proposed, including:
[0036] Step S1: Train a target detection model and a semantic segmentation model using a traffic participant dataset. The target detection model is used to detect foreground targets, and the semantic segmentation model is used to segment the road surface area.
[0037] First, construct a traffic participant dataset containing a total of n images. Where, x i Let y be the set of label bounding boxes for the i-th image. i Let be the semantic label of the road surface region in the i-th image.
[0038] For example, constructing a traffic participant dataset containing 1500 images. The labeled targets include vehicles, pedestrians, bicycles, motorcycles, roadblocks, and crash barriers, all categorized as "foreground targets." Furthermore, no projectile targets are included in any of the images.
[0039] Then, the training samples are input into the object detection model respectively. and semantic segmentation model The model parameters are updated using gradient descent until training is complete. The training methods all use default settings and will not be elaborated further. The object detection model... This could be a YOLOv7 model, responsible for detecting foreground objects in the image; or a semantic segmentation model. It can be DeepLabv3++, which is responsible for segmenting the road surface region in the image.
[0040] Step S2: Use the trained object detection model to detect candidate boxes for the unknown category dataset and the known category dataset respectively, retain the candidate boxes with an intersection-union ratio greater than the first threshold, calculate the perceptual parameters corresponding to the retained candidate boxes, fit the perceptual parameter set with the probability density function, and generate the unknown category distribution and the known category distribution.
[0041] This step is used to generate the unknown class distribution θ. unk and known category distribution θ kno The two distributions are generated in the same way. For a known class distribution θ... kno First, using the model Detecting known category datasets Candidate boxes with a predicted intersection-union ratio greater than 0.5 are retained, and the perceptual parameters p of these retained candidate boxes are calculated. uccp (f) forms the set of perception parameters. Then, various probability density functions are used to fit the set of perception parameters, and the probability density function with the best fit, known class distribution θ, is selected using the AICc index. kno .
[0042] For an unknown class distribution θ unkThe unknown class distribution θ is generated using the same method as the known class distributions mentioned above. The difference lies in how it is generated when generating the unknown class distribution θ. unk In this case, an unknown category dataset was used, and only the projectiles were labeled to obtain the unknown category perception parameters, also using p. uccp (f) represents this. Then, various probability density functions are used to fit the set of sensing parameters, and the probability density function with the best fit, θ, is selected as the unknown class distribution using the AICc index. unk .
[0043] To generate a known class distribution θ kno For example, to calculate the perceptual parameters corresponding to the retained candidate boxes, first, a dataset containing m images of known categories is constructed. in, Let be the set of label bounding boxes for the i-th image.
[0044] For example, construct a dataset containing 500 images of known categories. The labeled targets include vehicles, pedestrians, bicycles, motorcycles, roadblocks, and crash barriers, all categorized as "foreground targets." Furthermore, no projectile targets are included in any of the images.
[0045] Then, using the model Candidate boxes are detected, and those with an intersection-union ratio (IU) greater than a first threshold are retained. The perceptual parameters corresponding to the retained candidate boxes are calculated using the following formula:
[0046] p uccp (f)=ζ di ·ρ ue
[0047] Where, candidate box f belongs to the set of retained candidate boxes, ζ di ρ is the logical value corresponding to the candidate box f. ve Let f be the maximum feature value of the candidate box. Define the logical value ζ. di as follows:
[0048]
[0049] Among them, g logit (o pu ) represents the purity o corresponding to the candidate box f. pu The logical value of g logit (o in ) represents the completeness o corresponding to the candidate box f. in The logical value. Define the largest eigenvalue ρ. ve as follows:
[0050]
[0051] Among them, f maxThis indicates taking the maximum value of the vector within the parentheses, where w is the width of the candidate box f, h is the height of the candidate box f, and (x c y c () represents the center coordinates of the candidate box f. Let f be the region of interest for the candidate box. Obtaining the region of interest for candidate boxes is a well-established technique; for example, the RoI Pooling method in Faster R-CNN can be used. These will not be elaborated upon here.
[0052] In one specific embodiment, a probability density function is used to fit the set of sensing parameters to generate unknown category distributions and known category distributions. The known category distribution is used as an example for illustration below.
[0053] For example, if we fit the set of sensing parameters using the gamma probability density function, Weibull probability density function, Gaussian probability density function, and log-normal probability density function, respectively, and the AICc indices are 3310, 3567, 3670, and 3542, then we choose the gamma probability density function with the smallest AICc index as the known class distribution θ for fitting. kno .
[0054] Step S3: Input the image to be recognized into the trained object detection model and semantic segmentation model to obtain the foreground candidate box set and the road surface semantic segmentation result matrix, and filter out the target candidate box set from the foreground candidate box set.
[0055] For the image to be identified, input the image into the object detection model. Get a set of foreground candidate boxes of size n1 Input the same image into the semantic segmentation model Obtain the semantic segmentation result matrix of the road surface Then, from the set The middle brush selects the target candidate boxes, forming a target candidate box set of size n2.
[0056] The step of filtering the target candidate box set from the foreground candidate box set includes:
[0057] Step S3.1: For any foreground candidate box, expand it according to the expansion coefficient with the center of the foreground candidate box as the origin to generate a new candidate box.
[0058] For any foreground candidate box Represent it as (x c y c (x, w, h). c y cLet x be the center coordinate, w be the width, and h be the height. Using the center of the foreground candidate box f as the origin, expand both its length and width by a certain proportion to generate a new candidate box f′, and represent it as (x...). c y c , w′, h′):
[0059] w′=(1+δ)·w
[0060] h′=(1+δ)·h
[0061] Where δ is the expansion coefficient, which takes a value greater than 0 and less than 1.
[0062] Step S3.2: Based on the road surface semantic segmentation result matrix, calculate the confidence of the new candidate box located in the roadside area. When the confidence is greater than the second threshold, it is determined to be a target candidate box. All target candidate boxes are combined into a target candidate box set.
[0063] Define the semantic segmentation result matrix of the road surface as follows:
[0064]
[0065] Among them, a i，j The value can only be 0 or 1. When a i，j When the value is 0, it means Pixels are segmented into "non-road surfaces"; while when a i，j When the value is 1, it means The pixels are divided into "road surfaces". Furthermore, and These are the images to be detected. Width and height.
[0066] Then, the confidence level of the foreground candidate box f located in the road surface region is defined as:
[0067]
[0068] Among them, f sum (A f′ ) indicates that the matrix The sum of all values within the range of the upper candidate box f′, f sum (A f ) indicates that the matrix The sum of all values within the candidate box f is used to calculate the area of candidate box f'. w′·h′ represents the area of candidate box f', and w·h represents the area of candidate box f.
[0069] Finally, for any foreground candidate box f, the confidence τ of its location in the road surface region is calculated using the method described above. r When the confidence level τ rWhen the value is greater than the threshold δ, it is determined to be a target candidate box, and all target candidate boxes are combined into a set.
[0070] Step S4: Using the known and unknown category distributions, perceive unknown category targets from the target candidate box set to obtain a suspected litter candidate box set.
[0071] This step utilizes known and unknown class distributions to select target candidate boxes from the set. Identify candidate bounding boxes for suspected spilled materials and form a set of candidate bounding boxes of size n³.
[0072] Specifically, using the known class distribution θ kno and unknown category distribution θ unk To detect targets of unknown categories. There are target candidate boxes. Determine whether it belongs to the "unknown category" based on the following conditions:
[0073]
[0074] Where, k = p ucpp (f) represents the perceptual parameter magnitude of the target candidate box f, where γ is a minimum value. When the above conditions are met, the target candidate box f is determined to belong to the "unknown category"; when the above conditions are not met, the target candidate box f is determined not to belong to the "unknown category". From the set of target candidate boxes... The candidate boxes that meet the above conditions are selected to form a set of suspected spilled object candidate boxes.
[0075] In another embodiment, the road spill detection method based on unknown category perception described in this application further includes:
[0076] Suppression operations are performed on the suspected debris candidate box set to suppress non-debris targets, and candidate boxes that have not been suppressed are selected to obtain the final debris candidate box set.
[0077] This step utilizes a non-spraying suppression method to select from the set of suspected spraying candidate boxes. Candidate bounding boxes for spilled materials are identified, resulting in a set of candidate bounding boxes for spilled materials. And output the prediction.
[0078] The described projectile suppression method is responsible for suppressing non-projectile targets.
[0079] First, for a candidate box for suspected spilled material It will be suppressed if the following conditions are met:
[0080]
[0081] in, This is the segmentation result of road surface semantic segmentation. Let y be the width of the image to be detected. c η is the height of the center coordinates of the candidate box f. ssio is the scale parameter. w is the width of the candidate box f, and h is the height of the candidate box f.
[0082] Then, for any candidate box of suspected spilled material It will be suppressed if the following conditions are met:
[0083]
[0084] Among them, y ty Let η be the y-axis value of the top-left corner of candidate box f, h be the height of candidate box f, and η be the height of candidate box f. ssro The size is defined as:
[0085]
[0086] in, For matrix 5η ssro The sum of all values for the following height, and For matrix The sum of all values in the expression.
[0087] Finally, from the set of candidate boxes for suspected spills The middle brush selects the candidate boxes that are not suppressed, resulting in a set of candidate boxes for the projectiles. And output the prediction.
[0088] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for detecting road debris based on unknown category perception, characterized in that, The method for detecting road debris based on unknown category perception includes: A target detection model and a semantic segmentation model are trained using a traffic participant dataset. The target detection model is used to detect foreground targets, and the semantic segmentation model is used to segment the road surface area. The trained object detection model is used to detect candidate boxes for the unknown category dataset and the known category dataset respectively. Candidate boxes with an intersection-union ratio greater than the first threshold are retained. The perceptual parameters corresponding to the retained candidate boxes are calculated. The perceptual parameter set is fitted with a probability density function to generate the unknown category distribution and the known category distribution. The image to be identified is input into the trained object detection model and semantic segmentation model to obtain a set of foreground candidate boxes and a road surface semantic segmentation result matrix, and the set of target candidate boxes is selected from the set of foreground candidate boxes. By utilizing known and unknown class distributions, unknown class targets are perceived from the target candidate box set, resulting in a suspected object candidate box set. The calculation of the perception parameters corresponding to the candidate boxes is performed using the following formula: ； Among them, candidate boxes It belongs to the set of retained candidate boxes. Candidate boxes The corresponding logical value, Candidate boxes The largest eigenvalue; The method of perceiving unknown category targets from the target candidate box set using known and unknown category distributions includes: There are target candidate boxes Determine whether it is an "unknown category" based on the following conditions: ； in, For target candidate boxes The magnitude of the sensing parameters, It is a local minimum value; when the above conditions are met, the target candidate box is determined. It belongs to the "unknown category"; when the above conditions are not met, the target candidate box is determined. It does not belong to the "unknown category". Given the category distribution, For unknown category distributions, from the target candidate box set The candidate boxes that meet the above conditions are selected to form a set of suspected spilled object candidate boxes.
2. The method for detecting road spills based on unknown category perception according to claim 1, characterized in that, The road debris detection method based on unknown category perception also includes: Suppression operations are performed on the suspected debris candidate box set to suppress non-debris targets, and candidate boxes that have not been suppressed are selected to obtain the final debris candidate box set.
3. The method for detecting road debris based on unknown category perception according to claim 1, characterized in that, The step of filtering the target candidate box set from the foreground candidate box set includes: For any foreground candidate box, expand it according to the expansion coefficient with the center of the foreground candidate box as the origin to generate a new candidate box; Based on the road surface semantic segmentation result matrix, the confidence of new candidate boxes located in the roadside area is calculated. When the confidence is greater than the second threshold, it is determined to be a target candidate box. All target candidate boxes are combined into a target candidate box set.
4. The method for detecting road spills based on unknown category perception according to claim 2, characterized in that, The suppression operation on the suspected projectile candidate box set, suppressing non-projectile targets, includes: For a candidate box of suspected litter It will be suppressed if the following conditions are met: ； in, This is the segmentation result of road surface semantic segmentation. The width of the image to be identified. Candidate boxes The height of the center coordinates This refers to the scale ratio parameter; Candidate boxes width, Candidate boxes Height; Then, for any candidate box of suspected spilled material It will be suppressed if the following conditions are met: ； in, Candidate boxes Top left corner coordinates Axis value, The size is defined as: ； in, For matrix middle The sum of all values for the following height, and For matrix The sum of all values in the expression.