An intelligent edge cleaning and road cleanliness determination integrated method

Abstract

This invention discloses an integrated method for intelligent edge sweeping and road surface cleanliness determination, relating to the fields of 3D vision, feature matching, neural networks, and target recognition. The method includes the following steps: Step 1: Acquiring road surface information in front of and behind the sweeping vehicle using its front and rear cameras respectively; Step 2: Processing the video frames acquired by the front camera using visual odometry; Step 3: Extracting the road edge based on the visual odometry results, obtaining the sweeping area and the distance from the sweeper to the road edge; Step 4: Processing the images acquired by the rear camera using a target detection algorithm; Step 5: Scoring the road surface cleanliness using a road surface cleanliness scoring algorithm based on the road debris detection results. This method not only achieves the extraction of the sweeping area but also the perception of the distance from the sweeper to the road edge, enabling effective scoring of road surface cleanliness in various complex scenarios.

Description

Technical Field

[0001] This invention relates to the fields of 3D vision, feature matching, neural networks, and target recognition, and in particular to an integrated method for intelligent edge cleaning and road surface cleanliness determination. Background Technology

[0002] Computer vision technology enables computers to simulate human visual processes, giving them the ability to perceive their environment. It integrates image processing, artificial intelligence, and pattern recognition technologies. Computer vision research has two main goals: one is to develop image understanding systems that automatically construct scene descriptions from input image data; the other is to understand human vision so that machines can perform tasks that are difficult for humans to accomplish. Currently, computer vision is also a very active and fruitful research topic within artificial intelligence and robotics.

[0003] Street cleaning is a tedious, time-consuming, and energy-intensive task for sanitation vehicles. Currently, sanitation vehicles rely primarily on manual drivers to observe the road surface, judge the distance between the sweeper and the road edge, and assess the type and area of ​​litter to control the cleaning area and intensity. This method results in low cleaning efficiency, frequent missed areas or rapid brush wear, driver fatigue, inaccurate road cleanliness assessment, and high energy consumption. Intelligent edge cleaning and road cleanliness assessment are urgent problems that intelligent sanitation devices need to solve. Currently, there are three main solutions:

[0004] The first approach, as described in Chinese Invention Patent 202011172360.1, provides a method for determining road surface cleanliness and automatic cleaning based on semantic segmentation. This system includes: collecting and labeling a dataset of road debris identification and segmentation; training a road debris semantic segmentation network; porting the network to an embedded vehicle platform to predict real-time road surface data input from a vehicle camera; classifying road surface cleanliness using a metric algorithm based on the debris prediction results; and adjusting the cleaning intensity of the sweeper's upper structure based on the road surface cleanliness assessment results to achieve intelligent cleaning. Based on the results of semantic instance segmentation of debris, feedback is provided to road cleaning personnel according to the debris category, enabling the recycling of usable debris. This approach can be applied to road cleaning vehicles to classify and recycle identified debris, achieving road surface cleanliness determination and intelligent road cleaning. However, this approach has the following problems: First, it lacks a cleaning area extraction function, making it prone to misidentifying objects outside the road, such as dead leaves in green belts, leading to inaccurate road surface cleanliness determination. Second, this solution can only adjust the sweeping intensity of the sweeper's upper structure, and is ineffective in addressing issues caused by the sweeper brushes being too far from the curb. Third, the waste detection network used is not optimized for small-sized waste on the road surface, leading to missed detections.

[0005] The second approach, as described in Chinese invention patent 202010663905.2, provides a method for assessing road surface cleanliness for intelligent road cleaning. It extracts road surface areas using a Yolact road segmentation network, then uses a Yolov3 object detection network to obtain the location and category of large debris, while simultaneously employing an MCNN density estimation network to obtain the density distribution of small debris. Finally, it weights and sums all debris to obtain a road surface cleanliness index. This approach uses transfer learning and image enhancement algorithms to improve the performance of the trained model, successfully estimating road surface cleanliness by combining image segmentation, object detection, and density estimation algorithms. However, this approach has the following problems: First, while it extracts the cleaning area using the Yolact segmentation network, it suffers from inaccurate segmentation in complex environments, exhibiting poor environmental stability. Second, although it can segment road edges, it lacks distance information, failing to address missed areas caused by the sweeper being too far from the curb. Third, using two networks to detect debris of different sizes undoubtedly increases computational costs and complicates deployment on vehicle platforms.

[0006] The third approach, as described in Chinese Invention Patent 202010322110.5, provides a machine vision-based intelligent assessment method for road surface cleanliness, relating to the field of artificial intelligence. This approach mainly includes: inputting a video frame to be detected; processing the image in the video frame using a detection algorithm model; whereby the detection algorithm model uses different layers of convolution to obtain a feature pyramid on the input image; and inputting the processed result into a scoring algorithm model. This approach innovates and improves upon Faster R-CNN based on machine vision detection algorithms, proposing a feature pyramid network to achieve the recognition of small targets. Simultaneously, it proposes a set of road surface cleanliness scoring criteria and formulas, enabling road surface cleanliness scoring in complex scenarios. However, this approach has the following problems: First, it lacks a cleaning area extraction function, making it prone to misidentifying objects outside the road surface, such as dead leaves in green belts, leading to inaccurate road surface cleanliness judgments. Second, the proposed solution uses an improved Faster-R-CNN network to identify garbage of different sizes. However, Faster-R-CNN is a two-stage object detection network, which has a slow computation speed and poor real-time performance, making it difficult to guarantee detection results when deployed on an in-vehicle platform. Summary of the Invention

[0007] The purpose of this invention is to provide an integrated method for intelligent edge cleaning and road surface cleanliness determination. This method not only extracts the cleaning area but also senses the distance between the sweeper and the curb, resulting in better accuracy, real-time performance, and fault tolerance. Furthermore, it proposes a road surface cleanliness calculation formula based on information such as the weight of garbage and the cleaning area, enabling road surface cleanliness scoring in various complex scenarios.

[0008] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0009] A method integrating intelligent edge cleaning and road surface cleanliness determination includes the following steps:

[0010] Step 1: Collect road information in front of and behind the vehicle using the front and rear cameras of the cleaning vehicle, respectively;

[0011] Step 2: Process the video frames captured by the front-facing camera using visual odometry.

[0012] Step 3: Based on the results of the visual odometry calculation method, extract the road edge and obtain the cleaning area and the distance from the brush to the road edge;

[0013] Step 4: Based on the cleaning area extracted in Step 3, process the image captured by the rear camera using a target detection algorithm;

[0014] Step 5: Based on the road surface litter detection results, use the road surface cleanliness scoring algorithm to score the road surface cleanliness;

[0015] Step 6: Based on the distance from the brush to the curb obtained in Step 3 and the cleanliness assessment results obtained in Step 5, display or provide voice prompts.

[0016] Furthermore, in step 3, the visual odometry method is used to reconstruct the road edge in three dimensions. While accurately segmenting and fitting the road edge, the distance from the sweeper to the road edge is calculated, thereby realizing the detection of the cleaning area and the distance from the sweeper to the curb.

[0017] Furthermore, the specific methods for detecting the cleaning area and the distance from the brush to the curb in step 3 include:

[0018] Step 3.1: Perform point cloud processing on the reconstructed point cloud, extract road boundary points, and fit road edge curves;

[0019] Step 3.2: Based on the obtained point cloud data, road edge curves, and calibration parameters of the front and rear cameras, calculate the distance from the sweeper brush to the road edge;

[0020] Step 3.3: Extract the cleaning area based on the obtained road edge curve, the calibration parameters of the front and rear cameras, and the prior information of the vehicle.

[0021] Furthermore, the method for processing the image in step 4 includes:

[0022] Step 4.1: Based on the cleaning area extracted from the front camera's field of view, the feature matching algorithm aligns the cleaning areas in the fields of view of the front and rear cameras to complete the extraction of the cleaning area after the cleaning operation.

[0023] Step 4.2: Perform target detection on the obtained cleaning area to obtain the location and category of large and small garbage.

[0024] Furthermore, the feature matching algorithm in step 4.1 uses ORB features to extract ORB features from the images captured by the front camera and the rear camera respectively, and then performs matching by calculating the Hamming distance between the descriptions.

[0025] Furthermore, after matching, based on the relative pose between the front and rear cameras... The system extracts the cleaning area from the front-facing camera and aligns and corrects the image captured by the rear-facing camera to extract the cleaning area after the cleaning operation.

[0026] Furthermore, step 4.2 employs the one-stage object detection algorithm YOLOv5, which locates and identifies objects by laying out prior boxes on the map and regressing the center point, length, width, and category of the objects.

[0027] Furthermore, the YOLOv5 framework uses CSPDarknet53 as the feature extraction network, which consists of layers 0 to 33. It is composed of a series of 1x1 and 3x3 convolutional layers, each followed by a BN layer and a Leaky-ReLU layer. CSPDarknet53 also employs residual connections.

[0028] Layers 34 through 70 of the CSPDarknet53 network are feature interaction layers of the YOLO network, which predict bounding boxes at three different scales.

[0029] In the first scale, the feature map is downsampled by 32 times to detect the first target in the image; in the second scale, the feature map is downsampled by 16 times to detect the second target in the image; in the third scale, the feature map is downsampled by 8 times to detect the third target in the image; the size of the first target, the second target, and the third target decreases in that order.

[0030] Furthermore, the waste identification dataset was collected using road surface information captured by the vehicle's front and rear cameras. This dataset includes multiple road surface waste image samples, and the targets in each image are reliably labeled.

[0031] A road debris detection network was trained by setting the hyperparameters for neural network training and then porting the network to an embedded platform to enable online prediction of real-time road data input from vehicle-mounted cameras.

[0032] Furthermore, the method for scoring road surface cleanliness in step 5 includes:

[0033] Step 5.1: Statistically analyze and measure the garbage classification on the road surface, assign different weights to each category, and then calculate the category measurement value;

[0034] Step 5.2: Calculate the area of ​​the cleaning area and use the road surface cleanliness scoring algorithm to score the road surface cleanliness;

[0035] The scoring formula is: ,in, Representing the The weighting coefficient of the waste type to which each piece of waste belongs. This represents the area to be cleaned. The representative is the cleanliness correction factor. This refers to the cleanliness score of the road surface.

[0036] In summary, the present invention has the following beneficial effects:

[0037] (1) This invention uses visual mileage calculation method and improves it to realize three-dimensional reconstruction of road edge. After point cloud filtering and curve fitting, the detection of road edge is realized. It not only realizes the extraction of cleaning area, but also realizes the perception of distance from the brush to the curb. The algorithm has better accuracy, real-time performance and fault tolerance, effectively improving cleaning quality and reducing the driver's workload.

[0038] (2) The present invention uses a single-stage target detection and image segmentation algorithm to locate and identify garbage on the road surface. This scheme can effectively identify garbage of various scales and densities, and has good accuracy and real-time performance.

[0039] (3) For the calculation of road surface cleanliness, a set of road surface cleanliness calculation formula based on information such as the weight of garbage and the cleaning area is proposed. This formula realizes the scoring of road surface cleanliness in various complex scenarios. Attached Figure Description

[0040] Figure 1 This is a flowchart of the algorithm in an embodiment of the present invention;

[0041] Figure 2 This is a flowchart of the cleaning area detection method in an embodiment of the present invention;

[0042] Figure 3 This is a flowchart of the visual odometry (DSO) in an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the radius filtering algorithm in an embodiment of the present invention;

[0044] Figure 5 This is an example diagram of road edge detection in an embodiment of the present invention;

[0045] Figure 6 This is a schematic diagram of the adjacent point slope method in an embodiment of the present invention;

[0046] Figure 7 This is a schematic diagram of the improved algorithm in an embodiment of the present invention;

[0047] Figure 8 This is a schematic diagram of the corner extraction algorithm in an embodiment of the present invention;

[0048] Figure 9 This is an example diagram of the visual odometry calculation method in an embodiment of the present invention;

[0049] Figure 10 This is a diagram of the Yolov5 network structure of the detection algorithm in this embodiment of the invention;

[0050] Figure 11 This is an example diagram of waste detection in an embodiment of the present invention. Detailed Implementation

[0051] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. These embodiments do not constitute a limitation on the present invention.

[0052] An integrated method for intelligent edge cleaning and road surface cleanliness assessment, such as... Figure 1 As shown, it includes the following steps:

[0053] Step 1: Turn on the fully installed and calibrated camera on the vehicle to collect test images of the road surface before and after the vehicle is swept; collect road surface information in front of and behind the vehicle using the front and rear cameras of the sweeping vehicle, respectively.

[0054] Step 2: Process the video frames captured by the front-facing camera using visual odometry.

[0055] Step 2.1: As Figure 2 As shown, the video captured by the front-facing camera is processed and a local map is constructed using the visual odometry method.

[0056] For ease of description, we assume that the camera coordinate system conforms to a right-handed coordinate system. The axis is the optical axis of the camera, and the origin of the coordinate system is the optical center of the camera.

[0057] Step 2.1.1: Use visual odometry to track the video stream. In the initial stage, to improve the initialization success rate, use the corner extraction strategy during initialization to build a local map. Then, extract corners according to the preset region of interest and continue tracking.

[0058] Step 2.1.2: Since visual odometry does not maintain a global map, and the optimization algorithm typically only has 5 to 7 frames of data and requires further iterative optimization, which is insufficient to extract road boundaries, a local map is constructed. This local map is responsible for maintaining the well-optimized and marginalized frames and their corresponding map points within the visual odometry sliding window. The size of this local map can be limited as needed, and unnecessary frame data can be deleted. The reference coordinate system of the local map is the camera coordinate system of the first frame in the local map. ;

[0059] Step 2.2: Filter and transform the constructed local map, and classify the local point cloud data based on the region of interest to construct ground point cloud data and edge detection point cloud data.

[0060] Step 2.2.1: In visual odometry, the error of 3D point cloud data increases with the depth of the point cloud. Therefore, each frame of data in the local map is filtered based on the point cloud depth value, i.e. When that happens, delete the map point, where, Representative in the camera Map points in coordinate system of axis coordinate values, This represents the depth threshold of the point cloud; removing large-depth point cloud data improves the accuracy of edge detection.

[0061] Step 2.2.2: Perform coordinate transformation on the filtered local map to unify it to the camera coordinate system corresponding to the first frame of the local map. The transformation formula is: ,in, Representative in the camera The first coordinate system The coordinates of a map point Represents camera pose in the world coordinate system Representing the world coordinate system to the camera Coordinate system transformation relationships, The above map points represent the camera. The coordinates in the coordinate system limit the cumulative error of visual odometry to the local map range. Reducing the size of the local map can reduce the impact of the cumulative error of visual odometry on subsequent algorithms.

[0062] Step 2.2.3: Based on the preset region of interest (ROI) division, the point cloud in the local map is segmented into ground point cloud data and edge detection point cloud data, ensuring a sufficient number of point clouds to meet computational requirements. Since the local map contains a large amount of point cloud data, including many that are ineffective for edge detection, to reduce the computational load of the algorithm and mitigate the adverse effects of irrelevant point clouds on edge detection, a region of interest is preset based on the 3D coordinates of the point cloud in the camera coordinate system. This initially segments the point cloud in the local map into ground point cloud data and edge detection point cloud data. To ensure subsequent computational needs, the number of extracted point clouds will be controlled. If the number is insufficient to meet computational requirements, the corner extraction grid will be dynamically adjusted to control the number of corner points.

[0063] Step 2.3: Use the plane fitting algorithm based on the Random Sample Consensus Algorithm (RANSAC) to perform plane fitting on the ground point cloud and adjust the coordinate system of the edge detection point cloud data according to the plane equation.

[0064] Step 2.3.1: Use a RANSAC-based plane fitting algorithm to fit the ground point cloud to the plane. To improve the stability of the fitting algorithm, the prior road surface normal vector is used to constrain the fitting algorithm. The road surface normal vector obtained after fitting is: The direction is from the ground to the sky; based on the calibrated camera extrinsic parameters, the normal vector of the horizontal road surface in the camera coordinate system can be calculated. Therefore, in order to improve the stability of the fitting algorithm, the prior road surface normal vector parameters are used to constrain the fitting algorithm, thereby fitting the plane equation of the road surface.

[0065] Step 2.3.2: Due to the influence of camera extrinsic parameters and road surface tilt, the obtained ground point cloud data is often tilted in the reference frame, making it difficult to perform structured processing of the point cloud and road edge detection. Therefore, using the fitted plane equation, a pure rotation adjustment is performed on the reference coordinate system of the local map to provide a basis for subsequent spatiotemporal tiling of the point cloud data. The specific calculation process includes: [Calculation steps are missing from the provided text]. Vectors in local map reference frame Obtain the rotation matrix Perform a pure rotation transformation on the local map point cloud. The transformation formula is as follows: ,in, This refers to the coordinates of the map points obtained in step 3.2 within the local map reference frame. The coordinates of map points after rotation transformation ultimately make the local map reference coordinate system... The axis is parallel to the normal vector of the plane.

[0066] Step 3: Based on the point cloud data obtained by the visual odometry method, extract the road edge, obtain the cleaning area, output the calculated distance between the sweeper and the curb to the display screen, and provide voice reminders based on the safety threshold.

[0067] The detection algorithms specifically include:

[0068] Step 3.1: Perform point cloud processing on the reconstructed point cloud, extract road boundary points, and fit road edge curves;

[0069] Specifically, this includes structuring the edge detection point cloud data, followed by radius filtering and centroid calculation after projection.

[0070] Step 3.1.1: Since visual odometry acquires unstructured point cloud data, and the extracted point cloud data has a certain degree of randomness, it is necessary to perform structured processing on the edge detection point cloud data. Based on the characteristics of the point cloud constructed by visual odometry, firstly, based on map points... The depth value is used to segment it into... Parts. That is, based on the point cloud, respectively in shaft and The distance threshold under the axis is used to perform two-dimensional mesh slicing, and the number of point clouds in each slice is judged to remove slices with too few point clouds.

[0071] Step 3.1.2: Since even a single map point with abnormal height can affect the edge detection algorithm, effective removal of interfering points is essential. Traditional 3D radius filtering algorithms are not suitable for unstructured and sparse point cloud data, easily removing a large amount of valid data, resulting in insufficient data for edge detection. Therefore, to address this deficiency, the first step is to filter each slice... Point cloud projection in Below the plane, the originally sparse ground point clouds are... It becomes denser in the axial direction, while at the same time... Sparse map points with height anomalies are preserved on the axis. Two-dimensional radius filtering is applied to map points in a plane to remove points with abnormal heights. Due to the unstructured and randomly distributed nature of the acquired point cloud data, directly applying three-dimensional radius filtering to the point cloud would lead to the accidental deletion of a large amount of valid data. Therefore, the point cloud in each slice is projected onto a plane... A two-dimensional radius filter is then applied below the plane to effectively remove interference points in the height direction while preserving the effective point cloud on the ground, thus improving the algorithm's robustness against interference. A schematic diagram of the algorithm is shown below. Figure 3 Each point represents a map point after projection, with darker points indicating map points with abnormal heights. The radius is a parameter.

[0072] Step 3.1.3: To improve the robustness of the algorithm and further reduce the influence of outliers, the centroid coordinates of the point cloud clusters in each slice are calculated as the features of that point cloud cluster. Based on the point cloud... The distance threshold on the axis is used to create a two-dimensional mesh slice, and each slice point cloud cluster is denoted as . And calculate the centroid coordinates of the point cloud in each slice, where the point cloud clusters are located in... The formula for calculating the coordinate values ​​on the axis is: ,in, The first in the representative point cloud cluster The elevation value of a map point. For example... Figure 4 As shown, this is the point cloud data after structuring, projection, and filtering. The point cloud at the lower level is the point cloud data on the motor vehicle lane, and the point cloud at the higher level is the point cloud data on the sidewalk. The height difference at the boundary of the motor vehicle lane is effectively extracted.

[0073] Step 3.1.4: Using the idea of ​​smooth features, the traditional neighbor slope method is improved. The centroid of the structured point cloud is used to extract the road boundary points, and the curve fitting algorithm based on RANSAC is used to fit the curve of the road edge.

[0074] Because real-world road surfaces often have undulations or small debris, the traditional neighbor-point slope method cannot effectively filter out these situations. The algorithm diagram is shown below. Figure 5 Small foreign objects appeared on the ground At this time, if there are ground points nearby According to the adjacent point slope method, the straight line The slope will likely be greater than the actual boundary point. The slope at that point leads to a high false positive rate.

[0075] Therefore, improvements were made to address the shortcomings of the neighbor slope method in road edge detection, as illustrated in the following diagram. Figure 6 The slope of the centroid of each slice is calculated, and points exceeding a certain threshold are considered boundary points. Compared to the neighboring point slope method, the optimized algorithm calculates the average ground height based on the detected ground points. And using this height as a baseline, a series of reference points were set, namely... Figure 6 The smaller circles, these points have the same height and are at the same horizontal distance from the corresponding point cloud. This allows for the calculation of the slope between the point and the corresponding point cloud. When a small foreign object appears on the ground... At this time, if there are ground points nearby According to the optimized algorithm, the straight line will be calculated. and The slope of the boundary point is significantly smaller than that of the true boundary point. The slope at that point effectively filters out interference points and improves the algorithm's adaptability.

[0076] The formula for calculating the slope is: ,in, Representing the The height value of the centroid of a point cloud cluster. Represents the average height of the ground. This represents the horizontal distance from the reference point to the corresponding centroid. The correction coefficient representing the sensitivity of road boundary point perception. This refers to the slope value of the centroid of each cloud cluster. The higher the value, the more likely it is to be a road boundary point.

[0077] The reference height of the reference point is calculated based on the ground point cloud. The slope of the centroid of each slice is calculated. When it is greater than a certain threshold, it is considered a boundary point. Finally, the extracted boundary points are fitted with curves using a RANSAC-based curve fitting algorithm.

[0078] Step 3.1.5: Update the region of interest based on the extracted edge curves, use a dynamic mesh strategy to adaptively extract corner points in different regions, realize continuous detection of road edges, and finally extract the area to be cleaned by the sweeper based on the detected road edge curves.

[0079] The extracted edge curves are transformed according to the camera model to obtain the corresponding pixel coordinates, thereby rasterizing the entire image and dividing it into regions of interest (ROI) and non-ROI. Traditional visual odometry often extracts corner points from the entire image, making it difficult to balance the trade-off between computational speed and point cloud density. Directly extracting corner points from a portion of the image may lead to degradation in visual odometry calculations, compromising tracking robustness. Therefore, an improvement has been made to the original odometry method, proposing an adaptive corner point extraction strategy based on dynamic grids. This strategy divides the entire image into grids, such as... Figure 7 As shown, where The area is The size of the non-interest region, its left side is The size of the region of interest is determined, and the region of interest is further subdivided into... Size Region and Size Area, area of ​​non-interest The corner extraction strategy involves calculating the gradient value of each pixel and selecting the corner point that exceeds a gradient threshold and is the maximum value. This corner point is denoted as... The corner point extraction strategy for the region of interest is to calculate... The gradient values ​​of each pixel in the region are analyzed, and the corner point that exceeds the gradient threshold and is the maximum value is selected and denoted as [corner point name]. ,therefore, A maximum of 4 corner points can be extracted from a region, and a maximum of 16 corner points can be extracted from this region of interest. That is, the number of corner points extracted from the region of interest grid and the non-region of interest grid can differ by a maximum of 16 times. The grid size will be adjusted according to the total number of corner points extracted, while ensuring that corner points can be extracted in all directions, thus improving the stability of the algorithm.

[0080] To extract as much effective point cloud data as possible for edge detection while ensuring the stability of visual odometry, an improvement was made to the traditional indiscriminate corner extraction strategy. Specifically, for the updated region of interest (ROI), a dynamic mesh strategy was used to adaptively extract corners in different regions: dense corner extraction in the ROI and sparse corner extraction in the non-ROI regions, thus completing the visual odometry tracking. The extracted edge curves were transformed according to the camera model to obtain the corresponding pixel coordinates, thereby meshing the entire image and updating the ROI. The dynamic mesh strategy was then used to adaptively extract corners in different regions to complete the visual odometry tracking and achieve continuous detection of road edges.

[0081] Step 3.2: Based on the obtained point cloud data, road edge curves, and calibration parameters of the front and rear cameras, calculate the distance from the sweeper brush to the road edge;

[0082] Step 3.3: Extract the cleaning area based on the obtained road edge curve, the calibration parameters of the front and rear cameras, and the prior information of the vehicle;

[0083] Specifically, based on continuously updated road edge curves and the size information of sanitation vehicles, the area to be cleaned by the sweeper is accurately extracted. According to the continuously updated road edge curves, calibration parameters of the front and rear cameras, and prior information about the sanitation vehicles, including the width of the vehicles, the width of the sweeping brushes, and the driving speed, a precise cleaning area is extracted.

[0084] A local map is constructed using visual odometry, and the point cloud data is classified based on regions of interest (ROIs). Point cloud tiling and centroid calculation are used to achieve gridding of the point cloud. An improved neighbor slope method is employed to extract road boundary points, and a dynamic grid extraction strategy based on ROIs is used for adaptive corner point extraction. This balances the trade-off between computational speed and accuracy, achieving precise segmentation of road edges and thus accurate extraction of the cleaning area for sanitation vehicles. The final detection algorithm is deployed on the Horizon X3M core board, enabling real-time perception of road edges by vehicles in various complex road surface scenarios.

[0085] It uses visual odometry to reconstruct the road edge in three dimensions, accurately segmenting and fitting the road edge while calculating the distance from the sweeper to the road edge, thus realizing the detection of the cleaning area and the distance from the sweeper to the curb. It includes DSO, ORB-SLAM2, and SVO.

[0086] In this embodiment, as Figure 8 As shown, the visual odometry method uses the DSO framework, which is a sparse direct method for visual odometry. In contrast, SVO is a semi-direct method, while ORB-SLAM2 is a pure feature method. DSO integrates data association and pose estimation into a unified nonlinear optimization problem, with its objective function being to minimize photometric error. While maintaining the same or higher accuracy, it can achieve five times the speed of the traditional feature point method.

[0087] From the backend perspective, DSO uses a sliding window consisting of 5 to 7 keyframes as its backend. This window persists throughout the entire VO process, maintaining and updating the Hessian matrix and b-vector in the Levenberg-Marquardt method, and optimizing the pose and map points through edge-mapping operations. Furthermore, to enhance the algorithm's robustness, DSO proposes photometric calibration and dynamically estimates photometric parameters during optimization, resulting in better performance for image brightness variations caused by different camera exposures.

[0088] The aforementioned DSO framework can be used to obtain a point cloud map of the forward view, and the road edge can be fitted using a road edge extraction algorithm to extract the cleaning area, such as... Figure 9 As shown, L1 represents the vehicle trajectory, and L2 represents the road edge. Finally, a voice prompt is given based on the distance between the sweeper and the road edge. If the distance is too large or too small, a voice alarm is triggered and the driver is reminded to control the direction of the sweeper.

[0089] Step 4: Based on the cleaning area extracted in Step 3, process the image captured by the rear camera using a target detection algorithm;

[0090] The detection algorithms include:

[0091] Step 4.1: Based on the cleaning area extracted from the front camera's field of view, the feature matching algorithm aligns the cleaning areas in the fields of view of the front and rear cameras to complete the extraction of the cleaning area after the cleaning operation.

[0092] The feature matching algorithm uses ORB features, extracting ORB features from images captured by the front and rear cameras respectively. Then, matching is performed by calculating the Hamming distance between the features. To avoid the adverse effects of mismatches, three consecutive successful matches are required between consecutive image frames. After matching, further matching can be performed based on the relative pose between the front and rear cameras. The system extracts the cleaning area from the front-facing camera and aligns and corrects the image captured by the rear-facing camera to extract the cleaning area after the cleaning operation.

[0093] Step 4.2: Perform target detection on the obtained cleaning area to obtain the location and category of large and small garbage; specifically, use a convolutional network to convolve the input image, and then input the feature pyramids obtained from different layers of convolution into the candidate region generation network and classifier. Finally, input the classification results into the scoring algorithm to achieve the scoring of road surface cleanliness.

[0094] Since there are few existing datasets in this field, the garbage identification dataset was collected by using the front and rear cameras of vehicles to collect road information. The collected and labeled road garbage identification dataset is used for data analysis and model training. It contains 5,000 road garbage image samples, and the targets in each image are reliably labeled.

[0095] In this embodiment, the rectangular bounding box annotation format for garbage target detection is (class,x,y,w,h), where the first parameter class represents the category of the annotation content, the second parameter x represents the x-coordinate of the normalized target center point, the third parameter y represents the y-coordinate of the normalized target center point, the fourth parameter w represents the width of the normalized target bounding box, and the fifth parameter h represents the height of the normalized target bounding box.

[0096] A road debris detection network was trained, and hyperparameters for neural network training were set, primarily including the learning rate (0.0001), the number of training iterations (1 million), and the optimizer (Adam). Training was performed offline on an NVIDIA GPU server. The network was then ported to an embedded platform to enable online prediction of real-time road data input from an onboard camera.

[0097] During the network porting process, the network model trained under the PyTorch framework was converted and ported to the embedded vehicle platform using the general neural network framework ONNX. The vehicle neural network hardware prediction platform used was Horizon X3M.

[0098] The Horizon Robotics X3M is a modular core board that provides the performance and power efficiency required for visual computing applications. It integrates Horizon Robotics' advanced Bernoulli 2.0 architecture AI chip, delivering an equivalent computing power of 5 TOPS. With its 64-bit CPU, 4K@60fps video encoding and decoding performance, and the ability to simultaneously handle video input from 4 to 8 cameras of different resolutions, the X3M is an excellent system for deep learning, computer vision, and image processing.

[0099] Its garbage detection network uses the one-stage object detection algorithm YOLOv5 to identify and locate multiple targets of interest in digital images or videos. It achieves target localization and identification by laying prior boxes on the image and regressing the center point, length, width and category of the object. It also uses an image segmentation algorithm to improve the ability of the YOLOv5 object detection network to identify small targets, and finally obtains the location and category of large and small garbage.

[0100] like Figure 10 As shown, the YOLOv5 framework uses CSPDarknet53 as the feature extraction network, which consists of layers 0 to 33. It is composed of a series of 1x1 and 3x3 convolutional layers, each followed by a BN layer and a Leaky-ReLU layer. CSPDarknet53 also uses residual connections.

[0101] Layers 34 through 70 of the CSPDarknet53 network are feature interaction layers of the YOLO network, which predict bounding boxes at three different scales.

[0102] In the first scale, the feature map is downsampled by 32 times to detect the first target in the image, suitable for detecting larger targets. In the second scale, the feature map is downsampled by 16 times to detect the second target, with a medium-sized receptive field, suitable for detecting medium-sized targets. In the third scale, the feature map is downsampled by 8 times to detect the third target, with a higher resolution, suitable for detecting smaller targets. The size of the first, second, and third targets decreases in that order. The final result is output by performing predictions at these three scales.

[0103] The aforementioned object detection network can be used to obtain the bounding boxes and labels of trash in the image, such as... Figure 11 As shown, this allows us to obtain information about the distribution of waste.

[0104] Step 5: Based on the road surface litter detection results, use the road surface cleanliness scoring algorithm to score the road surface cleanliness;

[0105] The detection algorithms include:

[0106] Step 5.1: Statistically analyze and measure the garbage on the road surface, and assign different weights to leaves, cigarette butts, packaging bags, plastic bottles, plastic bags, and paper in turn, and then calculate the category measurement value;

[0107] Step 5.2: Calculate the area of ​​the cleaning area and use the road surface cleanliness scoring algorithm. Based on the weight of the garbage and the cleaning area, the road surface cleanliness is scored.

[0108] The scoring formula is: ,in, Representing the The weighting coefficient of the waste type to which each piece of waste belongs. This represents the area to be cleaned. The representative is the cleanliness correction factor (which can be set as needed; in this embodiment, 0 < 0). <1), This refers to the cleanliness score of the road surface.

[0109] The weighting coefficients for different types of waste are defined as follows:

[0110]

[0111] The road surface cleanliness score is obtained through the aforementioned calculation formula. When the cleanliness value is less than a certain threshold, the road surface cleanliness is poor, and a voice alarm is triggered to remind the driver to control the sweeping brush and driving speed.

[0112] Step 6: Based on the distance between the sweeper and the curb obtained in Step 3 and the road debris detection results, the system displays the information on the in-vehicle screen and provides voice prompts based on the distance between the sweeper and the curb and the cleanliness assessment results obtained in Step 5, thus achieving intelligent cleaning.

[0113] When the spacing value is not within the effective range, a voice alarm will be triggered and the driver will be reminded to control the direction of the sweeper. When the cleanliness value is lower than a certain threshold, the cleaning quality is poor, and a voice alarm will be triggered and the driver will be reminded to control the sweeping brush and driving speed.

[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within the scope of its essence and protection. Such modifications or equivalent substitutions should also be considered to fall within the protection scope of the present invention.

Claims

1. A method integrating intelligent edge cleaning and road surface cleanliness determination, characterized in that, Includes the following steps: Step 1: Collect road information in front of and behind the vehicle using the front and rear cameras of the cleaning vehicle, respectively; Step 2: Process the video frames captured by the front-facing camera using visual odometry. Step 3: Based on the results of the visual odometry calculation method, extract the road edge and obtain the cleaning area and the distance from the brush to the road edge; Step 4: Based on the cleaning area extracted in Step 3, process the image captured by the rear camera using a target detection algorithm; The methods for image processing include: Step 4.1: Based on the cleaning area extracted from the front camera's field of view, the feature matching algorithm aligns the cleaning areas in the fields of view of the front and rear cameras to complete the extraction of the cleaning area after the cleaning operation. Step 4.2: Perform target detection on the obtained cleaning area to obtain the location and category of large and small waste; Step 5: Based on the results of the road surface litter detection, use the road surface cleanliness scoring algorithm to score the road surface cleanliness; Step 6: Based on the distance from the brush to the curb obtained in Step 3 and the cleanliness assessment results obtained in Step 5, display or provide voice prompts.

2. The integrated method for intelligent edge cleaning and road surface cleanliness determination according to claim 1, characterized in that, In step 3, the visual odometry method is used to reconstruct the road edge in three dimensions. While accurately segmenting and fitting the road edge, the distance from the sweeper to the road edge is calculated, thereby realizing the detection of the cleaning area and the distance from the sweeper to the curb.

3. The integrated method for intelligent edge cleaning and road surface cleanliness determination according to claim 2, characterized in that, The specific methods for detecting the cleaning area and the distance from the brush to the curb in step 3 include: Step 3.1: Perform point cloud processing on the reconstructed point cloud, extract road boundary points, and fit road edge curves; Step 3.2: Based on the obtained point cloud data, road edge curves, and calibration parameters of the front and rear cameras, calculate the distance from the sweeper brush to the road edge; Step 3.3: Extract the cleaning area based on the obtained road edge curve, the calibration parameters of the front and rear cameras, and the prior information of the vehicle.

4. The integrated method for intelligent edge cleaning and road surface cleanliness determination according to claim 1, characterized in that, The feature matching algorithm in step 4.1 uses ORB features. ORB features are extracted from the images captured by the front camera and the rear camera respectively, and then the matching is performed by calculating the Hamming distance between the descriptions.

5. The integrated method for intelligent edge cleaning and road surface cleanliness determination according to claim 4, characterized in that, After matching, based on the relative pose between the front camera and the rear camera The system extracts the cleaning area from the front-facing camera and aligns and corrects the image captured by the rear-facing camera to extract the cleaning area after the cleaning operation.

6. The integrated method for intelligent edge cleaning and road surface cleanliness determination according to claim 1, characterized in that, In step 4.2, the one-stage target detection algorithm YOLOv5 is used. The target is located and identified by laying out prior boxes on the map and regressing the center point, length, width and category of the object.

7. The integrated method for intelligent edge cleaning and road surface cleanliness determination according to claim 6, characterized in that, The Yolov5 framework uses CSPDarknet53 as the feature extraction network, which consists of layers 0 to 33. It is composed of a series of 1x1 and 3x3 convolutional layers, each followed by a BN layer and a Leaky-ReLU layer. CSPDarknet53 also uses residual connections. Layers 34 through 70 of the CSPDarknet53 network are feature interaction layers of the YOLO network, which predict bounding boxes at three different scales. In the first scale, the feature map is downsampled by 32 times to detect the first target in the image; in the second scale, the feature map is downsampled by 16 times to detect the second target in the image; in the third scale, the feature map is downsampled by 8 times to detect the third target in the image; the size of the first target, the second target, and the third target decreases in that order.

8. The integrated method for intelligent edge cleaning and road surface cleanliness determination according to claim 1, characterized in that, The garbage identification dataset was collected using road surface information captured by the front and rear cameras of vehicles. It includes multiple road garbage image samples, and the targets in each image are reliably labeled. A road debris detection network was trained by setting the hyperparameters for neural network training and then porting the network to an embedded platform to enable online prediction of real-time road data input from vehicle-mounted cameras.

9. The integrated method for intelligent edge cleaning and road surface cleanliness determination according to claim 1, characterized in that, The methods for scoring road surface cleanliness in step 5 include: Step 5.1: Statistically analyze and measure the garbage classification on the road surface, assign different weights to each category, and then calculate the category measurement value; Step 5.2: Calculate the area of ​​the cleaning area and use the road surface cleanliness scoring algorithm to score the road surface cleanliness; The scoring formula is: ,in, Representing the The weighting coefficient of the waste type to which each piece of waste belongs. This represents the area to be cleaned. The representative is the cleanliness correction factor. This refers to the cleanliness score of the road surface.

Images