Road surface foreign matter processing method, system and equipment and computer readable storage medium

Abstract

The invention discloses a road surface foreign matter processing method, system and device and a computer readable storage medium, and relates to the field of sweeper intelligent control, and the road surface foreign matter processing method comprises the following steps: obtaining training data, training a pre-constructed terminal recognition model for recognizing road surface foreign matters according to the training data, and obtaining a road surface foreign matter recognition model; training a pre-constructed decision model of an output cleaning strategy according to the trained terminal identification model; road surface and road surface foreign matter data before each time of sweeping are obtained, and the road surface and road surface foreign matter data before each time of sweeping are input into the trained terminal recognition model to output corresponding foreign matter recognition information; and inputting the corresponding foreign matter identification information into a pre-constructed decision model to output a cleaning strategy, and cleaning the foreign matters on the road surface according to the cleaning strategy. According to the method, the road surface and the road surface foreign matter data are obtained, the corresponding sweeping strategies are output according to the road surface and the road surface foreign matter data, different foreign matters on the road can be swept in a targeted mode, the sweeping purpose is achieved, and resources are saved.

Description

Technical Field

[0001] This invention relates to the field of intelligent control of sweepers, specifically to methods, systems, equipment, and computer-readable storage media for handling foreign objects on the road surface. Background Technology

[0002] Current trends in the intelligentization of sweepers primarily focus on path planning. A common approach relies on a single sensor to perceive road conditions, such as a camera. The camera captures and identifies the road, allowing the sweeper to plan its cleaning path based on the actual road conditions.

[0003] However, this approach does not obtain information about foreign objects on the road surface. During the cleaning process, a uniform cleaning strategy is used to clean the road. The undifferentiated cleaning approach will result in areas that are difficult to clean not being thoroughly cleaned, while areas that are easy to clean will consume excessive resources and increase economic costs. Summary of the Invention

[0004] This invention provides a method, system, device, and computer-readable storage medium for handling foreign objects on the road surface, which can solve the problem that existing technologies cannot thoroughly clean areas that are difficult to clean, while excessive resource consumption and increased economic costs occur in areas that are easy to clean.

[0005] In a first aspect, embodiments of the present invention provide a method for handling foreign objects on a road surface, comprising the following steps: Acquire training data, train a pre-built terminal recognition model for identifying foreign objects on the road surface based on the training data, and train a pre-built decision model for output cleaning strategy based on the trained terminal recognition model. Acquire road surface and foreign object data before each cleaning, input the road surface and foreign object data before each cleaning into the trained terminal recognition model, and output the corresponding foreign object recognition information. The corresponding foreign object identification information is input into the trained decision model to output a cleaning strategy, and the foreign objects on the road are cleaned according to the cleaning strategy.

[0006] In conjunction with the first aspect, in one implementation, the step of inputting the corresponding foreign object identification information into the trained decision model to output a cleaning strategy, and cleaning the foreign objects on the road surface according to the cleaning strategy, includes the following steps: S31, acquire data on the road surface and foreign objects after the road surface is cleaned; S32, determine whether the data on the road surface and the foreign objects after the road surface is cleaned up meet the preset cleaning requirements; S33, if yes, then the cleaning is complete, and the foreign object identification information and the cleaning strategy are saved into the decision model; S34. If not, output a cleaning strategy again based on the road surface and road debris data after the road debris is cleaned, and clean the road debris according to the cleaning strategy. Repeat steps S31-S32 until the cleaning is completed, and save the debris identification information and multiple related cleaning strategies into the decision model.

[0007] In conjunction with the first aspect, in one implementation, the following steps are also included: The terminal recognition model is retrained based on the road surface and foreign object data before each cleaning. The decision model is then retrained based on the retrained terminal recognition model.

[0008] In conjunction with the first aspect, in one implementation, the step of retraining the terminal recognition model based on road surface and foreign object data before each cleaning includes the following steps: The standard identification information corresponding to the road surface and foreign objects data before each cleaning is measured; The terminal recognition model is then retrained based on the road surface and foreign object data before each cleaning and the corresponding standard recognition information.

[0009] In conjunction with the first aspect, in one implementation, if not, then based on the obtained road surface and road debris data after debris removal, a new cleaning strategy is output, and the road debris is cleaned according to this cleaning strategy. Steps S31-S32 are repeated until cleaning is completed, and the debris identification information and related multiple cleaning strategies are saved into the decision model, including the following steps: The corresponding recognition model is updated based on the trained terminal recognition model and the acquired road surface and foreign object data after road surface cleaning. The recognition model update gradient is uploaded to the global recognition model in the cloud. The global recognition model in the cloud is optimized using the recognition model update gradient. The optimized global recognition model is then used to update the terminal recognition model.

[0010] In conjunction with the first aspect, in one implementation, after acquiring training data and training a pre-built terminal recognition model for identifying foreign objects on the road surface based on the training data, the process includes the following steps: The trained terminal recognition model is compressed.

[0011] In conjunction with the first aspect, in one implementation, acquiring road surface and debris data before each cleaning session includes the following steps: The system acquires road surface images and 3D point cloud data of road debris before each cleaning operation using a visual sensor unit, a lidar unit, and a GNSS unit.

[0012] Secondly, embodiments of the present invention provide a road surface foreign object handling system, comprising: The training module is used to acquire training data, train a pre-built terminal recognition model for identifying foreign objects on the road surface based on the training data, and train a pre-built decision model for output cleaning strategy based on the trained terminal recognition model. The identification module is used to acquire road surface and foreign object data before each cleaning, input the road surface and foreign object data before each cleaning into the trained terminal identification model, and output the corresponding foreign object identification information. The execution module is used to input the corresponding foreign object identification information into the pre-built decision model to output a cleaning strategy, and clean the foreign objects on the road according to the cleaning strategy.

[0013] Thirdly, embodiments of the present invention provide a road surface foreign object handling device, the road surface foreign object handling device including a processor, a memory, and a road surface foreign object handling program stored in the memory and executable by the processor, wherein when the road surface foreign object handling program is executed by the processor, the steps of the road surface foreign object handling method are implemented.

[0014] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a road surface foreign object handling program, wherein when the road surface foreign object handling program is executed by a processor, it implements the steps of the road surface foreign object handling method.

[0015] The beneficial effects of the technical solutions provided by the embodiments of the present invention include: This invention discloses a method, system, device, and computer-readable storage medium for handling foreign objects on the road surface. The method includes the following steps: acquiring training data; training a pre-constructed terminal recognition model for identifying foreign objects on the road surface based on the training data; training a pre-constructed decision model for outputting a cleaning strategy based on the trained terminal recognition model; acquiring road surface and foreign object data before each cleaning cycle; inputting the road surface and foreign object data before each cleaning cycle into the trained terminal recognition model to output corresponding foreign object recognition information; inputting the corresponding foreign object recognition information into the pre-constructed decision model to output a cleaning strategy; and cleaning the road surface according to the cleaning strategy. This invention, by acquiring road surface and foreign object data and outputting corresponding cleaning strategies based on the data, can specifically clean different foreign objects on the road, achieving the cleaning purpose while saving resources. Attached Figure Description

[0016] Figure 1 This is a schematic flowchart of an embodiment of the road surface foreign object treatment method of the present invention; Figure 2 For the present invention Figure 1 A detailed flowchart of step S3; Figure 3 This is a schematic diagram of the functional modules of an embodiment of the road surface foreign object handling system of the present invention; Figure 4 This is a schematic diagram of the hardware structure of the road surface foreign object handling device involved in the embodiment of the present invention. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0019] like Figure 1 As shown, in a first aspect, embodiments of the present invention provide a method for handling foreign objects on a road surface, comprising the following steps: S1. Acquire training data, train a pre-built terminal recognition model for identifying foreign objects on the road based on the training data, and train a pre-built decision model for output cleaning strategy based on the trained terminal recognition model. S2, acquire road surface and foreign object data before each cleaning, input the road surface and foreign object data before each cleaning into the trained terminal recognition model and output the corresponding foreign object recognition information. S3: Input the corresponding foreign object identification information into the trained decision model to output a cleaning strategy, and clean the foreign objects on the road according to the cleaning strategy.

[0020] The pre-built terminal recognition model for identifying road foreign objects adopts a multi-modal fusion Transformer architecture. It uses a Vision Transformer (ViT) or Swing Transformer structure instead of a traditional CNN, improving the recognition accuracy and generalization ability for multi-scale and multi-category road foreign objects. Specifically, the terminal recognition model extracts features from the image data of the input road surface and foreign object data using the Swing Transformer backbone; it also extracts features from the voxelized point cloud data of the input road surface and foreign object data using VoxelNet or PointPillar methods. Features from different modalities are fused through cross-attention in the Transformer encoder, and finally, a corresponding foreign object recognition information, including the type, size, coverage, and location information of the foreign object, is output through a detection head (such as a Deformable DETR).

[0021] The decision model for the pre-built output cleaning strategy adopts an adaptive control strategy based on reinforcement learning (RL). The cleaning task is modeled as a Partially Observable Markov Decision Process (POMDP). The state space represents the currently perceived foreign object identification information; the action space represents the cleaning strategy, including: main brush speed, side brush deployment angle, water pressure valve opening, and water pump power; the reward function comprehensively considers cleanliness, water consumption, time consumption, and energy consumption.

[0022] The training data is from large, publicly available datasets, such as ImageNet and COCO.

[0023] The sanitation sweeper truck is equipped with a road debris handling system, featuring multiple sensor units deployed at the front, sides, and rear of the vehicle. These units include visual sensor units, lidar units, millimeter-wave radar, and GNSS units. These sensors enable multimodal perception of road surface and debris data. By integrating the visual sensor units, lidar units, millimeter-wave radar, and GNSS units, a three-dimensional environmental perception capability can be constructed, improving the robustness and all-weather adaptability of road debris detection.

[0024] The visual sensor unit can be a global shutter RGB camera with a resolution of 8 megapixels or higher, which can capture high-definition road surface images at a frame rate of 30fps.

[0025] The lidar unit can be a 16-line or 32-line mechanical or solid-state LiDAR (such as Hesai AT128) used to acquire three-dimensional point cloud data of foreign objects on the road surface, and accurately determine their volume and relative distance to vehicles.

[0026] The millimeter-wave radar may be a 77GHz forward-facing radar, used to detect obstacles and large foreign objects in adverse weather conditions such as rain, fog, and dust, to compensate for the deficiencies of the visual sensor unit and the lidar unit.

[0027] The PPS (pulses per second) signal provided by the GNSS unit is used for synchronization, ensuring that the data in each mode of the acquired road surface and foreign object data are time-aligned. This prevents distortion of the road surface and foreign object data due to timing discrepancies during acquisition.

[0028] For example, at 30s and 31s, the visual sensor unit, lidar unit, and millimeter-wave radar on the sanitation sweeper vehicle collect relevant data. At this time, the GNSS unit provides synchronization information, assigning a corresponding time sequence to the collected data. This allows the data collected by the visual sensor unit, lidar unit, and millimeter-wave radar at 30s to be combined to construct the road surface and debris data for 30s. Without the GNSS unit, the time sequence of the data collected by the visual sensor unit, lidar unit, and millimeter-wave radar in the constructed road surface and debris data will be out of order. For instance, the road surface and debris data from 30s and 31s might be used to construct the road surface and debris data.

[0029] When handling road debris, a pre-built terminal recognition model for identifying road debris is first trained using training data. This training utilizes massive amounts of labeled training data from large public datasets to improve the accuracy of the debris identification information output by the terminal recognition model. After training, the terminal recognition model is "frozen" (its parameters are not updated) in a digital twin simulation environment and used as an environmental sensor. The decision-making model (reinforcement learning agent) learns decision-making strategies by interacting with the simulation environment. This approach decouples perception and decision-making errors, resulting in more stable training. After both the terminal recognition model and the decision-making model are trained, multiple sensor units acquire road surface and debris data before cleaning. This data is input into the trained terminal recognition model, which outputs corresponding debris identification information. This debris identification information is then input into the trained decision-making model, which outputs a cleaning strategy, and the debris is cleaned according to this strategy.

[0030] This invention acquires road surface and foreign object data, and outputs corresponding cleaning strategies based on the data. This allows for targeted cleaning of different foreign objects on the road, achieving the cleaning purpose while saving resources.

[0031] like Figure 2As shown, in one embodiment, the step of inputting the corresponding foreign object identification information into the trained decision model to output a cleaning strategy, and cleaning the foreign objects on the road surface according to the cleaning strategy, includes the following steps: S31, acquire data on the road surface and foreign objects after the road surface is cleaned; S32, determine whether the data on the road surface and the foreign objects after the road surface is cleaned up meet the preset cleaning requirements; S33, if yes, then the cleaning is complete, and the foreign object identification information and the cleaning strategy are saved into the decision model; S34. If not, output a cleaning strategy again based on the road surface and road debris data after the road debris is cleaned, and clean the road debris according to the cleaning strategy. Repeat steps S31-S32 until the cleaning is completed, and save the debris identification information and multiple related cleaning strategies into the decision model.

[0032] In a single cleaning operation, the decision model outputs a cleaning strategy based on the road surface and debris data before cleaning. After cleaning the debris, it re-evaluates whether the road surface and debris data after cleaning meet the preset cleaning requirements. Based on the cleaning results, it determines whether cleaning should end or continue, and saves the relevant cleaning data to the decision model.

[0033] For example, when the terminal recognition model identifies the road surface and debris data as: {There is an oil stain 2 meters ahead, with an area of ​​approximately 0.5 square meters and a confidence level of 0.95}, the decision model outputs a cleaning strategy based on this data: {Adjust the water pressure to 8 Bar, the main brush speed to 120 rpm, and reduce the vehicle speed to 3 km / h}. After cleaning according to this strategy, the terminal recognition model will acquire the cleaned road surface and debris data and determine whether the cleaned road surface and debris data meet the preset cleaning requirements to determine whether the cleaning is complete.

[0034] Furthermore, determining whether cleaning is complete means that when the road debris handling system determines that the debris in the current area has been removed (i.e., the road surface and debris data no longer contain the target debris after cleaning, or the debris coverage rate is lower than a certain threshold), and the expected reward for continuing cleaning no longer increases, the road debris handling system will choose to end the current task or switch to execute the next cleaning task.

[0035] The reward is a comprehensive quantification of indicators during the cleaning process. During cleaning, indicators such as cleanliness, water consumption, time consumption, and energy consumption are quantified into corresponding numerical scores. In one embodiment, the "reward function" is: Total reward = (Cleanliness score) + (Water resource penalty) + (Time penalty) + (Energy consumption penalty) The "cleanliness" rating corresponds to a "big reward" of +50 points, "consuming 1 liter of water" corresponds to a "small penalty" of -1 point, "taking 1 second" corresponds to a "small penalty" of -0.1 points, and "increased overall energy consumption" corresponds to a "small penalty" of -0.5 points.

[0036] During a cleaning process: the road debris removal system identifies an oil stain with an area of ​​2 units, and the road debris removal system has two corresponding cleaning strategies: Cleaning Strategy A: Conservative rinsing (time-consuming, water-intensive but effective), using medium water pressure, rinsing for 15 seconds, consuming 5 liters of water and 3 units of electricity. The grease is completely removed.

[0037] Calculate the reward: Cleanliness score: +50 * 2 = +100 points (2 unit areas were cleaned) Water resource penalty: -1 * 5 = -5 points Time penalty: -0.1 * 15 = -1.5 points Energy consumption penalty: -0.5 * 3 = -1.5 points Total reward: 100 - 5 - 1.5 - 1.5 = +92 points Cleaning Strategy B: Aggressive flushing (fast, water-saving but may not be clean), using high water pressure, flushing for 5 seconds, consuming 2 liters of water and 4 units of electricity.

[0038] Result: Only 80% of the oil stain was removed (0.4 unit area remained).

[0039] Calculate the reward: Cleanliness Bonus: +50 * 1.6 = +80 points (1.6 units of area cleaned) Water resource penalty: -1 * 2 = -2 points Time penalty: -0.1 * 5 = -0.5 points Energy consumption penalty: -0.5 * 4 = -2 points Total reward for this round: 80 - 2 - 0.5 - 2 = +75.5 points The road debris handling system will discover that cleaning strategy B scores lower than cleaning strategy A. Ultimately, the internal decision-making logic of the final settlement model will be adjusted so that when encountering similar oil stains in the future, it will be more likely to choose high-reward actions like "cleaning strategy A".

[0040] By balancing the reward function, the core objective of "cleaning up" and the constraint objective of "saving resources" are unified into an optimizable mathematical objective, so that the road debris handling system can choose a cleaning strategy that is both resource-saving and effective during the cleaning process.

[0041] Saving foreign object identification information and related cleaning strategies into the decision model is a reinforcement learning process. Its purpose is to adjust the parameters within the decision model based on the actual cleaning results and the road surface and foreign object data after cleaning, so that when encountering similar cleaning tasks in the future, it will be more inclined to choose cleaning strategies with higher cumulative rewards (considering not only immediate rewards but also the sum of potential future rewards) to complete the cleaning task.

[0042] For example, if the "high-pressure cleaning of oil stains" is effective and saves water, the decision-making module will remember it and be more likely to use a similar strategy when encountering oil stains in the future; if the effect is poor and wastes water, it will adjust the strategy and use a different cleaning strategy next time.

[0043] This invention determines whether the road surface and debris data after road debris removal meet the preset cleaning requirements and saves the cleaning strategy to the decision model. This allows the decision model to learn during the actual cleaning process and update the cleaning strategy trained in the simulation training with the cleaning strategy that has good actual cleaning effect, thereby gradually optimizing the decision model.

[0044] In one embodiment, the method further includes the following steps: retraining the terminal recognition model based on road surface and foreign object data before each cleaning; and retraining the decision model based on the retrained terminal recognition model.

[0045] In one embodiment, the step of retraining the terminal recognition model based on the road surface and foreign object data before each cleaning includes the following steps: measuring the standard recognition information corresponding to the road surface and foreign object data before each cleaning; and retraining the terminal recognition model based on the road surface and foreign object data before each cleaning and its corresponding standard recognition information.

[0046] The road surface and debris data collected before each cleaning are preprocessed. Image data undergoes automatic white balance, distortion correction, and HDR synthesis; point cloud data undergoes downsampling and noise reduction.

[0047] Then, pre-fusion technology is used to associate foreign object pixels in the image with LiDAR point cloud clusters and radar targets, and annotation tools (such as CVAT and Supervisely) are used for unified annotation. The annotation information includes the type of foreign object on the road (such as plastic bottles, oil stains, fallen leaves, bricks), bounding box, pixel-level segmentation mask, and 3D dimensions.

[0048] Generative adversarial networks (GANs), such as StyleGAN3, are used to generate high-fidelity images of extremely rare scenes (such as large areas of icy roads or mixed mud) to expand the training dataset.

[0049] Digital twin scenes can be built using simulation environments (such as NVIDIA DRIVE Sim, Carla) to generate a large amount of synthetic data with precise labels, thereby covering long-tail scenes at low cost.

[0050] The above steps can measure the standard identification information corresponding to the road surface and foreign objects data before each cleaning. The standard identification information is paired, and the image data and lidar point cloud data collected at the same time and in the same scene must be strictly aligned (time and space synchronization); and labeled, each target foreign object in the data is accurately labeled by manual or semi-automatic tools (such as bounding boxes and category labels).

[0051] Road surface and foreign object data, along with their corresponding standard identification information, form the standard. When training the terminal recognition model, the road surface and foreign object data are input, and the model outputs the corresponding foreign object identification information. Comparing the foreign object identification information corresponding to the same road surface and foreign object data with the standard identification information reveals errors in the terminal recognition model's identification. By adjusting the parameters of the terminal recognition model, the output foreign object identification information is aligned with the standard identification information, further increasing the accuracy of the terminal recognition model. The larger the amount of road surface and foreign object data and its corresponding standard identification information, the better the training effect of the terminal recognition model, ultimately training it until the output foreign object identification information is indistinguishable from the standard identification information. After the terminal recognition model training is complete, the trained model is "frozen" (its parameters are not updated) and used as an environmental sensor. The decision model is then retrained.

[0052] In one embodiment, if not, a new cleaning strategy is output based on the obtained road surface and foreign object data after the foreign object removal, and the foreign object is removed according to the cleaning strategy. Steps S31-S32 are repeated until the cleaning is completed, and the foreign object identification information and related multiple cleaning strategies are saved in the decision model. This includes the following steps: outputting the corresponding identification model update gradient based on the trained terminal identification model and the obtained road surface and foreign object data after the foreign object removal; uploading the identification model update gradient to the global identification model in the cloud; optimizing the global identification model in the cloud using the identification model update gradient; and updating the terminal identification model using the optimized global identification model.

[0053] The road debris handling system collects all difficult case data during the cleaning process. The difficult case data refers to cases where cleaning is unsuccessful, including: 1. Perception failure, where the terminal recognition model makes a mistake or has too low confidence; 2. Decision failure, where the cleaning effect does not meet expectations; 3. Novelty / abnormality, where an unknown pattern or abnormal system status is detected.

[0054] The road debris handling system outputs corresponding recognition model update gradients based on difficult case data and the trained terminal recognition model. While ensuring data privacy, the recognition model update gradients collected from multiple road debris handling systems are uploaded to a cloud platform. On the cloud platform, the global recognition model is continuously iterated and optimized, enabling the optimized global recognition model to resolve cases that were previously unsuccessful during cleaning.

[0055] The cloud platform manages the simulation environment, the global recognition model training pipeline (MLOps), and the federated learning coordination. The road surface foreign object handling system periodically uploads the updated gradients of the recognition model to the cloud platform. The cloud platform automatically triggers global recognition model optimization and pushes the optimized global recognition model to the road surface foreign object handling system via OTA (Over-The-Air) to update the terminal recognition model.

[0056] This approach, which deploys a lightweight model in a road surface foreign object handling system and continuously trains and updates the model on a cloud platform, represents an edge-cloud collaborative computing architecture. Based on this architecture, the road surface foreign object handling system possesses the capability for continuous iterative optimization, enabling it to adapt to new road surface scenarios and foreign object types, thus extending the technology's lifecycle. This edge-cloud collaborative computing architecture involves deploying a lightweight model in the road surface foreign object handling system for real-time inference; and continuously training and updating the global recognition model on the cloud platform, supporting federated learning to protect data privacy.

[0057] This invention improves the cleaning level of the road debris handling system by collecting difficult case data during the cleaning process and using the difficult case data to optimize the global recognition model.

[0058] In one embodiment, after acquiring training data and training a pre-constructed terminal recognition model for identifying foreign objects on the road surface based on the training data, the following steps are included: compressing the trained terminal recognition model.

[0059] The model compression uses knowledge distillation technology to distill a lightweight student recognition model (such as MobileNetV3+SSD) from the terminal recognition model as the teacher recognition model, in order to adapt to the computing power limitations of the on-board computing platform of the sanitation sweeper.

[0060] The in-vehicle computing platform uses a high-performance in-vehicle domain controller as its computing core. NVIDIA Jetson AGX Orin or Qualcomm Snapdragon Ride platforms are recommended, providing at least 200 TOPS of AI computing power. The distilled terminal recognition model is accelerated using TensorRT or ONNX Runtime and deployed on the domain controller, ensuring inference latency is below 100ms.

[0061] The terminal recognition model is a computationally and storage-intensive deep neural network that must be compressed to run in real time on the automotive chip. The global recognition model on the cloud platform is a teacher recognition model. The knowledge from the massive teacher recognition model on the cloud platform is distilled into a small student recognition model, which is then pruned and quantized to ultimately form a terminal recognition model that can run in real time on the automotive chip.

[0062] This invention compresses the terminal recognition model, reducing its size to a size that can run in real time on an in-vehicle chip, thus adapting to the computing power limitations of the in-vehicle computing platform.

[0063] In one embodiment, acquiring road surface and foreign object data before each cleaning includes the following steps: acquiring road surface images and three-dimensional point cloud data of foreign objects before each cleaning using a visual sensor unit, a lidar unit, and a GNSS unit.

[0064] The PPS signal provided by the GNSS unit is used for synchronization, ensuring that the data in each mode of the road surface and road debris data acquired by the visual sensor unit and the lidar unit are aligned in time. This prevents distortion of the road surface and road debris data due to timing errors during the acquisition process.

[0065] This invention synchronizes the visual sensor unit and the lidar unit through the GNSS unit, ensuring that the acquired road surface and road debris data are synchronized in time, thereby further ensuring the accuracy of the data.

[0066] like Figure 3 As shown, in a second aspect, embodiments of the present invention also provide a road surface foreign object handling system, comprising: a training module, which is used to acquire training data, train a pre-constructed terminal recognition model for recognizing road surface foreign objects based on the training data, and train a pre-constructed decision model for outputting a cleaning strategy based on the trained terminal recognition model; an identification module, which is used to acquire road surface and road surface foreign object data before each cleaning, input the road surface and road surface foreign object data before each cleaning into the trained terminal recognition model to output corresponding foreign object recognition information; and an execution module, which is used to input the corresponding foreign object recognition information into the pre-constructed decision model to output a cleaning strategy, and clean the road surface foreign objects according to the cleaning strategy.

[0067] The functions of each module in the above-mentioned road surface foreign object handling system correspond to the steps in the above-mentioned road surface foreign object handling method embodiment, and their functions and implementation processes will not be described in detail here.

[0068] Thirdly, embodiments of the present invention provide a road surface foreign object handling device, which can be a personal computer (PC), laptop computer, server or other device with data processing capabilities.

[0069] Reference Figure 4 , Figure 4 This is a schematic diagram of the hardware structure of a road surface foreign object handling device according to an embodiment of the present invention. In this embodiment, the road surface foreign object handling device may include a processor, a memory, a communication interface, and a communication bus.

[0070] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0071] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used to interconnect devices within the road surface foreign object handling equipment, as well as interfaces used to interconnect the road surface foreign object handling equipment with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0072] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0073] The processor can be a general-purpose processor, which can call a road surface foreign object handling program stored in memory and execute the road surface foreign object handling method provided in the embodiments of the present invention. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the road surface foreign object handling program is called can be referred to in various embodiments of the road surface foreign object handling method of the present invention, and will not be repeated here.

[0074] Those skilled in the art will understand that Figure 4 The hardware structure shown does not constitute a limitation of the invention and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0075] Fourthly, embodiments of the present invention also provide a computer-readable storage medium.

[0076] The present invention provides a computer-readable storage medium storing a road surface foreign object handling program, wherein when the road surface foreign object handling program is executed by a processor, it implements the steps of the road surface foreign object handling method described above.

[0077] The method implemented when the road surface foreign object handling procedure is executed can be referred to in various embodiments of the road surface foreign object handling method of the present invention, and will not be repeated here.

[0078] It should be noted that the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0079] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0080] In the description of the embodiments of the present invention, terms such as "exemplary," "for example," or "for instance" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design that is described as "exemplary," "for example," or "for instance" in the embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0081] In the description of the embodiments of the present invention, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present invention, "multiple" means two or more.

[0082] In some processes described in the embodiments of the present invention, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of the present invention, or may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0083] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of the present invention.

[0084] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for handling foreign objects on a road surface, characterized in that, Includes the following steps: Acquire training data, train a pre-built terminal recognition model for identifying foreign objects on the road surface based on the training data, and train a pre-built decision model for output cleaning strategy based on the trained terminal recognition model. Acquire road surface and foreign object data before each cleaning, input the road surface and foreign object data before each cleaning into the trained terminal recognition model, and output the corresponding foreign object recognition information. The corresponding foreign object identification information is input into the trained decision model to output a cleaning strategy, and the foreign objects on the road are cleaned according to the cleaning strategy.

2. The method for handling foreign objects on a road surface according to claim 1, characterized in that, The step of inputting the corresponding foreign object identification information into the trained decision model to output a cleaning strategy, and cleaning the foreign objects on the road according to the cleaning strategy, includes the following steps: S31, acquire data on the road surface and foreign objects after the road surface is cleaned; S32, determine whether the data on the road surface and the foreign objects after the road surface is cleaned up meet the preset cleaning requirements; S33, if yes, then the cleaning is complete, and the foreign object identification information and the cleaning strategy are saved into the decision model; S34. If not, output a cleaning strategy again based on the road surface and road debris data after the road debris is cleaned, and clean the road debris according to the cleaning strategy. Repeat steps S31-S32 until the cleaning is completed, and save the debris identification information and multiple related cleaning strategies into the decision model.

3. The method for handling foreign objects on a road surface according to claim 1, characterized in that, It also includes the following steps: The terminal recognition model is retrained based on the road surface and foreign object data before each cleaning. The decision model is then retrained based on the retrained terminal recognition model.

4. The method for handling foreign objects on a road surface according to claim 3, characterized in that: The process of retraining the terminal recognition model based on road surface and debris data before each cleaning includes the following steps: The standard identification information corresponding to the road surface and foreign objects data before each cleaning is measured; The terminal recognition model is then retrained based on the road surface and foreign object data before each cleaning and the corresponding standard recognition information.

5. A method for handling foreign objects on a road surface according to claim 2, characterized in that, If not, then based on the obtained road surface and foreign object data after cleaning, another cleaning strategy is output, and the foreign objects are cleaned according to this cleaning strategy. Steps S31-S32 are repeated until cleaning is completed, and the foreign object identification information and multiple related cleaning strategies are saved into the decision model, including the following steps: The corresponding recognition model is updated based on the trained terminal recognition model and the acquired road surface and foreign object data after road surface cleaning. The recognition model update gradient is uploaded to the global recognition model in the cloud. The global recognition model in the cloud is optimized using the recognition model update gradient. The optimized global recognition model is then used to update the terminal recognition model.

6. The method for handling foreign objects on a road surface according to claim 1, characterized in that, After acquiring training data and training a pre-built terminal recognition model for identifying foreign objects on the road surface based on the training data, the process includes the following steps: The trained terminal recognition model is compressed.

7. The method for handling foreign objects on a road surface according to claim 1, characterized in that, The process of acquiring road surface and debris data before each cleaning includes the following steps: The system acquires road surface images and 3D point cloud data of road debris before each cleaning operation using a visual sensor unit, a lidar unit, and a GNSS unit.

8. A road surface foreign object handling system, characterized in that... include: The training module is used to acquire training data, train a pre-built terminal recognition model for identifying foreign objects on the road surface based on the training data, and train a pre-built decision model for output cleaning strategy based on the trained terminal recognition model. The identification module is used to acquire road surface and foreign object data before each cleaning, input the road surface and foreign object data before each cleaning into the trained terminal identification model, and output the corresponding foreign object identification information. The execution module is used to input the corresponding foreign object identification information into the pre-built decision model to output a cleaning strategy, and clean the foreign objects on the road according to the cleaning strategy.

9. A road surface foreign object handling device, characterized in that, The road surface foreign object handling device includes a processor, a memory, and a road surface foreign object handling program stored in the memory and executable by the processor, wherein when the road surface foreign object handling program is executed by the processor, it implements the steps of the road surface foreign object handling method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a road surface foreign object handling program, wherein when the road surface foreign object handling program is executed by a processor, it implements the steps of the road surface foreign object handling method as described in any one of claims 1 to 7.

Images