Forward road surface detection and riding early warning method and device based on image and radar fusion, electronic equipment and computer readable storage medium

Abstract

The invention relates to the technical field of intelligent driving and riding assistance, in particular to a forward road surface detection and riding early warning method and device based on image and radar fusion, electronic equipment and a computer readable storage medium. The method provided by the invention comprises the following steps: collecting forward road surface data through a millimeter wave radar and a monocular camera; carrying out fusion processing on the point cloud data acquired by the millimeter wave radar and the image data acquired by the monocular camera; according to the detected road condition type, the road condition area, the road condition distance and the current vehicle speed, early warning information is generated in combination with an early warning strategy defined by a user; early warning is sent to a rider through an early warning device of the vehicle machine system, and meanwhile road condition early warning information is reported to a cloud platform. According to the method, the accuracy and the reliability of road surface detection are improved, personalized early warning configuration is realized, the timeliness and the effectiveness of early warning are guaranteed, cloud sharing and continuous optimization of data are realized, and the method has good adaptability and expansibility.

Description

Technical Field

[0001] This invention relates to the fields of intelligent driving and cycling assistance technology, and in particular to a method, device, electronic device, and computer-readable storage medium for forward road detection and cycling warning based on image and radar fusion. Background Technology

[0002] With the increasing popularity of small vehicles such as electric two-wheelers and tricycles, riding safety has become a growing concern. In actual riding, road conditions are complex and varied, presenting numerous potential hazards such as potholes, puddles, ice, oil stains, and sand. Currently, riders primarily rely on visual observation to identify these road risks, a method with significant limitations: in rainy or foggy weather, at night, or in poor lighting conditions, visual observation capabilities are greatly reduced, making it difficult to accurately identify road anomalies; even when anomalies are observed, riders can only rely on personal experience for risk assessment, lacking scientific basis and prone to misjudgment.

[0003] Existing technologies include several road surface detection solutions, but most have significant shortcomings. Some solutions use a single sensor; for example, pure vision solutions are unreliable in adverse weather conditions, while pure radar solutions cannot accurately identify road surface materials. While some solutions employ multi-sensor fusion, they are primarily designed for four-wheeled vehicles, resulting in complex sensor configurations and high costs, making them unsuitable for smaller vehicles such as two-wheeled and three-wheeled vehicles. Furthermore, existing solutions lack effective warning mechanisms and personalized user configuration functions, failing to meet the actual needs of different riders. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a forward road surface detection and cycling warning method, device, electronic device and computer-readable storage medium based on image and radar fusion. Through low-cost sensor combination and intelligent fusion algorithm, it can realize real-time and accurate road condition detection and personalized warning, thereby improving cycling safety.

[0005] In a first aspect, the present invention provides a forward road surface detection and cycling warning method based on image and radar fusion, comprising the following steps: Forward road surface data is collected by a millimeter-wave radar and a monocular camera installed on an electric two-wheeler or three-wheeler. The millimeter-wave radar collects point cloud data, including distance, azimuth, pitch, radial velocity and reflection intensity information, and the monocular camera collects image data. The point cloud data acquired by the millimeter-wave radar and the image data acquired by the monocular camera are fused, including: associating and mapping the 3D coordinate point cloud of the millimeter-wave radar with the 2D pixel coordinates of the monocular camera through a joint calibration method, performing normalization calculation to obtain fusion parameters, and projecting the radar point cloud onto the image bounding box or back-projecting from the image region onto the point cloud space based on the fusion parameters to detect the road condition type of the road surface ahead, wherein the road condition type includes at least one of potholes, water accumulation, ice, oil stains, and sand. Based on the detected road condition type, road area, road distance, and current vehicle speed, and combined with the user-defined warning strategy, a warning message is generated. The warning strategy is configured through external devices and synchronized to the vehicle system, including road condition level classification and corresponding warning measures. The warning measures include at least one of voice broadcast, screen display, and beeping sound. The vehicle's infotainment system sends warnings to riders and simultaneously reports road condition warnings to the cloud platform. These warnings include road condition type, level, location, area, and timestamp.

[0006] In an optional implementation, the fusion processing of the point cloud data acquired by the millimeter-wave radar and the image data acquired by the monocular camera includes: The range, angle, and reflection intensity information acquired by the millimeter-wave radar are processed using 3D-FFT to generate a target point cloud. The range calculation formula is as follows: R = (c × n) r ) / (2×B×Fs r ); The formula for calculating azimuth is: θ = arcsin((λ×n) a ) / (d×n)); The formula for calculating reflection intensity is: I = 20 × log 10 (|RDMA) Cube [r, d, a]|)+g; Where R is the target distance, c is the speed of light, and n r Fs represents the peak range of the FFT, B represents the radar frequency modulation bandwidth, and Fs represents the peak range of the FFT. r Where n is the sampling frequency, θ is the target angle, and n is the target angle. a denoted as FFT angle peak, d as receiving antenna spacing, n as number of antennas, I as reflection intensity, r as distance index, d as Doppler index, a as antenna index, and g as calibration number, determined through laboratory calibration. Road surface materials can be distinguished based on the range of characteristic values ​​of reflection intensity; Image data captured by a monocular camera is input into a pre-trained model for road condition recognition, training data is labeled, and optimization is performed for electric two-wheeled or three-wheeled vehicle scenarios, outputting the road condition type and bounding box in the image; By fusing parameters, radar point clouds and image bounding boxes are mapped bidirectionally: when the radar detects abnormal reflection intensity, the corresponding point cloud is projected onto the image bounding box, and the model is used to verify the road condition type; when the model detects a suspected road condition, the image bounding box is back-projected onto the point cloud space to check whether the reflection intensity is abnormal, so as to confirm the road condition type.

[0007] In an optional implementation, the road condition type detection further includes calculating road area and road distance based on fused data: Road distance is calculated directly from the spatial location of millimeter-wave radar point clouds; The road area is calculated by the relationship between the number of pixels in the image bounding box and spatial projection. Combined with camera calibration parameters and radar ranging data, the actual physical area is obtained.

[0008] In an optional implementation, the user-defined early warning strategy configuration includes: The system provides a road condition definition and configuration interface through external devices. Users can add road condition types and associate them with road surface area, water depth, friction coefficient range, operating speed and hazard level. The friction coefficient is preset by the system based on laboratory test data. Users cannot modify it but can provide accuracy scores and feedback. The system provides an interface for configuring early warning measures through external devices. Users can set whether to enable early warning, screen light effect warning, custom sound effect warning, buzzer sound warning, and whether to repeat warnings within a specified time period for different road condition levels. After configuration, the traffic warning policy is sent to the vehicle system via the synchronized traffic warning policy to vehicle button.

[0009] In an optional implementation, the method further includes: The vehicle-mounted system reports road condition warnings to the cloud platform, including road condition type, level, location, area, timestamp, and raw image data; The cloud platform uses geolocation information to retrieve the cycling tracks of subscribed users within the past week. If the track matches the warning location, a warning message is pushed to the subscribed user. The cloud platform collects and reports image data, which is then labeled and corrected by the annotation team. The model is then retrained using the corresponding vehicle model, and the updated model version is pushed to the vehicle through the management system.

[0010] In an optional implementation, the generation of the warning information includes: Based on road condition type, road area, road distance, and current vehicle speed, a local strategy model is used to match the warning level. If the road conditions are potholes and the area is larger than the preset area, or the water depth is greater than the preset depth, or the ice surface friction coefficient is lower than the preset coefficient, a high-level warning will be triggered. Depending on the warning level and the warning measures configured by the user, the system may trigger voice broadcast content, screen color changes, or beeping frequency.

[0011] In an optional implementation, the method is applicable to cycling scenarios with a vehicle speed of no more than 25 km / h, and the acquisition, fusion, and early warning processing of the millimeter-wave radar and monocular camera are completed in 3-4 business cycles within 1680 milliseconds to ensure real-time performance.

[0012] Secondly, the present invention provides a forward road surface detection and cycling warning device based on image and radar fusion, comprising: The data acquisition module is used to acquire forward road surface data through a millimeter-wave radar and a monocular camera installed on an electric two-wheeler or three-wheeler. The millimeter-wave radar acquires point cloud data, including distance, azimuth angle, pitch angle, radial velocity and reflection intensity information, and the monocular camera acquires image data. The data fusion module is used to fuse the point cloud data acquired by the millimeter-wave radar and the image data acquired by the monocular camera. The fusion process includes: mapping the 3D coordinate point cloud of the millimeter-wave radar to the 2D pixel coordinates of the monocular camera through a joint calibration method, performing normalization calculation to obtain fusion parameters, and projecting the radar point cloud onto the image bounding box or back-projecting it from the image region to the point cloud space based on the fusion parameters to detect the road condition type of the road surface ahead. The road condition type includes at least one of potholes, water accumulation, ice, oil stains, and sand. The warning generation module is used to generate warning information based on the detected road condition type, road area, road distance and current vehicle speed, combined with user-defined warning strategies. The warning strategies are configured through external devices and synchronized to the vehicle system, including road condition level classification and corresponding warning measures. The warning measures include at least one of voice broadcast, screen display and beeping sound. The warning execution module is used to issue warnings to riders through the warning device of the vehicle system; The cloud communication module is used to report traffic warning information to the cloud platform, wherein the traffic warning information includes traffic type, level, location, area and timestamp.

[0013] Thirdly, the present invention provides an electronic device including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the forward road detection and cycling warning method based on image and radar fusion as described in any of the foregoing embodiments.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the forward road surface detection and cycling warning method based on image and radar fusion as described in any of the foregoing embodiments.

[0015] Compared with existing technologies, the technical advantages of the forward road surface detection and cycling warning method based on image and radar fusion provided by this invention are as follows: By fusing millimeter-wave radar and a monocular camera, the system comprehensively leverages the advantages of radar in distance detection and material recognition, as well as the strengths of cameras in visual recognition and type differentiation, achieving complementary advantages. Especially in adverse environments such as rain, fog, and poor lighting conditions, it can accurately identify various road conditions including potholes, water accumulation, ice, oil stains, and sand, overcoming the limitations of single-sensor solutions and improving the accuracy and reliability of road surface detection. Warning strategies can be configured via external devices (such as a mobile app) and synchronized to the vehicle's infotainment system, allowing users to customize road condition levels and corresponding warning measures (voice broadcast, screen display, beeping sounds, etc.) according to their riding habits and risk preferences. This personalized configuration greatly enhances the targeting and practicality of warnings, improves the user experience, and enables personalized warning settings. Based on multi-dimensional information such as detected road condition type, area, distance, and current speed, combined with the user-configured warning strategies, warning information is generated and promptly issued to the rider through the vehicle's infotainment system. This comprehensive warning mechanism, which considers multiple factors, provides riders with more scientific and accurate decision support, ensuring the timeliness and effectiveness of warnings. Reporting road condition warning information (including type, level, location, area, timestamp, etc.) to the cloud platform not only provides warning information sharing for other riders but also provides data support for continuous system optimization, forming a virtuous cycle of data and a system evolution mechanism, achieving cloud-based data sharing and continuous optimization. This method is specifically designed for small vehicles such as electric two-wheelers and three-wheelers, with a simple and practical sensor configuration, controllable cost, and good adaptability to the installation needs of different vehicle types. At the same time, the cloud-based data sharing and update mechanism provides a solid foundation for the system's functional expansion.

[0016] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A system service module diagram provided for embodiments of the present invention; Figure 2 A system flowchart provided for embodiments of the present invention; Figure 3 This is a schematic diagram of the device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0019] Icons: 310 - Data Acquisition Module; 320 - Data Fusion Module; 330 - Early Warning Generation Module; 340 - Early Warning Execution Module; 350 - Cloud Communication Module; 400 - Memory; 401 - Processor; 402 - Bus; 403 - Communication Interface. Detailed Implementation

[0020] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0022] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0023] Furthermore, the technical solutions of the various embodiments can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0024] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings.

[0025] This embodiment provides a forward road surface detection and cyclist warning method based on image and radar fusion, including the following steps: Forward road surface data is collected using a millimeter-wave radar and a monocular camera mounted on an electric two-wheeler or three-wheeler. The millimeter-wave radar collects point cloud data, including distance, azimuth, pitch, radial velocity, and reflection intensity information, while the monocular camera collects image data. The point cloud data collected by the millimeter-wave radar and the image data collected by the monocular camera are fused, including: associating and mapping the 3D coordinate point cloud of the millimeter-wave radar with the 2D pixel coordinates of the monocular camera using a joint calibration method, performing normalization calculations to obtain fusion parameters, and projecting the radar point cloud onto the image bounding box based on the fusion parameters. Alternatively, the image region can be back-projected into the point cloud space to detect the road condition type of the road ahead, which includes at least one of potholes, water accumulation, ice, oil stains, and sand. Based on the detected road condition type, road area, road distance, and current vehicle speed, and combined with a user-defined warning strategy, a warning message is generated. The warning strategy is configured through external devices and synchronized to the vehicle's infotainment system, including road condition level classification and corresponding warning measures, which include at least one of voice broadcast, screen display, and beeping. The warning is issued to the rider through the warning device of the vehicle's infotainment system, and the road condition warning information is reported to the cloud platform, which includes road condition type, level, location, area, and timestamp.

[0026] In this embodiment, the fusion processing of millimeter-wave radar and a monocular camera comprehensively utilizes the advantages of radar in distance detection and material recognition, as well as the strengths of cameras in visual recognition and type differentiation, achieving complementary advantages. Especially in adverse environments such as rain, fog, and poor lighting conditions, it can still accurately identify various road conditions such as potholes, water accumulation, ice, oil stains, and sand, overcoming the limitations of single-sensor solutions and improving the accuracy and reliability of road surface detection. Warning strategies are configured via external devices (such as a mobile app) and synchronized to the vehicle's infotainment system, allowing users to customize road condition levels and corresponding warning measures (voice broadcast, screen display, beeping sounds, etc.) according to their riding habits and risk preferences. This personalized configuration greatly enhances the targeting and practicality of warnings, improves user experience, and enables personalized warning configuration. Based on multi-dimensional information such as detected road condition type, area, distance, and current vehicle speed, combined with the user-configured warning strategies, warning information is generated and promptly issued to the rider through the vehicle's infotainment system. This warning mechanism, which comprehensively considers multiple factors, provides riders with more scientific and accurate decision support, ensuring the timeliness and effectiveness of warnings. Reporting road condition warning information (including type, level, location, area, timestamp, etc.) to the cloud platform not only provides warning information sharing for other riders but also provides data support for continuous system optimization, forming a virtuous cycle of data and a system evolution mechanism, achieving cloud-based data sharing and continuous optimization. This method is specifically designed for small vehicles such as electric two-wheelers and three-wheelers, with a simple and practical sensor configuration, controllable cost, and good adaptability to the installation needs of different vehicle types. At the same time, the cloud-based data sharing and update mechanism provides a solid foundation for the system's functional expansion.

[0027] In this embodiment, a fusion scheme of 20Hz millimeter-wave radar and 30FPS monocular camera is adopted, without introducing lidar or polarized light camera. Under controllable cost, it can cover 95% of daily cycling road conditions.

[0028] Specifically, the system hardware configuration includes: one 20Hz millimeter-wave radar, one 30FPS monocular camera, and one smartphone. The millimeter-wave radar and monocular camera are installed near the headlights at a height of approximately 0.8m above the ground and a downward tilt angle of approximately 5°, covering a ground area of ​​5–25m in front. The system is suitable for riding scenarios with a vehicle speed of ≤25km / h, in which the vehicle travels 6.94m per second, and the sensing range is 5–25m away from the vehicle.

[0029] Forward road surface data is collected using millimeter-wave radar and a monocular camera mounted on an electric two-wheeler or three-wheeler. The millimeter-wave radar collects point cloud data at a frequency of 20Hz, including five categories of information: distance, azimuth, elevation, radial velocity, and reflection intensity. Distance is used to determine the proximity of the risk location, azimuth / elevation angles determine the target's direction, radial velocity is used to determine the target's relative speed, and reflection intensity is used to preliminarily infer the target's material. The monocular camera collects image data at a frequency of 30FPS. Under the condition that the electric two-wheeler or three-wheeler is used at a speed ≤25km / h in 95% of the scenarios, the vehicle travels 6.94m per second. From the start of the warning to the rider's reaction and the triggering of pressing the brake lever, the entire process can be completed within approximately 1.2 seconds, which is equivalent to traveling 8.33m at a speed of 6.94m / s.

[0030] In this embodiment, the fusion processing of point cloud data acquired by millimeter-wave radar and image data acquired by a monocular camera includes millimeter-wave radar detection processing. Millimeter-wave radar detection is divided into three stages: transmission, reception, and analysis. The information received in the reception stage includes five categories: distance, azimuth, elevation, radial velocity, and reflection intensity. Distance is used to determine the distance to the risk location; azimuth / elevation angle is used to determine the direction of the target; radial velocity is used to determine the relative speed of the target (a core indicator in collision warning scenarios); and reflection intensity is used to preliminarily infer the target material (high intensity indicates metal, ice, etc.; low intensity indicates soft asphalt, oil, plastic, loose sand, etc.).

[0031] Specifically, the point cloud data acquired by millimeter-wave radar and the image data acquired by a monocular camera are fused. This includes: performing 3D-FFT processing on the distance, angle, and reflection intensity information acquired by the millimeter-wave radar to generate a target point cloud. That is, the target point cloud is generated using CFAR technology by performing 3D-FFT on the radar data (i.e., the distance, angle, and intensity information from the previous step). The distance calculation formula is: R = (c × n) r ) / (2×B×Fs r The formula for calculating the azimuth angle is: θ = arcsin((λ × n)). a The formula for calculating reflection intensity is: I = 20 × log (d × n) / (d × n). 10 (|RDMA) Cube [r, d, a]|)+g; where R is the target distance, c is the speed of light, nr is the FFT distance peak, B is the radar frequency modulation bandwidth, Fsr is the sampling frequency, θ is the target angle, na is the FFT angle peak, d is the receiving antenna spacing, n is the number of antennas, I is the reflection intensity, r is the distance index, d is the Doppler index, a is the antenna index, and g is the calibration number, determined through laboratory calibration; the road surface material is distinguished based on the range of reflection intensity characteristic values; the range of relative reflection intensity (compared to ordinary dry asphalt road surface) characteristic values ​​is shown in the table below:

[0032] Image data captured by a monocular camera is input into a pre-trained model for road condition recognition. Training data is labeled and optimized for electric two-wheelers or three-wheelers, outputting the road condition type and bounding box in the image. A bidirectional mapping is performed between the radar point cloud and the image bounding box using fusion parameters: when the radar detects abnormal reflection intensity, the corresponding point cloud is projected onto the image bounding box, and the model is used to verify the road condition type; when the model detects a suspected road condition, the image bounding box is back-projected onto the point cloud space to check if the reflection intensity is abnormal, thus confirming the road condition type. Road condition type detection also includes calculating the road condition area and distance based on the fused data: the road condition distance is directly calculated from the spatial location of the millimeter-wave radar point cloud; the road condition area is calculated from the pixel count of the image bounding box and the spatial projection relationship, combined with camera calibration parameters and radar ranging data to obtain the actual physical area.

[0033] Specifically, the process involves fusing point cloud data acquired by millimeter-wave radar and image data acquired by a monocular camera, including image recognition processing. Building upon radar detection, the camera captures road surface images at a specified frequency. These images are then fed into a self-trained YOLO recognition model. The YOLO model selected is the YOLO11s NPU version, which reduces the number of parameters and uses parallel convolutional design, improving the detection accuracy and recognition time for complex small targets. In the single-camera scenario of electric two-wheeled vehicles, it supports the detection of complex road conditions while effectively balancing latency and power consumption. The LabelImg annotation tool is used to annotate the ground truth bounding boxes of the road condition images. The annotated data is then used to train the YOLO11s model, generating a YOLO11s model for multi-target road condition recognition. camera The road condition detection model is based on the positional fusion of radar point cloud data and image recognition pixel data. The forward-facing millimeter-wave radar and camera are mounted in fixed positions on a rigid vehicle frame to avoid relative displacement during vehicle vibration. For different vehicle models, the relative positions and angles of the two devices are maintained. Based on the fixed millimeter-wave radar and camera, a joint calibration method is used to map the radar's (x, y, z) axis points to the camera's image pixels (u, v). OpenCV is used to normalize and calculate the fusion parameters for the 3D radar coordinate point cloud and 2D pixel coordinates. During real-time driving, the 3D position of targets with abnormal reflected signals detected by the radar is obtained by transforming the projection to obtain the bounding box of the 2D image. The image data within the bounding box is processed using YOLOv11s. camera The model performs image detection, merges the two outputs to classify road conditions, or when YOLOv11s... cameraWhen the model detects a "suspected puddle", it obtains the point cloud information of the 3D radar by transforming the projection of the image pixel area, detects whether the reflection intensity of the corresponding point is abnormal, and fuses the two outputs to classify the road condition.

[0034] In the optional technical solution of this embodiment, the user-defined early warning strategy configuration includes: providing a road condition definition configuration interface through an external device, where users can add road condition types and associate them with road surface area, water depth, friction coefficient range, vehicle speed, and hazard level. The friction coefficient is preset by the system based on laboratory test data and cannot be modified by the user, but the user can provide accuracy feedback; providing an early warning measure configuration interface through an external device, where users can set whether to enable early warning, screen light effect warning, custom sound effect warning, buzzer sound warning, and whether to repeat warning within a specified time period for different road condition levels; after configuration, the strategy is sent to the vehicle system via a button to synchronize the road condition early warning strategy to the vehicle.

[0035] In an optional implementation, the method further includes: the vehicle system reporting road condition warning information to a cloud platform, including road condition type, level, location, area, timestamp, and raw image data; the cloud platform retrieving the cycling trajectories of subscribed users within the past week based on geographic location information, and if the trajectory matches the warning location, pushing the warning information to the subscribed user; the cloud platform collecting the reported image data, having the annotation team perform correction annotations, retraining the model using the corresponding vehicle model recognition model, and pushing the updated model version to the vehicle through the management system.

[0036] In an optional implementation, the generation of the warning information includes: matching the warning level using a local strategy model based on the road condition type, road area, road distance, and current vehicle speed; triggering a high-level warning if the road condition is a pothole with an area greater than a preset area, or the water depth is greater than a preset depth, or the ice surface friction coefficient is lower than a preset coefficient; and triggering voice broadcast content, screen color changes, or beeping frequency based on the warning level and the warning measures configured by the user.

[0037] In an optional implementation, the method is applicable to cycling scenarios with a vehicle speed of no more than 25 km / h, and the acquisition, fusion, and early warning processing of the millimeter-wave radar and monocular camera are completed in 3-4 business cycles within 1680 milliseconds to ensure real-time performance.

[0038] The detailed design of the user-defined configuration function is as follows: Figure 1As shown, the user-configurable functions are divided into three categories. The first category is the road condition warning strategy configuration, which includes road condition levels and warning measures. This configuration is made through the user's mobile APP, and the configured strategy takes effect in the vehicle's infotainment system. The second category is the YOLO 11s model configuration. The cloud platform monitors the vehicle model and hardware information pushed by the BOM production line system, maintains the corresponding vehicle model's software version information, generates OTA upgrade tasks for the user's vehicle through the OTA management system, and manages the model version OTA upgrade process through the state machine. The third category is the "user subscribes to road condition warning information" configuration. This configuration is made through the user's mobile APP, and after configuration, it is pushed to users point-to-point through in-app message reminders or SMS channels.

[0039] like Figure 2As shown, the detailed system process steps are as follows: The intelligent system listens to the vehicle model hardware messages pushed by the production line BOM system, establishes vehicle model and component version library information, and manages the static association data of twin vehicles for vehicle model hardware and software; it collects different road condition data for different vehicle model hardware capabilities, performs corresponding software recognition model data annotation and model training, and generates recognition sub-models for different road conditions for corresponding hardware versions; the user logs into the mobile APP and binds the vehicle information for mission authentication; enters the APP's road condition warning event subscription function page, and subscribes to road condition warning information through the "Subscribe to frequently used road section road condition warning information" button; enters the APP's road condition definition configuration page, creates a new road condition definition item, and selects the corresponding road type (icy road) under the road condition item. The system configures road condition warning measures for various road types, including uneven surfaces, potholes, oil-stained surfaces, and loose surfaces. Specific parameters are configured based on the road type, including road area, water depth, friction coefficient (derived from the road type, with different friction coefficient ranges abstracted for different road conditions; this coefficient is derived from extensive laboratory and real-world testing and cannot be modified by the user, but accuracy feedback is available), vehicle speed, and road condition hazard level. Users can then access the APP's road condition warning configuration page, select the added road condition level, and choose from options such as whether to issue a warning, screen light effect warning, custom sound effect warning, buzzer sound warning, and whether to issue a repeated warning within a specified time period. After configuring the mobile APP, users can synchronize the road condition warning strategy to the vehicle's infotainment system by clicking the "Synchronize Road Condition Warning Strategy to Vehicle" configuration button. The system displays the latest vehicle infotainment model for the current vehicle on the mobile app's vehicle model configuration management page. Users can trigger an OTA (Over-The-Air) event by clicking "Update Vehicle Infotainment Model" to update the road condition recognition model to the vehicle system. When the user starts and drives the vehicle, the system loads the local recognition model. During the ride, the millimeter-wave radar collects data such as Doppler intensity and reflection intensity at its designated frequency, while the camera collects road image information at its designated frequency. The vehicle model then identifies the collected data, combining the millimeter-wave radar reflection results and camera image recognition results to obtain the corresponding road condition type, point cloud data, and image segmentation data. Based on the millimeter-wave radar detection distance and image pixel segmentation, a high-confidence road condition type and road condition classification are calculated. Information such as distance and road area is collected; based on the road condition type and current vehicle speed, a local strategy model is used to match the warning level and issue warnings according to the corresponding emergency measures; if the warning measure is a custom light effect warning, the light warning is issued through the vehicle's interactive screen according to the preset light effect; if the warning measure is a custom sound effect warning, the sound effect warning is played through the vehicle's speaker according to the preset sound effect; if the warning measure is a sharp buzzing sound warning, the sound effect warning is broadcast through the vehicle's speaker according to the preset buzzing sound effect; at the same time as triggering the vehicle-side warning measures, the vehicle system organizes the identified road condition risk level, area, type, vehicle location, system time, raw image data and other information, and reports the warning information to the cloud platform through the T-Box;The cloud platform receives road condition warnings reported by vehicles. Based on the geographical location information of the reported events, it performs a geographic search on the recent week's trajectories of users who have subscribed to road condition warnings. If a match is found, information such as the road condition warning type, level, location, area, and recommended measures is pushed to the subscribed users. The cloud platform also receives raw image data reported by vehicles. The model annotation team corrects and annotates the data, then uses the annotated data to train the corresponding hardware recognition model. The trained model undergoes laboratory testing and road testing. Once the testing is successful, the model version is updated to the OTA version system, and an OTA update event is pushed to users of the corresponding vehicle models. If the user has not customized settings for the road condition hazard level, the system's built-in road condition definitions and corresponding warning methods are used by default. Users only need to set whether to enable or disable warnings.

[0040] Based on the above embodiments, this invention provides a forward road surface detection and cycling warning device based on image and radar fusion, see [link to relevant documentation]. Figure 3 The illustrated embodiment of the present invention provides a structural schematic diagram of a forward road surface detection and cycling warning device based on image and radar fusion. The device includes: a data acquisition module 310, used to acquire forward road surface data via a millimeter-wave radar and a monocular camera mounted on an electric two-wheeler or three-wheeler. The millimeter-wave radar acquires point cloud data, including distance, azimuth, pitch, radial velocity, and reflection intensity information; the monocular camera acquires image data. A data fusion module 320 is used to fuse the point cloud data acquired by the millimeter-wave radar and the image data acquired by the monocular camera. This fusion process includes: mapping the 3D coordinate point cloud of the millimeter-wave radar to the 2D pixel coordinates of the monocular camera using a joint calibration method, performing normalization calculations to obtain fusion parameters, and projecting the radar point cloud onto the image boundary based on the fusion parameters. A bounding box or back-projection from an image region onto a point cloud space is used to detect the road condition type of the road ahead, wherein the road condition type includes at least one of potholes, water accumulation, ice, oil stains, and sand; a warning generation module 330 is used to generate warning information based on the detected road condition type, road condition area, road condition distance, and current vehicle speed, combined with a user-defined warning strategy, wherein the warning strategy is configured through an external device and synchronized to the vehicle system, including road condition level classification and corresponding warning measures, wherein the warning measures include at least one of voice broadcast, screen display, and buzzer sound; a warning execution module 340 is used to issue a warning to the rider through the warning device of the vehicle system; a cloud communication module 350 is used to report the road condition warning information to the cloud platform, wherein the road condition warning information includes road condition type, level, location, area, and timestamp.

[0041] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the forward road surface detection and cycling warning device based on image and radar fusion described above can be referred to the corresponding process in the aforementioned embodiments of the forward road surface detection and cycling warning method based on image and radar fusion, and will not be repeated here.

[0042] This invention also provides an electronic device for running a forward road detection and cycling warning method based on image and radar fusion; see [link to related documentation]. Figure 4 The schematic diagram of an electronic device provided by the embodiment of the present invention shown below includes a memory 400 and a processor 401. The memory 400 is used to store one or more computer instructions, which are executed by the processor 401 to realize the above-mentioned forward road detection and cycling warning method based on image and radar fusion.

[0043] Furthermore, Figure 4 The electronic device shown also includes a bus 402 and a communication interface 403. The processor 401, the communication interface 403 and the memory 400 are connected via the bus 402.

[0044] The memory 400 may include high-speed random access memory (RAM) 400, and may also include non-volatile memory 400, such as at least one disk storage device 400. Communication between this system network element and at least one other network element is achieved through at least one communication interface 403 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 402 may be an ISA bus 402, a PCI bus 402, or an EISA bus 402, etc. The bus 402 can be divided into an address bus 402, a data bus 402, a control bus 402, etc. For ease of representation, Figure 4 The symbol is represented by only one double-headed arrow, but this does not mean that there is only one bus 402 or one type of bus 402.

[0045] Processor 401 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed through integrated logic circuits in the hardware of processor 401 or through software instructions. The processor 401 may be a general-purpose processor 401, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor 401 may be a microprocessor 401, or it may be any conventional processor 401. The steps of the method disclosed in the embodiments of the present invention can be directly manifested as being executed by the hardware decoding processor 401, or executed by a combination of hardware and software modules in the decoding processor 401. The software modules can reside in a random access memory 400, flash memory, read-only memory 400, programmable read-only memory 400, electrically erasable programmable memory 400, registers, or other mature storage media in the art. This storage medium is located in the memory 400, and the processor 401 reads information from the memory 400 and, in conjunction with its hardware, completes the steps of the method in the aforementioned embodiments.

[0046] This invention also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by the processor 401, they cause the processor 401 to implement the aforementioned forward road detection and cycling warning method based on image and radar fusion. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0047] The computer program products of the forward road detection and cycling warning method, device and electronic device based on image and radar fusion provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0048] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and / or device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0049] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0050] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A forward road surface detection and cycling warning method based on image and radar fusion, characterized in that, Including the following steps: Forward road surface data is collected by a millimeter-wave radar and a monocular camera installed on an electric two-wheeler or three-wheeler. The millimeter-wave radar collects point cloud data, including distance, azimuth, pitch, radial velocity and reflection intensity information, and the monocular camera collects image data. The point cloud data acquired by the millimeter-wave radar and the image data acquired by the monocular camera are fused, including: associating and mapping the 3D coordinate point cloud of the millimeter-wave radar with the 2D pixel coordinates of the monocular camera through a joint calibration method, performing normalization calculation to obtain fusion parameters, and projecting the radar point cloud onto the image bounding box or back-projecting from the image region onto the point cloud space based on the fusion parameters to detect the road condition type of the road surface ahead, wherein the road condition type includes at least one of potholes, water accumulation, ice, oil stains, and sand. Based on the detected road condition type, road area, road distance, and current vehicle speed, and combined with the user-defined warning strategy, a warning message is generated. The warning strategy is configured through external devices and synchronized to the vehicle system, including road condition level classification and corresponding warning measures. The warning measures include at least one of voice broadcast, screen display, and beeping sound. The vehicle's infotainment system sends warnings to riders and simultaneously reports road condition warnings to the cloud platform. The road condition warning information includes road condition type, level, location, area, and timestamp.

2. The forward road surface detection and cycling warning method based on image and radar fusion according to claim 1, characterized in that, The fusion processing of the point cloud data acquired by the millimeter-wave radar and the image data acquired by the monocular camera includes: The range, angle, and reflection intensity information acquired by the millimeter-wave radar are processed using 3D-FFT to generate a target point cloud. The range calculation formula is as follows: R=(c×n r ) / (2×B×Fs r ); The formula for calculating azimuth is: θ=arcsin((λ×n) a ) / (d×n); The formula for calculating reflection intensity is: I=20×log 10 (|RDMA Cube [r,d,a]|)+g; Where R is the target distance, c is the speed of light, and n r Fs represents the peak range of the FFT, B represents the radar frequency modulation bandwidth, and Fs represents the peak range of the FFT. r Where n is the sampling frequency, θ is the target angle, and n is the target angle. a denoted as FFT angle peak, d as receiving antenna spacing, n as number of antennas, I as reflection intensity, r as distance index, d as Doppler index, a as antenna index, and g as calibration number, determined through laboratory calibration. Road surface materials can be distinguished based on the range of characteristic values ​​of reflection intensity; Image data captured by a monocular camera is input into a pre-trained model for road condition recognition, training data is labeled, and optimization is performed for electric two-wheeled or three-wheeled vehicle scenarios, outputting the road condition type and bounding box in the image; By fusing parameters, radar point clouds and image bounding boxes are mapped bidirectionally: when the radar detects abnormal reflection intensity, the corresponding point cloud is projected onto the image bounding box, and the model is used to verify the road condition type; when the model detects a suspected road condition, the image bounding box is back-projected onto the point cloud space to check whether the reflection intensity is abnormal, so as to confirm the road condition type.

3. The forward road surface detection and cycling warning method based on image and radar fusion according to claim 2, characterized in that, The road condition type detection also includes calculating road condition area and road condition distance based on fused data: Road distance is calculated directly from the spatial location of millimeter-wave radar point clouds; The road area is calculated by the relationship between the number of pixels in the image bounding box and spatial projection. Combined with camera calibration parameters and radar ranging data, the actual physical area is obtained.

4. The forward road surface detection and cycling warning method based on image and radar fusion according to claim 1, characterized in that, The user-defined early warning strategy configuration includes: The system provides a road condition definition and configuration interface through external devices. Users can add road condition types and associate them with road surface area, water depth, friction coefficient range, operating speed and hazard level. The friction coefficient is preset by the system based on laboratory test data. Users cannot modify it but can provide accuracy scores and feedback. The system provides an interface for configuring early warning measures through external devices. Users can set whether to enable early warning, screen light effect warning, custom sound effect warning, buzzer sound warning, and whether to repeat warnings within a specified time period for different road condition levels. After configuration, the traffic warning policy is sent to the vehicle system via the synchronized traffic warning policy to vehicle button.

5. The forward road surface detection and cycling warning method based on image and radar fusion according to claim 1, characterized in that, The method further includes: The vehicle-mounted system reports road condition warnings to the cloud platform, including road condition type, level, location, area, timestamp, and raw image data; The cloud platform uses geolocation information to retrieve the cycling tracks of subscribed users within the past week. If the track matches the warning location, a warning message is pushed to the subscribed user. The cloud platform collects and reports image data, which is then labeled and corrected by the annotation team. The model is then retrained using the corresponding vehicle model, and the updated model version is pushed to the vehicle through the management system.

6. The forward road surface detection and cycling warning method based on image and radar fusion according to claim 1, characterized in that, The generation of the early warning information includes: Based on road condition type, road area, road distance, and current vehicle speed, a local strategy model is used to match the warning level. If the road conditions are potholes and the area is larger than the preset area, or the water depth is greater than the preset depth, or the ice surface friction coefficient is lower than the preset coefficient, a high-level warning will be triggered. Depending on the warning level and the warning measures configured by the user, the system may trigger voice broadcast content, screen color changes, or beeping frequency.

7. The forward road surface detection and cycling warning method based on image and radar fusion according to claim 1, characterized in that, The method is applicable to cycling scenarios with a speed not exceeding 25km / h, and the acquisition, fusion, and early warning processing of the millimeter-wave radar and monocular camera are completed within 1680 milliseconds for 3-4 business cycles to ensure real-time performance.

8. A forward-facing road surface detection and cycling warning device based on image and radar fusion, characterized in that, include: The data acquisition module (310) is used to acquire forward road surface data by means of a millimeter-wave radar and a monocular camera installed on an electric two-wheeler or three-wheeler, wherein the millimeter-wave radar acquires point cloud data, including distance, azimuth angle, pitch angle, radial velocity and reflection intensity information, and the monocular camera acquires image data; The data fusion module (320) is used to fuse the point cloud data acquired by the millimeter-wave radar and the image data acquired by the monocular camera. The fusion process includes: mapping the 3D coordinate point cloud of the millimeter-wave radar to the 2D pixel coordinates of the monocular camera through a joint calibration method, performing normalization calculation to obtain fusion parameters, and projecting the radar point cloud onto the image bounding box or back-projecting it from the image region to the point cloud space based on the fusion parameters to detect the road condition type of the road surface in front, wherein the road condition type includes at least one of potholes, water accumulation, ice, oil stains, and sand. The warning generation module (330) is used to generate warning information based on the detected road condition type, road area, road distance and current vehicle speed, combined with the user-defined warning strategy. The warning strategy is configured through an external device and synchronized to the vehicle system, including road condition level classification and corresponding warning measures. The warning measures include at least one of voice broadcast, screen display and beeping sound. The warning execution module (340) is used to issue a warning to the rider through the warning device of the vehicle system; The cloud communication module (350) is used to report traffic warning information to the cloud platform, wherein the traffic warning information includes traffic type, level, location, area and timestamp.

9. An electronic device, characterized in that, It includes a processor (401) and a memory (400), the memory (400) storing computer-executable instructions that can be executed by the processor (401), the processor (401) executing the computer-executable instructions to implement the forward road detection and cycling warning method based on image and radar fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when invoked and executed by the processor (401), cause the processor (401) to implement the forward road surface detection and cycling warning method based on image and radar fusion as described in any one of claims 1 to 7.

Images