Road surface abnormal target detection and safety prompting system and method based on multi-sensor fusion

Abstract

The invention aims to provide a road surface abnormal target detection and prompt system and method based on multi-sensor fusion, so as to solve the problem that detection and friendly prompt for occurred road anomalies (such as animal remains, falling objects, road surface damage and the like) are lacked in the prior art. While the detection precision of the vehicle on the road abnormal target in the driving process is improved, the system emphasizes a human-friendly prompting mode which takes traffic safety as guidance: when the abnormal condition of the front road surface is detected, the system can remind a driver to pay attention to avoidance with a neutral and safe prompt word; the use of blessing languages that may cause emotional fluctuations or distraction is avoided. The invention also aims to reduce the misjudgment rate through multi-sensor information fusion, dynamically adjust the prompt strategy according to the vehicle driving state and the driver load, and provide effective warning on the premise of not interfering normal driving. In addition, data recording and automatic reporting of road abnormal events are expected to be achieved, reference is provided for road maintenance and traffic management, and obstacles are removed in time to eliminate potential safety hazards.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving assistance technology for vehicles, and in particular to an in-vehicle system and method for detecting abnormal targets on the road surface and providing human-centered safety prompts to the driver during vehicle operation. Background Technology

[0002] Current autonomous driving and advanced driver assistance systems (ADAS) primarily focus on detecting and avoiding pedestrians, vehicles, and conventional obstacles, with relatively insufficient handling of unusual targets such as animals and falling objects on the road. Some existing technologies have proposed using onboard sensors to detect animals on the road and take avoidance measures. For example, patent CN118434614A discloses an autonomous driving vehicle technology capable of detecting animals on the road based on perception data and maintaining a predetermined safe distance when passing them. Additionally, Sharma et al. used image algorithms in a driver assistance system to detect animals such as cows on highways, achieving a detection accuracy of approximately 80%. However, these technologies mainly focus on collision prevention, lacking detection and feedback mechanisms for animals already run over or obstacles left behind, and also failing to provide user-friendly prompts for driver safety and psychological considerations.

[0003] On the other hand, relying solely on a single sensor (such as a camera) for animal or obstacle detection is prone to false alarms. Shadows, plastic bags, or puddles on the road can all be misjudged as dangerous targets. Multi-sensor fusion technology combines data from cameras with radar and other sensors, significantly improving detection accuracy and reducing false triggers. Research shows that multi-sensor fusion target detection can improve system robustness, reduce false alarms and false negatives, and enhance the completeness of environmental perception. Therefore, there is an urgent need for a new in-vehicle system and method that can not only integrate multi-sensor information to improve the accuracy of identifying abnormal targets on the road, but also provide timely and appropriate safety warnings to the driver based on actual driving conditions, thereby improving driving safety and the driving experience. Summary of the Invention

[0004] The purpose of this invention is to provide a road anomaly detection and alert system and method based on multi-sensor fusion, addressing the lack of detection and user-friendly alerts for existing road anomalies (such as animal remains, fallen objects, and road surface damage). While improving the accuracy of vehicle detection of road anomalies during driving, this invention emphasizes a human-centered, safety-oriented alert approach: when an anomaly is detected ahead, the system can remind the driver to take evasive action with neutral and safe language, avoiding potentially emotionally charged or distracting language. This invention also aims to reduce the false alarm rate through multi-sensor information fusion and dynamically adjust the alert strategy based on vehicle driving status and driver load, providing effective warnings without interfering with normal driving. Furthermore, this invention aims to achieve data recording and automatic reporting of road anomaly events, providing a reference for road maintenance and traffic management, and promptly removing obstacles and eliminating safety hazards. Technical solution

[0005] To achieve the above objectives, this invention provides a vehicle-mounted road anomaly detection and alert system and method. The system hardware includes a forward-facing camera, millimeter-wave radar, a processing unit, a storage unit, and an alert output unit, which work together to detect and warn of abnormal targets on the road surface. Specifically:

[0006] Multi-sensor data acquisition: During vehicle operation, a forward-facing camera acquires real-time images of the road ahead; a millimeter-wave radar scans for targets ahead, obtaining measurement data such as their distance and speed. The processing unit receives synchronous sensing information from the camera and radar.

[0007] Fusion Recognition Algorithm: The processing unit employs a fusion algorithm to jointly analyze camera images and radar data to identify suspected abnormal targets on the road ahead. For example, image recognition algorithms are used to detect suspicious target areas on the road surface, which are then verified by combining distance and stationary features measured by radar. When the classification confidence of a visual candidate target is higher than a preset threshold (e.g., η=0.8) and the radar detects a nearly stationary object at the corresponding location (e.g., radial velocity), the recognition algorithm is applied. If the radar cross-section (ε = 0.5 m / s) is within a certain range, it is considered a genuine anomalous target. If no radar point corresponds to the candidate area detected by the camera in either the spatial or temporal neighborhood, it is considered a false detection caused by shadows, reflections, etc., and is therefore filtered out. Non-dangerous objects can be further filtered out by setting thresholds for target size, shape features, etc. The multi-sensor fusion strategy effectively combines the high-resolution recognition capability of the camera with the precise ranging and velocity measurement capability of the radar, improving the reliability of detecting abnormal obstacles such as animal remains and fallen objects in complex scenes.

[0008] Human-centered design prompts: The system dynamically adjusts the form and intensity of prompts based on the vehicle's current driving parameters (such as speed, acceleration, and steering angle). When an abnormal road condition is detected ahead, the processing unit selects a safety prompt from a preset prompt library and alerts the driver via voice, dashboard, or head-up display with messages such as "Abnormal road conditions ahead, please be cautious." The prompt library defaults to neutral, safety-oriented prompts, avoiding religious or plea-like expressions to minimize negative impact on the driver's mood. For high-load driving conditions such as high-speed driving, the system can reduce the voice volume or only use a flashing HUD icon for low-interference warnings; for situations with greater safety margins, such as low speeds or parking, a full voice prompt can be played supplemented with text information to ensure the driver is fully aware of the situation ahead. Through human-centered design prompt strategies, this invention warns of driving risks while minimizing disruption to driving operations.

[0009] Event Recording and Linkage: Upon detecting abnormal road events, the system of this invention records and stores information such as the time of occurrence, geographical location, target type, and identification result in the vehicle's onboard storage unit. If the vehicle has communication capabilities, the relevant information can be automatically reported to the road maintenance or traffic management platform via the onboard wireless communication module for subsequent obstacle removal and accident prevention analysis. This automatic reporting mechanism enables highway maintenance departments to promptly obtain information on abnormal road conditions, quickly organize cleanup operations, and reduce the duration of road safety hazards.

[0010] In summary, the present invention improves the accuracy and robustness of abnormal target detection on roads through multi-sensor fusion, employs a human-friendly dynamic prompting method to ensure the prompting effect without interfering with normal driving, and introduces event logging and network reporting functions to promote traffic safety management. This system and method are highly practical, significantly improving driving safety and providing data support for road maintenance and animal protection.

[0011] Technical Effects: This invention utilizes a combined camera and radar sensing method to effectively improve the accuracy and reliability of detecting abnormal road conditions. Through an adaptive alert strategy based on driving status and driver workload, it avoids the interference caused by harsh alarms, enhancing the system's safety and user-friendliness. The alert language library defaults to neutral safety alerts while also catering to the personalized needs of users from different cultural backgrounds, maintaining a calm tone while highlighting dangers. The event information recording and automatic reporting functions provide data support and linkage possibilities for road safety maintenance and animal protection. In summary, this invention has significant innovation and positive effects in improving vehicle driving safety and road environment management. Attached Figure Description

[0012] Figure 1This is a schematic diagram of the vehicle-mounted road anomaly detection and warning system described in this invention. 1 is a forward-facing camera for real-time acquisition of images of the road surface ahead of the vehicle; 2 is a millimeter-wave radar for acquiring distance and speed information of targets ahead; 3 is a processing unit for fusing and analyzing camera and radar data and performing target recognition and decision-making; 4 is a storage unit for storing recognition algorithm programs, threshold parameters, and warning message libraries; 5 is a warning output unit for outputting selected warning information to the driver via voice broadcast, instrument display, or other means. The system also acquires vehicle speed, acceleration, steering wheel angle, and other driving parameters through the vehicle's internal network (such as a CAN bus) for use by the processing unit. All components work together to achieve the detection and warning function for abnormal road events.

[0013] Figure 2 This is a flowchart of the road anomaly detection and alert method described in this invention, including steps S1 to S9. The specific process is as follows: S1 System startup and sensor data acquisition; S2 Data preprocessing; S3 Multi-sensor fusion detection and identification of abnormal targets on the road ahead; S4 Target temporal consistency verification; S5 Target morphology and texture feature analysis and filtering; S6 Driver load estimation; S7 Alert output method selection; S8 Alert information generation and output; S9 Abnormal event recording and reporting. Detailed Implementation

[0014] The system and method of the present invention will be further described below with reference to the accompanying drawings and embodiments. The system structure of this embodiment is as follows: Figure 1 As shown, the vehicle's onboard system includes a camera 1, a millimeter-wave radar 2, a processing unit 3, a storage unit 4, and a prompt output unit 5. The functions of each unit are as described above. In practical applications, the system operates as follows:

[0015] Step S1: System Startup and Data Acquisition. After the vehicle starts, the system activates the multi-sensor perception module. The forward-facing camera 1 begins continuously acquiring image frames of the road ahead, while the millimeter-wave radar 2 simultaneously scans the area ahead to obtain information such as the distance and speed of targets. In preparation for subsequent data fusion, the processing unit timestamps the data from the camera and radar and performs spatial coordinate calibration, mapping it uniformly to the vehicle coordinate system or the ground coordinate system to ensure consistency of data from different sensors in time and space.

[0016] Step S2: Data Preprocessing. The processing unit preprocesses the acquired multi-source data to improve information quality. Operations such as denoising, enhancement, and distortion correction are performed on the camera image data. For example, Gaussian filtering is used to smooth image noise, contrast is adjusted, and lens distortion is eliminated using camera calibration parameters. Point clouds or echo signals obtained from radar are filtered for denoising and subjected to cluster analysis (e.g., using the DBSCAN algorithm) to remove environmental noise and isolated point targets. After preprocessing, the camera and radar data are aligned, synchronized, and cleaned, providing a reliable input foundation for subsequent fusion and recognition.

[0017] Step S3: Multi-sensor fusion detection and identification. In this step, the processing unit combines the spatial information provided by the millimeter-wave radar with the visual features provided by the camera to detect and identify candidate abnormal targets on the road ahead. Specifically, based on the pre-calculated camera-radar coordinate transformation relationship, the target points detected by the radar are projected onto the image plane, obtaining the possible location regions of the targets in the camera image. Using the fused data, the system employs deep learning models (such as convolutional neural networks) or traditional machine vision algorithms (such as histogram of oriented gradients (HOG) features + cascaded classifiers) to detect targets on the road ahead. The visual algorithm outputs the bounding boxes of candidate targets and their category confidence scores Pᶜ. If the confidence score of an abnormal target in a candidate region is higher than a preset threshold η (e.g., η = 0.8), it is preliminarily determined to be a possible abnormal object. At the same time, the radar module identifies stationary targets at close range and with extremely low relative speeds in front of the vehicle based on the echo signals for auxiliary verification. For example, by detecting the radial velocity of the radar target. If satisfied If ε = 0.5 m / s, the target is considered essentially stationary (possibly an obstacle left on the road). Through the aforementioned camera and radar detection, a series of visual candidate boxes and radar target points are obtained. Next, the processing unit matches the two spatially: based on the calibration relationship between the camera and radar, each radar target is projected onto the image coordinate system to obtain the corresponding point. Check if it falls within a certain visual candidate box. Internal. If there exists a radar point j that satisfies... If a candidate region Bᵢ is found to be a real target, then it is considered to have radar confirmation and can be determined as such. Conversely, if no corresponding radar target is found in a visual candidate region at adjacent times and locations, then the candidate region is considered to be a false detection caused by shadows, reflections, etc., and is therefore eliminated. This multi-sensor cross-validation significantly improves the accuracy of abnormal target detection and reduces the probability of false alarms caused by shadows, plastic bags, and other false targets. During the fusion judgment process, the system can also calculate a comprehensive confidence score S to quantify the probability that the target is a real obstacle. For example, combining visual detection confidence... Radar echo intensity Environmental factor weights Factors such as the target's relative velocity normalized value v̂ and the target's temporal stability parameter τ are used to construct a fusion scoring function, and the score S is calculated using normalization functions such as the Sigmoid function. When S exceeds a threshold θ, the target is determined to be a real road surface anomaly. The above fusion detection strategy fully utilizes the advantages of multiple sensors, significantly improving the system's ability to perceive road anomalies under complex weather and lighting conditions.

[0018] Step S4: Target Temporal Consistency Verification. To avoid interference from transient noise, the system performs multi-frame temporal verification on each detected target candidate. The processing unit tracks the positional changes of candidate targets in consecutive video frames. Only when a target appears in multiple consecutive frames and its position remains stable is it considered to have temporal stability and retained. If a target appears only in a single frame or its position fluctuates, it is considered transient interference and ignored. Through this multi-frame temporal matching, interference factors that appear briefly and disappear quickly on the road surface (such as fleeting pedestrian shadows, vehicle reflections, etc.) can be filtered out, preventing false alarms caused by instantaneous image noise. Formally, a multi-frame stability determination module can be set in the processing unit to calculate the intersection-union ratio (IoU) of the bounding boxes of the target in K consecutive frames and determine whether the IoU is higher than a predetermined threshold δ and the morphological changes are within the allowable range. If the conditions are met, the target is considered to have stable temporal consistency.

[0019] Step S5: Target Shape and Texture Feature Filtering. For candidate targets that pass the time-series verification, the system further analyzes their shape contours and surface textures to eliminate those that clearly do not conform to the characteristics of real objects. Generally, real animal remains or fallen objects have relatively complete shape contours and rich surface textures, while false targets such as shadows and puddles on the road surface are usually scattered in shape, have blurred edges, and have simple textures. The processing unit can constrain and judge the targets based on preset morphological and texture feature thresholds. For example, setting a minimum area threshold for the target region. and texture complexity threshold Only when the pixel area of ​​the candidate target region And texture feature values Only when the condition is met is the target considered a genuine road anomaly; otherwise, it is identified as a false alarm and filtered out. Here, texture feature values ​​can be represented by the variance of the gray-level distribution in the region or some statistical measure of the texture gradient. If necessary, the system can also dynamically adjust the above thresholds or introduce an attention mechanism based on ambient lighting to reduce the interference of strong light reflection and shadow areas on detection, ensuring that only targets that conform to the morphological characteristics of actual obstacles are judged as valid road anomaly events.

[0020] Step S6: Once a real abnormal target is confirmed, the system acquires the vehicle's current driving state parameters to assess the driver's instantaneous workload. The processing unit continuously collects data such as vehicle speed v, acceleration a, and steering wheel angle change Δθ, and inputs this data into a pre-established driver cognitive load assessment model. For example, a linear weighted model can be used to calculate the driver's load index L:

[0021] in and The normalized baseline values ​​for changes in vehicle speed, acceleration, and steering angle (e.g., typical upper limits for each parameter under normal driving conditions) are used, with α, β, and γ as weighting coefficients to balance the impact of different factors on driving load. The load index L calculated by the above model reflects the intensity of the current driving task. A high L value indicates that the driver is under high load conditions such as high-speed driving or emergency maneuvers; a low L value indicates that the current driving is relatively easy. The processing unit uses this to determine the degree of permissible interference from the driver.

[0022] Step S7: Selecting the Prompt Output Method. Based on the estimated driver load index, the system adaptively selects an appropriate prompt output method and intensity, aiming to alert the driver to abnormalities ahead without causing unnecessary interference. When the driver load L is low, the system prioritizes direct and obvious prompts to ensure timely detection. For example, the system can clearly announce the prompt through the car audio system, or highlight a warning icon on the head-up display (HUD). If necessary, it can also supplement this with slight vibrations of the seat or steering wheel (tactile cues) to alert the driver to abnormalities ahead through multiple sensory channels. When the driver load L is high (such as during complex road conditions or emergency operations), the system selects a low-intrusion prompt method to avoid increasing the driver's burden. In this case, it can provide a silent visual warning (such as a flashing icon) on the instrument panel or HUD, or use a short audio prompt at a reduced volume, or use slight steering wheel vibration to prompt the driver, without forcibly playing a long audio message. The intensity (such as sound volume, icon brightness, vibration amplitude) and duration T of the prompt method are also limited according to the load situation to ensure that they are within a predetermined range (not exceeding the preset maximum intensity and duration). Dynamic adjustment. Through the aforementioned human-factor adaptive prompt output module, the system can flexibly select prompt methods in different driving situations, improving the warning effect while minimizing interference with driving operations.

[0023] Step S8: Generation and Output of Prompt Messages. After determining the prompt mode, the processing unit selects specific prompt content from the prompt message library in the storage unit. By default, the system prioritizes neutral statements with a safety warning nature, such as "Abnormal road conditions ahead, please be aware and avoid them." These prompts are objectively worded, reminding drivers to pay attention to road conditions and take measures, while avoiding excessive emotional appeal or religious overtones to prevent emotional fluctuations in the driver. Of course, some reassuring or personalized statements can also be pre-set in the prompt message library according to user needs, but it must be ensured that they do not interfere with safety while driving. The selected prompt information is provided to the driver in an appropriate form through the prompt output unit 5: for example, converted into speech by the speech synthesis module and broadcast through the in-vehicle speaker, while / or displayed as a text reminder on the dashboard or HUD. In this embodiment, when the vehicle is traveling at high speed, the system only prompts "There is a foreign object ahead" through a flashing icon on the HUD, and after the vehicle slows down, it supplements the message with "There is an obstacle ahead, please slow down and avoid it." Through this hierarchical prompting, the driver can obtain the necessary information in a timely manner, improve alertness, and safely deal with abnormal situations ahead.

[0024] Step S9: Abnormal Event Recording and Reporting (Optional). Whenever the system detects an abnormal road event, it records and archives the relevant information. On one hand, the processing unit saves data such as the event's timestamp, GPS location information, abnormal target type (e.g., animal remains, fallen objects), image evidence, and the vehicle's driving status at the time to the onboard storage unit 4 for future reference in accident analysis, liability determination, or algorithm improvement. On the other hand, if the vehicle has network connectivity or is connected to a cloud platform, the system can automatically generate an event report and upload it to road maintenance or traffic management departments via a wireless communication module. The report includes key information such as the event location, time, and target description, facilitating timely notification and action by relevant departments. For example, upon receiving the report, the road maintenance unit can quickly dispatch personnel to clear obstacles or repair damaged road surfaces, thus eliminating potential safety hazards at their inception. Through the automatic reporting mechanism, this invention achieves linkage between vehicle-side detection and management, helping relevant departments to quickly intervene and improve road safety management efficiency.

[0025] The above steps are executed sequentially or triggered cyclically under specific conditions, collectively constituting the road anomaly detection and alert method of this invention. Through the organic combination of the above steps, this invention can accurately identify abnormal obstacles on the road, such as run-over animal remains and fallen objects, and provide timely warnings to the driver in an appropriate manner based on the vehicle's driving environment and the driver's condition, thereby significantly improving driving safety. Simultaneously, the recorded and reported event data also provides strong support for road maintenance and traffic safety analysis, demonstrating the application value of this invention in real-world road traffic scenarios.

Claims

1. A method for detecting and alerting abnormal road surface targets during vehicle operation, characterized in that, include: The vehicle's onboard camera collects image data of the road ahead, while the radar sensor collects distance and speed data of targets ahead. The camera image data and radar data are fused using a fusion algorithm to identify suspected abnormal targets on the road ahead. Thresholds are set based on the size or shape characteristics of the targets to filter out non-dangerous targets. The form and intensity of the prompt output are dynamically adjusted according to the vehicle's driving status parameters. When an abnormal target is determined to be present, a prompt statement is selected from a preset prompt statement library and output to the driver through voice, display, or other means to remind them to avoid the target.

2. The method according to claim 1, characterized in that, Also includes: The relevant information of the detected abnormal events (including the time and location of the event and the identified target type or result) is recorded and stored, and reported to the road maintenance or traffic management system when the vehicle is connected to the network, for subsequent obstacle removal and analysis.

3. A vehicle-mounted road surface anomaly detection and alert system, characterized in that, include: Cameras are used to collect image information of the road surface in front of the vehicle; Radar sensors are used to detect the distance and speed of targets in front of the vehicle; A processing unit, connected to the camera and radar sensor, is used to execute the steps of the method of claim 1; The storage unit is used to store the program algorithm, threshold parameters, and prompt language library data required to execute the method; the prompt output unit is used to output the prompt information to the driver, and the prompt output unit includes a vehicle audio system and / or a display device to prompt abnormal situations ahead in voice and / or visual form.

4. The system according to claim 3, characterized in that, The processing unit dynamically adjusts the prompting method and intensity based on vehicle speed, acceleration, and steering angle: when the vehicle is driving smoothly, it uses voice broadcasts or obvious visual / tactile prompts; when the driving conditions are complex, it uses silent images or weak tactile prompts to reduce interference with the driver.

5. The system according to claim 3, characterized in that, It also includes an event logging module, which records information such as the time, location, target type, and vehicle condition of detected abnormal road events, and provides a data interface to upload the information to an external management platform.

6. A computer-readable storage medium having stored thereon computer-executable instructions that, when executed by the processing unit of claim 3, enable the processing unit to perform the steps of the method of claim 1.

7. The method according to claim 1 or 2, characterized in that, The fusion algorithm includes the following steps: (1) Based on the spatial calibration relationship between the camera and the radar, the coordinates of the point cloud data collected by the radar are projected onto the camera image coordinate system to achieve consistent matching of the target's spatial position; (2) According to the target's image classification confidence score Pᶜ, radar echo intensity Pʳ, environmental weight λw, normalized velocity parameter v̂ and temporal consistency parameter τ, the target's fusion confidence score S is calculated and normalized by the Sigmoid function. When the score S is greater than the preset threshold θ, the target is determined to be a real road surface anomaly.

8. The system according to claim 3 or 4, characterized in that, The processing unit includes: (1) a multi-frame stability determination module, used to calculate the intersection-over-union ratio (IoU) of the bounding boxes of the target in K consecutive frames of images, and determine whether the IoU is higher than the threshold δ and the morphological feature constraint parameters are satisfied. (1) Meets the preset conditions to filter out real targets that are stable in time sequence; (2) Human factor adaptive output module, used to calculate the intensity I and duration T of the prompt output based on the vehicle operating status. The intensity I is determined by factors such as vehicle speed v, acceleration a, steering angular velocity ω and cabin noise level N, and the presentation mode of the prompt is dynamically adjusted within the limited range of I and T, so as to reduce the interference to driving operation while ensuring the warning effect.

Images