A multi-sensor fusion lane departure warning method and system

Abstract

The application discloses a lane departure warning method and system based on multi-sensor fusion, adopts a visual sensor to acquire visual data for lane line identification and warning, checks and corrects the visual data acquired by the visual sensor, and performs a departure warning based on the lane line data identified after the checking and correction. The application has the advantages that the single camera scheme is compensated for the disadvantage of low identification rate in bad weather, and the multi-sensor fusion and timing consistency verification improve the reliability of the identification result.

Description

Technical Field

[0001] This invention relates to the field of lane departure warning control, and in particular to a lane departure warning method and system based on multi-sensor fusion. Background Technology

[0002] Current Lane Departure Warning Systems (LDWS) primarily employ a monocular vision camera solution, using image processing algorithms to identify lane lines and issue a warning when the vehicle unintentionally deviates from its lane. The system operates at speeds between 60-140 km / h, displaying a warning image and emitting an alarm sound on the instrument panel or central control screen.

[0003] Existing technologies have poor environmental adaptability. In adverse weather conditions (heavy rain, heavy snow, heavy fog, sandstorms), at night, in strong light, backlight, etc., the imaging capabilities of cameras are severely limited, and the recognition rate drops significantly. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a lane departure warning method and system based on multi-sensor fusion to improve the detection accuracy in adverse environments: solving the problem of lane line recognition under extreme conditions such as rain, snow, fog, haze, night, and strong light.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A multi-sensor fusion lane departure warning method is proposed, which uses a visual sensor to acquire visual data for lane line recognition and warning; the visual data acquired by the visual sensor is verified and corrected, and lane departure warning is issued based on the lane line data identified after verification and correction.

[0007] The visual data is verified and corrected using data detected by vehicle-mounted radar.

[0008] The verification and correction methods include hierarchical verification, where the first level of verification is the internal data verification of the sensor; the second level of verification is cross-sensor verification, which involves spatial registration and consistency checks of data from different sensors.

[0009] The hierarchical verification also includes a third level verification, which is a temporal consistency verification. This third level verification introduces kinematic logic in the time dimension to verify the rationality of lane line recognition fusion results after single-sensor internal verification and cross-sensor cross verification.

[0010] Single-sensor internal validation is used to analyze and verify data acquired by a single sensor.

[0011] A single sensor uses data comparison between preceding and subsequent time-series acquisitions, along with data stability analysis, to filter data and remove outliers. Multiple sensors include vision sensors, millimeter-wave radar, and lidar.

[0012] For visual sensors, multi-frame continuity verification is employed: by comparing the lane line position and curvature changes detected in consecutive image frames, abnormal jump data caused by instantaneous reflections or occlusions are eliminated; for millimeter-wave radar, multi-target tracking verification is performed: the motion trajectory of each detected target is tracked using Kalman filtering, and if the appearance and disappearance of the detected target does not satisfy kinematic continuity, it is judged as noise or false alarm and filtered out; for lidar, point cloud stability verification is performed: the point cloud density and distribution pattern of the same target in consecutive scanning cycles are analyzed, discrete noise points are identified and removed.

[0013] Cross-sensor cross-validation includes: utilizing the complementary physical characteristics of different sensors to perform spatial geometric consistency checks on the sensing results, and validating the data collected by multiple sensors based on the complementary validation results;

[0014] Complementary verification includes geometric consistency verification, feature matching verification, and distance consistency verification.

[0015] The cross-validation of vision and radar adopts geometric consistency verification: the lane line equation recognized by the vision sensor is projected onto the radar sensor coordinate system and compared with the road edge detected by the radar sensor. If the two deviate too much in lateral distance and direction, the confidence level is lowered or rematching is triggered.

[0016] Feature matching verification between visual sensors and LiDAR: Extract lane line corner points and edge features from the image, and register them with the corresponding intensity and height change regions in the LiDAR point cloud to confirm the real existence of visual features in three-dimensional space;

[0017] Range consistency verification between millimeter-wave radar and lidar: For a target that is detected by both, compare the ranging results of radar ranging and lidar point cloud clustering, and determine whether the error between the two is within the set threshold. If not, discard the measurement results.

[0018] The temporal consistency verification includes using historical trajectory prediction to estimate the expected lane position at the current moment based on the vehicle motion state of the previous few frames. If the deviation between the current fusion result and the predicted value exceeds a set threshold range, the sensor data is re-detected and the predicted value is used as the final recognition and perception result for lane line warning control.

[0019] Re-detecting sensor data includes:

[0020] Kinematic model verification: Based on vehicle dynamics constraints, calculate whether the currently detected deviation speed is within the deviation speed threshold range; if the deviation speed exceeds the set deviation threshold range and no corresponding steering wheel angle change is detected, it is determined to be a sensor false detection, and the warning is suppressed at this time;

[0021] Driver Intent Analysis: Input the steering wheel angle, turn signal, accelerator, and brake signals from the most recent historical data into the LSTM network, and output the current driving intent; if the intent is to actively change lanes or avoid obstacles in an emergency, then the warning is disabled; if the intent is to unintentionally deviate, then the warning is triggered normally.

[0022] In cross-sensor verification, feature matching verification is used for visual sensors and LiDAR. This includes: extracting lane line corner points and edge features from the image and registering them with the corresponding intensity and height change regions in the LiDAR point cloud to confirm the real existence of visual features in three-dimensional space.

[0023] The pixels at the edge of the lane line are extracted from the image, and the corresponding ray is generated by calibration parameters. In the LiDAR point cloud, points with abrupt changes in height or reflection intensity are searched along the ray. If a matching point is found and the point cloud distribution is consistent with the edge direction of the image, the verification is successful; otherwise, the confidence of the visual detection result is reduced.

[0024] If the timing consistency verification fails, the lane departure warning output is frozen until the verification succeeds; if the cross-sensor cross-verification fails, the optimal estimate is made based on historical data and kinematic model, and the lane departure warning is issued based on the optimal estimate data.

[0025] A warning system for a lane departure warning method using multi-sensor fusion, characterized in that: the warning system includes a data sensing sensor, a data verification module, a data processing module, and a warning module;

[0026] The data sensing sensor is used to collect visual data and radar data, and its output is connected to the data verification module.

[0027] The data verification module uses a three-level verification method to verify the perceived data. The data that passes the verification is sent to the data processing module. The data processing module obtains the corresponding lane line recognition data based on the verified perceived data and sends it to the warning module. The warning module issues a lane departure warning based on the lane line recognition data.

[0028] The advantages of this invention are: it overcomes the disadvantage of low recognition rate in adverse weather conditions caused by single-camera solutions, and the reliability of recognition results is improved by multi-sensor fusion and temporal consistency verification. The core advantage of this invention lies in the construction of a three-level progressive perception verification mechanism, which systematically solves the shortcoming of insufficient reliability of single-sensor solutions in complex environments.

[0029] First, the multi-sensor fusion architecture physically compensates for the disadvantage of single-camera solutions, which suffer from a sharp drop in recognition rate under adverse weather conditions such as rain, fog, and backlight. Millimeter-wave radar is unaffected by light and rain / fog attenuation, while lidar provides accurate three-dimensional geometric information, strongly complementing the dense semantic capabilities of vision and ensuring a stable baseline for all-weather perception.

[0030] Building upon this foundation, the invention employs a unique three-tiered progressive verification system to filter the fusion results layer by layer: The first tier, single-sensor internal verification, utilizes visual multi-frame continuity, radar kinematic tracking, and lidar point cloud stability analysis to filter out instantaneous noise and false alarms at the source; the second tier, cross-sensor verification, through visual-radar geometric consistency, visual-lidar feature matching, and radar-lidar distance consistency checks, eliminates or downweights results with spatial geometric inconsistencies; the third tier, temporal consistency verification, predicts the current perception result based on historical trajectories, captures and replaces abnormal jumps that violate motion logic with predicted values, ensuring a smooth and reliable final output. This progressive verification and correction, from data quality and spatial consistency to motion logic, enables the perception system to not only possess extremely strong transient anti-interference capabilities but also provide a safety net through redundancy and prediction when some sensors fail, significantly improving the overall intelligent driving system's recognition reliability, decision-making accuracy, and functional safety under complex conditions. Attached Figure Description

[0031] The following is a brief explanation of the contents of each of the accompanying drawings and the markings in the drawings:

[0032] Figure 1 This is a schematic diagram of the early warning system of the present invention. Detailed Implementation

[0033] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and the description of the preferred embodiments.

[0034] Existing technology uses a single camera to identify lane lines and then implements lane departure warnings. However, due to the complexity of the driving environment, this leads to problems such as high false alarm rates and delays. Specific problems include:

[0035] 1. High false alarm rate: The false alarm rate of mainstream products is as high as 2.3 times per thousand kilometers at night. The false alarm rate is even higher on road sections with unclear or incomplete lane markings and many road joints.

[0036] 2. Response delay: Traditional systems require approximately 0.3-0.5 seconds from detection to warning, which may result in insufficient reaction time at high speeds.

[0037] 3. Limited functionality: Functionality is limited or malfunctions in special scenarios such as sharp turns, slopes, bumpy roads, and construction zones.

[0038] 4. The visual sensor detection has no calibration and its accuracy is not high.

[0039] To address these issues, this solution utilizes radar sensor data for cross-verification, building upon existing camera technology. This three-level verification method validates lane line perception and recognition data, improving data reliability and thus enhancing the reliability of lane departure warning.

[0040] like Figure 1 As shown in this embodiment, a multi-sensor fusion lane departure warning method uses a visual sensor to acquire visual data for lane line recognition and warning. The visual data acquired by the visual sensor is verified and corrected, and lane departure warning is issued based on the verified and corrected lane line data. In this embodiment, data detected by onboard radar is used to verify and correct the visual data. The radar can be millimeter-wave radar or lidar. Verification and correction are performed by combining radar-detected data with data collected by the visual sensor itself, through data interaction, improving data accuracy. When the data is accurate and reliable, the lane line warning will be more reliable.

[0041] In this embodiment, the verification and correction method includes hierarchical verification. The first level of verification is internal sensor data verification; the second level of verification is cross-sensor cross-verification, which performs spatial registration and consistency checks on data from different sensors. The third level of verification is temporal consistency verification, which introduces kinematic logic in the time dimension to verify the rationality of lane line recognition fusion results after single-sensor internal verification and cross-sensor cross-verification. The first, second, and third levels of verification are performed according to the temporal sequence. Data that passes the third level of verification is used as input data for lane line warning control.

[0042] Single-sensor internal verification is used to analyze and verify data collected by a single sensor. For vision sensors, multi-frame continuity verification is employed: by comparing the lane line position and curvature changes detected in consecutive image frames, abnormal jump data caused by instantaneous reflections or occlusions is eliminated. For millimeter-wave radar, multi-target tracking verification is performed: the motion trajectory of each detected target is tracked using Kalman filtering; if the appearance and disappearance of a detected target does not satisfy kinematic continuity, it is judged as noise or a false alarm and filtered out. For lidar, point cloud stability verification is performed: the point cloud density and distribution pattern of the same target in consecutive scanning cycles are analyzed, discrete noise points are identified and removed. Single-sensor internal verification refers to analyzing and verifying the original detection results using only the temporal and spatial redundancy and physical constraints of continuously collected data from a single sensor, without introducing external sensor information, thereby achieving noise suppression, false alarm filtering, and abnormal jump repair. This mechanism is the core link of "front-end purification" in the autonomous driving perception stack, which can greatly reduce the pressure on back-end fusion and decision-making. The following sections will take vision, millimeter-wave radar, and lidar as examples to elaborate on the mechanism and implementation of their internal verification. Visual perception is extremely sensitive to changes in lighting, occlusion, and texture. In single-frame images, anomalies such as lane line position jumps and object box flickering often occur due to specular reflections, shadow cutting, or raindrop adhesion. Multi-frame continuity verification utilizes a temporally smooth prior to the road environment, ensuring that the actual lane line curvature and vehicle trajectory do not change abruptly within millisecond intervals. The system describes the lane lines detected in consecutive image frames in a parameterized form and performs temporal filtering on coefficients such as lateral offset and curvature through inter-frame matching to establish a constant curvature or constant speed motion model. A Kalman filter is then used to predict and update the lane line state. When the residual between the detected curvature and the predicted value in a new frame exceeds a dynamic threshold (e.g., 3 times the standard deviation) and the confidence level drops sharply, it is determined that the detection was caused by instantaneous reflection or local occlusion. The system discards this outlier and smoothly fills in the gap with the predicted value. For object detection boxes, IoU and appearance features can be used for inter-frame correlation. If an object box flickers intermittently (appears, disappears, and reappears) across multiple consecutive frames, and the vehicle's own motion cannot explain the change, it is considered a false alarm and is removed. In this way, the continuity and geometric consistency of the visual output are guaranteed.

[0043] Millimeter-wave radar measures the distance, velocity, and angle of targets through electromagnetic wave reflection, but its raw point cloud is filled with clutter: multipath reflections from guardrails and manhole covers, ground stray echoes, and sidelobe effects from metallic targets, easily producing "ghost" targets. The essence of multi-target tracking verification is to utilize the kinematic characteristics of targets to accumulate evidence over time; only targets whose behavior conforms to the laws of physical motion are confirmed as valid. The core is a target tracker based on Kalman filtering or extended Kalman filtering. Each potential track maintains a state vector (position, velocity, acceleration) and a covariance matrix. In each frame, detection points are correlated with existing tracks (e.g., nearest neighbor or joint probability data correlation). Successfully correlated detection points are used for filter updates, predicting the target's position in the next frame and narrowing the search threshold. Track management follows a strict birth and death logic: a new track must be correlated with a detection point N times in M ​​consecutive scans (i.e., M / N logic) to be converted into a confirmed track and output; a confirmed track that fails to be correlated with a detection point multiple times consecutively is eliminated. If a strong reflection point suddenly appears and then disappears, failing to form a trajectory that satisfies kinematic continuity (e.g., a sudden speed change exceeding the vehicle's physical limits), it is judged as noise or a false alarm and not output. Through dual verification of existence continuity and consistent motion parameters, millimeter-wave radar can reduce the false alarm rate by one to two orders of magnitude, retaining only truly stable road participants. LiDAR directly acquires the 3D point cloud of the environment, but its data is also affected by atmospheric scattering, multipath effects, target surface materials, and vehicle vibration, resulting in suspended discrete noise points and transient false targets. Point cloud stability verification starts with the spatiotemporal existence and geometric distribution characteristics of targets, filtering out unreliable measurements. Within a single cycle, a statistical outlier removal algorithm is used for density stability verification: for each point, the average distance to its K nearest neighbors is calculated. Assuming this distance follows a Gaussian distribution, points with distances much greater than the sum of the mean and standard deviation are considered discrete noise points and removed. Simultaneously, analyzing the distribution pattern of targets reveals that the real vehicle surface point cloud exhibits a typical L-shaped or planar distribution. If a point cloud distribution is irregular and the voxel occupancy rate changes drastically between consecutive frames, it is judged as a false target and removed. After point cloud stability verification, the target list and point cloud environment model output by the lidar will significantly reduce transient interference.

[0044] Cross-sensor cross-validation includes: utilizing the complementary physical characteristics of different sensors to check the spatial geometric consistency of perception results. The cross-validation of vision and radar adopts geometric consistency verification: the lane line equation recognized by vision is projected onto the radar coordinate system and compared with the road edge detected by radar. If the two deviate too much in lateral distance and direction, the confidence level is lowered or rematched.

[0045] Visual and LiDAR feature matching verification: Extract lane line corner points and edge features from the image, and register them with the corresponding intensity and height change regions in the LiDAR point cloud to confirm the real existence of visual features in three-dimensional space;

[0046] Range consistency verification between millimeter-wave radar and lidar: For a target that is detected by both, compare the ranging results of radar ranging and lidar point cloud clustering, and determine whether the error between the two is within the set threshold. If not, discard the measurement results.

[0047] The temporal consistency check includes using historical trajectory prediction to estimate the expected lane position at the current moment based on the vehicle's motion state in previous frames. If the deviation between the current fusion result and the predicted value exceeds a set threshold, the sensor data is re-detected, and the predicted value is used as the final recognition and perception result for lane line warning control. In this embodiment, the predicted value is only used as the perception result for lane line warning within a set time range. If the time exceeds the set time threshold and the temporal consistency check still cannot be satisfied, a lane line fault warning is issued to the driver, reminding the driver that the current lane departure function is abnormal.

[0048] In this embodiment, re-detecting the sensor data includes:

[0049] Kinematic model verification: Based on vehicle dynamics constraints, calculate whether the currently detected deviation speed is within the deviation speed threshold range; if the deviation speed exceeds the set deviation threshold range and no corresponding steering wheel angle change is detected, it is determined to be a sensor false detection, and the warning is suppressed at this time;

[0050] Driver Intent Analysis: Input the steering wheel angle, turn signal, accelerator, and brake signals from the most recent historical data into the LSTM network, and output the current driving intent; if the intent is to actively change lanes or avoid obstacles in an emergency, then the warning is disabled; if the intent is to unintentionally deviate, then the warning is triggered normally.

[0051] The visual and lidar feature matching verification also includes: extracting the pixels of the lane line edge from the image, generating the corresponding ray through calibration parameters, and searching for points with height abrupt changes or reflection intensity abrupt changes along the ray in the lidar point cloud. If a matching point is found and the point cloud distribution is consistent with the edge direction of the image, the verification is successful; otherwise, the confidence of the visual detection result is reduced.

[0052] If the timing consistency verification fails, the lane departure warning output is frozen until the verification succeeds; if the cross-sensor cross-verification fails, the optimal estimate is made based on historical data and kinematic model, and the lane departure warning is issued based on the optimal estimate data.

[0053] A warning system for a lane departure warning method using multi-sensor fusion, characterized in that: the warning system includes a data sensing sensor, a data verification module, a data processing module, and a warning module;

[0054] The data perception sensor is used to collect visual and radar data, and its output is connected to the data verification module. The data verification module uses a three-level verification method to verify the perception data. Data that passes verification is sent to the data processing module. The data processing module obtains the corresponding lane line recognition data based on the verified perception data and sends it to the warning module. The warning module issues lane departure warnings based on the lane line recognition data. The hardware of this solution specifically includes:

[0055] Visual sensor: A high dynamic range (HDR) global shutter camera with a resolution of 1280×720 and a frame rate of 30fps is installed on the inside of the windshield to capture images of the road in front of the vehicle.

[0056] Millimeter-wave radar: It adopts a 77GHz long-range radar with a maximum detection range of 200m and a horizontal field of view of ±60°. It is installed in the center of the front bumper and is used to detect road guardrails, curbs and obstacles in front.

[0057] LiDAR: A 16-line mechanical LiDAR with a horizontal field of view of 360° and a vertical field of view of ±15° is installed on the roof of the vehicle to acquire high-precision 3D point cloud data and assist in lane line feature extraction.

[0058] Inertial Measurement Unit (IMU): Integrates a three-axis accelerometer and gyroscope, with a sampling frequency of 100Hz, and is installed at the center of the vehicle chassis to measure the vehicle's yaw rate, lateral acceleration, and attitude angle.

[0059] CAN bus interface: Connects to the vehicle's CAN network via the OBD interface to obtain signals such as steering wheel angle, turn signal status, vehicle speed, and accelerator pedal opening in real time.

[0060] Central Processing Unit: Adopts automotive-grade multi-core SoC (such as NVIDIA Orin or TI TDA4), integrating GPU, DSP and dedicated deep learning acceleration unit, responsible for running all algorithms.

[0061] Level 3 verification includes:

[0062] Level 1: Internal Verification for Single Sensors. This aims to ensure the reliability of raw data from each sensor from the outset. For vision sensors, multi-frame continuity verification is used: by comparing the lane line position and curvature changes detected in consecutive image frames, abnormal jumps caused by instantaneous reflections or occlusions are eliminated, ensuring the temporal stability of the visual output. For millimeter-wave radar, multi-target tracking verification is performed: Kalman filtering is used to track the motion trajectory of each detected target. If the appearance and disappearance of a target lacks kinematic continuity, it is judged as noise or a false alarm and filtered out. For lidar, point cloud stability verification is performed: the point cloud density and distribution pattern of the same target within consecutive scanning cycles are analyzed, removing discrete noise caused by environmental interference (such as rain, fog, dust), and retaining stable geometric features.

[0063] Level 2: Cross-sensor cross-validation. Utilizing the complementary physical characteristics of different sensors, spatial geometric consistency checks are performed on the perception results. Visual and radar cross-validation focuses on geometric consistency: the lane line equations recognized by vision are projected onto the radar coordinate system and compared with the road edges detected by radar (such as guardrails and curbs). If the deviations in lateral distance and direction are too large, a confidence level reduction or re-matching is triggered. Visual and LiDAR feature matching validation: lane line corner points and edge features are extracted from the image and registered with corresponding intensity and height abrupt change regions in the LiDAR point cloud to confirm the real existence of visual features in three-dimensional space. Millimeter-wave radar and LiDAR distance consistency validation: for jointly detected targets (such as vehicles ahead or road obstacles), the radar ranging results and the ranging results after LiDAR point cloud clustering are compared. The error between the two must be within a preset threshold; otherwise, measurements with large deviations are discarded.

[0064] Level 3: Temporal Consistency Verification. Kinematic logic with a time dimension is introduced to perform a final rationality check on the fusion result. Through historical trajectory prediction, the expected lane position at the current moment is calculated using the vehicle motion state of previous frames. If the current fusion result deviates from the predicted value by more than a certain range, abrupt changes need to be checked. Kinematic model verification, based on vehicle dynamics (such as maximum lateral acceleration and yaw rate limits), determines whether the currently detected vehicle-lane relative motion is physically achievable, eliminating non-realistic deviations caused by sensor delays or misalignments. Driver intent analysis integrates temporal information such as turn signals, steering wheel angle, and accelerator pedal input to assess whether the current driving behavior matches the warning conditions, avoiding false alarms due to temporal logic misjudgments when the driver actively changes lanes or avoids obstacles.

[0065] In this embodiment, to ensure the accuracy and reliability of the multi-sensor fusion results, a three-level progressive verification mechanism is designed, which verifies and corrects the perceived information layer by layer, from single-sensor data quality and multi-sensor spatial consistency to temporal motion logic.

[0066] Level 1: Internal Verification for Single Sensors. This aims to ensure the reliability of raw data from each sensor from the outset. For vision sensors, multi-frame continuity verification is used: by comparing the lane line position and curvature changes detected in consecutive image frames, abnormal jumps caused by instantaneous reflections or occlusions are eliminated, ensuring the temporal stability of the visual output. For millimeter-wave radar, multi-target tracking verification is performed: Kalman filtering is used to track the motion trajectory of each detected target. If the appearance and disappearance of a target lacks kinematic continuity, it is judged as noise or a false alarm and filtered out. For lidar, point cloud stability verification is performed: the point cloud density and distribution pattern of the same target within consecutive scanning cycles are analyzed, removing discrete noise caused by environmental interference (such as rain, fog, dust), and retaining stable geometric features.

[0067] Level 2: Cross-sensor cross-validation. Utilizing the complementary physical characteristics of different sensors, spatial geometric consistency checks are performed on the perception results. Visual and radar cross-validation focuses on geometric consistency: the lane line equations recognized by vision are projected onto the radar coordinate system and compared with the road edges detected by radar (such as guardrails and curbs). If the deviations in lateral distance and direction are too large, a confidence level reduction or re-matching is triggered. Visual and LiDAR feature matching validation: lane line corner points and edge features are extracted from the image and registered with corresponding intensity and height abrupt change regions in the LiDAR point cloud to confirm the real existence of visual features in three-dimensional space. Millimeter-wave radar and LiDAR distance consistency validation: for jointly detected targets (such as vehicles ahead or road obstacles), the radar ranging results and the ranging results after LiDAR point cloud clustering are compared. The error between the two must be within a preset threshold; otherwise, measurements with large deviations are discarded.

[0068] Level 3: Temporal Consistency Verification. Kinematic logic with a time dimension is introduced to perform a final rationality check on the fusion result. Through historical trajectory prediction, the expected lane position at the current moment is calculated using the vehicle motion state of previous frames. If the current fusion result deviates from the predicted value by more than a certain range, abrupt changes need to be checked. Kinematic model verification, based on vehicle dynamics (such as maximum lateral acceleration and yaw rate limits), determines whether the currently detected vehicle-lane relative motion is physically achievable, eliminating non-realistic deviations caused by sensor delays or misalignments. Driver intent analysis integrates temporal information such as turn signals, steering wheel angle, and accelerator pedal input to assess whether the current driving behavior matches the warning conditions, avoiding false alarms due to temporal logic misjudgments when the driver actively changes lanes or avoids obstacles.

[0069] like Figure 1 As shown, the lane departure warning system in this embodiment consists of the following hardware:

[0070] Visual sensor: A high dynamic range (HDR) global shutter camera with a resolution of 1280×720 and a frame rate of 30fps is installed on the inside of the windshield to capture images of the road in front of the vehicle.

[0071] Millimeter-wave radar: It adopts a 77GHz long-range radar with a maximum detection range of 200m and a horizontal field of view of ±60°. It is installed in the center of the front bumper and is used to detect road guardrails, curbs and obstacles in front.

[0072] LiDAR: A 16-line mechanical LiDAR with a horizontal field of view of 360° and a vertical field of view of ±15° is installed on the roof of the vehicle to acquire high-precision 3D point cloud data and assist in lane line feature extraction.

[0073] Inertial Measurement Unit (IMU): Integrates a three-axis accelerometer and gyroscope, with a sampling frequency of 100Hz, and is installed at the center of the vehicle chassis to measure the vehicle's yaw rate, lateral acceleration, and attitude angle.

[0074] CAN bus interface: Connects to the vehicle's CAN network via the OBD interface to obtain signals such as steering wheel angle, turn signal status, vehicle speed, and accelerator pedal opening in real time.

[0075] Central Processing Unit: Adopts automotive-grade multi-core SoC (such as NVIDIA Orin or TI TDA4), integrating GPU, DSP and dedicated deep learning acceleration unit, responsible for running all algorithms.

[0076] The software algorithm flow in this embodiment includes:

[0077] The system software is divided into a data acquisition layer, a preprocessing layer, a feature extraction layer, a fusion verification layer, and a decision output layer. The overall process is as follows:

[0078] Data Acquisition and Time Synchronization: Each sensor acquires raw data at its own frequency and adds a precise timestamp. The central processing unit aligns all data to the same time reference (e.g., using the camera frame rate as a reference, interpolating and aligning radar and lidar point cloud data) through hardware triggering or software synchronization mechanisms, forming synchronized multimodal data frames.

[0079] Level 1 Validation: Single Sensor Internal Validation: Data from each sensor is first independently internally verified to eliminate obvious anomalies and ensure the reliability of the data entering the fusion process.

[0080] Visual multi-frame continuity verification: Track lane lines detected in N consecutive frames (N≥5) of images. A Kalman filter is used to predict the position and curvature of the lane line in the current frame, and the actual detection result is compared with the predicted value. If the lateral position deviation exceeds a threshold (e.g., 0.2m) or the curvature change exceeds physical limits, the current frame is considered abnormal, marked as low confidence, and compensated using the predicted value. If multiple consecutive frames show abnormalities, a sensor reset or cleaning prompt is triggered.

[0081] Radar multi-target tracking verification: A trajectory is established for each target detected by the radar (such as guardrails and curb points). The Joint Probabilistic Data Association (JPDA) algorithm is used to associate the detection points in the current frame with historical trajectories. If a target point cannot be associated with any stable trajectory and appears isolated for more than 3 frames, it is judged as a false alarm noise point and is removed; otherwise, the trajectory is updated and the target speed and position are output.

[0082] LiDAR point cloud stability verification: Ground segmentation and clustering are performed on the LiDAR point cloud. For each cluster target, the number of points, distribution variance, and reflection intensity consistency are statistically analyzed. If the number of points is too small (<5 points), the spatial distribution is discrete, or the intensity changes drastically, it is judged as an unstable target (such as rain, fog, or noise points) and marked as pending; only clusters that are stable for multiple consecutive frames are retained.

[0083] The data that passed the first-level verification are denoted as DcamDcam, DradarDradar, and DlidarDlidar, respectively, and each carries a confidence level label.

[0084] Level 2 verification: cross-sensor verification, which involves spatial registration and consistency checks of data from different sensors to ensure physical consistency.

[0085] Visual and radar geometric consistency check: First, the rotation and translation matrices of the camera and radar are obtained through joint calibration, and the road edge points detected by the radar are projected onto the image coordinate system. The projected points are compared with the visual lane line detection results: the lateral distance from the projected points to the lane line curves is calculated. If the average distance is less than a threshold (e.g., 0.3m), the two are considered to be consistent; otherwise, confidence reduction is triggered, and a rematch is attempted (e.g., adjusting the correspondence between radar targets and different lane lines).

[0086] Visual and LiDAR feature matching verification: Pixels at the edge of lane lines are extracted from the image, and corresponding rays are generated using calibration parameters. In the LiDAR point cloud, points with abrupt changes in height (e.g., road curbs) or reflection intensity (e.g., lane line paint) are searched along this ray. If a matching point is found, and the point cloud distribution is consistent with the edge direction of the image, the verification passes; otherwise, the confidence of the visual detection result is reduced, as false detections may occur due to reflections or shadows.

[0087] Millimeter-wave radar and lidar distance consistency verification: For the same target (such as a vehicle in front or a guardrail) within a shared coverage area, compare the radar ranging value RradarRradar with the lidar point cloud cluster center distance RlidarRlidar. If the absolute difference is less than 0.5m, the consistency is good; if the difference is too large, weighted fusion is performed based on the confidence level of each sensor, or obviously abnormal data is discarded.

[0088] After the second-level verification, the fused multimodal perception result PfusionPfusion is obtained, which includes information such as lane line equations, road boundaries, and vehicle status.

[0089] Level 3 verification: Temporal consistency verification: Introducing a time dimension, using vehicle kinematics and driver behavior to logically verify the fusion results.

[0090] Historical trajectory prediction: Based on the vehicle status (vehicle speed, yaw rate) and fused lane lines of previous frames, the vehicle-lane relative position of the current frame is predicted using a constant yaw rate model. The predicted value is compared with the current fusion result. If the lateral deviation exceeds 0.5m or the heading angle deviation exceeds 3°, inconsistency may be caused by sensor delay or jump. The data synchronization should be checked and the predicted value should be adopted as appropriate.

[0091] Kinematic model verification: Based on vehicle dynamics constraints, calculate whether the currently detected deviation speed (such as lateral speed) is within a reasonable range (e.g., the maximum lateral speed typically does not exceed 1.5 m / s). If the deviation speed is abnormally high and there is no corresponding change in steering wheel angle, it is determined to be a sensor false detection, and the warning is suppressed.

[0092] Driver Intent Analysis: Steering wheel angle, turn signal, accelerator, and brake signals from the past 3 seconds are input into the LSTM network, which outputs the current driving intent (normal cruise, active lane change, cornering, unintentional deviation, emergency obstacle avoidance). If the intent is active lane change or emergency obstacle avoidance, the warning should be disabled even if the time-to-leave (TLC) is short; if the intent is unintentional deviation, the warning should be triggered normally. The intent recognition result serves as a key input for the final decision.

[0093] After all three levels of verification are passed, a high-confidence final perception result, Rfinal, is obtained.

[0094] Exception handling and degradation strategy: If a certain level of verification fails, the system adopts a degradation strategy:

[0095] Single sensor internal verification failure: If only one sensor data is abnormal, the other sensors are used for fusion, but the output confidence is reduced and the warning threshold is appropriately relaxed.

[0096] Cross-sensor cross-validation failure: If there is a conflict between data from multiple sensors, the optimal estimate is made based on historical data and kinematic models, and a system status prompt is issued.

[0097] Timing verification failed: If there is a kinematic inconsistency, it may be due to sensor delay. The system will briefly freeze the warning output and wait for the data to stabilize.

[0098] Warning Decision and Execution: Finally, the time-to-leave (TLC) is calculated. If the TLC is less than the dynamic threshold (1.2 seconds for clear roads, 1.8 seconds for wet roads), and the driver's intention is an unintentional deviation, a warning is triggered. Warnings are divided into three levels:

[0099] After the above-described solution in this embodiment is implemented, its effects include:

[0100] In real-vehicle testing, the system improved lane detection accuracy by 35% and reduced false alarm rate by 70% in adverse scenarios such as heavy rain, night, and strong backlighting compared to traditional solutions, while keeping system latency below 100ms, meeting the safety requirements for high-speed driving.

[0101] The core advantage of this invention lies in the construction of a three-level progressive perception and verification mechanism, which systematically solves the shortcoming of insufficient reliability of single sensor solutions in complex environments.

[0102] First, the multi-sensor fusion architecture physically compensates for the disadvantage of single-camera solutions, which suffer from a sharp drop in recognition rate under adverse weather conditions such as rain, fog, and backlight. Millimeter-wave radar is unaffected by light and rain / fog attenuation, while lidar provides accurate three-dimensional geometric information, strongly complementing the dense semantic capabilities of vision and ensuring a stable baseline for all-weather perception.

[0103] Building upon this foundation, the invention employs a unique three-tiered progressive verification system to filter the fusion results layer by layer: The first tier, single-sensor internal verification, utilizes visual multi-frame continuity, radar kinematic tracking, and lidar point cloud stability analysis to filter out instantaneous noise and false alarms at the source; the second tier, cross-sensor verification, through visual-radar geometric consistency, visual-lidar feature matching, and radar-lidar distance consistency checks, eliminates or downweights results with spatial geometric inconsistencies; the third tier, temporal consistency verification, predicts the current perception result based on historical trajectories, captures and replaces abnormal jumps that violate motion logic with predicted values, ensuring a smooth and reliable final output. This progressive verification and correction, from data quality and spatial consistency to motion logic, enables the perception system to not only possess extremely strong transient anti-interference capabilities but also provide a safety net through redundancy and prediction when some sensors fail, significantly improving the overall intelligent driving system's recognition reliability, decision-making accuracy, and functional safety under complex conditions.

[0104] Obviously, the specific implementation of this invention is not limited to the above-described methods. Any non-substantial improvements made using the inventive concept and technical solution of this invention are within the protection scope of this invention.

Claims

1. A lane departure warning method using multi-sensor fusion, employing a visual sensor to acquire visual data for lane line recognition and warning; characterized in that: The visual data acquired by the vision sensor is verified and corrected, and lane line data identified after verification and correction is used to issue lane departure warnings.

2. The lane departure warning method based on multi-sensor fusion as described in claim 1, characterized in that: The visual data is verified and corrected using data detected by vehicle-mounted radar.

3. The lane departure warning method based on multi-sensor fusion as described in claim 2, characterized in that: The verification and correction methods include hierarchical verification, where the first level of verification is the internal data verification of the sensor; The second level of verification is cross-sensor verification, which involves spatial registration and consistency checks on data from different sensors.

4. The lane departure warning method based on multi-sensor fusion as described in claim 3, characterized in that: The hierarchical verification also includes a third level verification, which is a temporal consistency verification. This third level verification introduces kinematic logic in the time dimension to verify the rationality of lane line recognition fusion results after single-sensor internal verification and cross-sensor cross verification.

5. The lane departure warning method based on multi-sensor fusion as described in claim 3, characterized in that: Single-sensor internal validation is used to analyze and verify data acquired by a single sensor. A single sensor uses data comparison between preceding and subsequent time-series acquisitions and data stability analysis to filter data and remove abnormal data; Cross-sensor cross-validation includes: utilizing the complementary physical characteristics of different sensors to perform spatial geometric consistency checks on the sensing results, and validating the data collected by multiple sensors based on the complementary validation results; Complementary verification includes geometric consistency verification, feature matching verification, and distance consistency verification.

6. The lane departure warning method based on multi-sensor fusion as described in claim 4, characterized in that: The temporal consistency verification includes using historical trajectory prediction to estimate the expected lane position at the current moment based on the vehicle motion state of the previous few frames. If the deviation between the current fusion result and the predicted value exceeds a set threshold range, the sensor data is re-detected and the predicted value is used as the final recognition and perception result for lane line warning control.

7. The lane departure warning method based on multi-sensor fusion as described in claim 6, characterized in that: Re-examining sensor data includes kinematic model validation and driver intent analysis; Kinematic model verification: Based on vehicle dynamics constraints, calculate whether the currently detected deviation speed is within the deviation speed threshold range; if the deviation speed exceeds the set deviation threshold range and no corresponding steering wheel angle change is detected, it is determined to be a sensor false detection, and the warning is suppressed at this time; Driver intent analysis: Input the steering wheel angle, turn signal, accelerator, and brake signals from recent historical data into the LSTM network, and output the current driving intent; If the intention is to proactively change lanes or avoid obstacles in an emergency, then the warning will be disabled. If the intention is an unintentional deviation, a warning will be triggered normally.

8. The lane departure warning method based on multi-sensor fusion as described in claim 5, characterized in that: In cross-sensor verification, feature matching verification is used for visual sensors and LiDAR, including: extracting lane line corner points and edge features from the image and registering them with the corresponding intensity change and height change regions in the LiDAR point cloud to confirm the real existence of visual features in three-dimensional space; The pixels at the edge of the lane line are extracted from the image, and the corresponding ray is generated by calibration parameters. In the LiDAR point cloud, points with abrupt changes in height or reflection intensity are searched along the ray. If a matching point is found and the point cloud distribution is consistent with the edge direction of the image, the verification is successful; otherwise, the confidence of the visual detection result is reduced.

9. The lane departure warning method based on multi-sensor fusion as described in claim 4, characterized in that: If the timing consistency verification fails, the lane departure warning output is frozen until the verification succeeds; if the cross-sensor cross-verification fails, the optimal estimate is made based on historical data and kinematic model, and the lane departure warning is issued based on the optimal estimate data.

10. A warning system for operating the lane departure warning method based on multi-sensor fusion as described in any one of claims 1-9, characterized in that: The early warning system includes a data sensing sensor, a data verification module, a data processing module, and an early warning module; The data sensing sensor is used to collect visual data and radar data, and its output is connected to the data verification module. The data verification module uses a three-level verification method to verify the perceived data. The data that passes the verification is sent to the data processing module. The data processing module obtains the corresponding lane line recognition data based on the verified perceived data and sends it to the warning module. The warning module issues a lane departure warning based on the lane line recognition data.

Images