Intelligent lane departure early warning system based on multi-sensor fusion

Abstract

The invention discloses an intelligent lane departure early warning system based on multi-sensor fusion, which relates to the technical field of auxiliary driving and comprises a lane information acquisition module, a driving behavior acquisition module, a multi-sensor data preprocessing module, an intention recognition module, a multi-data processing module and an early warning interaction module. By fusing the visual sensor, the millimeter-wave radar sensor and the solid-state laser radar, the limitation of a single sensor in a complex environment is overcome, the accuracy and robustness of lane line detection are remarkably improved, the intention recognition module is combined with the steering wheel angle sensor and driver behaviors, the early warning threshold can be dynamically adjusted, and the early warning effect is improved. Compared with the prior art, the system has the advantages that the false alarm rate is effectively reduced, the driving experience of a driver is improved, meanwhile, the system can still keep high detection performance under the condition of severe weather or poor illumination conditions, and higher environmental adaptability is achieved.

Description

Technical Field

[0001] The present invention relates to the field of assisted driving technology, and particularly to an intelligent lane departure warning system based on multi-sensor fusion. Background Art

[0002] The lane departure warning system can intelligently sense through a variety of sensor devices, collect data in real time, and issue danger warnings in a timely manner, thereby reducing traffic accidents caused by lane departure of vehicles.

[0003] Traditional lane departure warning systems mostly collect data from a single sensor for warning control, but there are deficiencies. For example, a single vision sensor may fail when the light is poor or the lane lines are blurred, and only using a millimeter-wave radar may not be able to accurately identify lane lines. And there is a lack of active behavior prediction for drivers during warning, resulting in false warnings, and generally the warning form is single, which will directly affect the driving experience. In certain situations, excessive warning behavior will instead affect safe driving. Therefore, the present invention proposes an intelligent lane departure warning system based on multi-sensor fusion to solve the deficiencies in the prior art. Summary of the Invention

[0004] In view of the above problems, the purpose of the present invention is to provide an intelligent lane departure warning system based on multi-sensor fusion. Through the intention recognition module combined with the steering wheel angle sensor and driver behavior, the warning threshold can be dynamically adjusted, effectively reducing the false alarm rate and improving the driving experience of the driver. At the same time, the system of the present invention can still maintain high detection performance under bad weather or poor lighting conditions, and has stronger environmental adaptability.

[0005] To achieve the above purpose, the present invention provides the following technical solutions:

[0006] An intelligent lane departure warning system based on multi-sensor fusion, including a lane information collection module, a driving behavior collection module, a multi-sensor data preprocessing module, an intention recognition module, a multi-data processing module, and a warning interaction module;

[0007] The lane information collection module collects lane information on the vehicle driving road through a vision sensor, a millimeter-wave radar sensor, and a solid-state lidar;

[0008] The driving behavior collection module collects driver behavior data through an eye tracker installed in the vehicle, and collects vehicle self-steering data through a steering wheel angle sensor;

[0009] The multi-sensor data preprocessing module is used to perform time synchronization and spatial calibration on the data collected by the lane information collection module;

[0010] The intention recognition module is used to integrate the data of the steering wheel angle sensor and the eye tracker data, and construct an intention recognition network model to predict the driving intention of the driver;

[0011] The multi-data processing module is used to perform fusion calculation on the multi-sensor information that has undergone data processing to obtain a lane departure signal;

[0012] The warning interaction module is used to receive the lane departure signal and the driving intention of the driver for fusion calculation, and finally perform a progressive warning prompt according to the calculation result.

[0013] A further improvement lies in that: the vision sensor is a high-resolution front view camera installed in the rearview mirror area of the vehicle windshield, which is used to collect the road image in front of the vehicle in real time and extract lane line information.

[0014] A further improvement lies in that: the millimeter wave radar sensor and the solid-state lidar are both installed at the front of the vehicle, which are used to detect the relative distance, speed and angle between the vehicle and the obstacles and lane lines in front in real time, and provide redundant lane line position information.

[0015] A further improvement lies in that: the eye tracker is used to collect the eye movement data of the driver, and the steering wheel angle sensor is used to collect the steering angle and steering wheel inclination data of the vehicle itself in real time.

[0016] A further improvement lies in that: when the multi-sensor data preprocessing module performs time synchronization and spatial calibration on the collected data, it includes downsampling, ground segmentation and feature extraction on the data collected by the solid-state lidar, including lane line projection on the data collected by the vision sensor, and dynamic target filtering and road boundary clustering on the data collected by the millimeter wave radar sensor.

[0017] A further improvement lies in that: the multi-sensor data preprocessing module also includes timestamp alignment and motion compensation for the multi-sensor data, and uniformly converts the data of each sensor to the vehicle coordinate system, and establishes a state space model to perform spatial calibration on the data of each sensor.

[0018] A further improvement lies in that: when the intention recognition module constructs an intention recognition network model to predict the driving intention of the driver, if the eye tracker data shows that the driver is looking at the rearview mirror and the turn signal is on, it is determined that the driver has an active intention to change lanes.

[0019] A further improvement lies in that: the multi-data processing module calculates the lateral offset and heading angle deviation of the vehicle based on multi-sensor information, and different lateral offsets and heading angle deviations form different lane departure signals. Specifically: a middle threshold range for the lateral offset and heading angle deviation is set. The lateral offset and heading angle deviation below the middle threshold range form a first-level lane departure signal, those within the middle threshold range form a second-level lane departure signal, and those above the middle threshold range form a third-level lane departure signal.

[0020] A further improvement lies in that: when the warning interaction module performs fusion calculation and progressive warning prompts, different levels of lane departure signals correspond to different levels of warning forms. Specifically: under the first-level lane departure signal, visual prompts are used for warning, and the lane lines are displayed as changing color, flashing or arrow indication through the instrument panel or HUD (head-up display) to remind the driver that the vehicle has deviated from the lane; under the second-level lane departure signal, the steering wheel or seat vibrates to provide a more intuitive warning signal; under the third-level lane departure signal, a sound alarm is issued to further remind the driver.

[0021] A further improvement lies in that: when the vehicle still has a lane departure within 1 s after the warning interaction module gives a warning prompt, a higher-level warning form is actively switched for warning, and when currently in the third-level warning mode, the warning is given by increasing the alarm volume.

[0022] The beneficial effects of the present invention are as follows: by integrating a vision sensor, a millimeter-wave radar sensor and a solid-state lidar, the present invention overcomes the limitations of a single sensor in a complex environment, significantly improves the accuracy and robustness of lane line detection. The intention recognition module combines the steering wheel angle sensor and the driver's behavior, can dynamically adjust the warning threshold, effectively reduces the false alarm rate, and improves the driving experience of the driver. At the same time, the system of the present invention can still maintain high detection performance under bad weather or poor lighting conditions, and has stronger environmental adaptability;

[0023] By performing data processing such as timestamp alignment, motion compensation and spatial calibration on multi-sensor data, the problems of false alarms and missed detections in complex road scenarios can be solved, and the warning accuracy rate can be effectively improved. Brief Description of the Drawings

[0024] Figure 1 It is a schematic diagram of the system framework of the present invention;

[0025] Figure 2 It is a schematic diagram of the progressive warning process of the present invention. Detailed Embodiment

[0026] To deepen the understanding of the present invention, the following will further elaborate on the present invention in conjunction with embodiments. These embodiments are only used to explain the present invention and do not constitute a limitation on the protection scope of the present invention.

[0027] Embodiment 1

[0028] According to Figure 1 As shown, this embodiment proposes an intelligent lane departure warning system based on multi-sensor fusion, including a lane information acquisition module, a driving behavior acquisition module, a multi-sensor data preprocessing module, an intention recognition module, a multi-data processing module, and a warning interaction module;

[0029] The lane information acquisition module acquires lane information on the vehicle's driving road through a vision sensor, a millimeter-wave radar sensor, and a solid-state lidar;

[0030] The driving behavior acquisition module acquires driver behavior data through an eye tracker set inside the vehicle, and acquires the vehicle's own steering data through a steering wheel angle sensor;

[0031] The multi-sensor data preprocessing module is used to synchronize the time and calibrate the space of the data acquired by the lane information acquisition module;

[0032] The intention recognition module is used to integrate the data of the steering wheel angle sensor and the eye tracker data, and construct an intention recognition network model to predict the driver's driving intention;

[0033] The multi-data processing module is used to perform fusion calculations on the multi-sensor information that has undergone data processing to obtain a lane departure signal;

[0034] The warning interaction module is used to receive the lane departure signal and the driver's driving intention for fusion calculation, and finally perform a progressive warning prompt according to the calculation result.

[0035] In the present invention, the vision sensor, millimeter-wave radar sensor, solid-state lidar, steering wheel angle sensor data, and eye tracker and other hardware devices are connected to the in-vehicle processing unit (including the multi-sensor data preprocessing module, intention recognition module, multi-data processing module) through the CAN bus or Ethernet. The warning interaction module realizes the warning interaction function through devices such as a display screen, a vibration module, and a speaker.

[0036] Embodiment 2

[0037] According to Figure 1 、 2 As shown, this embodiment proposes an intelligent lane departure warning system based on multi-sensor fusion, including a lane information acquisition module, a driving behavior acquisition module, a multi-sensor data preprocessing module, an intention recognition module, a multi-data processing module, and a warning interaction module;

[0038] The visual sensor is a high-resolution front-view camera installed in the rearview mirror area of the vehicle windshield, which is used to collect road images in front of the vehicle in real time and extract lane line information. The millimeter-wave radar sensor and the solid-state lidar are both installed at the front of the vehicle, which are used to detect the relative distance, speed and angle between the vehicle and the obstacles and lane lines in front in real time, and provide redundant lane line position information. The eye tracker is used to collect the eye movement data of the driver, and the steering wheel angle sensor is used to collect the steering angle and steering wheel inclination data of the vehicle itself in real time.

[0039] When the multi-sensor data preprocessing module synchronizes the collected data in time and calibrates it in space, it includes downsampling, ground segmentation and feature extraction for the data collected by the solid-state lidar, lane line projection for the data collected by the visual sensor, and dynamic target filtering and road boundary clustering for the data collected by the millimeter-wave radar sensor. The multi-sensor data preprocessing module also includes timestamp alignment and motion compensation for the multi-sensor data, as well as uniformly converting the data of each sensor to the vehicle coordinate system and establishing a state space model to calibrate the data of each sensor in space. The specific processing process is as follows:

[0040] Downsampling: Perform voxel filtering on the original 128-line lidar point cloud to reduce it to less than 50,000 points;

[0041] Ground segmentation: Use the RANSAC algorithm to extract the ground plane and separate the road surface from non-ground objects (such as vehicles and guardrails);

[0042] Feature extraction: Extract edge feature points (such as lane lines and curbs) based on curvature calculation and retain key geometric features;

[0043] Dynamic target filtering: Remove the reflection points of moving objects through Doppler velocity detection (in the range of ±200 km / h);

[0044] Road boundary clustering: Use the DBSCAN algorithm to cluster the static point cloud and extract the continuous reflection points on both sides of the road;

[0045] Lane line projection: Project the lane lines (2D pixel coordinates) recognized by the front-view camera to the lidar coordinate system through the calibration matrix to generate a virtual reference point cloud;

[0046] Timestamp alignment: Add a hardware-level synchronous timestamp (accuracy ±1 ms) to the data of each sensor;

[0047] Motion compensation: Calculate the vehicle pose change in real time through the 6-axis IMU data (100 Hz), perform reverse motion compensation on the point cloud, and eliminate the influence of vehicle displacement during sensor acquisition.

[0048] When the intention recognition module constructs an intention recognition network model to predict the driving intention of the driver, if the eye movement data shows that the driver is looking at the rearview mirror and the turn signal is on, it is determined that the driver has the active intention to change lanes. When the eye movement data shows that the driver is looking at the rearview mirror but the turn signal is not on, it is determined that the driver has no intention to change lanes.

[0049] The multi-data processing module includes calculating the lateral offset and heading angle deviation of the vehicle based on multi-sensor information. Different lateral offsets and heading angle deviations form different lane departure signals. Specifically: set the intermediate threshold range of the lateral offset and heading angle deviation. The lateral offset and heading angle deviation below the intermediate threshold range form a first-level lane departure signal, the lateral offset and heading angle deviation within the intermediate threshold range form a second-level lane departure signal, and the lateral offset and heading angle deviation above the intermediate threshold range form a third-level lane departure signal. When the warning interaction module performs fusion calculation and progressive warning prompts, different levels of lane departure signals correspond to different levels of warning forms. Specifically: under the first-level lane departure signal, visual prompts are used for warning, and the lane lines are displayed as colored, flashing, or arrow-indicated through the instrument panel or HUD (head-up display) to remind the driver that the vehicle has deviated from the lane; under the second-level lane departure signal, the steering wheel or seat vibrates to provide a more intuitive warning signal; under the third-level lane departure signal, a sound alarm is issued to further remind the driver. When the vehicle still has a lane departure within 1 s after the warning interaction module issues a warning prompt, it warns by actively switching to a higher-level warning form, and when it is currently in the third-level warning mode, it warns by increasing the alarm volume.

[0050] In the above lane departure warning process, the lane departure judgment is based on the prediction result of the driver's driving intention, that is: when there are lateral offset and heading angle deviation and the driver has no intention to change lanes, subsequent warning control operations are performed.

[0051] The present invention combines a vision sensor, a millimeter-wave radar sensor, and a solid-state lidar, overcomes the limitations of a single sensor in a complex environment, significantly improves the accuracy and robustness of lane line detection. The intention recognition module combines the steering wheel angle sensor and driver behavior, can dynamically adjust the warning threshold, effectively reduces the false alarm rate, and improves the driving experience of the driver. At the same time, the system of the present invention can still maintain high detection performance in case of bad weather or poor lighting conditions, and has stronger environmental adaptability; by performing data processing such as timestamp alignment, motion compensation, and spatial calibration on multi-sensor data, the problems of false alarms and missed detections in complex road scenarios can be solved, and the warning accuracy rate can be effectively improved.

[0052] The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and what is described in the above embodiments and the specification is only to illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. An intelligent lane departure warning system based on multi-sensor fusion, characterized in that: It includes a lane information acquisition module, a driving behavior acquisition module, a multi-sensor data preprocessing module, an intention recognition module, a multi-data processing module, and a warning interaction module; The lane information acquisition module acquires lane information on the vehicle's driving road through a vision sensor, a millimeter-wave radar sensor, and a solid-state lidar; The driving behavior acquisition module acquires driver behavior data through an eye tracker installed in the vehicle, and acquires the vehicle's own steering data through a steering wheel angle sensor; The multi-sensor data preprocessing module is used to perform time synchronization and spatial calibration on the data acquired by the lane information acquisition module; The intention recognition module is used to integrate the data of the steering wheel angle sensor and the eye tracker data, and construct an intention recognition network model to predict the driver's driving intention; The multi-data processing module is used to perform fusion calculation on the multi-sensor information that has undergone data processing to obtain a lane departure signal; The warning interaction module is used to receive the lane departure signal and the driver's driving intention for fusion calculation, and finally perform a progressive warning prompt according to the calculation result.

2. The intelligent lane departure warning system based on multi-sensor fusion according to claim 1, characterized in that: The vision sensor is a high-resolution front view camera installed in the rearview mirror area of the vehicle's windshield, which is used to collect the road image in front of the vehicle in real time and extract lane line information.

3. An intelligent lane departure warning system based on multi-sensor fusion according to claim 2, characterized in that: The millimeter-wave radar sensor and the solid-state lidar are both installed at the front of the vehicle, which are used to detect the relative distance, speed, and angle between the vehicle and the front obstacle and lane line in real time, and provide redundant lane line position information.

4. An intelligent lane departure warning system based on multi-sensor fusion according to claim 3, characterized in that: The eye tracker is used to collect the eye movement data of the driver, and the steering wheel angle sensor is used to collect the vehicle's own steering angle and steering wheel inclination data in real time.

5. An intelligent lane departure warning system based on multi-sensor fusion according to claim 4, characterized in that: When the multi-sensor data preprocessing module performs time synchronization and spatial calibration on the collected data, it includes downsampling, ground segmentation, and feature extraction of the data collected by the solid-state lidar, lane line projection of the data collected by the vision sensor, and dynamic target filtering and road boundary clustering of the data collected by the millimeter-wave radar sensor.

6. The intelligent lane departure warning system based on multi-sensor fusion according to claim 5, characterized in that: The multi-sensor data preprocessing module also includes timestamp alignment and motion compensation for the multi-sensor data, and uniformly converting the data of each sensor to the vehicle coordinate system, and establishing a state space model to perform spatial calibration on the data of each sensor.

7. An intelligent lane departure warning system based on multi-sensor fusion according to claim 1, characterized in that: When the intention recognition module constructs an intention recognition network model to predict the driver's driving intention, if the eye tracker data shows that the driver is looking at the rearview mirror and the turn signal is on, it is judged that the driver has an active intention to change lanes.

8. An intelligent lane departure warning system based on multi-sensor fusion according to claim 1, characterized in that: The multi-data processing module includes calculating the lateral offset and heading angle deviation of the vehicle according to the multi-sensor information. Different lateral offsets and heading angle deviations form different lane departure signals. Specifically: set the intermediate threshold range of the lateral offset and heading angle deviation. The lateral offset and heading angle deviation below the intermediate threshold range form a first-level lane departure signal, the lateral offset and heading angle deviation within the intermediate threshold range form a second-level lane departure signal, and the lateral offset and heading angle deviation above the intermediate threshold range form a third-level lane departure signal.

9. An intelligent lane departure warning system based on multi-sensor fusion according to claim 8, characterized in that: When the warning interaction module performs fusion calculation and progressive warning prompts, different levels of lane departure signals correspond to different levels of warning forms, specifically: under the first-level lane departure signal, visual prompts are used for warning, and the lane lines are displayed on the instrument panel or HUD (head-up display) to change color, flash, or arrow indication to remind the driver that the vehicle has deviated from the lane; under the second-level lane departure signal, the steering wheel or seat vibrates to provide a more intuitive warning signal; under the third-level lane departure signal, a sound alarm is issued to further remind the driver.

10. An intelligent lane departure warning system based on multi-sensor fusion according to claim 9, characterized in that: When the vehicle still has a lane departure within 1 s after the warning interaction module gives a warning prompt, it warns by actively switching to a higher-level warning form, and when it is currently in the third-level warning mode, it warns by increasing the alarm volume.

Images