Driverless automobile drivable area detection system and method and storage medium
By integrating lidar, visible camera and IMU data, the driving area detection and positioning problems of driverless cars in complex environments are solved, and the driving experience and driving stability are improved.
Patent Information
- Application Number
- CN202510720732.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-30
- Publication Date
- 2025-08-26
AI Technical Summary
The existing driverless cars have insufficient accuracy and robustness in difficult scenarios such as road and curb texture, shadow occlusion, and light changes, and are inaccurate in positioning in scenarios such as loss of GPS signals or weak signal, resulting in poor driving experience and unstable driving.
The laser radar and visible camera data are fused, and the driving area detection and positioning is combined with IMU data. Deep learning methods are used to detect uneven road surfaces, and the SLAM algorithm is used to perform bicycle positioning when GPS signals are lost. Terrain recognition and spatial positioning are performed through the laser radar and IMU data.
In complex environments, precise detection and stable positioning of the driving area are achieved, improving user driving experience and cargo stability, and ensuring safe passage of the car.
Smart Images
Figure CN120539741A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of unmanned driving technology, and in particular to a system, method and storage medium for detecting a drivable area of an unmanned vehicle. Background Art
[0002] In recent years, with the continuous advancement of communications and chip technologies, the trend toward intelligent vehicles has accelerated rapidly. With the development of other technologies, such as automatic control and artificial intelligence, intelligent driving vehicles have become a research hotspot in the industry, with numerous technology companies and universities investing heavily in research. Currently, by analyzing and processing perceived environmental information through computer vision and machine learning algorithms, vehicles can achieve autonomous route planning and decision-making, reducing the risk of traffic accidents while providing users with a more convenient and comfortable travel experience. Furthermore, intelligent driving vehicles can select the optimal route based on real-time traffic information and predicted road conditions, avoiding congestion and accidents and significantly improving traffic efficiency.
[0003] In autonomous driving, environmental perception is a fundamental building block of intelligent vehicles. The accuracy and robustness of drivable area detection and positioning technology are directly related to the vehicle's posture estimation and distance to obstacles, which in turn directly impacts the accuracy of subsequent path planning and the timing of action execution. Furthermore, when a vehicle enters weak or even no signal environments, such as tunnels or underground garages, positioning technologies that rely on external signals are nearly ineffective. In autonomous driving, the accuracy and robustness of the vehicle's positioning are directly related to its driving safety. Therefore, comprehensive consideration of intelligent vehicle positioning in various environments is a must-address issue in practical engineering projects. Furthermore, in autonomous vehicles carrying people or goods, road speed bumps, cracks in the ground, and raised or sunken manhole covers often cause vehicle jolting, resulting in a poor driving experience and damage to transported goods. Therefore, these issues also require attention. At the same time, it should be noted that the existing autonomous driving has pain points in the environmental perception level, that is, traditional cameras, lidars, etc. are easily affected by harsh environments. High-precision maps and V2X communications, as supplementary means to the insufficient environmental perception of automobiles, can achieve high-quality unmanned driving. However, high-precision maps can only cover some highways and elevated roads and cannot be used in parking lots, rural roads and other places, which seriously restricts the driving area of unmanned vehicles. V2X communication is still in the development stage, and its construction scale needs to be further improved. Summary of the Invention
[0004] In view of this, the purpose of the present invention is to provide a system, method and storage medium for detecting the drivable area of an unmanned vehicle. The invention can realize drivable area detection by fusing lidar and visible light camera data, and solve the problem of drivable area detection in difficult scenarios such as similar textures of roads and curbs, shadow occlusion, and lighting changes; by coupling lidar and IMU data, lidar and SLAM algorithms can be used to perform terrain recognition and spatial positioning without relying on satellite navigation, and obtain the real-time position of the unmanned vehicle; by fusing lidar data, visible light camera data and IMU vehicle posture data, uneven road conditions in the drivable area can be detected, and by controlling the car to slow down or stop, the user's driving experience and cargo stability can be improved, ensuring the safe passage of the car.
[0005] To achieve the above-mentioned objectives, the present invention provides the following technical solutions: a system for detecting drivable areas of unmanned vehicles, comprising an environmental perception module, a decision-making and planning module, and a control execution module, wherein the environmental perception module integrates lidar data and visible light camera data, and adopts a deep learning method to detect the drivable areas of the vehicle. At the same time, the lidar data, visible light camera data, and vehicle posture data of the inertial measurement unit (IMU) are integrated to perform a flatness detection on the road surface in the drivable area of the vehicle, and obtain a driving map of the unmanned vehicle; the decision-making and planning module performs path planning and behavior decision-making based on the environmental perception data and the established driving map of the unmanned vehicle, wherein the behavior decision determines the specific actions of the vehicle according to the current traffic conditions and goals, and the actions include at least: following the vehicle, overtaking, stopping, accelerating, and decelerating; the control execution module completes the planned actions and accurately executes them.
[0006] Furthermore, when the laser radar and visible light camera perception of the environmental perception module are limited, the decision-making and planning module performs a rough match in the driving map based on the GPS location information of the car, and performs precise position matching based on the limited perception data of the laser radar and camera and the car posture change curve data recorded by the IMU, to obtain the current environment of the car and perform path planning and behavior decisions.
[0007] Furthermore, when the GPS signal of the driverless car is weak or there is no signal, the decision-making planning module uses inertial navigation calculation based on the GPS position information before the signal is lost and the IMU data to obtain the rough position of the car and request a map of the driving area within the set range of the current position. Through the lidar and IMU data, the SLAM algorithm is used to perform terrain recognition, and by matching the relationship between the built graphics and the real-time graphics, the vehicle's positioning and then the car's current environment are determined, and path planning and behavior decisions are executed.
[0008] At the same time, the present invention also provides a method for detecting a drivable area of an unmanned vehicle. The method detects the drivable area of an unmanned vehicle based on laser radar data and visible light camera data, comprising: (1) Scene point cloud map reconstruction: project the laser point cloud collected by the lidar onto the depth range image collected and processed by the visible light camera. For ground point clouds, a plane model fitting method is used for extraction. For non-ground point clouds, a density-based and Euclidean distance-based method is used for clustering. Noise points are filtered out before extraction. (2) Driving area detection: Based on the sensor platform that integrates lidar and visible light camera, the point cloud data is projected onto the camera imaging plane to obtain a sparse depth map. A dense depth map is obtained through a depth completion algorithm. The surface normal is fitted based on the dense depth to obtain the fused road feature information. A deep learning algorithm is then used to detect the driving area based on the fused feature information.
[0009] Preferably, the method for detecting the drivable area of the unmanned vehicle is also based on the fused laser radar and visible light camera feature data, and adopts the target detection deep learning method to detect road unevenness, and marks the influencing factors that may affect the smooth passage of the car in the unmanned vehicle driving map, including at least: road speed bumps, raised or sunken manhole covers, road bulges, road potholes, and road cracks.
[0010] Preferably, for uneven roads that are difficult to perceive with laser radar and visible light cameras, the method for detecting the drivable area of the unmanned vehicle is also based on the posture information measured by the vehicle IMU, and draws a curve of the vehicle posture changing with position. When the peak height of the peak or trough of the changing curve exceeds the set threshold thh, the method for detecting the drivable area of the unmanned vehicle marks the current position in the unmanned vehicle driving map to affect the smooth passage of the vehicle.
[0011] In particular, the method for detecting the drivable area of the unmanned vehicle also includes a method for storing a driving map of the unmanned vehicle, specifically: The autonomous vehicle driving map storage method periodically updates the perceived map data to the cloud for autonomous driving sharing. At the same time, it records the user's usage frequency f of the autonomous vehicle driving map of a certain area, the user's collection attribute c of the driving map of a certain area, and the time interval dt between the driving map data collected and updated by the vehicle and the current moment. When the driving map of a certain area is updated to the cloud, it is also saved locally in the car for quick query. Among them, c=1 means it has been collected, c=0 means it has not been collected, f0 is the set driving map usage frequency threshold, and dt0 is the time interval threshold between the driving map data collected and updated by the vehicle and the current moment.
[0012] Finally, the present invention also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the unmanned vehicle drivable area detection method.
[0013] Compared with the prior art, the present invention has the following beneficial effects:
[0014] 1. For structured and unstructured roads in autonomous vehicle driving scenarios, this invention can detect drivable areas by fusing lidar and visible light camera data, solving the problem of drivable area detection in difficult scenarios such as similar road and curb textures, shadow occlusion, and changing lighting.
[0015] 2. By coupling lidar and IMU data, the present invention can use lidar and SLAM algorithms to perform terrain recognition and spatial positioning without relying on satellite navigation, and obtain the real-time position of the unmanned vehicle.
[0016] 3. By fusing lidar data, visible light camera data, and IMU vehicle posture data, the present invention can detect uneven road conditions in the drivable area. By controlling the vehicle to slow down or stop, the user's driving experience and cargo stability can be improved, ensuring the safe passage of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 This is a schematic diagram of the overall framework of a system and method for detecting a drivable area for an unmanned vehicle according to the present invention; Figure 2 This is a schematic diagram of the detection effect of the present invention fusing lidar and visible light camera data to detect the drivable area; Figure 3 The figure is a schematic diagram of an implementation process of the laser radar-based self-positioning of an unmanned vehicle according to the present invention. DETAILED DESCRIPTION
[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0019] like Figure 1-3As shown in the figure, this solution mainly considers the accuracy and robustness of drivable area detection in difficult scenarios such as road and curb textures being similar, shadows being blocked, and lighting changes, as well as the problem of accurate vehicle positioning in special scenarios where the GPS signal is lost or interfered with. At the same time, it considers the comfort of passengers and the passability of the vehicle during driving. It aims to solve the problem of obvious bumps when the driverless car passes over speed bumps and potholes without adding more hardware equipment, thereby improving the user experience.
[0020] 1. Obtaining a driving map for autonomous vehicles based on environmental perception data
[0021] Regarding structured and unstructured roads in autonomous vehicle driving scenarios, the current algorithm's accuracy and robustness in detecting drivable areas need to be further improved in difficult scenarios such as road and curb textures being similar, shadows being obscured, and lighting changes. At the same time, existing algorithms pay little attention to road surface irregularities such as speed bumps, manhole covers, potholes, and cracks on the road sections where cars are driving, and insufficient consideration is given to user riding comfort, cargo transportation stability, and vehicle passability. This invention adopts a solution that fuses lidar and camera sensors, using clustering methods and deep learning-based visual detection methods for point cloud and image information to obtain obstacle data of different modalities. A two-stage fusion strategy is then used to fuse obstacle information and detect drivable areas. At the same time, the vehicle's position data from the inertial measurement unit (IMU) is integrated to describe the road surface smoothness in the drivable area, thereby obtaining a vehicle driving map with road surface smoothness, specifically:
[0022] (1) Scene point cloud map reconstruction
[0023] Using the onboard computer's built-in 3D simulation software, virtual targets are generated relative to the virtual scene. Using point cloud coloring and ground extraction algorithms, the collected laser point cloud is projected onto the depth range image and colored. In actual driving scenarios, ground point clouds play a negative role in positioning and require initial extraction. This extraction method can use plane model fitting. For non-ground point clouds, a method combining density and Euclidean distance is used for clustering to filter out noise points and prepare for feature extraction.
[0024] (2) Driving area detection
[0025] Based on the sensor platform fused with LiDAR and visible light camera, joint calibration is performed to obtain the spatial position relationship between sensors. The point cloud data is then projected onto the camera imaging plane to obtain a sparse depth map. The improved PENet Algorithm (IM_PENet) depth completion algorithm is used to obtain a dense depth map. Surface normal fitting is performed based on the dense depth to obtain fused road feature information. A deep learning algorithm is then used to detect the drivable area based on the fused feature information. Feature extraction and feature fusion are performed sequentially to avoid feature interference and feature conflict. The detection effect is as follows: Figure 2 shown.
[0026] (3) Road surface smoothness detection
[0027] Based on the fused feature data of lidar and visible light camera, the target detection deep learning method is used to detect speed bumps, raised or sunken manhole covers, road ridges, potholes, road cracks, etc., and mark the influencing factors that may affect the smooth passage of cars.
[0028] At the same time, for uneven roads that are difficult for radar and cameras to perceive, a curve of the vehicle's posture changing with position is drawn based on the posture information measured by the vehicle's IMU. When the peak height or trough height of the change curve exceeds the set threshold thh, it is considered that the current road surface causes the vehicle to be bumpy enough to affect the driving experience and cargo stability, or affect the vehicle's own passability (bumping into the vehicle chassis). The vehicle should slow down or stop, and mark the current position as a factor that may affect the vehicle's smooth passage.
[0029] (4) Establishing a driving map for autonomous vehicles
[0030] Based on the car's own position, the road's drivable area, and road surface smoothness information, a driving map for the autonomous vehicle is established, providing a basis for the autonomous vehicle's next step of planning and decision-making.
[0031] 2. Autonomous Vehicle Driving Map Storage
[0032] For each vehicle's driving experience, the system uses this data to perform real-time detection of drivable areas. Simultaneously, based on driving map matching, the vehicle's own positioning is achieved when GPS signals are weak, providing precise autonomous navigation and control. Furthermore, historical driving map data is updated. Therefore, it is necessary to balance driving map query and storage costs. Accordingly, the present invention stores driving maps based on the user's frequency of use (f) of driving maps for a particular area, the user's regional favorites (c), and the time interval (dt) between the current moment and the current time interval between the vehicle's collected and updated driving map data.
[0033] Among them, the driving maps with high usage frequency, user favorited areas, and short time intervals from the current time are considered to have a higher probability of being used again in the short term, i.e. Therefore, in addition to being saved in the cloud for sharing with other driverless cars, this type of driving map data is also retained locally in the car for quick access. For areas with low usage frequency, areas not collected by users, and driving maps that are far from the current time, only the driving maps are stored in the cloud to update the driving maps and reduce local storage costs.
[0034] 3. Unmanned Driving Control
[0035] During driving, the unmanned driving method of the present invention uses laser radar and visible light cameras to perform environmental perception and real-time detection of drivable areas, obtaining the legal area in which the vehicle can drive. Simultaneously, it combines IMU posture data, the vehicle's current position, and a driving map to obtain information about obstacles and road smoothness within the drivable area. When obstacles such as people, vehicles, or traffic cones are present in the drivable area, the unmanned vehicle replans its route and detours or stops to ensure driving safety. When uneven road conditions such as speed bumps, raised or recessed manhole covers, cracks, or potholes are present in the drivable area, the unmanned vehicle executes deceleration or parking commands in advance to ensure stable and safe driving. Furthermore, based on the vehicle posture data obtained in real time by the IMU, the unmanned vehicle executes a power increase command when the vehicle is currently in a head-up posture, i.e., when the vehicle is going uphill; and executes a deceleration command when the vehicle is in a head-down posture, i.e., when the vehicle is going downhill.
[0036] Among them, when the perception of radar or camera is limited, the present invention performs coarse matching in the driving map based on the GPS location information of the car, and performs precise position matching based on the limited perception data of radar and camera and the car posture change data recorded by IMU.
[0037] When a self-driving car enters a tunnel or underground garage, where the signal is weak or absent, it may face inaccurate positioning. Accordingly, the present invention uses inertial navigation calculations based on GPS location information before signal loss and IMU data to obtain the car's approximate location and request a map of the driving area within a certain range of the current location. Using LiDAR and IMU data, the system uses LiDAR and SLAM algorithms to identify terrain without relying on satellite navigation. The vehicle's position is determined by matching the relationship between pre-built graphics and real-time graphics. The requested map of the driving area within a certain range of the current location can be local to the car (if available) or a shared map of the driving area in the cloud.
[0038] In principle, SLAM positioning utilizes the concept of point curvature from Lidar Odometry and Mapping in Real-time (LOAM). Features are extracted by calculating the roughness of a point within its local area. The calculated curvature data is compared with a threshold to distinguish point types and evenly extract feature information from the global data. The range image is horizontally divided into multiple equal sub-images. The points on the image are sorted based on their roughness values, and a threshold is used to distinguish different types of features. From each row of the sub-image, several points with the maximum roughness that do not belong to the ground are selected as edge feature points, and several points with the minimum roughness are selected as plane feature points. Finally, the real-time rotation and translation matrices and coordinate transformation relationships are calculated to determine the positioning result.
[0039] like Figure 3 As shown in the figure, a specific positioning process example based on SLAM is as follows: first, the constructed point cloud map is loaded, and the point cloud is downsampled to reduce the computational complexity required for registration. When the initial pose is known, the point cloud and map are registered in real time through the lidar to output the positioning pose.
[0040] Since the present invention performs matching based on features, the influence of interference points can be eliminated, representative features can be extracted from the environment, less computing resources can be occupied, and the real-time requirements of the system can be met.
[0041] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A system for detecting drivable areas of unmanned vehicles, comprising an environment perception module, a decision-making and planning module, and a control execution module, characterized in that: The environmental perception module integrates lidar data and visible light camera data, and adopts deep learning methods to detect the area where the car can be driven. At the same time, it integrates lidar data, visible light camera data and vehicle posture data from the inertial measurement unit (IMU) to detect the flatness of the road surface in the area where the car can be driven, and obtains a driving map for the unmanned vehicle. The decision-making and planning module performs path planning and behavior decision-making based on the environmental perception data and the established driving map for the unmanned vehicle. The behavior decision determines the specific action of the vehicle according to the current traffic conditions and goals, and the actions include at least: following the vehicle, overtaking, stopping, accelerating, and decelerating. The control execution module completes the precise execution of the planned actions.
2. The unmanned vehicle drivable area detection system according to claim 1, characterized in that: When the laser radar and visible light camera of the environmental perception module have limited perception, the decision-making and planning module performs a rough match in the driving map based on the GPS location information of the car, and performs precise position matching based on the limited perception data of the laser radar and camera and the car posture change curve data recorded by the IMU, to obtain the current environment of the car and perform path planning and behavior decisions.
3. The autonomous vehicle drivable area detection system according to claim 2, characterized in that: When the GPS signal of the driverless car is weak or there is no signal, the decision-making planning module uses inertial navigation calculation based on the GPS position information before the signal is lost and the IMU data to obtain the rough position of the car and request a map of the driving area within the set range of the current position. Through the lidar and IMU data, the SLAM algorithm is used to perform terrain recognition, and by matching the relationship between the built graphics and the real-time graphics, the vehicle's positioning and then the car's current environment are determined, and path planning and behavior decisions are executed.
4. The method for detecting a drivable area of an unmanned vehicle used in the drivable area detection system of any one of claims 1 to 3 is characterized in that: The method for detecting the drivable area of an unmanned vehicle performs drivable area detection based on laser radar data and visible light camera data, and includes: (1) Scene point cloud map reconstruction: project the laser point cloud collected by the lidar onto the depth range image collected and processed by the visible light camera. For ground point clouds, a plane model fitting method is used for extraction. For non-ground point clouds, a density-based and Euclidean distance-based method is used for clustering. Noise points are filtered out before extraction. (2) Driving area detection: Based on the sensor platform that integrates lidar and visible light camera, the point cloud data is projected onto the camera imaging plane to obtain a sparse depth map. A dense depth map is obtained through a depth completion algorithm. The surface normal is fitted based on the dense depth to obtain the fused road feature information. A deep learning algorithm is then used to detect the driving area based on the fused feature information.
5. The method for detecting a drivable area of an unmanned vehicle according to claim 4, wherein: The method for detecting the drivable area of an unmanned vehicle is also based on the fused feature data of a lidar and a visible light camera, and adopts a target detection deep learning method to detect road unevenness, and marks the factors that may affect the smooth passage of the vehicle on the unmanned vehicle driving map, including at least: road speed bumps, raised or sunken manhole covers, road ridges, road potholes, and road cracks.
6. The method for detecting a drivable area of an unmanned vehicle according to claim 5, wherein: For uneven roads that are difficult to perceive with lidar and visible light cameras, the method for detecting the drivable area of an unmanned vehicle also draws a curve of the vehicle's posture changing with position based on the posture information measured by the vehicle's IMU. When the peak height of the peak or trough of the changing curve exceeds the set threshold thh, the method for detecting the drivable area of an unmanned vehicle marks the current position in the unmanned vehicle driving map, which affects the smooth passage of the vehicle.
7. The method for detecting a drivable area of an unmanned vehicle according to claim 6, wherein: The method for detecting the drivable area of an unmanned vehicle further includes a method for storing a driving map of the unmanned vehicle, specifically: The autonomous vehicle driving map storage method periodically updates the perceived map data to the cloud for autonomous driving sharing. At the same time, it records the user's usage frequency f of the autonomous vehicle driving map of a certain area, the user's collection attribute c of the driving map of a certain area, and the time interval dt between the driving map data collected and updated by the vehicle and the current moment. When the driving map of a certain area is updated to the cloud, it is also saved locally in the car for quick query. Among them, c=1 means it has been collected, c=0 means it has not been collected, f0 is the set driving map usage frequency threshold, and dt0 is the time interval threshold between the driving map data collected and updated by the vehicle and the current moment.
8. A storage medium, wherein the storage medium is a computer-readable storage medium and is used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the method for detecting the drivable area of an unmanned vehicle as described in any one of claims 5 to 7.