Lane boundary detection methods, devices, media, and products

Abstract

This application discloses a method, apparatus, medium, and product for detecting lane boundaries, relating to the field of vehicle control technology. The method includes: acquiring image data of the vehicle's environment using a camera; and scanning point cloud data of the vehicle's environment using a lidar scanner; identifying lane boundaries in the image data to obtain pixel information of the lane boundaries in the image data, the pixel information including pixel positions; mapping the image data to the point cloud data to obtain fused point cloud data, the fused point cloud data indicating the mapping result of the lane boundary pixel information in the point cloud data; and based on the fused point cloud data, obtaining a distance value sequence corresponding to the lane boundary, the distance value sequence indicating the lane width. By acquiring environmental data of different dimensions, the stability and accuracy of boundary detection results under different driving environments are improved.

Description

Technical Field

[0001] This application relates to the field of vehicle control technology, and in particular to a method, device, medium, and product for detecting lane boundaries. Background Technology

[0002] While driving, you must follow the instructions of road signs, which include lane boundaries, which are represented by lane lines of different colors and types.

[0003] In related technologies, lane boundary information is obtained through single-sensor detection methods. For example, after capturing environmental images with a camera, lane boundary information is obtained through a graph neural network algorithm; or, after acquiring point cloud signals with LiDAR, lane boundary information is obtained through a classification algorithm. Lane boundary information is the basis for vehicle driving control.

[0004] However, the aforementioned boundary detection methods have low accuracy and safety when implemented in complex driving environments. Summary of the Invention

[0005] This application provides a method, apparatus, medium, and product for detecting lane boundaries. The technical solution provided by this application includes the following aspects.

[0006] According to one aspect of the embodiments of this application, a method for detecting lane boundaries is provided, the method comprising: The vehicle acquires image data of its environment using a camera, and scans point cloud data of the environment using a lidar scanner. Identify the lane boundary in the image data to obtain the pixel information of the lane boundary in the image data, the pixel information including the pixel position; The image data is mapped to the point cloud data to obtain fused point cloud data, which is used to indicate the mapping result of the pixel information of the lane boundary in the point cloud data; Based on the fused point cloud data, a distance value sequence corresponding to the lane boundary is obtained, and the distance value sequence is used to indicate the lane width.

[0007] According to one aspect of the embodiments of this application, a lane boundary detection device is provided, the device comprising: The acquisition module is configured to acquire image data of the vehicle's environment via a camera, and to scan point cloud data of the vehicle's environment via a lidar. The recognition module is configured to recognize lane boundaries in the image data and obtain pixel information of the lane boundaries in the image data, the pixel information including pixel positions; The processing module is configured to map the image data to the point cloud data to obtain fused point cloud data, wherein the fused point cloud data is used to indicate the mapping result of the pixel information of the lane boundary in the point cloud data; The acquisition module is configured to acquire a distance value sequence corresponding to the lane boundary based on the fused point cloud data, and the distance value sequence is used to indicate the lane width.

[0008] According to one aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer program, which is loaded and executed by a processor to implement the lane boundary detection method described above.

[0009] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer program to implement the above-described lane boundary detection method.

[0010] The technical solutions provided in this application can bring at least the following beneficial effects: By acquiring image data and point cloud data of the vehicle's environment respectively, the pixel data corresponding to the lane boundary in the image data is identified, and information such as depth, color, and shape of the lane boundary around the vehicle is obtained, thus expanding the data sources for boundary detection. Furthermore, by mapping the image data to the point cloud data, fused point cloud data is obtained, avoiding reliance on a single environmental data during the detection process, and improving the stability and accuracy of boundary detection results under different driving environments. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the implementation environment of the lane boundary detection method provided in the embodiments of this application; Figure 2 This is a flowchart of a lane boundary detection method provided in an exemplary embodiment of this application; Figure 3 This is a schematic diagram of a lane boundary detection process provided in an exemplary embodiment of this application; Figure 4 This is a flowchart of a lane boundary detection method provided in another exemplary embodiment of this application; Figure 5 This is a schematic diagram of a joint calibration process provided in an exemplary embodiment of this application; Figure 6 This is a block diagram of a lane boundary detection device provided in an exemplary embodiment of this application; Figure 7This is a structural block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0013] While driving on the road, vehicles must follow road signs, including lane boundaries. Lane boundaries are represented by lane lines of different colors and types, such as white and yellow, and solid and dashed lines. Different colors and types of lane lines indicate different driving rules. Correspondingly, in vehicle active safety and autonomous driving technologies, the driving decision-making process includes processing such as lane boundary detection to perceive the vehicle's driving environment. Lane boundary detection is implemented using one of the following methods: 1. Camera-based vision solutions acquire two-dimensional (2D) image data using a camera to obtain the geometric model corresponding to the lane boundaries, and then obtain the relative distance to the lane boundaries through 2D image recognition processing. However, camera-based vision solutions have limitations, including but not limited to at least one of the following: 1.1 The detection basis of the vision solution, namely the image data collected by the camera, lacks depth information between the camera and the lane boundary; the lane boundary information obtained by detecting based on image data is the relative distance in the image.

[0014] 1.2 In the process of obtaining the distance information corresponding to the lane boundary through 2D images, the method relies on triangulation on the plane, and the measurement error increases with the increase of distance.

[0015] 1.3 Image data acquired by cameras is highly sensitive to the driving environment. Specifically, low-light environments such as nighttime and tunnels; low-visibility weather environments such as rain, snow, and dense fog; information loss such as lane line fading causing lane information to be blurred or obscured in the image; and false edges such as shadows and guardrails interfering with lane boundaries in the environmental image. The above complex vehicle driving environment leads to a decrease in the accuracy of lane boundary detection results of camera-based vision solutions.

[0016] 2. A point cloud solution based on LiDAR: This solution uses LiDAR to scan objects in the environment surrounding the vehicle, obtaining complete point cloud data of the surrounding environment. The point cloud data includes the 3D coordinates and reflection intensity of the objects. Based on this point cloud data, algorithms such as plane segmentation and clustering are used to determine the point cloud data corresponding to the lane boundaries, obtaining information such as the distance to the lane boundaries.

[0017] However, the detection basis of this scheme, namely the point cloud data obtained by LiDAR scanning, only includes the three-dimensional coordinates and reflection intensity of the object, which has semantic gaps. When it is necessary to distinguish the color and type of lane lines, it can only rely on the difference in reflection intensity, resulting in a large detection error. In addition, when the lane boundary is far away from the vehicle, the laser signal weakens, the corresponding point cloud data is sparse, and the accuracy of lane boundary recognition decreases. Furthermore, complex driving environments cause the laser signal to weaken, resulting in low efficiency in detecting lane boundaries. For example, when the road surface is covered by water, the laser reflectivity is reduced or even absent.

[0018] To address the aforementioned issues, this application provides a lane boundary detection method. By acquiring image data and point cloud data of the vehicle's environment, the method identifies the pixel data corresponding to the lane boundary in the image data and obtains information such as depth, color, and shape of the lane boundary around the vehicle, thus expanding the data sources for boundary detection. Furthermore, by mapping the image data to the point cloud data, fused point cloud data is obtained, avoiding reliance on a single environmental data during the detection process. This improves the stability of boundary detection under different driving environments and provides a safety basis for the vehicle control system, thereby improving the accuracy of vehicle control.

[0019] This application provides a method for detecting lane boundaries. Please refer to... Figure 1 The diagram illustrates the implementation environment of the method provided in this application embodiment. This implementation environment may include: an Intelligent Driving Module (IDM) 11, a camera 12, a lidar 13, a central control screen 14, a vehicle audio system 15, a Blind Spot Detection (BSD) system 16, an Electric Power Steering (EPS) system 17, an Integrated Electronic Brake Module (IEBM) 18, and a Vehicle Central Control Module (VCCM) 19.

[0020] Optionally, cameras 12 and lidar 13 are installed around the vehicle to collect environmental information around the vehicle and send it to the IDM 11. Specifically, camera 12 collects image data of the vehicle's environment and sends it to the IDM 11, while lidar 13 acquires point cloud data of the vehicle's environment and sends it to the IDM 11. The IDM 11 implements a lane boundary detection method, including but not limited to: receiving environmental information collected by cameras 12 and lidar 13; based on the environmental information, the IDM 11 identifies lane boundaries in the image data and maps the image data to point cloud data to obtain fused point cloud data; based on the fused point cloud data, it obtains a distance value sequence corresponding to the lane boundary and generates control instructions for vehicle movement. These control instructions include, but are not limited to, at least one of the following: 1. Display Instructions: Indicatively, the central control screen 14 is used to display the distance relationship between the current vehicle and the lane boundary, or the vehicle's driving path planning, in the form of images or text, according to the display instructions generated by IDM11.

[0021] 2. Warning instructions: Indicatively, the vehicle audio system 15 is used to broadcast the current distance relationship between the vehicle and the lane boundary according to the warning instructions of the IDM11, such as broadcasting "The vehicle is deviating from the lane, please hold the steering wheel firmly"; or, the BSD16 is used to confirm the status of parallel vehicles in the adjacent blind spot when the vehicle is changing lanes or turning back to center, and to provide a warning by flashing indicator lights according to the warning instructions of the IDM11.

[0022] 3. Control command: Indicatively, EPS17 is used to receive control commands sent by IDM11 and output assist torque. The assist torque is used to turn the steering wheel to ensure that the vehicle always follows the center line of the lane.

[0023] For example, IEBM18 is used to receive the acceleration to be executed in the control command sent by IDM11, convert the acceleration to be executed into assist torque, and then send the assist torque to VCCM19, which controls the vehicle's movement. In one possible implementation, IEBM18 is also used to receive the braking request in the control command sent by IDM11, convert the deceleration value corresponding to the braking request into braking torque, and then send the braking torque to VCCM19, which controls the vehicle's deceleration. IDM11, camera 12, lidar 13, central control screen 14, vehicle audio system 15, BSD 16, EPS 17, IEBM18, and VCCM19 establish a communication connection via a wired or wireless network.

[0024] Based on the above Figure 1 The implementation environment shown in this application provides a method for detecting lane boundaries. Figure 2As shown, taking the application of this method to IDM as an example, the method includes at least one of steps 210 to 240.

[0025] Step 210: Collect image data of the vehicle's environment using a camera, and scan point cloud data of the vehicle's environment using a lidar.

[0026] The IDM (Integrated Device Manager) observes the environment around the vehicle using cameras and lidar. The cameras acquire image data, and the lidar acquires point cloud data. The cameras and lidar are installed around the vehicle. This application does not limit the number of cameras and lidars; optionally, taking four cameras and four lidars as an example, the cameras and lidars can be installed on the front, rear, left, and right sides of the vehicle, respectively.

[0027] Image data of the vehicle's environment is acquired via a camera. This image data includes, but is not limited to, color information such as RGB (Red, Green, Blue) values ​​and location information such as pixel coordinates. The process of scanning the vehicle's environment with a LiDAR scanner includes, but is not limited to, the following: the LiDAR measures the emitted laser signal and receives the laser signal reflected by objects in the vehicle's environment; based on the time difference between the emitted and received electromagnetic waves and the beam pointing angle, the LiDAR obtains the slant range and azimuth angle of objects in the vehicle's environment relative to the LiDAR, thereby obtaining the location information of the objects and obtaining the point cloud data corresponding to the vehicle's environment. The point cloud data is a set of three-dimensional coordinates of the reflection points of each object in the spatial environment.

[0028] In some embodiments, the point cloud data includes coordinate data and reflection intensity. The reflection intensity indicates the magnitude of the echo energy of the electromagnetic wave from the surface of the object reflecting the laser signal, and is related to at least one characteristic such as the material of the object surface, its roughness, and the incident direction of the laser signal. Optionally, based on the reflection intensity in the point cloud data, it is possible to distinguish object categories, estimate the surface roughness of objects, or detect concealed targets. Schematic, the reflection intensity is the amplitude of the energy ratio between the reflected signal and the emitted signal; a higher reflection intensity means a greater echo energy of the electromagnetic wave from the object surface, correspondingly indicating a higher conductivity or dielectric constant of the object surface material reflecting the laser signal, a smoother surface, or a more perpendicular incident angle of the laser signal.

[0029] Step 220: Identify the lane boundaries in the image data and obtain the pixel information of the lane boundaries in the image data.

[0030] Pixel information includes pixel location. In some embodiments, an edge pixel set corresponding to the lane boundary in the image data is identified, the edge pixel set being used to indicate the contour features of the lane boundary near the vehicle side; the edge pixel set is fitted to obtain the pixel information of the lane boundary in the image data.

[0031] Methods for identifying the set of edge pixels corresponding to lane boundaries in image data include, but are not limited to: identifying regions of the same color in image data through RGB color segmentation algorithms; detecting straight lines or curves in image data through edge detection algorithms, such as Canny Edge Detection and Hough Transform; and classifying pixels in the image through semantic segmentation algorithms, such as Fully Convolutional Network (FCN) and Spatial Convolutional Neural Network (SCNN), to identify lane boundaries when adjacent pixels have different categories.

[0032] In some embodiments, the process of identifying lane boundaries in image data includes, but is not limited to, inputting the image data into the YOLOv8 (You Only Look Once) detection algorithm framework to obtain the mask region corresponding to the lane boundary. The YOLOv8 detection framework is used for image segmentation, that is, assigning a category label to each pixel in the image data to distinguish the semantic boundaries of different objects or regions.

[0033] In the process of obtaining the mask region corresponding to the lane boundary, based on the input image data, the YOLOv8 detection framework outputs at least two detection boxes, where each detection box is a rectangular bounding box used to output the coordinates and size of the detection box to indicate the approximate position and range of the object in the image. Optionally, redundant detection boxes are removed using the Non-Maximum Suppression (NMS) method. Schematic, the candidate detection boxes output by the YOLOv8 detection framework are sorted by confidence level, where confidence level indicates the reliability of the detection result. The candidate detection box with the highest confidence level is selected, and the overlap between this detection box and other candidate detection boxes is calculated. Candidate detection boxes with an overlap level exceeding a preset overlap threshold are deleted. The above steps are repeated until all candidate detection boxes output by the YOLOv8 detection framework have been processed. Alternatively, by setting a confidence threshold, the target detection box corresponding to the lane boundary is determined, thereby obtaining the mask region corresponding to the lane boundary based on the target detection box corresponding to the lane boundary. Schematic, the confidence threshold is set to 0.85, that is, if the confidence level of the detection box corresponding to the lane line target reaches 0.85, then the target lane line is considered.

[0034] In some embodiments, when a mask region corresponding to the lane boundary is obtained, an edge pixel set is extracted based on the mask region. The edge pixel set is used to indicate the contour features of the lane boundary near the vehicle side. For example, for a straight lane boundary on the right side of the vehicle, the edge pixel set corresponds to the left edge of the straight lane boundary in the image data. A modified Random Sample Consensus (RANSAC) algorithm is used to perform straight line or curve fitting on the edge pixel set. Based on the fitting result, the pixel information corresponding to the lane boundary is obtained. This pixel information can be implemented as a sequence of pixel coordinates obtained based on the fitting equation of the edge pixel set. Optionally, the method of performing fitting processing on the edge pixel set is determined by the lane boundary type. The lane boundary type includes, but is not limited to, straight lane boundaries and curved lane boundaries, wherein curved lane boundaries include, but are not limited to, curves.

[0035] Schematic illustration: For straight lane boundaries, outliers are eliminated through random sampling. Outliers include, but are not limited to, pixels that fail to detect due to image shadows or dirt. The set of edge pixels after eliminating outliers is fitted to obtain the corresponding straight line equation. For curved lane boundaries, piecewise quadratic polynomial fitting is used, and a fitting error threshold is set. If the fitting error exceeds this threshold, the curve parameters are recalculated for the remaining pixel data. This process is iterated until the fitting error falls below the threshold, thus obtaining the curve equation corresponding to the edge pixel set. Schematic illustration: The pixel edge set is divided into at least two 5-meter segments, and quadratic polynomial fitting is performed. If the fitting result does not exceed the fitting error threshold of 0.3 meters, the pixel curve equation corresponding to the lane boundary in the image data is obtained.

[0036] By converting discrete sets of edge pixels into continuously parameterized lane boundary models, such as straight lines or curves, data complexity is reduced, abnormal fluctuations are smoothed, and recognition errors or noise interference are reduced. Irregular pixel objects are expressed using a unified mathematical model, which facilitates subsequent processing of the pixel information of lane boundaries in image data and improves computational efficiency.

[0037] Step 230: Map the image data to the point cloud data to obtain fused point cloud data.

[0038] The point cloud data is fused to indicate the pixel information of the lane boundary in the mapping result of the point cloud data.

[0039] In some embodiments, image data is mapped to point cloud data to determine the target region corresponding to the lane boundary in the point cloud data. The target region is used to indicate the mapping result of pixel position in the point cloud data. Based on the pixel information and the point cloud data of the target region, fused point cloud data is obtained.

[0040] In some embodiments, based on a pre-acquired transformation matrix between a first coordinate system and a second coordinate system, the pixel information of the lane boundary obtained from identifying the lane boundary in the image data is back-projected to the second coordinate system using a coordinate transformation formula to generate an initial Region of Interest (ROI) for the lane boundary in the space corresponding to the point cloud data. Here, the first coordinate system is the coordinate system corresponding to the image data, and the second coordinate system is the coordinate system corresponding to the point cloud data. Based on this initial ROI, the target region corresponding to the lane boundary in the point cloud data is obtained, and the pixel information and the corresponding point cloud data are merged and converted into PointXYZHSV format point cloud data containing three-dimensional coordinates and HSV (Hue, Saturation, Value) color information using a Point Cloud Library (PCL). The hue, saturation, and brightness information corresponding to HSV values ​​are more consistent with human intuitive color descriptions than the red, green, and blue information corresponding to RGB values, resulting in higher accuracy during color segmentation, adjustment, and feature extraction. Based on the PointXYZHSV format point cloud data including information such as reflection intensity, pixel position, and pixel color, fused point cloud data is obtained through color thresholding.

[0041] Schematic, the process of obtaining the target region corresponding to the lane boundary in the point cloud data based on the initial ROI includes, but is not limited to: expanding the initial ROI along the vehicle's driving direction and limiting the lateral and height ranges of the ROI to obtain the three-dimensional target region corresponding to the lane boundary. Optionally, the initial ROI can be expanded to 50m along the vehicle's driving direction, with a lateral range of ±6m and a height limited to 0-0.3m above the ground, forming a three-dimensional region of interest, i.e., the target region.

[0042] In some embodiments, when acquiring point cloud data in PointXYZHSV format, the point cloud data corresponding to the lane boundaries is segmented by color thresholding by setting layer thresholds for different types of lane boundaries, and then fused point cloud data is obtained through downsampling and noise reduction processing. Illustratively, the steps for acquiring fused point cloud data include, but are not limited to, at least one of the following steps: 1. The threshold range corresponding to the white lane boundary is H∈[0,20]∪[340,360], S∈[0,0.2], V∈[0.8,1.0]; the threshold range corresponding to the yellow lane boundary is H∈[25,35], S∈[0.7,1.0], V∈[0.6,0.9].

[0043] 2. After segmentation by color thresholding, voxel filtering is used to downsample the point cloud data containing pixel information. Voxel filtering divides the 3D point cloud space into a uniform cubic grid, replacing at least one point within the same voxel with the centroid of that voxel or a randomly specified point. Each cube in the grid represents a corresponding voxel. This reduces the amount of data and improves processing efficiency while maintaining the overall shape of the point cloud. Optionally, the voxel size is 0.05m × 0.05m × 0.05m.

[0044] 3. Based on the downsampled fused point cloud data, radius filtering is used to denoise the point cloud data containing pixel information, thereby obtaining fused point cloud data. For each data point in the point cloud data containing pixel information, the number of neighboring points within a preset radius is obtained; if the number of neighboring points is lower than a preset point count threshold, the point is determined to be an isolated noise point and is removed. Illustratively, the preset radius for radius filtering is implemented within 0.1m, and the point count threshold is implemented at 8. That is, within 0.1m of each point cloud data point, if the number of neighboring points is less than 8, the isolated noise point is removed.

[0045] Before fusing the image data mapped to the point cloud data with the point cloud data, the target area corresponding to the lane boundary in the point cloud data is determined, focusing on key areas, reducing the processing burden of irrelevant data, improving computational efficiency, and eliminating background noise interference in the point cloud data. This reduces matching errors caused by spatial misalignment or redundancy of multimodal data, improves the accuracy of fusion processing, and thus improves the accuracy of lane boundary detection.

[0046] Step 240: Based on the fused point cloud data, obtain the distance value sequence corresponding to the lane boundary.

[0047] The distance value sequence is used to indicate lane width. Optionally, the distance value sequence includes a left distance sequence of the vehicle's distance from the left lane boundary and a right distance sequence of the vehicle's distance from the right lane boundary.

[0048] In some embodiments, after dividing and fusing the point cloud data, smoothing is performed to obtain first fused point cloud data and second fused point cloud data. The first fused point cloud data is used to indicate the data corresponding to the lane boundary on the left side of the vehicle, and the second fused point cloud data is used to indicate the data corresponding to the lane boundary on the right side of the vehicle. Sampling processing is performed on the first fused point cloud data and the second fused point cloud data respectively to obtain a distance value sequence.

[0049] Schematic illustration: With the second coordinate system implemented as the coordinate system corresponding to the point cloud data, using the origin of the second coordinate system, i.e., the vehicle's centroid, as the reference, the area along the positive X-axis is divided into the right lane boundary region, and the area along the negative X-axis is divided into the left lane boundary region, thus initially separating the fused point cloud data corresponding to the left and right lane boundaries. The initially separated point cloud data is then smoothed and sampled to obtain a distance value sequence. The steps for obtaining the distance value sequence include, but are not limited to, at least one of the following: 1. Perform smoothing processing on the segmented fused point cloud data to obtain the first fused point cloud data corresponding to the lane boundary on the left side of the vehicle, and the second fused point cloud data corresponding to the lane boundary on the right side of the vehicle.

[0050] Optionally, a statistical filtering method is applied to the segmented fused point cloud data to obtain the mean μ and standard deviation σ corresponding to the fused point cloud data on both sides of the vehicle, where the standard deviation σ is used to measure the dispersion of the fused point cloud data distribution. Outlier data points outside μ±3σ in the fused point cloud data on both sides of the vehicle are identified and removed. Then, a sliding window mean filter is applied to obtain a pre-set fixed-width sliding window, which slides along the fused point cloud data on both sides of the vehicle point cloud point by point. The average value of at least two data points within the sliding window is taken, and this average value replaces the point cloud data point corresponding to the center of the sliding window. This calculation is repeated after the sliding window moves, until the fused point cloud data on that side has been traversed. Indicatively, the sliding window size is set to 10 point cloud data points.

[0051] 2. Sampling processing is performed on the smoothed first and second fused point cloud data respectively to obtain distance value sequences. Schematic, the first and second fused point cloud data are divided into at least two sub-data segments, and basic spline (B-spline) curve fitting is used to generate two continuous three-dimensional curve models, respectively indicating the lane boundaries on both sides of the vehicle; based on the curve models and sampling processing, the distance value sequence between the lane boundaries on both sides of the vehicle is obtained.

[0052] Schematic, the first and second fused point cloud data are divided into at least two sub-data segments using 3m as the segmentation standard, where the data points in each sub-data segment belong to the same distance interval. B-spline fitting is used to obtain fitting models corresponding to both sides of the vehicle. Along the vehicle's driving direction, i.e., the Z-axis in the second coordinate system, at least two sets of coordinate pairs are obtained based on the above fitting models, where each coordinate pair includes data points on the lane boundaries on both sides of the vehicle. The Euclidean distance between each coordinate pair is calculated to obtain a distance value sequence, which indicates the distance to the lane boundaries on both sides of the vehicle, i.e., the lane width. Optionally, sampling points are obtained at 0.2m intervals to obtain at least two sets of coordinate pairs in the fitting model.

[0053] By distinguishing the first fused point cloud data and the second fused point cloud data corresponding to the drivable boundaries on both sides of the vehicle, the road structure is clarified, avoiding confusion during boundary detection. Furthermore, by fitting the lane boundaries on the left and right sides of the vehicle respectively, the lane width and the vehicle's center position can be described more accurately, reducing unilateral interference, such as the impact of temporary obstacles or occlusions on boundary detection. This improves the accuracy of boundary detection and localization in complex driving scenarios.

[0054] In some embodiments, the distance value sequence is corrected using a preset system model to obtain a corrected distance value sequence. The system model is used to indicate the change pattern of the distance value sequence over time during vehicle travel. Optionally, the distance value sequence is dynamically corrected using Kalman filtering. Kalman filtering is a recursive correction algorithm that uses a prediction and update loop to fuse the prediction model and observation data, and controls vehicle travel based on the corrected distance value sequence.

[0055] Schematic, Kalman filtering, based on historical state estimation and the state equation, obtains the prior probability distribution of the current state. The vehicle control system acquires the current observation distribution through sensors and obtains the correction result based on the prior probability distribution and the current observation distribution. Optionally, the state equation is set as a first-order Markov process, and the observation noise covariance is dynamically adjusted according to the point cloud density. This observation noise covariance is used to indicate the degree of uncertainty of sensor observations. High point cloud data density indicates high observation reliability, so the observation noise covariance is reduced to increase the contribution of the observation data in boundary detection; the opposite is true when the point cloud data density is low. For example, the observation noise covariance R = 0.0004 is set when there are 50 data points per 1m², and R = 0.0001 is set when there are 100 data points per 1m².

[0056] In some embodiments, after obtaining the corrected distance value sequence, the corrected distance value sequence is transmitted to the planning and control system via the Controller Area Network (CAN) bus, and the distance value sequence is used as the planning and control basis to guide the vehicle's driving.

[0057] By correcting the distance value sequence through a preset system model, the system can dynamically correct the distance value sequence in real time during driving. It integrates observation data and prediction data corresponding to the pattern prediction model to improve the stability of boundary detection, thereby improving the stability of vehicle control based on detection results.

[0058] For illustrative purposes, please refer to the following: Figure 3 , Figure 3 This is a schematic diagram of a lane boundary detection process provided in an exemplary embodiment of this application, as shown below. Figure 3As shown, based on pre-set calibration parameters 310, image data of the vehicle's environment is acquired by camera 321, and point cloud data of the vehicle's environment is acquired by lidar 322. Image recognition 331 identifies the pixel information 332 in the image data corresponding to the lane boundary; schematically, the image data is input into the YOLOv8 detection framework to obtain pixel information 332. Furthermore, based on the mapping of the image data to the point cloud data, a target region 340 corresponding to the lane boundary in the point cloud data is determined. The target region 340 indicates the mapping result of the pixel position corresponding to pixel information 332 in the point cloud data; that is, the target region 340 is the region corresponding to the lane boundary in the point cloud data.

[0059] like Figure 3 As shown, based on the point cloud data corresponding to pixel information 330 and target region 340, and with the transformation matrix pre-acquired, a Perspective-n-Point (PnP) problem 350 is constructed. Solving this PnP problem yields a target transformation matrix 360 corresponding to pixel information 330 and target region 340. This target transformation matrix 360 corresponds to the image data and point cloud data acquired in the current driving scene. Based on the target transformation matrix 360, the image data is mapped to the point cloud data. First, point cloud data with pixel information 370 is obtained by fusing the point cloud data corresponding to pixel information 330 and target region 340. Then, downsampling and noise reduction are performed on the point cloud data 370 with pixel information to obtain fused point cloud data 380. Based on the fused point cloud data 380, a distance value sequence 390 is obtained through sampling and smoothing. Optionally, the distance value sequence 390 is corrected based on a preset system model.

[0060] In summary, the method provided in this application expands the data sources for boundary detection by acquiring image data and point cloud data of the vehicle's environment, respectively, identifying pixel data corresponding to lane boundaries in the image data, and obtaining information such as depth, color, and shape of lane boundaries around the vehicle. Furthermore, by mapping image data to point cloud data to obtain fused point cloud data, the method avoids relying on single environmental data during the detection process, thereby improving the stability and accuracy of boundary detection results under different driving environments.

[0061] Another embodiment of this application provides a lane boundary detection method as follows: Figure 4 As shown, taking the application of this method to IDM as an example, the method includes the following steps. Optionally, step 410 is included before step 210 above.

[0062] Step 410: Obtain the transformation matrix between the first coordinate system corresponding to the image data and the second coordinate system corresponding to the point cloud data.

[0063] The transformation matrix is ​​used to indicate the mapping of image data to point cloud data. In some embodiments, the image data is two-dimensional data and the point cloud data is three-dimensional data; correspondingly, the first coordinate system is a two-dimensional coordinate system and the second coordinate system is a three-dimensional coordinate system.

[0064] In some embodiments, reference image data is acquired by a camera and reference point cloud data is scanned by a lidar based on preset joint calibration parameters. The reference image data corresponds to a first coordinate system and the reference point cloud data corresponds to a second coordinate system. A transformation matrix is ​​obtained based on the joint calibration parameters, the reference image data, and the reference point cloud data.

[0065] In some embodiments, the process of obtaining the transformation matrix is ​​implemented during the testing or training process. Optionally, the joint calibration uses a checkerboard calibration board, with the intersection points of the squares, i.e., the corner points, as two-dimensional feature points, thereby establishing a coordinate correspondence by extracting the pixel coordinates of the corner points in the image data and their corresponding world coordinates. Schematably, the checkerboard calibration board is implemented with 20×20 squares, each square being 70mm in size. Optionally, the calibration board is controlled to change different poses within the sensor's field of view, wherein the poses include, but are not limited to, translational poses, rotational poses, and tilt angle change poses.

[0066] Simultaneously, reference image data of the test vehicle's environment is acquired via a camera, and reference point cloud data of the same environment is scanned via LiDAR. During the acquisition of environmental information corresponding to the test vehicle's environment, it is ensured that the calibration board is completely and unobstructedly exposed within the scanning range of the LiDAR and the field of view of the camera. The reference image data acquired by the camera and the reference point cloud data acquired by the LiDAR are processed to determine the data corresponding to the corner points of the calibration board within the reference data.

[0067] In some embodiments, an edge-related corner extraction algorithm is used on the reference point cloud data. A plane segmentation algorithm, such as RANSAC, is used to separate the calibration board plane from the reference point cloud data. Edge point cloud data corresponding to the calibration board is extracted based on boundary point clustering. Furthermore, the three-dimensional coordinates (X, Y, Z) corresponding to at least four corner points of the calibration board are located by combining the geometric constraints corresponding to the checkerboard calibration board. Geometric constraints refer to rules that reduce unknown parameters by utilizing prior knowledge of spatial geometry. Prior knowledge includes, but is not limited to, planarity, orthogonality, parallelism, and known dimensions. During joint calibration using the checkerboard calibration board, geometric constraints include, but are not limited to: 1. Planar constraint: all corner points are located on the same physical plane; 2. Mesh size constraint: the horizontal or vertical spacing between adjacent corner points is known and fixed; 3. Orthogonality constraint: the rows and columns of the checkerboard are perpendicular to each other; 4. Linear constraint: corner points in the same row or column are collinear. This application embodiment does not limit these constraints.

[0068] In some embodiments, a corner detection algorithm, such as the Harris corner detection algorithm, is used on the reference image data to extract the pixel coordinates (u,v) corresponding to the corners of the calibration board, and the accuracy of determining the corner positions is improved by methods such as iterative least squares fitting.

[0069] It is worth noting that the detection processing of the reference image data and reference point cloud data described above is only illustrative of the distance, and the specific method for determining the corner points of the calibration board is not limited in the embodiments of this application.

[0070] By collecting reference image data and reference point cloud data under the specifications of preset joint calibration parameters, and the transformation matrix between the first coordinate system and the second coordinate system, and pre-determining the transformation matrix that conforms to the actual test scenario in at least one set of training data, the positioning accuracy and transformation accuracy between image data and point cloud data are improved, thereby improving the accuracy of lane boundary detection.

[0071] In some embodiments, the transformation matrix is ​​obtained by at least one of the following steps: based on the aforementioned reference image data and reference point cloud data, constructing a PnP problem and solving for the extrinsic parameter matrix M, which serves as the transformation matrix between the first and second coordinate systems. Illustratively, the formula corresponding to the PnP problem is as follows:

[0072]

[0073] Where (u,v) is the mapping of the reference point cloud data in the second coordinate system onto the first coordinate system; ) are the scale factors in the x-axis and y-axis directions of the first coordinate system, respectively; (X,Y,Z) represents the principal point coordinates corresponding to the camera. These principal point coordinates are preset joint calibration parameters and are inherent parameters of the camera. They are used to indicate the position of the intersection of the camera's optical axis and the image sensor in the first coordinate system; R is the rotation matrix; T is the translation matrix; and (X,Y,Z) represents the three-dimensional coordinates of the reference point cloud data in the second coordinate system. to These are the 12 parameters in the transformation matrix M; s is the preset scaling factor.

[0074] Schematic, the mapped coordinates of the reference point cloud data in the second coordinate system in the first coordinate system can be represented as:

[0075]

[0076] In summary, given at least one set of reference image data and reference point cloud data to obtain corresponding coordinate sets (u,v) and (X,Y,Z), the transformation matrix is ​​obtained by constructing and solving a PnP problem. Illustratively, the formula corresponding to the PnP problem is converted to:

[0077] Thus, the extrinsic parameter matrix M, also known as the transformation matrix, is calculated. The extrinsic parameter matrix M, or transformation matrix, can be expressed as:

[0078] For illustrative purposes, please refer to the following: Figure 5 , Figure 5 This is a schematic diagram of a joint calibration process provided in an exemplary embodiment of this application, as shown below. Figure 5 As shown, reference image data 521 is acquired by camera 511, and reference point cloud data 522 is scanned by lidar 512. Given the acquired reference image data 521 and reference point cloud data 522, based on preset joint calibration parameters, it is determined whether the reference image data 521 and reference point cloud data 522 have data corresponding to common calibration board corner points. A judgment step 530 is then executed, confirming the existence of common calibration board data. This common calibration board data indicates that the observation data from different sensors, such as image data and point cloud data acquired by the camera and lidar respectively, are associated with the same set of calibration board corner points through the world coordinate system corresponding to the calibration board.

[0079] like Figure 5 As shown, if the reference image data 521 and the reference point cloud data 522 do not have common calibration board data, the above data acquisition steps are repeated, that is, the reference image data 521 is acquired by the camera 511, and the reference point cloud data 522 is scanned by the lidar 512; if the reference image data 521 and the reference point cloud data 522 have common calibration board data, step 540 is first executed to obtain the corner coordinates corresponding to the reference image data; optionally, a corner detection algorithm is used to extract the pixel coordinates corresponding to the corners of the calibration board. Step 551 is executed to preprocess the reference point cloud data; wherein, the preprocessing of the reference point cloud data includes, but is not limited to, noise reduction processing of the reference point cloud data by filtering, and separation of the calibration board plane from the reference point cloud data by a plane segmentation algorithm.

[0080] like Figure 5 As shown, given the preprocessed reference point cloud data, step 552 is executed to obtain the corner coordinates corresponding to the common data in the reference point cloud data. Based on constructing and solving the PnP problem, the transformation matrix 560 is obtained.

[0081] In summary, the method provided in this application expands the data sources for boundary detection by acquiring image data and point cloud data of the vehicle's environment, respectively, identifying pixel data corresponding to lane boundaries in the image data, and obtaining information such as depth, color, and shape of lane boundaries around the vehicle. Furthermore, by mapping image data to point cloud data to obtain fused point cloud data, the method avoids relying on single environmental data during the detection process, thereby improving the stability and accuracy of boundary detection results under different driving environments.

[0082] The method provided in this application establishes a correlation between image data and point cloud data by obtaining the transformation matrix between the first coordinate system and the second coordinate system, thereby achieving spatial alignment and avoiding the accumulation of errors in layered calibration through direct mathematical transformation. Based on the transformation matrix, the method fuses vehicle environment information obtained by LiDAR and camera respectively, thereby improving the computational efficiency of the mapping process and enhancing the reliability of boundary detection under different driving environments.

[0083] Figure 6 This is a structural block diagram of a lane boundary detection device provided in an exemplary embodiment of this application, as shown below. Figure 6 As shown, the device includes: The acquisition module 610 is configured to acquire image data of the vehicle's environment via a camera, and to scan point cloud data of the vehicle's environment via a lidar. The recognition module 620 is configured to recognize lane boundaries in image data and obtain pixel information of the lane boundaries in the image data, including pixel positions. The processing module 630 is configured to map image data to point cloud data to obtain fused point cloud data, which is used to indicate the mapping result of pixel information of lane boundaries in the point cloud data. The acquisition module 640 is configured to acquire a distance value sequence corresponding to the lane boundary based on fused point cloud data. The distance value sequence is used to indicate the lane width.

[0084] In an optional embodiment, the processing module 630 is further configured to map image data to point cloud data and determine a target region corresponding to a lane boundary in the point cloud data, the target region being used to indicate the mapping result of pixel positions in the point cloud data; The acquisition module 640 is also configured to acquire fused point cloud data based on pixel information and point cloud data of the target area.

[0085] In an optional embodiment, the recognition module 620 is further configured to recognize a set of edge pixels corresponding to a lane boundary in the image data, the set of edge pixels being used to indicate the contour features of the lane boundary near the vehicle side. The processing module 630 is also configured to perform fitting processing on the edge pixel set to obtain the pixel information of the lane boundary in the image data.

[0086] In an optional embodiment, the acquisition module 640 is further configured to obtain first fused point cloud data and second fused point cloud data by smoothing after dividing the fused point cloud data. The first fused point cloud data is used to indicate the data corresponding to the lane boundary on the left side of the vehicle, and the second fused point cloud data is used to indicate the data corresponding to the lane boundary on the right side of the vehicle. The processing module 630 is also configured to perform sampling processing on the first fused point cloud data and the second fused point cloud data respectively to obtain a distance value sequence.

[0087] In an optional embodiment, after the acquisition module 640 acquires the distance value sequence corresponding to the lane boundary based on the fused point cloud data, the processing module 630 is further configured to correct the distance value sequence through a preset system model to obtain a corrected distance value sequence. The system model is used to indicate the change pattern of the distance value sequence over time during vehicle travel.

[0088] In an optional embodiment, before the processing module 630 maps the image data to the point cloud data to obtain the fused point cloud data, the acquisition module 640 is further configured to acquire a transformation matrix between the first coordinate system corresponding to the image data and the second coordinate system corresponding to the point cloud data, the transformation matrix being used to indicate mapping the image data to the point cloud data.

[0089] In an optional embodiment, the acquisition module 610 is further configured to acquire reference image data via a camera and scan reference point cloud data via a lidar based on preset joint calibration parameters, wherein the reference image data corresponds to a first coordinate system and the reference point cloud data corresponds to a second coordinate system. The acquisition module 640 is also configured to acquire the transformation matrix based on joint calibration parameters, reference image data, and reference point cloud data.

[0090] In summary, the apparatus provided in this application expands the data sources for boundary detection by acquiring image data and point cloud data of the vehicle's environment, respectively, identifying pixel data corresponding to lane boundaries in the image data, and obtaining information such as depth, color, and shape of lane boundaries around the vehicle. Furthermore, by mapping image data to point cloud data to obtain fused point cloud data, it avoids relying on a single environmental data during the detection process, thereby improving the stability and accuracy of boundary detection results under different driving environments.

[0091] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0092] Please refer to Figure 7 This diagram illustrates a structural block diagram of a computer device 700 provided in one embodiment of this application. The computer device 700 may be... Figure 1 The IDM shown is used to implement the lane boundary detection method provided in the above embodiments. Specifically: Typically, computer device 700 includes a processor 701 and a memory 702.

[0093] Processor 701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 701 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 701 may also include a main processor and a coprocessor. The main processor, also known as a central processing unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 701 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 701 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0094] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 702 are used to store a computer program configured to be executed by one or more processors to implement the lane boundary detection method described above.

[0095] In some embodiments, the computer device 700 may also optionally include other components 703: a peripheral device interface and at least one peripheral device. The processor 701, memory 702, and peripheral device interface can be connected via a bus or signal lines. Each peripheral device can be connected to the peripheral device interface via a bus, signal lines, or a circuit board. Specifically, the peripheral device includes at least one of: radio frequency circuitry, a display screen, audio circuitry, and a power supply.

[0096] Those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the computer device 700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0097] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein a computer program is stored in the storage medium, and the computer program, when executed by a processor, implements the lane boundary detection method described above. Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0098] In an exemplary embodiment, a computer program product is also provided, the computer program product including a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the lane boundary detection method described above.

[0099] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions when applied in practice. Informed consent or separate consent from the personal information subject is obtained, and subsequent data use and processing are conducted within the scope authorized by laws, regulations, and the personal information subject. For example, the image data and point cloud data involved in this application were obtained with full authorization.

[0100] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0101] Furthermore, the step numbers described herein are merely illustrative of one possible execution order between steps. In some other embodiments, the steps may not be executed in the order of their numbers, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0102] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting lane boundaries, characterized in that, The method includes: The vehicle acquires image data of its environment using a camera, and scans point cloud data of the environment using a lidar scanner. Identify the lane boundary in the image data to obtain the pixel information of the lane boundary in the image data, the pixel information including the pixel position; The image data is mapped to the point cloud data to obtain fused point cloud data, which is used to indicate the mapping result of the pixel information of the lane boundary in the point cloud data; Based on the fused point cloud data, a distance value sequence corresponding to the lane boundary is obtained, and the distance value sequence is used to indicate the lane width.

2. The method according to claim 1, characterized in that, The step of mapping the image data to the point cloud data to obtain fused point cloud data includes: The image data is mapped to the point cloud data to determine the target area corresponding to the lane boundary in the point cloud data. The target area is used to indicate the mapping result of the pixel position in the point cloud data. Based on the pixel information and the point cloud data of the target area, fused point cloud data is obtained.

3. The method according to claim 1, characterized in that, The process of identifying lane boundaries in the image data and obtaining pixel information of the lane boundaries in the image data includes: Identify the set of edge pixels corresponding to the lane boundary in the image data, wherein the set of edge pixels is used to indicate the contour features of the lane boundary near the vehicle side; The edge pixel set is fitted to obtain the pixel information of the lane boundary in the image data.

4. The method according to any one of claims 1 to 3, characterized in that, The step of obtaining the distance value sequence corresponding to the lane boundary based on the fused point cloud data includes: After dividing the fused point cloud data, smoothing is performed to obtain first fused point cloud data and second fused point cloud data. The first fused point cloud data is used to indicate the data corresponding to the lane boundary on the left side of the vehicle, and the second fused point cloud data is used to indicate the data corresponding to the lane boundary on the right side of the vehicle. Sampling processing is performed on the first fused point cloud data and the second fused point cloud data respectively to obtain the distance value sequence.

5. The method according to any one of claims 1 to 3, characterized in that, After obtaining the distance value sequence corresponding to the lane boundary based on the fused point cloud data, the method further includes: The distance value sequence is corrected by a preset system model to obtain a corrected distance value sequence. The system model is used to indicate the change pattern of the distance value sequence over time during vehicle travel.

6. The method according to any one of claims 1 to 3, characterized in that, Before mapping the image data to the point cloud data to obtain the fused point cloud data, the method further includes: Obtain the transformation matrix between the first coordinate system corresponding to the image data and the second coordinate system corresponding to the point cloud data. The transformation matrix is ​​used to indicate how to map the image data to the point cloud data.

7. The method according to claim 6, characterized in that, The step of obtaining the transformation matrix between the first coordinate system corresponding to the image data and the second coordinate system corresponding to the point cloud data includes: Based on preset joint calibration parameters, reference image data is acquired through the camera, and reference point cloud data is scanned through the lidar. The reference image data corresponds to the first coordinate system, and the reference point cloud data corresponds to the second coordinate system. The transformation matrix is ​​obtained based on the joint calibration parameters, the reference image data, and the reference point cloud data.

8. A lane boundary detection device, characterized in that, The device includes: The acquisition module is configured to acquire image data of the vehicle's environment via a camera, and to scan point cloud data of the vehicle's environment via a lidar. The recognition module is configured to recognize lane boundaries in the image data and obtain pixel information of the lane boundaries in the image data, the pixel information including pixel positions; The processing module is configured to map the image data to the point cloud data to obtain fused point cloud data, wherein the fused point cloud data is used to indicate the mapping result of the pixel information of the lane boundary in the point cloud data; The acquisition module is configured to acquire a distance value sequence corresponding to the lane boundary based on the fused point cloud data, and the distance value sequence is used to indicate the lane width.

9. A non-transitory computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the lane boundary detection method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, and a processor reads from and executes the computer program to implement the lane boundary detection method as described in any one of claims 1 to 7.

Images