Road three-dimensional lane line detection method and system based on binocular vision

Abstract

The invention provides a road three-dimensional lane line detection method and system based on binocular vision, and relates to the technical field of automatic driving, and the method comprises the steps: carrying out the online calibration of internal and external parameters of a camera and the time sequence alignment of the binocular image and IMU attitude data through collecting the forward binocular image and IMU attitude data of a vehicle; processing the binocular image after time sequence alignment, and determining a dense disparity map; generating a road point cloud based on the dense disparity map and camera parameters, and constructing an adaptive terrain model; constructing a depth enhanced BEV feature map according to the depth feature of the road point cloud and the texture feature of the binocular image, predicting candidate parameters of a three-dimensional lane line, and screening through various constraints; and then executing extended Kalman filtering and trajectory optimization to obtain three-dimensional lane line parameters. According to the invention, rapid and accurate detection of the three-dimensional lane line can be realized, the accuracy of three-dimensional lane line detection under a complex terrain is improved, and the deployment cost is reduced.

Description

A method and system for detecting 3D lane lines on roads based on binocular vision Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method and system for detecting three-dimensional lane lines on roads based on binocular vision. Background Technology

[0002] In the field of autonomous driving environmental perception, 3D lane line detection is a core technology to ensure accurate vehicle positioning and safe driving. Especially in road scenarios without high-precision map coverage, its detection accuracy directly affects the reliability of path planning and control decisions. Currently, the industry urgently needs a detection solution that takes into account environmental adaptability, real-time performance, and cost-effectiveness.

[0003] In existing technologies, monocular vision solutions map 2D lane detection results to a 3D coordinate system through an inverse projection matrix and acquire depth information based on camera calibration, without requiring additional hardware costs; while lidar solutions, with their direct depth perception capabilities, provide high-precision 3D environmental data and have become one of the mainstream choices for high-level autonomous driving.

[0004] However, monocular vision solutions generally rely on the assumption of a flat road, which cannot adapt to the slope undulations and lateral tilt changes of the actual road surface, resulting in misalignment or abnormal curvature in the 3D lane line mapping. Although lidar solutions have excellent accuracy, they are expensive, bulky, and susceptible to interference from rain and fog, making it difficult to deploy them on a large scale in low- and medium-cost vehicles. They suffer from poor accuracy and high cost. Summary of the Invention

[0005] This invention provides a method and system for detecting three-dimensional lane lines on roads based on binocular vision. It can achieve rapid and accurate detection of three-dimensional lane lines, taking into account environmental adaptability, real-time performance and cost-effectiveness, improving the accuracy of three-dimensional lane line detection in complex terrain and reducing deployment costs.

[0006] To achieve the above objectives, embodiments of the present invention employ the following technical solutions: Firstly, a method for detecting three-dimensional lane lines on roads based on binocular vision is provided. This method includes: simultaneously acquiring binocular images and IMU attitude data of a vehicle traveling forward; performing online calibration of camera intrinsic and extrinsic parameters on the binocular images and IMU attitude data to correct intrinsic and extrinsic parameter drift; performing temporal alignment to correct timestamp deviations, obtaining temporally aligned binocular images and IMU attitude data; performing distortion correction, illumination compensation preprocessing, and multi-scale feature extraction on the temporally aligned binocular images; determining a dense disparity map based on IMU attitude data constraints and left-right consistency constraints; generating a road point cloud based on the dense disparity map and camera parameters; and performing local ground plane fitting and quadratic curve... An adaptive terrain model incorporating height, slope, and lateral tilt is constructed using surface expansion and normal clustering. Depth features of the road point cloud and texture features of the binocular images are mapped to a bird's-eye view to construct a depth-enhanced BEV feature map. A 3D anchor box detection network, combined with the geometric prior of the adaptive terrain model, is used to detect 3D lane line candidate parameters based on the depth-enhanced BEV feature map. Valid 3D lane line candidate parameters are then selected from these candidates through reprojection error constraints and road prior constraints. Extended Kalman filtering and trajectory optimization are performed on the valid 3D lane line candidate parameters in consecutive frames to obtain the 3D lane line parameters. Trajectory optimization ensures that the 3D lane line parameters satisfy position consistency constraints, parameter continuity constraints, and road geometric constraints.

[0007] The beneficial effects of this invention are as follows: The method provided by this invention simultaneously acquires binocular images and IMU attitude data, combines online calibration and temporal alignment of camera intrinsic and extrinsic parameters to correct parameter drift and timestamp deviation, and then proceeds through a complete process of disparity map generation, adaptive terrain modeling, BEV feature fusion, 3D anchor frame detection, and temporal optimization. This overcomes the limitations of existing monocular vision models that rely on planar assumptions, while avoiding the real-time insufficiency and false detection problems caused by existing neural network models. It improves data reliability through binocular vision and IMU data fusion, dynamically adapts to complex undulating terrain through adaptive terrain models, enriches feature expression through depth-enhanced BEV feature maps, improves candidate accuracy through 3D anchor frame detection and multi-constraint screening, and ensures parameter stability through temporal optimization. Ultimately, it achieves high-precision, high-robustness, and real-time 3D lane line detection under complex terrain, providing a reliable environmental perception basis for autonomous driving.

[0008] In one possible implementation of the first aspect, a dense disparity map is determined based on IMU pose data constraints and left-right consistency constraints, including: extracting the relative camera pose based on temporally aligned IMU pose data and calculating the fundamental matrix using camera intrinsic parameters to obtain epipolar constraint rules; defining the feature matching range of the left and right images included in the binocular images based on the epipolar constraint rules, constructing a disparity cost body containing all disparity candidates, wherein a penalty term is added to disparity candidates that violate the epipolar constraint rules, and the constraint weights are dynamically adjusted according to the confidence level of the IMU pose data; and solving the disparity cost body using the soft Argmin method. Integer disparity values ​​are obtained, and subpixel offsets are obtained through quadratic curve fitting. These are combined to form subpixel-level disparity values. Based on the subpixel-level disparity values, the disparity values ​​of each pixel in the left image and the corresponding pixels in the right image are obtained. A left-right consistency check is performed to determine whether the deviation between the two is within a preset threshold range. Pixels within the preset threshold range are marked as valid pixels, and those outside the preset threshold range are marked as invalid pixels. If there are valid pixels around an invalid pixel, a weighted interpolation method based on the surrounding valid pixels is used for correction. If there are no valid pixels around an invalid pixel, the invalid pixel is removed, and filtering and noise reduction are performed to obtain a dense disparity map.

[0009] The method provided by this invention determines a dense disparity map through IMU attitude data constraints and left-right consistency constraints. First, the relative camera attitude is extracted based on the IMU attitude, and epipolar constraint rules are obtained by combining the camera intrinsic parameters, which effectively narrows the matching range of left and right image features and reduces invalid matching calculations. Penalty terms are added to disparity candidates that violate epipolar constraints, and the constraint weights are dynamically adjusted to further improve matching accuracy. The sub-pixel level disparity values ​​are solved by the soft Argmin method, which significantly improves the disparity calculation accuracy compared to integer disparity. Then, valid and invalid pixels are marked by left-right consistency checks, and invalid pixels are corrected or removed by weighted interpolation. Combined with filtering and noise reduction, a high-quality dense disparity map is finally obtained, which effectively suppresses mismatches and noise interference, and provides high-precision data support for subsequent road point cloud generation and terrain modeling.

[0010] In one possible implementation of the first aspect, generating a road point cloud based on a dense disparity map and camera parameters includes: obtaining the focal length, principal point coordinates, and binocular baseline length of the left camera based on the camera intrinsic parameter matrix; performing normalization processing on the coordinates of each effective pixel in the dense disparity map to obtain the relative coordinates of each effective pixel in the camera coordinate system; calculating the true depth of the road point corresponding to each effective pixel based on the disparity value of each effective pixel; converting the relative coordinates into three-dimensional coordinates in the camera coordinate system based on the true depth; converting the three-dimensional coordinates of all effective pixels in the camera coordinate system into three-dimensional coordinates in the vehicle coordinate system; and generating a road point cloud based on the three-dimensional coordinates of each effective pixel in the vehicle coordinate system.

[0011] The method provided in this invention generates road point clouds based on dense disparity maps and camera parameters. It accurately obtains key parameters such as focal length, principal point coordinates, and binocular baseline length through the camera intrinsic parameter matrix. Normalization is performed on each valid pixel, and the true depth is calculated based on the disparity value. Then, coordinate transformation maps the valid pixels from the camera coordinate system to the vehicle coordinate system. Finally, the 3D coordinates of all valid pixels are aggregated to generate the road point cloud. The entire process follows geometric transformation logic, utilizing high-precision disparity data and camera parameters to ensure the accuracy of depth calculation and coordinate transformation. The generated road point cloud has uniform density and accurate positioning, providing a reliable 3D data foundation for the subsequent construction of adaptive terrain models and avoiding terrain modeling deviations caused by point cloud errors.

[0012] In one possible implementation of the first aspect, an adaptive terrain model including height, slope, and lateral tilt angle is constructed through local ground plane fitting, quadratic surface expansion, and normal clustering. This includes: dividing the road point cloud into grids of a preset size; performing local ground plane fitting on the point cloud data within each grid using the RANSAC algorithm to obtain the local ground plane equation and plane parameters for each grid; performing quadratic surface expansion on the local ground plane based on the local ground plane equation and point cloud data for each grid; solving for the quadratic surface coefficients using the least squares method to obtain the quadratic surface equation for each grid; calculating the normal vector of each grid based on its quadratic surface equation; normalizing the normal vector to obtain a standardized grid normal vector; and then using K... The -means clustering algorithm performs cluster analysis on the standardized normal vectors of all grids, classifying grids with similar normal vectors into the same terrain region, thus determining the terrain region classification corresponding to each grid. It obtains the quadratic surface coefficients and grid normal vectors of the current frame t, compares them with the quadratic surface coefficients and grid normal vectors of the corresponding grid in frame t-1, calculates the corresponding differences, and uses a preset threshold to determine if there are any terrain abrupt changes. For the detected terrain abrupt change grids, it combines Kalman filtering and sparse optimization algorithms to fuse terrain information from frame t-1 and multiple historical frames, correcting and adjusting the abrupt terrain parameters so that the corrected terrain parameters of frame t are continuously connected with the terrain parameters of historical frames, resulting in a continuous adaptive terrain model including height, slope, and lateral tilt angle.

[0013] The method provided in this invention constructs an adaptive terrain model through local ground plane fitting, quadratic surface expansion, and normal clustering, overcoming the limitations of the planar assumption in existing technologies. It first divides the road point cloud into a grid, fits the local ground plane using the RANSAC algorithm, then accurately captures road curvature changes through quadratic surface expansion, and uses normal clustering to classify terrain regions, comprehensively describing the terrain's height, slope, and lateral tilt. By calculating the differential detection of terrain parameters across consecutive frames, and combining Kalman filtering and sparse optimization to fuse historical information, the abrupt change parameters are corrected to ensure model continuity. This terrain model can dynamically adapt to complex terrains such as slopes and lateral tilts, providing accurate geometric priors for subsequent lane line detection, effectively reducing the 3D coordinate mapping error of lane lines caused by terrain undulations, and improving the accuracy of lane line detection in complex terrain.

[0014] In one possible implementation of the first aspect, constructing a depth-enhanced BEV feature map includes: dividing a BEV grid within a preset detection range in front of the vehicle, with each grid corresponding to a pixel in the BEV feature map; projecting the three-dimensional coordinates of the road point cloud onto the corresponding BEV grid, statistically analyzing the depth features of the point cloud within each grid, and generating a depth feature map; calculating the height coordinates of each BEV grid using interpolation based on an adaptive terrain model to obtain the complete three-dimensional coordinates of the grid; projecting the complete three-dimensional coordinates of the BEV grid onto the left and right images of the binocular image, respectively, extracting the texture features at the corresponding locations, and generating a texture feature map; and concatenating and fusing the depth feature map and the texture feature map by channel to obtain the depth-enhanced BEV feature map.

[0015] The method provided by this invention constructs a depth-enhanced BEV feature map. By dividing the area in front of the vehicle into a BEV grid within a pre-defined detection range, the depth statistical features of the road point cloud are transformed into a depth feature map. Simultaneously, an adaptive terrain model is used to supplement the grid's height coordinates. The grid's 3D coordinates are projected onto a binocular image to extract texture features. Finally, the two types of features are stitched and fused along their respective channels. This process fully combines the 3D spatial information of depth features with the detailed descriptive capabilities of texture features. The BEV perspective can globally perceive the road environment, avoiding the problem of insufficient information from a single feature. The generated depth-enhanced BEV feature map has richer information dimensions and stronger representational capabilities, significantly improving the accuracy of subsequent 3D anchor box detection networks in predicting lane line candidate parameters, and providing high-quality feature support for effective candidate selection.

[0016] In one possible implementation of the first aspect, a 3D anchor frame detection network is used in conjunction with the geometric prior of an adaptive terrain model to detect 3D lane line candidate parameters based on a depth-enhanced BEV feature map. This includes: inputting the depth-enhanced BEV feature map into the 3D anchor frame detection network; supplementing grid position information through learnable position encoding; aggregating global context features via a multi-head self-attention mechanism; and optimizing feature representation through a feedforward network and residual connections. Based on the height, slope, and lateral tilt parameters of the adaptive terrain model, 3D anchor frames adapted to the terrain are pre-defined on the BEV grid. Anchor frame parameters include center position, orientation angle, length, and width, and the slope and lateral tilt of the anchor frames are adapted to the terrain parameters of the corresponding grid. The 3D anchor frame detection network performs parameter regression and classification on each anchor frame, and combines the geometric prior to determine the adaptability of the anchor frame to the terrain, predicting 3D lane line candidate parameters. These 3D lane line candidate parameters include the 3D coordinates, width estimate, curvature estimate, slope estimate, lane line type, and confidence level of key nodes in the lane line center trajectory.

[0017] The method provided by this invention employs a 3D anchor box detection network combined with geometric priors from an adaptive terrain model to predict 3D lane line candidate parameters. First, a depth-enhanced BEV feature map is input into the network, and positional information is supplemented through learnable positional encoding. The feature representation is then optimized via a multi-head self-attention mechanism and a feedforward network. Next, based on parameters such as height and slope of the adaptive terrain model, pre-defined 3D anchor boxes adapted to the terrain are generated. Finally, the network performs parameter regression and classification on the anchor boxes. This approach achieves a higher degree of matching between the anchor boxes and the actual terrain, avoiding prediction bias caused by traditional anchor boxes being out of sync with the terrain. Simultaneously, the network accurately captures global lane line features through global feature aggregation, and combines geometric priors to determine anchor box adaptability, ultimately accurately predicting candidate parameters such as the 3D coordinates, curvature, and slope of the lane line center trajectory, providing high-quality foundational data for subsequent effective candidate selection.

[0018] In one possible implementation of the first aspect, effective 3D lane line candidate parameters are selected from 3D lane line candidate parameters through reprojection error constraints and road prior constraints. This includes: for each 3D lane line candidate, the 3D coordinates of key nodes are projected onto the binocular image in combination with camera intrinsic and extrinsic parameters to obtain the projected pixel coordinates of each node; image features around the projected pixel coordinates in the binocular image are extracted, and the deviation between the projected pixel coordinates and the pixel coordinates of the lane line texture features in the image is calculated to obtain the reprojection error of each node; if the reprojection error of any node exceeds a preset pixel threshold, the node is filtered; if the reprojection error of more than a preset proportion of nodes in the same 3D lane line candidate exceeds the standard, the 3D lane line candidate parameter is removed; for candidates that pass the reprojection error constraints, road prior constraint checks are performed, including lane line width constraints, curvature constraints, adjacent node continuity constraints, and adjacent lane line parallelism constraints; and 3D lane line candidate parameters that satisfy all constraints are retained as effective 3D lane line candidate parameters.

[0019] The method provided by this invention filters effective 3D lane line candidates through reprojection error constraints and road prior constraints. First, the 3D coordinates of the lane line candidate nodes are projected onto the left and right binocular images. The reprojection error is calculated, and nodes and candidates with excessive deviations are filtered out to ensure consistency between the candidates and image texture features. Then, road prior constraints such as lane line width, curvature, continuity, and parallelism are applied to eliminate candidates that do not conform to the physical laws of real roads. This dual constraint mechanism filters candidates from both visual matching consistency and physical laws perspectives, effectively filtering false positives caused by feature mismatches, terrain interference, etc., significantly reducing the proportion of invalid candidates, and obtaining high-quality, effective 3D lane line candidates. This provides a guarantee for subsequent time-series optimization to output stable and reliable lane line parameters.

[0020] In one possible implementation of the first aspect, trajectory optimization includes: constructing a trajectory optimization objective function, which includes a data term, a smoothing term, and a prior term. The data term is used to measure the consistency between the optimized position and the observed position, the smoothing term is used to ensure the continuity of parameters across frames, and the prior term is used to introduce road geometric constraints; iteratively optimizing the objective function using a nonlinear optimization algorithm, using the result of the extended Kalman filter as the initial value, and stopping when the change in the objective function is less than a preset threshold or the maximum number of iterations is reached; setting an optimization frequency, performing real-time optimization every frame to ensure real-time detection, and performing batch optimization every preset number of frames to improve the consistency of global parameters.

[0021] The method provided by this invention constructs a trajectory optimization objective function that includes data terms, smoothing terms, and prior terms. The data terms ensure that the optimized position is consistent with the observed position, the smoothing terms ensure the continuity of parameters across frames, and the prior terms introduce road geometric constraints. Then, a nonlinear optimization algorithm is used to iteratively optimize using the extended Kalman filter results as initial values, and a reasonable optimization frequency is set. This optimization method takes into account the accuracy of detection results, cross-frame continuity, and road adaptability. Real-time optimization ensures real-time detection, batch optimization improves the consistency of global parameters, and additional optimization is triggered when there are abrupt changes in terrain or a decrease in confidence, effectively avoiding lane line parameter jumps. This makes the optimized lane line parameters more stable and accurate, meeting the high-precision requirements of autonomous driving for lane line detection.

[0022] In one possible implementation of the first aspect, the extended Kalman filter and cross-frame continuity constraints include: using the effective three-dimensional lane line candidate parameters of consecutive frames as observations, combining vehicle speed and turning angle, and performing prediction and update steps through extended Kalman filtering to obtain initially stable lane line parameters; applying cross-frame continuity constraints, where the positional continuity constraint requires that the positional change of corresponding lane line points in adjacent frames does not exceed the product of the maximum permissible vehicle speed and the frame interval, and the parameter continuity constraint requires that the changes in lane line curvature and slope in adjacent frames do not exceed the product of the corresponding preset maximum rate of change and the frame interval; smoothing correction is performed on lane line parameters that violate the cross-frame continuity constraints, and the parameters are readjusted by combining the optimization results of historical frames with the observations of the current frame to ensure that there are no jumps in the cross-frame lane line parameters, thereby obtaining three-dimensional lane line parameters, which include the three-dimensional coordinates, curvature parameters, slope parameters, width parameters, lateral offset, confidence level, and lane line type of key nodes on the lane line center trajectory.

[0023] The method provided by this invention combines extended Kalman filtering with cross-frame continuity constraints. First, using effective 3D lane line candidate parameters as observations, prediction and updates are performed in conjunction with vehicle speed and turning angle to obtain initially stable lane line parameters. Then, positional continuity constraints limit the range of lane line point position changes between adjacent frames, while parameter continuity constraints limit the range of curvature and slope changes, smoothly correcting parameters that violate these constraints. This combined approach fully utilizes the predictive and updating capabilities of extended Kalman filtering and the stability guarantee provided by continuity constraints. It improves prediction accuracy through vehicle speed and turning angle, and avoids parameter jumps through constraint correction. By combining historical frame optimization results with current frame observations to adjust parameters, it ultimately outputs stable and continuous 3D lane line parameters, providing a reliable decision-making basis for autonomous driving upper-level control algorithms.

[0024] Secondly, the present invention provides a road 3D lane line detection system based on binocular vision, applied to the road 3D lane line detection method based on binocular vision according to any one of the first aspects above. The system includes: a data acquisition module, used to simultaneously acquire binocular images and IMU attitude data of the vehicle in the forward direction, perform online calibration of camera intrinsic and extrinsic parameters on the binocular images and IMU attitude data, and correct intrinsic and extrinsic parameter drift; and perform temporal alignment to correct timestamp deviation, thereby obtaining temporally aligned binocular images and IMU attitude data; a data processing module, used to perform distortion correction, illumination compensation preprocessing, and multi-scale feature extraction on the temporally aligned binocular images, and determine a dense disparity map based on IMU attitude data constraints and left-right consistency constraints; and a model building module, used to generate road point clouds based on the dense disparity map and camera parameters, and to obtain road point clouds through local horizons. The system employs surface fitting, quadratic surface expansion, and normal clustering to construct an adaptive terrain model incorporating height, slope, and lateral tilt. A feature map construction module maps the depth features of road point clouds and the texture features of binocular images to a bird's-eye view, constructing a depth-enhanced BEV feature map. A detection module uses a 3D anchor box detection network combined with the geometric prior of the adaptive terrain model to detect 3D lane line candidate parameters based on the depth-enhanced BEV feature map, and filters effective 3D lane line candidate parameters from these parameters through reprojection error constraints and road prior constraints. An optimization module performs extended Kalman filtering and trajectory optimization on the effective 3D lane line candidate parameters in consecutive frames to obtain 3D lane line parameters. Trajectory optimization ensures that the 3D lane line parameters satisfy position consistency constraints, parameter continuity constraints, and road geometric constraints.

[0025] Thirdly, an electronic device is provided, the electronic device including a memory and one or more processors; the memory is coupled to the processors; wherein the memory stores computer program code, the computer program code including computer instructions, which, when executed by the processor, cause the electronic device to perform the method as described in any implementation of the first aspect.

[0026] Fourthly, a computer-readable storage medium is provided, including computer instructions that, when executed on an electronic device, cause the electronic device to perform a method as described in any implementation of the first aspect.

[0027] Fifthly, a computer program product is provided that, when run on a computer, causes the computer to perform the method in any implementation of the first aspect.

[0028] Understandably, the beneficial effects achieved by the system of the second aspect, the electronic device of the third aspect, the computer-readable storage medium of the fourth aspect, and the computer program product of the fifth aspect provided above can be referred to with reference to the beneficial effects of the first aspect and any of its possible design embodiments, which will not be repeated here. Attached Figure Description

[0029] Figure 1 is a schematic diagram of an electronic device provided in an embodiment of the present invention; Figure 2 is a flowchart of a three-dimensional road lane line detection method based on binocular vision provided in an embodiment of the present invention; Figure 3 is a schematic diagram of a detection system provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings. In the description of the present invention, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. The "or" in the present invention is merely a description of the relationship between the related objects, indicating that three relationships can exist. For example, A or B can represent: A alone, A and B simultaneously, and B alone. A and B can be singular or plural. Furthermore, in the description of the present invention, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items.

[0031] Furthermore, to facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0032] In this embodiment of the invention, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this embodiment of the invention should not be construed as superior or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0033] In the field of autonomous driving environmental perception, 3D lane line detection is a core technology to ensure accurate vehicle positioning and safe driving. Especially in road scenarios without high-precision map coverage, its detection accuracy directly affects the reliability of path planning and control decisions. Currently, the industry urgently needs a detection solution that takes into account environmental adaptability, real-time performance, and cost-effectiveness.

[0034] In existing technologies, monocular vision solutions map 2D lane detection results to a 3D coordinate system through an inverse projection matrix and acquire depth information based on camera calibration, without requiring additional hardware costs; while lidar solutions, with their direct depth perception capabilities, provide high-precision 3D environmental data and have become one of the mainstream choices for high-level autonomous driving.

[0035] However, monocular vision solutions generally rely on the assumption of a flat road, which cannot adapt to the slope undulations and lateral tilt changes of the actual road surface, resulting in misalignment or abnormal curvature in the 3D lane line mapping. Although lidar solutions have excellent accuracy, they are expensive, bulky, and susceptible to interference from rain and fog, making it difficult to deploy them on a large scale in low- and medium-cost vehicles. They suffer from poor accuracy and high cost.

[0036] In view of this, embodiments of the present invention provide a method and system for detecting three-dimensional lane lines on roads based on binocular vision. The method includes: simultaneously acquiring binocular images and IMU attitude data of the vehicle in the forward direction; performing online calibration of camera intrinsic and extrinsic parameters on the binocular images and IMU attitude data to correct intrinsic and extrinsic parameter drift; performing temporal alignment to correct timestamp deviation and obtain temporally aligned binocular images and IMU attitude data; performing distortion correction, illumination compensation preprocessing, and multi-scale feature extraction on the temporally aligned binocular images; determining a dense disparity map based on IMU attitude data constraints and left-right consistency constraints; generating a road point cloud based on the dense disparity map and camera parameters; and performing local ground plane fitting, quadratic surface expansion, and normal clustering. An adaptive terrain model incorporating height, slope, and lateral tilt is constructed. The depth features of the road point cloud and the texture features of the binocular images are mapped to a bird's-eye view to construct a depth-enhanced BEV feature map. A 3D anchor box detection network combined with the geometric prior of the adaptive terrain model is used to detect 3D lane line candidate parameters based on the depth-enhanced BEV feature map. Valid 3D lane line candidate parameters are selected from these candidates through reprojection error constraints and road prior constraints. Extended Kalman filtering and trajectory optimization are performed on the valid 3D lane line candidate parameters in consecutive frames to obtain the 3D lane line parameters. Trajectory optimization is used to ensure that the 3D lane line parameters satisfy position consistency constraints, parameter continuity constraints, and road geometric constraints.

[0037] The method provided by this invention synchronously acquires binocular images and IMU attitude data, combines online calibration and temporal alignment of camera intrinsic and extrinsic parameters to correct parameter drift and timestamp deviations, and then proceeds through a complete process of disparity map generation, adaptive terrain modeling, BEV feature fusion, 3D anchor frame detection, and temporal optimization. This overcomes the limitations of existing monocular vision models that rely on planar assumptions, while avoiding the real-time insufficiency and false detection problems of existing neural network models. It improves data reliability through binocular vision and IMU data fusion, dynamically adapts to complex undulating terrain through adaptive terrain models, enriches feature representation through depth-enhanced BEV feature maps, improves candidate accuracy through 3D anchor frame detection and multi-constraint screening, and ensures parameter stability through temporal optimization. Ultimately, it achieves high-precision, highly robust, and real-time 3D lane line detection under complex terrain, providing a reliable environmental perception basis for autonomous driving.

[0038] In some embodiments, the road three-dimensional lane line detection method based on binocular vision provided by the present invention can be executed by a road three-dimensional lane line detection system 100 based on binocular vision (hereinafter referred to as detection system 100).

[0039] As an example, the detection system 100 can be any electronic device 200 with data processing capabilities, such as a general-purpose computer, personal computer, laptop computer, switch, or tablet computer. The specific implementation of the detection system 100 is not limited here.

[0040] Figure 1 shows a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. The electronic device 200 includes a processor 210, a memory 220, and a communication interface 230.

[0041] Processor 210 may include one or more processing cores. Processor 210 connects to various parts within electronic device 200 using various interfaces and lines, and performs various functions and processes data of electronic device 200 by running or executing instructions, programs, code sets, or instruction sets stored in memory 220, and by calling data stored in memory 220. Optionally, processor 210 may be implemented using at least one of the following hardware forms: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA).

[0042] The memory 220 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 220 may include a non-transitory computer-readable storage medium. The memory 220 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 220 may include a program storage area. This program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described above, etc.

[0043] Communication interface 230 is used to communicate with other devices, equipment or communication networks, such as data storage devices, image processing devices or Ethernet, wireless access network (RAN), wireless local area network (WLAN), etc.

[0044] In terms of physical implementation, the aforementioned devices (such as processor 210, memory 220, and communication interface 230) can each be devices within the same device (such as a laptop computer). Alternatively, at least two of these devices can be located within the same device, i.e., as different devices within the same device, similar to the deployment of devices or components in a distributed system.

[0045] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 200. In other embodiments of the present invention, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0046] The following description, in conjunction with the accompanying drawings, illustrates a method for detecting three-dimensional lane lines on roads based on binocular vision, as provided in an embodiment of the present invention.

[0047] Figure 2 is a flowchart of a road 3D lane line detection method based on binocular vision provided by an embodiment of the present invention. Optionally, this method can be executed by the electronic device 200 shown in Figure 1. The method may include the following steps: S1, synchronously acquiring binocular images and IMU attitude data of the vehicle in the forward direction, performing online calibration of camera intrinsic and extrinsic parameters on the binocular images and IMU attitude data to correct intrinsic and extrinsic parameter drift; and performing time alignment to correct timestamp deviation, obtaining time-aligned binocular images and IMU attitude data.

[0048] It should be noted that IMU attitude data is vehicle motion state data collected by the Inertial Measurement Unit, which includes the rotation matrix and translation vector of the camera (or vehicle). It can accurately characterize the vehicle's attitude information in three-dimensional space (such as attitude angle, motion trajectory, etc.), correct camera intrinsic and extrinsic parameter drift and timestamp deviation, and provide geometric basis for epipolar constraints of binocular images.

[0049] Among them, online calibration of camera intrinsic and extrinsic parameters is used to correct intrinsic and extrinsic parameter drift, and timing alignment is used to correct timestamp deviation.

[0050] Specifically, online calibration of camera intrinsic and extrinsic parameters includes intrinsic parameter calibration, extrinsic parameter calibration, and online calibration.

[0051] The intrinsic parameter calibration adopts Zhang Zhengyou's calibration method. By acquiring multiple sets of chessboard images, the camera intrinsic parameter matrix K (including the focal length f) is solved using the least squares method. x f y and principal point coordinates c x c y The calibration process involves calibrating the left and right cameras independently to obtain their respective intrinsic parameter matrices K. This includes the distortion coefficients (radial distortion k1, k2, k3 and tangential distortion p1, p2). L and K R Extrinsic parameter calibration is performed using binocular stereo calibration, solving for the rotation matrix R and translation vector T (baseline vector) between the left and right cameras. A feature-point matching-based calibration method is employed, extracting corner points on a calibration board to establish the correspondence between the left and right images. Extrinsic parameters are then obtained through PnP algorithm and BundleAdjustment optimization. The extrinsic parameter matrix is ​​represented as [R|T], where R is a 3×3 rotation matrix and T is a 3×1 translation vector. Online calibration, performed during system operation, involves fine-tuning the camera extrinsic parameters by detecting the calibration board or utilizing feature points from the natural scene, combined with IMU data, to compensate for parameter drift caused by factors such as temperature changes and vibration.

[0052] In one example, the timing alignment is specifically implemented through hardware synchronization, including: the left and right cameras adopt a hardware trigger mode, and the image acquisition timestamps are kept consistent through an external synchronization signal, with the time deviation controlled within 1ms.

[0053] S2. Perform distortion correction, illumination compensation preprocessing, and multi-scale feature extraction on the time-aligned binocular images. Determine the dense disparity map based on IMU pose data constraints and left-right consistency constraints.

[0054] In one possible implementation, the determination of the dense disparity map based on IMU pose data constraints and left-right consistency constraints includes: extracting the camera relative pose based on the temporally aligned IMU pose data and calculating the fundamental matrix using camera intrinsic parameters to obtain epipolar constraint rules; defining the feature matching range of the left and right images included in the binocular images based on the epipolar constraint rules, constructing a disparity cost body containing all disparity candidates, wherein a penalty term is added to disparity candidates that violate the epipolar constraint rules, and the constraint weights are dynamically adjusted according to the confidence level of the IMU pose data; and solving the disparity cost body using the soft Argmin method, first obtaining integer disparity values, and then using a second... Subpixel offsets are obtained through curve fitting and combined to form subpixel-level disparity values. Based on the subpixel-level disparity values, the disparity values ​​of each pixel in the left image and the corresponding pixels in the right image are obtained. The discrepancy between the two is checked by left and right consistency to determine whether it is within a preset threshold range. Pixels within the preset threshold range are marked as valid pixels, and those outside the preset threshold range are marked as invalid pixels. If there are valid pixels around an invalid pixel, a weighted interpolation method based on the surrounding valid pixels is used for correction. If there are no valid pixels around an invalid pixel, the invalid pixel is removed and filtered for noise reduction to obtain a dense disparity map. The camera relative pose includes the camera rotation matrix and translation vector.

[0055] It should be understood that the constraints on IMU attitude data specifically involve: firstly, extracting the rotation matrix R at the current moment from the IMU attitude data. imu (3×3) Translation vector T imu (3×1), which represents the pose of the camera coordinate system relative to the world coordinate system; during the construction of the disparity cost volume, geometric consistency constraints are applied to the feature matching of the left and right images using IMU pose data, that is, the left image pixel p L The corresponding point p in the right image R The polar constraint formula p must be satisfied. R^T •F•p L =0. Here, F is the fundamental matrix calculated based on camera intrinsic parameters and IMU pose; for disparity candidates d that violate this constraint, a penalty term is added to the cost body C(x,y,d), with the formula: C(x,y,d)=C(x,y,d)+λ geo • Penalty. λ geo λ represents the geometric constraint weights, ranging from 0.1 to 0.3, and penalty represents the degree of constraint violation. Simultaneously, λ is dynamically adjusted based on the IMU data confidence level (determined by the consistency between accelerometer and gyroscope data). geo Increase the weight when the confidence level is high and decrease the weight when the confidence level is low to avoid incorrect constraints affecting the matching results.

[0056] Furthermore, the subpixel-level disparity calculation and dense disparity map generation process is as follows: the cost volume is optimized at the subpixel level using the soft Argmin method. First, the integer disparity value d at the pixel position (x,y) is obtained through Argmin. int , and then in d int Quadratic curve fitting is performed in the vicinity to obtain the sub-pixel offset δ, which is then combined to form the sub-pixel disparity value d. sub =d int +δ (δ is solved by minimizing the cost function); the subpixel disparity result is the initial dense disparity map D(x,y), whose size is the same as the input image (H×W), and the disparity value range is 0-192 pixels (corresponding to a depth range of 0.5m-50m); finally, through post-processing operations such as median filtering and bilateral filtering, noise and outliers are removed to obtain the final high-quality dense disparity map.

[0057] It should also be noted that the specific implementation of the left-right consistency constraint is as follows: for the left image pixel p L Its disparity value is d L The corresponding matching point p in the right image R =p L -(d L ,0), calculate p R The disparity value d of the position R If |d L -d R If |>τ (τ is the threshold, ranging from 1 to 2 pixels), then the point is determined to not meet the constraint; for pixels that meet the constraint, a mask M(x,y)=1 is set, and for those that do not, M(x,y)=0 is set. This mask is used for subsequent point cloud generation and terrain modeling; for pixels that do not meet the constraint (invalid pixels), if there are surrounding pixels that meet the constraint (valid pixels), a weighted interpolation method is used to correct their disparity value; if there are no surrounding pixels that meet the constraint, they are marked as invalid points and do not participate in subsequent processing; the left-right consistency check is performed again on the corrected disparity map. If there are still points that do not meet the constraint, the correction process is repeated, with a maximum of 3 iterations, to ensure the consistency and accuracy of the disparity map.

[0058] As described above, the method provided in this embodiment of the invention determines a dense disparity map through IMU attitude data constraints and left-right consistency constraints. First, the relative camera attitude is extracted based on the IMU attitude, and epipolar constraint rules are obtained by combining the camera intrinsic parameters, which effectively narrows the matching range of left and right image features and reduces invalid matching calculations. Penalty terms are added to disparity candidates that violate epipolar constraints, and the constraint weights are dynamically adjusted to further improve matching accuracy. The sub-pixel level disparity values ​​are solved by the soft Argmin method, which significantly improves the disparity calculation accuracy compared to integer disparity. Then, valid and invalid pixels are marked by left-right consistency checks, and invalid pixels are corrected or removed by weighted interpolation. Combined with filtering and noise reduction, a high-quality dense disparity map is finally obtained, which effectively suppresses mismatches and noise interference, and provides high-precision data support for subsequent road point cloud generation and terrain modeling.

[0059] S3. Based on dense disparity maps and camera parameters, road point clouds are generated. An adaptive terrain model containing height, slope and lateral tilt angle is constructed through local ground plane fitting, quadratic surface expansion and normal clustering.

[0060] In some embodiments, generating a road point cloud based on a dense disparity map and camera parameters includes: obtaining the focal length, principal point coordinates, and binocular baseline length of the left camera based on the camera intrinsic parameter matrix; performing normalization processing on the coordinates of each effective pixel in the dense disparity map to obtain the relative coordinates of each effective pixel in the camera coordinate system; calculating the true depth of the road point corresponding to each effective pixel based on the disparity value of each effective pixel; converting the relative coordinates into three-dimensional coordinates in the camera coordinate system according to the true depth; converting the three-dimensional coordinates of all effective pixels in the camera coordinate system into three-dimensional coordinates in the vehicle coordinate system; and generating a road point cloud based on the three-dimensional coordinates of each effective pixel in the vehicle coordinate system.

[0061] In one example, the normalization process for the coordinates of each valid pixel in the dense disparity map is performed as follows: using the left image as the reference alignment object for the disparity map, for each valid pixel (u,v) in the dense disparity map, the formula is used... , Perform normalization processing.

[0062] Among them, f x f y c is the focal length of the left camera. x c y The principal point coordinates are all contained in the camera intrinsic parameter matrix K. This process can eliminate the influence of camera hardware parameters on pixel coordinates and obtain the relative coordinates of the pixel in the camera coordinate system.

[0063] The calculation process for the true depth of each valid pixel corresponding to a road point is as follows: combining the binocular baseline length and the focal length of the left camera in the x-direction, using the formula... Convert the parallax value to the true depth.

[0064] Where B is the binocular baseline length (unit: meters) and d is the disparity value of the effective pixels. This calculation process strictly follows the geometric imaging principle of binocular vision to ensure the accuracy of the depth value.

[0065] The process of converting relative coordinates to 3D coordinates in the camera coordinate system, based on the actual depth, is as follows: using the formula: Converting two-dimensional pixel coordinates to three-dimensional coordinates in the camera coordinate system (X). cam Y cam Z cam This completes the coordinate mapping from the image plane to three-dimensional space.

[0066] The process of generating road point clouds based on the 3D coordinates of each effective pixel in the vehicle coordinate system is as follows: the 3D coordinates of all effective pixels in the camera coordinate system are uniformly transformed to the vehicle coordinate system. The transformation process needs to be combined with the extrinsic parameter matrix (rotation matrix and translation vector) between the camera and the vehicle. The above complete transformation process is only performed on pixels that meet the left-right consistency constraint. Finally, all transformed 3D coordinates in the vehicle coordinate system are summarized to form effective road point cloud data with uniform density and accurate position.

[0067] The method provided in this invention generates road point clouds based on dense disparity maps and camera parameters. It accurately obtains key parameters such as focal length, principal point coordinates, and binocular baseline length through the camera intrinsic parameter matrix. Normalization is performed on each valid pixel, and the true depth is calculated based on the disparity value. Then, coordinate transformation maps the valid pixels from the camera coordinate system to the vehicle coordinate system. Finally, the 3D coordinates of all valid pixels are aggregated to generate the road point cloud. The entire process follows geometric transformation logic, utilizing high-precision disparity data and camera parameters to ensure the accuracy of depth calculation and coordinate transformation. The generated road point cloud has uniform density and accurate positioning, providing a reliable 3D data foundation for the subsequent construction of adaptive terrain models and avoiding terrain modeling deviations caused by point cloud errors.

[0068] In other embodiments, an adaptive terrain model incorporating height, slope, and lateral tilt angle is constructed through local ground plane fitting, quadratic surface expansion, and normal clustering. This includes: dividing the road point cloud into grids of a preset size; performing local ground plane fitting on the point cloud data within each grid using the RANSAC algorithm to obtain the local ground plane equation and plane parameters for each grid; performing quadratic surface expansion on the local ground plane based on the local ground plane equation and point cloud data for each grid; solving for the quadratic surface coefficients using the least squares method to obtain the quadratic surface equation for each grid, with a preset size of 0.2m × 0.2m; calculating the grid's normal vector based on the quadratic surface equation for each grid; and normalizing the normal vector to obtain a standardized grid normal vector. Next, the K-means clustering algorithm is used to perform cluster analysis on the standardized normal vectors of all grids, classifying grids with similar normal vectors into the same terrain region, and determining the terrain region classification corresponding to each grid. The quadratic surface coefficients and grid normal vectors of the current frame t are obtained and compared with the quadratic surface coefficients and grid normal vectors of the corresponding grid in frame t-1. The corresponding differences are calculated, and a preset threshold is used to determine whether there is a terrain abrupt change. For the detected terrain abrupt change grids, the terrain information of frame t-1 and multiple historical frames is fused by combining Kalman filtering and sparse optimization algorithms to correct and adjust the abrupt terrain parameters, so that the corrected terrain parameters of frame t are continuously connected with the terrain parameters of historical frames, resulting in a continuous adaptive terrain model that includes height, slope, and lateral tilt angle.

[0069] The method provided in this invention constructs an adaptive terrain model through local ground plane fitting, quadratic surface expansion, and normal clustering, overcoming the limitations of the planar assumption in existing technologies. It first divides the road point cloud into a grid, fits the local ground plane using the RANSAC algorithm, then accurately captures road curvature changes through quadratic surface expansion, and uses normal clustering to classify terrain regions, comprehensively describing the terrain's height, slope, and lateral tilt. By calculating the differential detection of terrain parameters across consecutive frames, and combining Kalman filtering and sparse optimization to fuse historical information, the abrupt change parameters are corrected to ensure model continuity. This terrain model can dynamically adapt to complex terrains such as slopes and lateral tilts, providing accurate geometric priors for subsequent lane line detection, effectively reducing the 3D coordinate mapping error of lane lines caused by terrain undulations, and improving the accuracy of lane line detection in complex terrain.

[0070] S4. Map the depth features of the road point cloud and the texture features of the binocular image to the bird's-eye view to construct a depth-enhanced BEV feature map.

[0071] Specifically, the BEV (Bird's Eye View) feature map is a feature map generated by mapping the depth features of road point clouds and the texture features of binocular images to the bird's-eye view. By dividing the detection range in front of the vehicle into grids, each grid corresponds to a pixel in the image. It integrates the depth statistical features of the point cloud (such as average depth and depth variance) and the texture features of the image (such as RGB information), and combines them with an adaptive terrain model to supplement the grid height coordinates, forming a depth-enhanced BEV feature map.

[0072] In one possible implementation, constructing a depth-enhanced BEV feature map includes: dividing the area within a preset detection range in front of the vehicle into BEV grids, with each grid corresponding to a pixel in the BEV feature map; projecting the 3D coordinates of the road point cloud onto the corresponding BEV grid, statistically analyzing the depth features of the point cloud within each grid, and generating a depth feature map; calculating the height coordinates of each BEV grid using interpolation based on an adaptive terrain model to obtain the complete 3D coordinates of the grid; projecting the complete 3D coordinates of the BEV grid onto the left and right images of the binocular image, respectively, extracting the texture features at the corresponding locations, and generating a texture feature map; and concatenating and fusing the depth feature map and the texture feature map by channel to obtain the depth-enhanced BEV feature map.

[0073] In one example, the construction of the depth-enhanced BEV feature map described above includes: firstly, dividing a pre-defined detection area in front of the vehicle (e.g., 0-50m in the X direction and -6m to 6m in the Y direction) into BEV grids, with a grid size of 0.2m × 0.2m, and each grid corresponding to a pixel in the BEV feature map; then, the three-dimensional coordinates p of each point in the road point cloud P are... i =(X i ,Y i, Z i The point cloud is projected onto the corresponding BEV grid, completing the spatial mapping from the vehicle coordinate system to the BEV coordinate system. For each BEV grid, all point cloud data projected onto that grid are collected. A depth feature map is generated by statistically analyzing the depth characteristics of the point cloud within the grid (such as average depth and depth variance), transforming the 3D spatial depth information of the point cloud into a feature representation from the BEV perspective, providing spatial dimensional support for subsequent fusion. Based on an adaptive terrain model, the height coordinates (Z-axis coordinates) of each BEV grid are calculated through interpolation, completing the grid's full 3D coordinates (X-axis coordinates). BEV ,Y_ BEV ,Z_ BEVThe complete 3D coordinates are projected onto the RGB color space of the left and right binocular images respectively. The corresponding pixel positions are located in the images, and the RGB texture features at those positions are extracted. The texture features of all meshes are then combined to generate a texture feature map, realizing the mapping of image texture information to the BEV coordinate system. The depth feature map and the texture feature map are concatenated and fused along the channels to obtain the initial BEV feature map, which has a size of 250×60×256 (H). BEV ×W BEV ×C); The feature map is input into the Transformer encoder, and the grid position information is supplemented by learnable position encoding. The global context features are aggregated by a multi-head self-attention mechanism with 8 attention heads (32 dimensions per head). The feature representation is then optimized by a two-layer fully connected feedforward network with dimensions ranging from 1024 to 256. A normalization layer and residual connection are added after each sub-layer to improve training stability.

[0074] The method provided by this invention constructs a depth-enhanced BEV feature map. By dividing the area in front of the vehicle into a BEV grid within a pre-defined detection range, the depth statistical features of the road point cloud are transformed into a depth feature map. Simultaneously, an adaptive terrain model is used to supplement the grid's height coordinates. The grid's 3D coordinates are projected onto a binocular image to extract texture features. Finally, the two types of features are stitched and fused along their respective channels. This process fully combines the 3D spatial information of depth features with the detailed descriptive capabilities of texture features. The BEV perspective can globally perceive the road environment, avoiding the problem of insufficient information from a single feature. The generated depth-enhanced BEV feature map has richer information dimensions and stronger representational capabilities, significantly improving the accuracy of subsequent 3D anchor box detection networks in predicting lane line candidate parameters, and providing high-quality feature support for effective candidate selection.

[0075] S5. A three-dimensional anchor frame detection network is adopted in combination with the geometric prior of the adaptive terrain model. Based on the depth-enhanced BEV feature map, candidate parameters of three-dimensional lane lines are detected. Valid candidate parameters of three-dimensional lane lines are selected from the candidate parameters of three-dimensional lane lines through reprojection error constraints and road prior constraints.

[0076] In another possible implementation, a 3D anchor box detection network is used in conjunction with the geometric prior of an adaptive terrain model to detect 3D lane line candidate parameters based on a depth-enhanced BEV feature map. This includes: inputting the depth-enhanced BEV feature map into the 3D anchor box detection network; supplementing grid position information through learnable position encoding; aggregating global context features via a multi-head self-attention mechanism; and optimizing feature representation through a feedforward network and residual connections. Based on the height, slope, and lateral tilt parameters of the adaptive terrain model, 3D anchor boxes adapted to the terrain are pre-defined on the BEV grid. Anchor box parameters include center position, orientation angle, length, and width, and the slope and lateral tilt of the anchor boxes are adapted to the terrain parameters of the corresponding grid. The 3D anchor box detection network performs parameter regression and classification on each anchor box, and combines the geometric prior to determine the adaptability of the anchor box to the terrain, predicting 3D lane line candidate parameters. These 3D lane line candidate parameters include the 3D coordinates, width estimate, curvature estimate, slope estimate, lane line type, and confidence score of key nodes in the lane line center trajectory.

[0077] Specifically, three-dimensional anchor boxes adapted to the terrain are defined on the BEV plane (each anchor box corresponds to a potential lane line). The anchor box parameters are set as follows: the center position is uniformly sampled on the BEV grid (5m interval in the X direction, covering -6m to 6m in the Y direction), the azimuth angle is from -90° to 90° (15° interval), the length is fixed at 20m, and the width is 0.1-0.3m (0.05m interval). Approximately 2000 anchor boxes are generated for the 250×60 BEV feature map. The network performs parametric regression and classification on each anchor box to predict the candidate parameters of the three-dimensional lane line, including: the three-dimensional coordinate sequence of the key nodes of the lane line center trajectory (N is usually 10-20), the lane line width w, and the left and right boundary offsets Δw. left / Δw right Curvature κ and rate of change of curvature dκ / ds, longitudinal slope θ long and lateral tilt angle θ lat Lane line type (solid / dashed / unknown, output activated by softmax), detection confidence score (0-1, output activated by sigmoid, combining visual feature response strength, geometric consistency with terrain model, and cross-frame temporal stability).

[0078] The method provided by this invention employs a 3D anchor box detection network combined with geometric priors from an adaptive terrain model to predict 3D lane line candidate parameters. First, a depth-enhanced BEV feature map is input into the network, and positional information is supplemented through learnable positional encoding. The feature representation is then optimized via a multi-head self-attention mechanism and a feedforward network. Next, based on parameters such as height and slope of the adaptive terrain model, pre-defined 3D anchor boxes adapted to the terrain are generated. Finally, the network performs parameter regression and classification on the anchor boxes. This approach achieves a higher degree of matching between the anchor boxes and the actual terrain, avoiding prediction bias caused by traditional anchor boxes being out of sync with the terrain. Simultaneously, the network accurately captures global lane line features through global feature aggregation, and combines geometric priors to determine anchor box adaptability, ultimately accurately predicting candidate parameters such as the 3D coordinates, curvature, and slope of the lane line center trajectory, providing high-quality foundational data for subsequent effective candidate selection.

[0079] In another possible implementation, valid 3D lane line candidate parameters are selected from the 3D lane line candidate parameters through reprojection error constraints and road prior constraints. This includes: for each 3D lane line candidate, the 3D coordinates of the key nodes are projected onto the stereo image in combination with the camera's intrinsic and extrinsic parameters to obtain the projected pixel coordinates of each node; image features around the projected pixel coordinates in the stereo image are extracted, and the deviation between the projected pixel coordinates and the pixel coordinates of the lane line texture features in the image is calculated to obtain the reprojection error of each node; if the reprojection error of any node exceeds a preset pixel threshold, the node is filtered; if more than a preset proportion of nodes in the same 3D lane line candidate have excessive reprojection errors, the 3D lane line candidate parameters are removed; for candidates that pass the reprojection error constraints, road prior constraint checks are performed, including lane line width constraints, curvature constraints, adjacent node continuity constraints, and adjacent lane line parallelism constraints; 3D lane line candidate parameters that satisfy all constraints are retained as valid 3D lane line candidate parameters.

[0080] The specific calculation of reprojection error is as follows: Projection transformation: For the three-dimensional coordinates p of the key node of the three-dimensional lane line candidate... 3d =(x,y,z), combined with the camera's intrinsic parameters (focal length, principal point coordinates, etc.) and extrinsic parameters (rotation matrix, translation vector), are mapped to the left and right binocular images respectively using the perspective projection formula to obtain the corresponding projected pixel coordinates (u). L ,v L (Left image) and (u) R ,v R(Right image); Image feature extraction: A fixed-size neighborhood (e.g., 3×3 or 5×5 pixels) is preset around the projected pixel coordinates of the left and right images. Image features within this area are extracted, including edge intensity (calculated using the Sobel and Canny operators) and color features (RGB channel grayscale values, hue saturation, etc.); Error calculation and judgment: The reprojection error is obtained by calculating the deviation between the image features at the location of the projected pixel coordinates and the typical features of the lane lines (e.g., high edge intensity, specific color distribution). The larger the deviation, the higher the error value. If the error value exceeds the preset pixel threshold (usually 2-3 pixels), the node is determined to be unreliable and is filtered out (it is not included in the subsequent lane line parameter fitting).

[0081] Specifically, the constraints include: Lane width constraint: Based on the physical dimensions of lane lines on real roads, the width range of candidate lane lines is limited (usually 0.1-0.3m). If the estimated width of a candidate lane line exceeds this range, it is deemed not to meet the constraint. Curvature constraint: Combining road design specifications and vehicle dynamics limitations, a maximum allowable value for lane line curvature is preset (e.g., |K|≤0.001m in highway scenarios). If the curvature parameter of a candidate lane line exceeds this threshold, it indicates that its curvature does not conform to the laws of real roads and is therefore eliminated. Adjacent node continuity constraint: The three-dimensional spatial distance between adjacent key nodes on the center trajectory of a candidate lane line is required not to exceed a preset threshold (usually 0.5m) to avoid situations such as node dispersion and trajectory breakage that do not conform to the continuity characteristics of lane lines. Adjacent lane line parallelism constraint: For multiple candidate lane lines in the same road scenario, the angle between the direction vectors of two adjacent lane lines is calculated. If the angle exceeds a preset threshold (usually 5°), it is deemed not to meet the parallelism requirement and the candidate lane line that does not meet the constraint is eliminated.

[0082] The method provided by this invention filters effective 3D lane line candidates through reprojection error constraints and road prior constraints. First, the 3D coordinates of the lane line candidate nodes are projected onto the left and right binocular images. The reprojection error is calculated, and nodes and candidates with excessive deviations are filtered out to ensure consistency between the candidates and image texture features. Then, road prior constraints such as lane line width, curvature, continuity, and parallelism are applied to eliminate candidates that do not conform to the physical laws of real roads. This dual constraint mechanism filters candidates from both visual matching consistency and physical laws perspectives, effectively filtering false positives caused by feature mismatches, terrain interference, etc., significantly reducing the proportion of invalid candidates, and obtaining high-quality, effective 3D lane line candidates. This provides a guarantee for subsequent time-series optimization to output stable and reliable lane line parameters.

[0083] S6. Perform extended Kalman filtering and trajectory optimization on the effective 3D lane line candidate parameters of consecutive frames to obtain the 3D lane line parameters.

[0084] Specifically, the three-dimensional lane line parameters include the three-dimensional coordinates, curvature parameters, slope parameters, width parameters, lateral offset, confidence level, and lane line type of key nodes on the lane line center trajectory.

[0085] In one possible implementation, trajectory optimization includes: constructing a trajectory optimization objective function, which includes a data term, a smoothing term, and a prior term. The data term is used to measure the consistency between the optimized position and the observed position, the smoothing term is used to ensure the continuity of parameters across frames, and the prior term is used to introduce road geometric constraints; iterating the objective function using a nonlinear optimization algorithm, using the result of the extended Kalman filter as the initial value, and stopping when the change in the objective function is less than a preset threshold or the maximum number of iterations is reached; setting an optimization frequency, performing real-time optimization every frame to ensure real-time detection, and performing batch optimization every preset number of frames to improve the consistency of global parameters.

[0086] Overall form of the objective function: For lane detection results of T consecutive frames, construct an optimization objective function E based on multi-constraint fusion. total =E data +λ smooth* E smooth +λ prior *E prior .

[0087] Where, λ smooth For the smoothing term weights, λ prior These are the weights of the prior terms, used to balance the influence of different constraints through weight allocation.

[0088] Data item E data The core function is to ensure the consistency between the optimized lane line position and the actual observation data, by calculating the optimized position p of the i-th lane line point in the t-th frame. t,i With the detection and observation position z t,i The sum of squared Euclidean distances is achieved, as shown in the formula: This data item can effectively reduce the impact of observation noise on lane line parameters.

[0089] Smoothing term E smooth To ensure the continuity of lane line parameters across frames and avoid parameter jumps, it includes two constraints: one is the continuity constraint of the corresponding lane line point positions in adjacent frames. ( The first is the optimized position of the lane line point in frame t-1, and the second is the continuity constraint of the lane line curvature between adjacent frames. ( (These are the lane line curvatures in frame t and frame t-1, respectively), and the smooth transition of the lane line trajectory is ensured through dual constraints.

[0090] Prior term E priorIntroducing prior constraints on road geometry further improves the rationality of lane line parameters; the formula is as follows: , where p terrain κ represents the three-dimensional coordinates of the corresponding location predicted by the adaptive terrain model. max w is the maximum curvature allowed by road design specifications. i To constrain the weights, this prior term ensures that the optimized lane line parameters conform to the geometry of real roads.

[0091] The method provided by this invention constructs a trajectory optimization objective function that includes data terms, smoothing terms, and prior terms. The data terms ensure that the optimized position is consistent with the observed position, the smoothing terms ensure the continuity of parameters across frames, and the prior terms introduce road geometric constraints. Then, a nonlinear optimization algorithm is used to iteratively optimize using the extended Kalman filter results as initial values, and a reasonable optimization frequency is set. This optimization method takes into account the accuracy of detection results, cross-frame continuity, and road adaptability. Real-time optimization ensures real-time detection, batch optimization improves the consistency of global parameters, and additional optimization is triggered when there are abrupt changes in terrain or a decrease in confidence, effectively avoiding lane line parameter jumps. This makes the optimized lane line parameters more stable and accurate, meeting the high-precision requirements of autonomous driving for lane line detection.

[0092] In another possible implementation, extended Kalman filtering and cross-frame continuity constraints are applied, including: using the effective 3D lane line candidate parameters of consecutive frames as observations, combining vehicle speed and turning angle, and performing prediction and update steps through extended Kalman filtering to obtain initially stable lane line parameters; applying cross-frame continuity constraints, where positional continuity constraints require that the positional change of corresponding lane line points in adjacent frames does not exceed the product of the maximum permissible vehicle speed and the frame interval, and parameter continuity constraints require that the changes in lane line curvature and slope in adjacent frames do not exceed the product of the corresponding preset maximum rate of change and the frame interval; and smoothly correcting lane line parameters that violate cross-frame continuity constraints, readjusting them by combining historical frame optimization results with current frame observations to ensure that there are no jumps in cross-frame lane line parameters, thus obtaining 3D lane line parameters.

[0093] It should be noted that the vehicle speed and turning angle mentioned above are only illustrative examples. The prediction and update steps can also be performed by combining other parameters or state variables through extended Kalman filtering. This embodiment of the invention does not impose any particular limitations on this.

[0094] The specific implementation of the prediction and update steps using extended Kalman filtering is as follows: Nonlinear optimization is performed using the Levenberg-Marquardt (LM) algorithm or the Gauss-Newton method. These algorithms combine the stability of gradient descent with the convergence speed of Newton's method, adapting to the needs of lane line parameter optimization. During the initialization phase, the preliminary stable parameters output by the extended Kalman filter are directly reused to avoid slow convergence caused by initial value deviations. During iteration, the lane line state variables (key node coordinates, curvature, slope, etc.) are dynamically updated by calculating the gradient vector and Hessian matrix of the objective function. Iteration stops when the change in the objective function is less than a preset threshold (e.g., 1e-6) or the number of iterations reaches the maximum limit (e.g., 50 times), ensuring a balance between optimization efficiency and parameter accuracy.

[0095] Flexible configuration of optimization frequency: Real-time optimization: Extended Kalman filtering is performed every frame to ensure the real-time requirements of autonomous driving scenarios with a processing frame rate of 30fps, and to quickly respond to dynamic changes in lane lines; Batch optimization: Batch optimization is performed every 5-10 frames to jointly optimize the lane line detection results of the past N frames, correct the local deviations of single-frame optimization, and improve the consistency of global parameters; Triggered optimization: When the terrain model detects abrupt changes (such as steep slopes or potholes) or the lane line detection confidence drops significantly (such as below 0.6), an additional optimization process is actively triggered to quickly correct parameter deviations and ensure detection stability in extreme scenarios.

[0096] Quantization of cross-frame continuity constraints: Positional continuity constraint: The three-dimensional spatial distance between corresponding lane line points in adjacent frames must satisfy the formula ||p t,i -p t−1,i || <v max *Δt, where v max The maximum permissible speed (usually 1.5 times the vehicle's real-time speed), Δt being the frame interval, ensures that lane line position changes conform to vehicle dynamics and avoid abrupt transitions; parameter continuity constraint: curvature changes must satisfy |κ t -κ t−1 |<Δκmax*Δt, the slope change must satisfy |θ t −θ t−1 |<Δθ max *Δt, where Δκ max (Maximum rate of change of curvature), Δθ max (Maximum rate of change of slope) is preset according to road design specifications to ensure that the changes in lane curvature and slope are smooth and conform to the geometric characteristics of real roads.

[0097] The method provided by this invention combines extended Kalman filtering with cross-frame continuity constraints. First, using effective 3D lane line candidate parameters as observations, prediction and updates are performed in conjunction with vehicle speed and turning angle to obtain initially stable lane line parameters. Then, positional continuity constraints limit the range of lane line point position changes between adjacent frames, while parameter continuity constraints limit the range of curvature and slope changes, smoothly correcting parameters that violate these constraints. This combined approach fully utilizes the predictive and updating capabilities of extended Kalman filtering and the stability guarantee provided by continuity constraints. It improves prediction accuracy through vehicle speed and turning angle, and avoids parameter jumps through constraint correction. By combining historical frame optimization results with current frame observations to adjust parameters, it ultimately outputs stable and continuous 3D lane line parameters, providing a reliable decision-making basis for autonomous driving upper-level control algorithms.

[0098] As described in S1-S6 above, the method provided by this invention synchronously acquires binocular images and IMU attitude data, combines online calibration and temporal alignment of camera intrinsic and extrinsic parameters to correct parameter drift and timestamp deviation, and then proceeds through a complete process of disparity map generation, adaptive terrain modeling, BEV feature fusion, 3D anchor frame detection, and temporal optimization. This overcomes the limitations of existing monocular vision's reliance on planar assumptions and avoids the real-time insufficiency and false detection problems caused by existing neural network models. It improves data reliability through binocular vision and IMU data fusion, dynamically adapts to complex undulating terrain using an adaptive terrain model, enriches feature expression with depth-enhanced BEV feature maps, improves candidate accuracy through 3D anchor frame detection and multi-constraint screening, and ensures parameter stability through temporal optimization. Ultimately, it achieves high-precision, high-robustness, and real-time 3D lane line detection under complex terrain, providing a reliable environmental perception basis for autonomous driving.

[0099] This can also be understood as follows: The method provided in this embodiment of the invention achieves high-precision 3D reconstruction by adaptively fusing binocular depth and terrain, overcoming the limitations of the planar assumption in monocular vision compared to monocular vision. Furthermore, compared to traditional binocular vision, this invention combines a pure vision-based BEV algorithm, integrating left and right reprojection errors, road topology priors, and temporal filtering to significantly reduce false detection rates and curvature jumps. Dynamic credibility and rematching strategies are introduced for special scenarios such as abrupt slope changes and weak textures in rain and fog, ensuring robustness. Finally, compared to LiDAR solutions, the overall solution can run in real-time on automotive-grade embedded platforms, offering a cost advantage.

[0100] In other words, this invention overcomes the limitations of the monocular planar assumption by simultaneously acquiring binocular images and vehicle posture, and aligning binocular intrinsic and extrinsic parameters and timestamps in real time. This achieves high-precision 3D reconstruction under complex terrain with undulating slopes, significantly improving the vehicle's environmental perception capabilities. Furthermore, it uses a binocular depth estimation module to output a dense disparity map, combined with a terrain modeling module to generate an adaptive road terrain model, achieving accurate 3D environment map reconstruction and meeting real-time and accuracy requirements. Additionally, it outputs 3D lane line candidates through feature fusion and lane line detection modules, and combines this with filtering and prediction from a temporal optimization module, effectively reducing false detection rates and curvature jumps, thus improving lane line detection accuracy. Finally, it employs an adaptive mechanism to handle complex terrain and severe weather conditions, fusing historical information through Kalman filtering and sparse optimization to ensure model continuity and stability, improving the system's robustness in extreme environments. The above mainly describes the scheme of this invention from a methodological perspective. It is understood that the detection system 100, in order to achieve the above functions, includes at least one of the hardware structures and software modules corresponding to each function. Those skilled in the art will readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of the present invention.

[0101] In this embodiment of the invention, the detection system 100 can be divided into functional units according to the above method example. For example, the detection system 100 can be divided into functional units corresponding to each function, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this embodiment of the invention is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0102] For example, Figure 3 shows a schematic diagram of the hardware structure of a detection system provided in an embodiment of the present invention. The detection system 100 includes: a data acquisition module 110, used to simultaneously acquire binocular images and IMU attitude data of the vehicle in the forward direction, perform online calibration of camera intrinsic and extrinsic parameters on the binocular images and IMU attitude data, correct intrinsic and extrinsic parameter drift; and perform time alignment to correct timestamp deviation, so as to obtain time-aligned binocular images and IMU attitude data; a data processing module 120, used to perform distortion correction, illumination compensation preprocessing and multi-scale feature extraction on the time-aligned binocular images, and determine a dense disparity map based on IMU attitude data constraints and left-right consistency constraints; and a model building module 130, used to generate road point clouds based on dense disparity maps and camera parameters, and construct a model containing height, slope and lateral tilt angle through local ground plane fitting, quadratic surface expansion and normal clustering. An adaptive terrain model is included; a feature map construction module 140 is used to map the depth features of the road point cloud and the texture features of the binocular image to the bird's-eye view to construct a depth-enhanced BEV feature map; a detection module 150 is used to detect 3D lane line candidate parameters based on the depth-enhanced BEV feature map by using a 3D anchor box detection network combined with the geometric prior of the adaptive terrain model, and to filter effective 3D lane line candidate parameters from the 3D lane line candidate parameters through reprojection error constraints and road prior constraints; an optimization module 160 is used to perform extended Kalman filtering and trajectory optimization on the effective 3D lane line candidate parameters of consecutive frames to obtain 3D lane line parameters. The trajectory optimization is used to ensure that the 3D lane line parameters satisfy the position consistency constraints, parameter continuity constraints, and road geometric constraints.

[0103] It should be understood that specific descriptions of the above-mentioned optional methods can be found in the foregoing method embodiments, and will not be repeated here. Furthermore, explanations of any of the detection systems 100 provided above, as well as descriptions of their beneficial effects, can be found in the corresponding method embodiments described above, and will not be repeated here.

[0104] This invention also provides a computer-readable storage medium storing at least one computer instruction, which is loaded and executed by a processor to implement the methods of the various embodiments described above. Explanations of the relevant content and descriptions of the beneficial effects of any of the computer-readable storage media provided above can be found in the corresponding embodiments described above, and will not be repeated here.

[0105] This invention also provides a chip. This chip integrates a control circuit for implementing the functions of the detection system 100 described above, and one or more ports. Optionally, the functions supported by this chip are as described above, and will not be repeated here.

[0106] Those skilled in the art will understand that the program for implementing all or part of the steps of the above embodiments, which can be executed by a program instructing related hardware, can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a random access memory, etc. The processing unit or processor mentioned above can be a central processing unit, a general-purpose processor, an application-specific integrated circuit (ASIC), a microprocessor (DSP), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0107] This invention also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform any of the methods described in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this invention is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD), etc.

[0108] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of the present invention, such as, but not limited to, the aforementioned memory, computer-readable storage medium, and communication chip, are all non-transitory. Those skilled in the art should recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or code on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium accessible to general-purpose or special-purpose computers.

[0109] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for detecting three-dimensional lane lines on roads based on binocular vision, characterized in that, The method includes: simultaneously acquiring binocular images and IMU attitude data of the vehicle's forward direction; performing online calibration of camera intrinsic and extrinsic parameters on the binocular images and IMU attitude data to correct intrinsic and extrinsic parameter drift; performing temporal alignment to correct timestamp deviations and obtain temporally aligned binocular images and IMU attitude data; performing distortion correction, illumination compensation preprocessing, and multi-scale feature extraction on the temporally aligned binocular images; determining a dense disparity map based on IMU attitude data constraints and left-right consistency constraints; generating a road point cloud based on the dense disparity map and camera parameters; and constructing an adaptive terrain model including height, slope, and lateral tilt angle through local ground plane fitting, quadratic surface expansion, and normal clustering. The depth features of the road point cloud and the texture features of the binocular image are mapped to a bird's-eye view to construct a depth-enhanced BEV feature map. A 3D anchor box detection network is used in conjunction with the geometric prior of the adaptive terrain model to detect 3D lane line candidate parameters based on the depth-enhanced BEV feature map. Valid 3D lane line candidate parameters are selected from the 3D lane line candidate parameters through reprojection error constraints and road prior constraints. Extended Kalman filtering and trajectory optimization are performed on the valid 3D lane line candidate parameters of consecutive frames to obtain 3D lane line parameters. The trajectory optimization is used to ensure that the 3D lane line parameters satisfy position consistency constraints, parameter continuity constraints, and road geometric constraints.

2. The method for detecting three-dimensional lane lines on roads based on binocular vision according to claim 1, characterized in that, Determining a dense disparity map based on IMU pose data constraints and left-right consistency constraints includes: extracting the relative camera pose based on the temporally aligned IMU pose data, calculating the fundamental matrix using camera intrinsic parameters, and obtaining epipolar constraint rules; defining the feature matching range of the left and right images included in the binocular images based on the epipolar constraint rules, constructing a disparity cost body containing all disparity candidates, wherein a penalty term is added to disparity candidates that violate the epipolar constraint rules, and the constraint weights are dynamically adjusted according to the confidence level of the IMU pose data; and solving the disparity cost body using the soft Argmin method to first obtain integer disparities. The values ​​are then obtained by fitting a quadratic curve to obtain subpixel offsets, which are combined to form subpixel-level disparity values. Based on the subpixel-level disparity values, the disparity values ​​of each pixel in the left image and the corresponding pixels in the right image are obtained. The discrepancy between the two is checked by left and right consistency to determine whether the discrepancy is within a preset threshold range. Pixels within the preset threshold range are marked as valid pixels, and those outside the preset threshold range are marked as invalid pixels. If there are valid pixels around an invalid pixel, a weighted interpolation method based on the surrounding valid pixels is used for correction. If there are no valid pixels around an invalid pixel, the invalid pixel is removed, and filtering and denoising are performed to obtain the dense disparity map.

3. The road three-dimensional lane line detection method based on binocular vision according to claim 2, characterized in that, Generating a road point cloud based on the dense disparity map and camera parameters includes: obtaining the focal length, principal point coordinates, and binocular baseline length of the left camera based on the camera intrinsic parameter matrix; performing normalization processing on the coordinates of each effective pixel in the dense disparity map to obtain the relative coordinates of each effective pixel in the camera coordinate system; calculating the true depth of the road point corresponding to each effective pixel based on the disparity value of each effective pixel; converting the relative coordinates into three-dimensional coordinates in the camera coordinate system according to the true depth; converting the three-dimensional coordinates of all effective pixels in the camera coordinate system into three-dimensional coordinates in the vehicle coordinate system; and generating a road point cloud based on the three-dimensional coordinates of each effective pixel in the vehicle coordinate system.

4. The method for detecting three-dimensional lane lines on roads based on binocular vision according to claim 1, characterized in that, An adaptive terrain model incorporating height, slope, and lateral tilt angle is constructed through local ground plane fitting, quadratic surface expansion, and normal clustering. The process includes: dividing the road point cloud into grids of preset size; performing local ground plane fitting on the point cloud data within each grid using the RANSAC algorithm to obtain the local ground plane equation and plane parameters for each grid; performing quadratic surface expansion on the local ground plane based on the local ground plane equation and point cloud data for each grid, solving for the quadratic surface coefficients using the least squares method to obtain the quadratic surface equation for each grid; calculating the normal vector of each grid based on its quadratic surface equation, normalizing the normal vector to obtain a standardized grid normal vector, and then using K-means clustering. The algorithm performs cluster analysis on the standardized normal vectors of all grids, classifying grids with similar normal vectors into the same terrain region, and determining the terrain region classification corresponding to each grid. It obtains the quadratic surface coefficients and grid normal vectors of the current frame t, and calculates the difference between them and the quadratic surface coefficients and grid normal vectors of the corresponding grid in frame t-1. It then uses a preset threshold to determine whether there are terrain abrupt changes. For the detected terrain abrupt change grids, it combines Kalman filtering and sparse optimization algorithms to fuse terrain information from frame t-1 and multiple historical frames, and corrects and adjusts the abrupt terrain parameters so that the corrected terrain parameters of frame t are continuously connected with the terrain parameters of historical frames, resulting in a continuous adaptive terrain model that includes height, slope, and lateral tilt angle.

5. The method for detecting three-dimensional lane lines on a road based on binocular vision according to claim 1, characterized in that, Constructing a depth-enhanced BEV feature map includes: dividing a BEV grid within a preset detection range in front of the vehicle, with each grid corresponding to one pixel in the BEV feature map; projecting the 3D coordinates of the road point cloud onto the corresponding BEV grid, statistically analyzing the depth features of the point cloud within each grid, and generating a depth feature map; calculating the height coordinates of each BEV grid using interpolation based on the adaptive terrain model to obtain the complete 3D coordinates of the grid; projecting the complete 3D coordinates of the BEV grid onto the left and right images of the binocular image, respectively, extracting texture features at the corresponding locations, and generating a texture feature map; and concatenating and fusing the depth feature map and the texture feature map by channel to obtain the depth-enhanced BEV feature map.

6. The method for detecting three-dimensional lane lines on a road based on binocular vision according to claim 1, characterized in that, A 3D anchor frame detection network, combined with the geometric prior of the adaptive terrain model, is used to detect 3D lane line candidate parameters based on the depth-enhanced BEV feature map. This includes: inputting the depth-enhanced BEV feature map into the 3D anchor frame detection network; supplementing grid position information through learnable position encoding; aggregating global context features via a multi-head self-attention mechanism; and optimizing feature representation through a feedforward network and residual connections. Based on the height, slope, and lateral tilt parameters of the adaptive terrain model, 3D anchor frames adapted to the terrain are pre-set on the BEV grid. Anchor frame parameters include center position, orientation angle, length, and width, and the slope and lateral tilt of the anchor frame are adapted to the terrain parameters of the corresponding grid. The 3D anchor frame detection network performs parameter regression and classification on each anchor frame, and combines the geometric prior to determine the adaptability of the anchor frame to the terrain, predicting 3D lane line candidate parameters. These 3D lane line candidate parameters include the 3D coordinates, width estimate, curvature estimate, slope estimate, lane line type, and confidence level of key nodes in the lane line center trajectory.

7. The method for detecting three-dimensional lane lines on a road based on binocular vision according to claim 1, characterized in that, The process of selecting effective 3D lane line candidate parameters from the candidate parameters by means of reprojection error constraints and road prior constraints includes: for each 3D lane line candidate, the 3D coordinates of key nodes are projected onto the stereo image in combination with camera intrinsic and extrinsic parameters to obtain the projected pixel coordinates of each node; image features around the projected pixel coordinates in the stereo image are extracted, and the deviation between the projected pixel coordinates and the pixel coordinates of the lane line texture features in the image is calculated to obtain the reprojection error of each node; if the reprojection error of any node exceeds a preset pixel threshold, the node is filtered; if more than a preset proportion of nodes in the same 3D lane line candidate have excessive reprojection errors, the 3D lane line candidate parameter is removed; for candidates that pass the reprojection error constraints, road prior constraint checks are performed, including lane line width constraints, curvature constraints, adjacent node continuity constraints, and adjacent lane line parallelism constraints; 3D lane line candidate parameters that satisfy all constraints are retained as effective 3D lane line candidate parameters.

8. The method for detecting three-dimensional lane lines on a road based on binocular vision according to claim 1, characterized in that, The trajectory optimization includes: constructing a trajectory optimization objective function, which includes a data term, a smoothing term, and a prior term. The data term is used to measure the consistency between the optimized position and the observed position, the smoothing term is used to ensure the continuity of parameters across frames, and the prior term is used to introduce road geometric constraints; iterating the objective function using a nonlinear optimization algorithm, using the result of the extended Kalman filter as the initial value, and stopping when the change in the objective function is less than a preset threshold or the maximum number of iterations is reached; setting an optimization frequency, performing real-time optimization every frame to ensure real-time detection, and performing batch optimization every preset number of frames to improve the consistency of global parameters.

9. The method for detecting three-dimensional lane lines on a road based on binocular vision according to claim 1, characterized in that, The extended Kalman filter and cross-frame continuity constraints include: using the effective three-dimensional lane line candidate parameters of consecutive frames as observations, combining vehicle speed and turning angle, and performing prediction and update steps through extended Kalman filtering to obtain initially stable lane line parameters; applying cross-frame continuity constraints, where the positional continuity constraint requires that the positional change of corresponding lane line points in adjacent frames does not exceed the product of the maximum allowable vehicle speed and the frame interval, and the parameter continuity constraint requires that the changes in lane line curvature and slope in adjacent frames do not exceed the product of the corresponding preset maximum rate of change and the frame interval; smoothing correction is performed on lane line parameters that violate the cross-frame continuity constraints, and readjustment is performed by combining the optimization results of historical frames with the observations of the current frame to ensure that there are no jumps in the cross-frame lane line parameters, thus obtaining the three-dimensional lane line parameters, which include the three-dimensional coordinates, curvature parameters, slope parameters, width parameters, lateral offset, confidence level, and lane line type of key nodes on the center trajectory of the lane line.

10. A road three-dimensional lane line detection system based on binocular vision, applied to the road three-dimensional lane line detection method based on binocular vision as described in any one of claims 1-9, characterized in that, The system includes: a data acquisition module for synchronously acquiring binocular images and IMU attitude data of the vehicle's forward direction; performing online calibration of camera intrinsic and extrinsic parameters on the binocular images and IMU attitude data to correct intrinsic and extrinsic parameter drift; and performing temporal alignment to correct timestamp deviations, resulting in temporally aligned binocular images and IMU attitude data; a data processing module for performing distortion correction, illumination compensation preprocessing, and multi-scale feature extraction on the temporally aligned binocular images, and determining a dense disparity map based on IMU attitude data constraints and left-right consistency constraints; and a model building module for generating road point clouds based on the dense disparity map and camera parameters, and constructing an adaptive terrain model including height, slope, and lateral tilt angle through local ground plane fitting, quadratic surface expansion, and normal clustering. The system comprises: a feature map construction module, used to map the depth features of the road point cloud and the texture features of the binocular image to a bird's-eye view to construct a depth-enhanced BEV feature map; a detection module, used to use a 3D anchor box detection network combined with the geometric prior of the adaptive terrain model to detect 3D lane line candidate parameters based on the depth-enhanced BEV feature map, and to filter effective 3D lane line candidate parameters from the 3D lane line candidate parameters through reprojection error constraints and road prior constraints; and an optimization module, used to perform extended Kalman filtering and trajectory optimization on the effective 3D lane line candidate parameters of consecutive frames to obtain 3D lane line parameters, wherein the trajectory optimization is used to ensure that the 3D lane line parameters satisfy position consistency constraints, parameter continuity constraints, and road geometric constraints.

Images