A three-degree-of-freedom dynamic external participation lane line structure combined perception method and system

Abstract

This invention relates to a three-degree-of-freedom dynamic extrinsic parameter and lane line structure joint perception method and system. The method includes: S1. Obtaining the pixel coordinates of all lane line points in the current frame from a 2D lane line detection network, and reading camera intrinsic parameters and static extrinsic parameters; S2. Initializing dynamic extrinsic parameters, initializing the cubic parametric curve coefficients and curvature parameters of each lane line; S3. Constructing a rotation matrix from the vehicle coordinate system to the ground coordinate system based on the current dynamic extrinsic parameter values; S4. Calculating the longitudinal coordinates of the landing point by backprojection along the camera rays, and constructing a depth residual with the longitudinal coordinates predicted by the lane line parametric equation. Compared with the implicit ground deviation processing methods in the prior art, this invention not only has a clearer physical meaning and more stable solution, but also introduces structured prior constraints for lane line curve fitting, which can still achieve stable solution even with large detection noise.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving perception technology, and in particular to a three-degree-of-freedom dynamic external participation lane line structure joint perception method and system. Background Technology

[0002] In autonomous driving scenarios, vehicles need to perceive 3D lane line information using a monocular camera to support path planning and control. The ground-plane assumption leads to large deviations in 3D lane line position: Existing methods often assume that the vehicle coordinate system coincides with the ground coordinate system (yaw, pitch, and roll angle errors are 0). When the vehicle is bumpy or in incline / descent areas, this assumption no longer holds, resulting in severe deviations in the 3D lane line position. Lack of explicit dynamic extrinsic parameter modeling: Current technologies do not formalize the three-degree-of-freedom Euler angle deviation between the vehicle and the ground as a "dynamic extrinsic parameter" and use it as a parameter to be solved, making it impossible to accurately describe the positional relationship between the real ground and the ideal ground.

[0003] Limited constraint types and insufficient robustness: Existing optimization methods only use reprojection error and parallelism error, without introducing structured constraints for lane line curve fitting, resulting in unstable solutions when lane line detection noise is high.

[0004] Uncoupled solution of dynamic extrinsic participants in lane line parameter estimation: Existing methods separate ground position estimation from lane line parameter estimation, and do not optimize them as joint variables simultaneously, resulting in error accumulation. Summary of the Invention

[0005] In view of the above problems, the present invention provides a three-degree-of-freedom dynamic external participation lane line structure joint perception method and system, which not only has a clearer physical meaning and more stable solution compared with the implicit processing of ground deviation in the prior art, but also introduces structured prior constraints for lane line curve fitting, which can still solve stably when the detection noise is large.

[0006] To achieve the above and other related objectives, the present invention provides the following technical solution: A three-degree-of-freedom dynamic externally involved lane line structure joint sensing method, the method comprising: S1. Obtain the pixel coordinates of all lane line points in the current frame from the 2D lane line detection network, and read the camera intrinsic parameters and static extrinsic parameters; S2. Initialize dynamic extrinsic parameters, and initialize the cubic parametric curve coefficients and curvature parameters of each lane line; S3. Based on the current dynamic external parameter values, construct the rotation matrix from the vehicle coordinate system to the ground coordinate system; S4. Calculate the longitudinal coordinates of the landing point by back-projection along the camera rays, and construct the depth residual by comparing the longitudinal coordinates predicted by the lane line parametric equations; S5. Calculate the cubic parametric curve fitting error and the consistency constraint error between the curvature parameter and the analytical curvature; S6. Calculate the consistency constraint error between the spacing between adjacent lane lines and the standard lane width; S7. The joint cost function is obtained by weighted summation of the three types of errors; S8. Determine whether the joint cost function has converged. If it has not converged, update the parameters and return to step S3. If it has converged, proceed to step S9. S9. Output the optimized 3D lane line parameter equations and dynamic extrinsic parameters.

[0007] Furthermore, the camera intrinsic parameters are the camera's intrinsic parameter matrix, and the camera static extrinsic parameters are the static extrinsic parameters from the camera to the vehicle coordinate system, including rotation matrices and translation vectors.

[0008] Furthermore, in step S3, the rotation matrix from the vehicle coordinate system to the ground coordinate system is constructed by defining the yaw angle deviation, pitch angle deviation, and roll angle deviation between the vehicle coordinate system and the ground coordinate system as dynamic extrinsic parameters.

[0009] Furthermore, in step S4, the predicted longitudinal coordinates are used to construct the depth residual. The depth residual is calculated based on the inverse projection depth residual. The depth residual between the longitudinal coordinates of the landing point in the ground coordinate system, which are inversely projected from the 2D lane line points along the camera rays, and the longitudinal coordinates predicted by the lane line parameter equation are calculated. The Huber robust kernel function is used to suppress outliers.

[0010] Furthermore, the use of the Huber robust kernel function to suppress outliers is defined as follows: Let the residual be e, and the loss function ρ(e) corresponding to the Huber robust kernel function is defined as follows: when |e|≤δ, ρ(e)=e 2 / 2; when |e|>δ, ρ(e)=δ|e|-δ 2 / 2 Where δ is an adjustment parameter that controls the boundary between the quadratic region and the linear region; the corresponding weight function is w(e)=ρ(e) / e, when |e|≤δ, w(e)=1; when |e|>δ, w(e)=δ / |e|, where δ and e are real numbers.

[0011] Furthermore, in step S5, the calculation of the cubic parametric curve fitting error and the consistency constraint error between the curvature parameter and the analytical curvature is carried out by using the lane line construction standard and adopting a spiral curve or a straight line, locally using cubic curve fitting, to construct the lane line cubic parametric curve fitting error, while introducing the consistency constraint between the curvature parameter and the analytical curvature to ensure the physical consistency of the curve parameters.

[0012] Furthermore, in step S6, the calculation of the consistency constraint error between the spacing between adjacent lane lines and the standard lane width is achieved by using the prior knowledge that the spacing between adjacent lane lines should meet the road construction standards to construct the lane spacing consistency constraint error, which replaces the traditional parallel constraint. In non-parallel scenarios such as forks, the constraint can be weakened by adaptively adjusting the weights.

[0013] Furthermore, in step S7, the weighted summation of the three types of errors to obtain the joint cost function is to form a joint cost function by weighting the depth residual, curve fitting and curvature constraint error and lane spacing consistency constraint error, and using the dynamic extrinsic parameters and lane line parameter equations as joint variables to be solved, and solving for the optimal parameters by minimizing the joint cost function.

[0014] Furthermore, the 3D lane line parameter equation is the cubic parameter equation and curvature parameter of the lane line in the ground coordinate system of the current frame, and the dynamic extrinsic parameter is the three-degree-of-freedom Euler angle deviation between the vehicle coordinate system and the ground coordinate system in the current frame.

[0015] To achieve the above and other related objectives, the present invention also provides a three-degree-of-freedom dynamic externally involved lane line structure joint perception system for implementing the aforementioned three-degree-of-freedom dynamic externally involved lane line structure joint perception method, the system comprising: The 2D lane detection module detects 2D lane lines from images taken by a monocular camera and outputs the pixel coordinates of the lane line points. The camera calibration parameter module provides the camera intrinsic parameter matrix and static extrinsic parameters from the camera to the vehicle coordinate system, providing a basis for projection calculation; The dynamic extrinsic parameter modeling module, connected to the camera calibration parameter module, is used to define the yaw angle deviation, pitch angle deviation, and roll angle deviation between the vehicle coordinate system and the ground coordinate system as dynamic extrinsic parameters, construct the rotation matrix from the vehicle coordinate system to the ground coordinate system, and the dynamic extrinsic parameters participate in the joint optimization as variables to be solved. The inverse projection depth residual construction module is connected to the 2D lane detection module, the interstellar calibration parameter module and the dynamic extrinsic parameter modeling module respectively. Based on the inverse projection depth residual, it calculates the depth residual between the longitudinal coordinates of the landing point in the ground coordinate system and the longitudinal coordinates predicted by the lane parameter equation when the 2D lane point is inversely projected along the camera ray. The Huber robust kernel function is used to suppress outliers. The quadratic curve fitting and curvature constraint module is connected to the 2D lane line detection module and the inverse projection depth residual construction module, respectively. It uses the lane line construction standard to construct the lane line cubic parametric curve fitting error, and introduces the consistency constraint between curvature parameters and analytical curvature to ensure the physical consistency of curve parameters. The lane spacing consistency constraint module is connected to the 2D lane line detection module and the quadratic curve fitting and curvature constraint module, respectively. It uses the prior knowledge that the spacing between adjacent lane lines should meet the road construction standards to construct the lane spacing consistency constraint error, which replaces the traditional parallel constraint. In non-parallel scenarios such as bifurcations, the constraint can be weakened by adaptively adjusting the weight. The joint cost optimization module, connected to the lane spacing consistency constraint module, is used to combine the depth residual, curve fitting and curvature constraint error and lane spacing consistency constraint error in a weighted manner to form a joint cost function. The dynamic extrinsic parameters and lane line parameter equations are used as joint variables to be solved, and the optimal parameters are solved by minimizing the joint cost function.

[0016] The present invention has the following positive effects: 1. This invention explicitly defines the three-degree-of-freedom Euler angle deviation between the vehicle coordinate system and the ground coordinate system as a dynamic extrinsic parameter and uses it as the parameter to be solved. Compared with the implicit ground deviation method in the prior art, the physical meaning is clearer and the solution is more stable.

[0017] 2. This invention introduces structured prior constraints for lane line curve fitting by simultaneously constructing three types of constraints: coordinate error, curve fitting error, and parallel constraint error. Compared with the existing technology that only uses reprojection error and parallel error, this invention can still solve stably even when the detection noise is large.

[0018] 3. This invention optimizes the lane line parameter equations by treating the dynamic external participants as joint variables to be solved simultaneously, thus avoiding the error accumulation caused by the step-by-step solution of first estimating the ground position and then calculating the lane line coordinates.

[0019] 4. This invention reduces system cost and deployment complexity by relying solely on 2D lane line detection results and camera calibration parameters, eliminating the need for additional sensors such as LiDAR and IMU. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the system framework of the present invention. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] Example: Figure 1 As shown, a three-degree-of-freedom dynamic externally involved lane line structure joint sensing method is described, the method comprising: S1. Obtain the pixel coordinates of all lane line points in the current frame from the 2D lane line detection network, and read the camera intrinsic parameters and static extrinsic parameters; S2. Initialize dynamic extrinsic parameters, and initialize the cubic parametric curve coefficients and curvature parameters of each lane line; S3. Based on the current dynamic external parameter values, construct the rotation matrix from the vehicle coordinate system to the ground coordinate system; S4. Calculate the longitudinal coordinates of the landing point by back-projection along the camera rays, and construct the depth residual by comparing the longitudinal coordinates predicted by the lane line parametric equations; S5. Calculate the cubic parametric curve fitting error and the consistency constraint error between the curvature parameter and the analytical curvature; S6. Calculate the consistency constraint error between the spacing between adjacent lane lines and the standard lane width; S7. The joint cost function is obtained by weighted summation of the three types of errors; S8. Determine whether the joint cost function has converged. If it has not converged, update the parameters and return to step S3. If it has converged, proceed to step S9. S9. Output the optimized 3D lane line parameter equations and dynamic extrinsic parameters.

[0023] In this embodiment, the camera intrinsic parameters are the camera's intrinsic parameter matrix, and the camera static extrinsic parameters are the static extrinsic parameters from the camera to the vehicle coordinate system. The static extrinsic parameters include a rotation matrix and a translation vector.

[0024] In this embodiment, in step S3, the rotation matrix from the vehicle coordinate system to the ground coordinate system is constructed by defining the yaw angle deviation, pitch angle deviation, and roll angle deviation between the vehicle coordinate system and the ground coordinate system as dynamic extrinsic parameters.

[0025] In this embodiment, in step S4, the predicted longitudinal coordinates are used to construct the depth residual. The depth residual is calculated based on the inverse projection depth residual. The depth residual between the longitudinal coordinates of the landing point in the ground coordinate system, which are inversely projected from the 2D lane line points along the camera rays, and the longitudinal coordinates predicted by the lane line parameter equation are calculated. The Huber robust kernel function is used to suppress outliers.

[0026] In this embodiment, the use of the Huber robust kernel function to suppress outliers is defined as follows: Let the residual be e, and the loss function ρ(e) corresponding to the Huber robust kernel function is defined as follows: when |e|≤δ, ρ(e)=e 2 / 2; when |e|>δ, ρ(e)=δ|e|-δ 2 / 2 where δ is an adjustment parameter that controls the boundary between the quadratic region and the linear region; the corresponding weight function is w(e)=ρ(e) / e, when |e|≤δ, w(e)=1; when |e|>δ, w(e)=δ / |e|, where δ and e are real numbers.

[0027] In this embodiment, in step S5, the calculation of the cubic parametric curve fitting error and the consistency constraint error between the curvature parameter and the analytical curvature is carried out by using the lane line construction standard and adopting a spiral curve or a straight line, locally using cubic curve fitting to construct the lane line cubic parametric curve fitting error, while introducing the consistency constraint between the curvature parameter and the analytical curvature to ensure the physical consistency of the curve parameters.

[0028] In this embodiment, in step S6, the calculation of the consistency constraint error between the spacing between adjacent lane lines and the standard lane width is to construct the lane spacing consistency constraint error by using the prior knowledge that the spacing between adjacent lane lines should meet the road construction standards, which replaces the traditional parallel constraint. In non-parallel scenarios such as forks, the constraint can be weakened by adaptively adjusting the weights.

[0029] In this embodiment, in step S7, the weighted summation of the three types of errors to obtain the joint cost function is to form a joint cost function by weighting the depth residual, curve fitting and curvature constraint error and lane spacing consistency constraint error. The dynamic extrinsic parameters and lane line parameter equations are the joint variables to be solved, and the optimal parameters are solved by minimizing the joint cost function.

[0030] In this embodiment, the 3D lane line parameter equation is the cubic parameter equation and curvature parameter of the lane line in the ground coordinate system of the current frame, and the dynamic extrinsic parameter is the three-degree-of-freedom Euler angle deviation between the vehicle coordinate system and the ground coordinate system in the current frame.

[0031] In this embodiment, as Figure 2 As shown, this invention provides a three-degree-of-freedom dynamic externally involved lane line structure joint perception system for implementing the aforementioned three-degree-of-freedom dynamic externally involved lane line structure joint perception method. The system includes: The 2D lane detection module detects 2D lane lines from images taken by a monocular camera and outputs the pixel coordinates of the lane line points. The camera calibration parameter module provides the camera intrinsic parameter matrix and static extrinsic parameters from the camera to the vehicle coordinate system, providing a basis for projection calculation; The dynamic extrinsic parameter modeling module, connected to the camera calibration parameter module, is used to define the yaw angle deviation, pitch angle deviation, and roll angle deviation between the vehicle coordinate system and the ground coordinate system as dynamic extrinsic parameters, construct the rotation matrix from the vehicle coordinate system to the ground coordinate system, and the dynamic extrinsic parameters participate in the joint optimization as variables to be solved. The inverse projection depth residual construction module is connected to the 2D lane detection module, the interstellar calibration parameter module and the dynamic extrinsic parameter modeling module respectively. Based on the inverse projection depth residual, it calculates the depth residual between the longitudinal coordinates of the landing point in the ground coordinate system and the longitudinal coordinates predicted by the lane parameter equation when the 2D lane point is inversely projected along the camera ray. The Huber robust kernel function is used to suppress outliers. The quadratic curve fitting and curvature constraint module is connected to the 2D lane line detection module and the inverse projection depth residual construction module, respectively. It uses the lane line construction standard to construct the lane line cubic parametric curve fitting error, and introduces the consistency constraint between curvature parameters and analytical curvature to ensure the physical consistency of curve parameters. The lane spacing consistency constraint module is connected to the 2D lane line detection module and the quadratic curve fitting and curvature constraint module, respectively. It uses the prior knowledge that the spacing between adjacent lane lines should meet the road construction standards to construct the lane spacing consistency constraint error, which replaces the traditional parallel constraint. In non-parallel scenarios such as bifurcations, the constraint can be weakened by adaptively adjusting the weight. The joint cost optimization module, connected to the lane spacing consistency constraint module, is used to combine the depth residual, curve fitting and curvature constraint error and lane spacing consistency constraint error in a weighted manner to form a joint cost function. The dynamic extrinsic parameters and lane line parameter equations are used as joint variables to be solved, and the optimal parameters are solved by minimizing the joint cost function.

[0032] In summary, compared with the implicit ground deviation methods in the prior art, the present invention not only has a clearer physical meaning and a more stable solution, but also introduces structured prior constraints for lane line curve fitting, which can still achieve stable solution even when the detection noise is large.

[0033] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A three-degree-of-freedom dynamic externally involved lane line structure joint sensing method, characterized in that, The method includes: S1. Obtain the pixel coordinates of all lane line points in the current frame from the 2D lane line detection network, and read the camera intrinsic parameters and static extrinsic parameters; S2. Initialize dynamic extrinsic parameters, and initialize the cubic parametric curve coefficients and curvature parameters of each lane line; S3. Based on the current dynamic external parameter values, construct the rotation matrix from the vehicle coordinate system to the ground coordinate system; S4. Calculate the longitudinal coordinates of the landing point by back-projection along the camera rays, and construct the depth residual by comparing the longitudinal coordinates predicted by the lane line parametric equations; S5. Calculate the cubic parametric curve fitting error and the consistency constraint error between the curvature parameter and the analytical curvature; S6. Calculate the consistency constraint error between the spacing between adjacent lane lines and the standard lane width; S7. The joint cost function is obtained by weighted summation of the three types of errors; S8. Determine whether the joint cost function has converged. If it has not converged, update the parameters and return to step S3. If it has converged, proceed to step S9. S9. Output the optimized 3D lane line parameter equations and dynamic extrinsic parameters.

2. The three-degree-of-freedom dynamic externally involved lane line structure joint sensing method according to claim 1, characterized in that: The camera intrinsic parameters are the camera's intrinsic parameter matrix, and the camera static extrinsic parameters are the static extrinsic parameters from the camera to the vehicle coordinate system. The static extrinsic parameters include rotation matrices and translation vectors.

3. The three-degree-of-freedom dynamic external participation lane line structure joint sensing method according to claim 1, characterized in that, In step S3, the rotation matrix from the vehicle coordinate system to the ground coordinate system is constructed by defining the yaw angle deviation, pitch angle deviation, and roll angle deviation between the vehicle coordinate system and the ground coordinate system as dynamic extrinsic parameters.

4. The three-degree-of-freedom dynamic externally involved lane line structure joint sensing method according to claim 1, characterized in that, In step S4, the predicted longitudinal coordinates are used to construct the depth residual. The depth residual is calculated based on the inverse projection depth residual. The depth residual between the longitudinal coordinates of the landing point in the ground coordinate system, which are inversely projected from the 2D lane line points along the camera rays, and the longitudinal coordinates predicted by the lane line parameter equation are calculated. The Huber robust kernel function is used to suppress outliers.

5. The three-degree-of-freedom dynamic externally involved lane line structure joint sensing method according to claim 4, characterized in that: The Huber robust kernel function is used to suppress outliers. Let the residual be e. The loss function ρ(e) corresponding to the Huber robust kernel function is defined as follows: when |e|≤δ, ρ(e)=e 2 / 2; when |e|>δ, ρ(e)=δ|e|-δ 2 / 2 Where δ is an adjustment parameter that controls the boundary between the quadratic region and the linear region; the corresponding weight function is w(e)=ρ(e) / e, when |e|≤δ, w(e)=1; when |e|>δ, w(e)=δ / |e|, where δ and e are real numbers.

6. The three-degree-of-freedom dynamic externally involved lane line structure joint sensing method according to claim 1, characterized in that, In step S5, the calculation of the cubic parametric curve fitting error and the consistency constraint error between the curvature parameter and the analytical curvature is carried out by using the lane line construction standard and adopting a spiral curve or a straight line, locally using cubic curve fitting to construct the lane line cubic parametric curve fitting error. At the same time, the consistency constraint between the curvature parameter and the analytical curvature is introduced to ensure the physical consistency of the curve parameters.

7. The three-degree-of-freedom dynamic externally involved lane line structure joint sensing method according to claim 1, characterized in that, In step S6, the calculation of the consistency constraint error between the spacing between adjacent lane lines and the standard lane width is achieved by using the prior knowledge that the spacing between adjacent lane lines should meet the road construction standards to construct the lane spacing consistency constraint error, which replaces the traditional parallel constraint. In non-parallel scenarios such as bifurcations, the constraint can be weakened by adaptively adjusting the weights.

8. The three-degree-of-freedom dynamic externally involved lane line structure joint sensing method according to claim 1, characterized in that, In step S7, the joint cost function obtained by weighted summation of the three types of errors is a joint cost function composed of weighted combination of depth residual, curve fitting and curvature constraint error and lane spacing consistency constraint error. The dynamic extrinsic parameters and lane line parameter equations are the joint variables to be solved, and the optimal parameters are solved by minimizing the joint cost function.

9. The three-degree-of-freedom dynamic externally involved lane line structure joint sensing method according to claim 1, characterized in that: The 3D lane line parametric equation is the cubic parametric equation and curvature parameter of the lane line in the ground coordinate system of the current frame, and the dynamic extrinsic parameter is the three-degree-of-freedom Euler angle deviation between the vehicle coordinate system and the ground coordinate system in the current frame.

10. A three-degree-of-freedom dynamic externally involved lane line structure joint sensing system, characterized in that, For implementing the three-degree-of-freedom dynamic externally involved lane line structure joint perception method according to any one of claims 1-9, the system comprises: The 2D lane detection module detects 2D lane lines from images taken by a monocular camera and outputs the pixel coordinates of the lane line points. The camera calibration parameter module provides the camera intrinsic parameter matrix and static extrinsic parameters from the camera to the vehicle coordinate system, providing a basis for projection calculation; The dynamic extrinsic parameter modeling module, connected to the camera calibration parameter module, is used to define the yaw angle deviation, pitch angle deviation, and roll angle deviation between the vehicle coordinate system and the ground coordinate system as dynamic extrinsic parameters, construct the rotation matrix from the vehicle coordinate system to the ground coordinate system, and the dynamic extrinsic parameters participate in the joint optimization as variables to be solved. The inverse projection depth residual construction module is connected to the 2D lane detection module, the interstellar calibration parameter module and the dynamic extrinsic parameter modeling module respectively. Based on the inverse projection depth residual, it calculates the depth residual between the longitudinal coordinates of the landing point in the ground coordinate system and the longitudinal coordinates predicted by the lane parameter equation when the 2D lane point is inversely projected along the camera ray. The Huber robust kernel function is used to suppress outliers. The quadratic curve fitting and curvature constraint module is connected to the 2D lane line detection module and the inverse projection depth residual construction module, respectively. It uses the lane line construction standard to construct the lane line cubic parametric curve fitting error, and introduces the consistency constraint between curvature parameters and analytical curvature to ensure the physical consistency of curve parameters. The lane spacing consistency constraint module is connected to the 2D lane line detection module and the quadratic curve fitting and curvature constraint module, respectively. It uses the prior knowledge that the spacing between adjacent lane lines should meet the road construction standards to construct the lane spacing consistency constraint error, which replaces the traditional parallel constraint. In non-parallel scenarios such as bifurcations, the constraint can be weakened by adaptively adjusting the weight. The joint cost optimization module, connected to the lane spacing consistency constraint module, is used to combine the depth residual, curve fitting and curvature constraint error and lane spacing consistency constraint error in a weighted manner to form a joint cost function. The dynamic extrinsic parameters and lane line parameter equations are used as joint variables to be solved, and the optimal parameters are solved by minimizing the joint cost function.

Images