Vehicle multi-camera external parameter calibration method and system based on three-dimensional lane line

Abstract

The invention discloses a vehicle multi-camera external parameter calibration method and system based on a three-dimensional lane line, and the system obtains a two-dimensional lane line instance detected by a plurality of cameras in the driving process of a vehicle, obtains the pixel depth corresponding to the two-dimensional lane line instance in a single-frame image through the laser projection alignment or binocular depth estimation, and carries out the calibration of the two-dimensional lane line instance. And according to the spatial information of the single-frame three-dimensional lane line instance and the binocular camera high-precision pose transformation trajectory of vehicle movement, reconstructing to obtain a multi-frame long-range three-dimensional lane line instance under a binocular system. And constructing a calibration mapping error between the multi-frame long-range three-dimensional lane line instance and the two-dimensional lane line instance of the camera to be calibrated, and performing optimization to obtain an external parameter rotation matrix from each camera to a vehicle coordinate system. By using the method, the extrinsic parameter rotation matrix of the multiple cameras relative to the vehicle coordinate system can be efficiently calibrated.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving technology, and in particular relates to a vehicle multi-camera extrinsic parameter calibration method and system based on three-dimensional lane lines. Background Art

[0002] When an intelligent driving vehicle leaves the factory after installing a camera, static calibration is usually required to obtain accurate internal and external parameters of the camera. However, after the vehicle is driven, the camera lens will become loose due to factors such as bumps, causing the original external parameters to change, affecting the intelligent driving perception performance. In addition, when testing intelligent driving solutions, the camera installation method is not always fixed, and there is no condition for factory calibration. Therefore, how to simply and effectively calibrate the camera external parameters is an important issue in the field of intelligent driving, and for multi-camera deployment solutions, multi-camera calibration becomes even more difficult. Summary of the invention

[0003] In view of this, the present invention provides a vehicle multi-camera extrinsic parameter calibration method and system based on three-dimensional lane lines, which uses the three-dimensional information of the lane lines passed by the vehicle and the lane line detection results perceived by each camera on the two-dimensional image to optimize the extrinsic parameter rotation matrix of each camera relative to the vehicle coordinate system, thereby improving the accuracy and efficiency of the camera extrinsic parameter calibration, and is specifically implemented by the following technical scheme.

[0004] In a first aspect, the present invention provides a method for calibrating vehicle multi-camera extrinsic parameters based on three-dimensional lane lines, comprising the following steps:

[0005] Acquire images captured by multiple cameras of a vehicle traveling, and perform single-frame lane line detection on the images to obtain two-dimensional lane line instances, wherein the two-dimensional lane line instances include single-frame two-dimensional lane line instances of a binocular camera and single-frame two-dimensional lane line instances of a camera to be calibrated;

[0006] Performing image depth analysis on the two-dimensional lane line instance to obtain single-frame three-dimensional lane line instance spatial information;

[0007] Acquire a high-precision vehicle motion posture transformation trajectory acquired by a binocular camera, and perform three-dimensional reconstruction based on the single-frame three-dimensional lane line instance spatial information and the high-precision vehicle motion posture transformation trajectory to obtain a multi-frame long-range three-dimensional lane line instance in a binocular coordinate system;

[0008] Construct a calibration mapping error of the multi-frame long-range three-dimensional lane line instance and the two-dimensional lane line instance of the camera to be calibrated, and optimize the calibration mapping error to obtain an extrinsic rotation matrix from each camera to the vehicle coordinate system.

[0009] As a preferred embodiment of the above technical solution, images captured by multiple cameras of a vehicle are obtained, and single-frame lane line detection is performed on the images to obtain a two-dimensional lane line instance, including:

[0010] Obtain lane line images captured by each camera on which the vehicle is traveling and the position and posture of each camera;

[0011] Single-frame lane line detection is performed on the lane line image according to the positions and postures of each camera to obtain a two-dimensional lane line instance.

[0012] As a preferred embodiment of the above technical solution, obtaining a high-precision vehicle motion posture transformation trajectory captured by a binocular camera includes:

[0013] Calculate the binocular camera coordinate system corresponding to a single-frame 2D lane line instance based on the position and posture of each camera;

[0014] A high-precision vehicle motion posture transformation trajectory captured by the binocular camera is determined according to the binocular camera coordinate system and the posture of each camera.

[0015] As a preferred embodiment of the above technical solution, performing image depth analysis on the two-dimensional lane line instance to obtain single-frame three-dimensional lane line instance spatial information includes:

[0016] Acquire feature points in the single-frame two-dimensional lane line instance, and perform feature matching on two single-frame two-dimensional lane line instances according to the feature points to obtain an instance matching pair;

[0017] Performing a depth image analysis on the instance matching pairs to obtain a pixel depth corresponding to a two-dimensional lane line instance in a single frame image, wherein the depth image analysis includes using laser projection alignment or binocular depth estimation;

[0018] The single-frame three-dimensional lane line instance spatial information corresponding to the two-dimensional lane line instance is restored based on the pixel depth.

[0019] As a preferred embodiment of the above technical solution, the spatial information of the single-frame three-dimensional lane line instance is spliced ​​according to the high-precision vehicle motion posture transformation trajectory to obtain multiple frames of long-range three-dimensional lane line instances.

[0020] As a preferred embodiment of the above technical solution, the spatial information of the single-frame three-dimensional lane line instance is spliced ​​according to the high-precision vehicle motion posture transformation trajectory to obtain a multi-frame long-range three-dimensional lane line instance, including:

[0021] The coordinates of the jth 3D point in the i-th frame in the vehicle coordinate system of the current frame are preset to (x, y, z) ij , construct the pose transformation matrix [R|T] of the vehicle coordinate system A of the i-th frame relative to the starting reference system B, where R is a 3×3 rotation matrix and T is a 3×1 translation matrix. The corresponding expression is:

[0022] (x B ,y B ,z B ) T=R A→B (x A ,y A ,z A ) T +T A→B (1)

[0023] Among them, (x B ,y B ,z B ) represents the three-dimensional coordinates of the starting reference system B, z A represents the three-dimensional coordinates of the vehicle coordinate system A, R A→B represents the rotation transformation from the vehicle coordinate system A to the starting reference system B, T A→B Represents the translation transformation from the vehicle coordinate system A to the starting reference system B;

[0024] Performing a posture transformation on the three-dimensional point according to the posture transformation matrix to obtain the three-dimensional coordinates of the three-dimensional point in the starting reference system B;

[0025] A mapping error function between the two-dimensional lane line instance and the three-dimensional lane line instance is constructed according to the three-dimensional coordinates in the starting reference system B, and the mapping error function is optimized to obtain an extrinsic rotation matrix from each camera to the vehicle coordinate system, wherein the three-dimensional lane line instance includes a multi-frame long-range three-dimensional lane line instance.

[0026] As a preferred embodiment of the above technical solution, constructing a mapping error function between the two-dimensional lane line instance and the three-dimensional lane line instance according to the three-dimensional coordinates in the starting reference system B includes:

[0027] The pixel position coordinate point set corresponding to the preset two-dimensional lane line instance is {(u0,v0),(u1,v1),(u2,v2)...,(u j ,v j ),...}, the corresponding two-dimensional lane line instance fitting equation is:

[0028] {u=a3*v 3 +a2*v 2 +a1*v+b,(v start ,v end )} (2)

[0029] Among them, v start and v end The v coordinates represent the starting point and the end point respectively, and a1, a2, a3 and b are constants;

[0030] The position coordinate point set of the vehicle coordinate system of the preset three-dimensional lane line instance {(x0,y0,z0),(x1,y1,z1),(x2,y2,z2)...,(x j ,y j,z j ),...}, the fitting equation corresponding to the three-dimensional lane line instance is:

[0031] {y=c3*x 3 +c2*x 2 +c1*x+c0,z=d3*x 3 +d2*x 2 +d1*x+d0,(x start ,x end )} (3)

[0032] Among them, c3, c2, c1, c0, d3, d2, d1, d0 are constants, x start and x end are the x coordinates of the starting point and the end point respectively;

[0033] Then the three-dimensional coordinates of a point W in the vehicle coordinate system (x w ,y w ,z w ) is transformed to the camera coordinate system through the matrix [R|T]:

[0034]

[0035] Point W in the camera coordinate system is transformed to the pixel coordinate system through the camera projection model. The intrinsic distortion parameters are known, and the corresponding extrinsic rotation matrix expression is:

[0036]

[0037] Where, f is the focal length of the camera, f x , f y Represents the camera focal length in the x and y directions.

[0038] As a preferred embodiment of the above technical solution, a perception result of a two-dimensional lane line instance perceived by a camera and projected onto a three-dimensional lane line instance is obtained, and a first point line distance error function of a mapping point on a fitting surface of the perception result to a fitting curve equation of a corresponding three-dimensional lane line instance is constructed;

[0039] Construct a line distance error function from the pixel point projected from the three-dimensional lane line instance on the camera image to the second point of the fitting curve equation corresponding to the two-dimensional lane line instance;

[0040] The mapping error function is optimized based on the first point-line distance error function and the second point-line distance error function to obtain an extrinsic rotation matrix from each camera to the vehicle coordinate system.

[0041] In a second aspect, the present invention further provides a vehicle multi-camera extrinsic parameter calibration system based on three-dimensional lane lines, comprising:

[0042] A lane line instance acquisition module is used to acquire images captured by multiple cameras of a vehicle, and perform single-frame lane line detection on the images to obtain two-dimensional lane line instances, wherein the two-dimensional lane line instances include single-frame two-dimensional lane line instances of a binocular camera and single-frame two-dimensional lane line instances of a camera to be calibrated;

[0043] A lane line instance detection module, used to perform image depth analysis on the two-dimensional lane line instance to obtain single-frame three-dimensional lane line instance spatial information;

[0044] A three-dimensional reconstruction module is used to obtain a high-precision vehicle motion posture transformation trajectory acquired by a binocular camera, and to perform three-dimensional reconstruction based on the single-frame three-dimensional lane line instance spatial information and the high-precision vehicle motion posture transformation trajectory to obtain a multi-frame long-range three-dimensional lane line instance in a binocular coordinate system;

[0045] The camera extrinsic calibration module is used to construct the calibration mapping error of the multi-frame long-range three-dimensional lane line instance and the two-dimensional lane line instance of the camera to be calibrated, and optimize the calibration mapping error to obtain the extrinsic rotation matrix of each camera to the vehicle coordinate system.

[0046] The present invention provides a vehicle multi-camera extrinsic parameter calibration method and system based on three-dimensional lane lines, which obtain images collected by multiple cameras of a traveling vehicle, perform single-frame lane line detection on the images to obtain a two-dimensional lane line instance, perform image depth analysis on the two-dimensional lane line instance to obtain single-frame three-dimensional lane line instance spatial information, obtain a high-precision vehicle motion posture transformation trajectory collected by a binocular camera, perform three-dimensional reconstruction according to the single-frame three-dimensional lane line instance spatial information and the high-precision vehicle motion posture transformation trajectory to obtain a multi-frame long-range three-dimensional lane line instance of a binocular coordinate system, construct a calibration mapping error of the multi-frame long-range three-dimensional lane line instance and the two-dimensional lane line instance of the camera to be calibrated, and optimize the calibration mapping error to obtain an extrinsic parameter rotation matrix from each camera to a vehicle coordinate system, and use three-dimensional lane lines to perform multi-camera extrinsic parameter calibration, which is more lightweight, can use pure vision configuration more flexibly, reduces the influence of time synchronization problems between different types of sensors, and effectively calibrates the extrinsic parameter rotation matrix of multiple cameras relative to the vehicle coordinate system, thereby improving calibration accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0048] Figure 1A flow chart of a vehicle multi-camera extrinsic parameter calibration method based on three-dimensional lane lines provided by the present invention;

[0049] Figure 2 A schematic diagram of the coordinate system of the posture conversion matrix provided by the present invention;

[0050] Figure 3 A schematic diagram of a coordinate system of a camera model provided by the present invention;

[0051] Figure 4 A curve diagram of the first point line distance error function provided by the present invention;

[0052] Figure 5 A curve diagram of the second point line distance error function provided by the present invention;

[0053] Figure 6 A structural block diagram of a vehicle multi-camera extrinsic calibration system based on three-dimensional lane lines provided by the present invention;

[0054] Figure 7 This is a diagram of the implementation process of the vehicle multi-camera extrinsic parameter calibration system based on three-dimensional lane lines provided by the present invention. DETAILED DESCRIPTION

[0055] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0056] See also Figure 1 The present invention provides a vehicle multi-camera extrinsic parameter calibration method based on three-dimensional lane lines, comprising the following steps:

[0057] S1: Acquire images captured by multiple cameras of a driving vehicle, and perform single-frame lane line detection on the images to obtain two-dimensional lane line instances, wherein the two-dimensional lane line instances include single-frame two-dimensional lane line instances of binocular cameras and single-frame two-dimensional lane line instances of cameras to be calibrated;

[0058] S2: performing image depth analysis on the two-dimensional lane line instance to obtain single-frame three-dimensional lane line instance spatial information;

[0059] S3: Performing three-dimensional reconstruction according to the single-frame three-dimensional lane line instance spatial information and the high-precision vehicle motion posture transformation trajectory to obtain a multi-frame long-range three-dimensional lane line instance in a binocular coordinate system;

[0060] S4: constructing a calibration mapping error of the multi-frame long-range three-dimensional lane line instance and the two-dimensional lane line instance of the camera to be calibrated, and optimizing the calibration mapping error to obtain an extrinsic rotation matrix from each camera to the vehicle coordinate system.

[0061] In this embodiment, images captured by multiple cameras of a vehicle are obtained, and single-frame lane line detection is performed on the images to obtain a two-dimensional lane line instance, including: obtaining lane line images captured by each camera of the vehicle and the position and posture of each camera; performing single-frame lane line detection on the lane line images according to the position and posture of each camera to obtain a two-dimensional lane line instance. The high-precision vehicle motion posture transformation trajectory captured by the binocular camera is obtained, including: calculating the binocular camera coordinate system corresponding to the single-frame two-dimensional lane line instance according to the position and posture of each camera; determining the high-precision vehicle motion posture transformation trajectory captured by the binocular camera according to the binocular camera coordinate system and the position and posture of each camera. The camera can be installed in different orientations on the vehicle, and can be a forward mid-range camera, a left camera, a right camera, a left rear camera, and a right rear camera. The corresponding two-dimensional lane line instances include two parallel lane lines, left and right side arcs, and single straight lines at the left rear and right rear. Usually, the lane lines are white and the road surface is black.

[0062] It should be noted that the single-frame lane line detection is based on the two-dimensional lane line detection technology to detect the lane line instances corresponding to each camera image, mainly including the images taken by the front-view medium-range binocular camera, the front-view long-range camera, the front-view short-range camera, the left camera, the right camera, the left rear camera and the right rear camera. The distance will affect the clarity of the image. The pose is the representation of the position and posture of a coordinate system relative to another coordinate system, which is described by rotation and translation in three-dimensional space. The high-precision vehicle motion pose transformation trajectory is the pose transformation trajectory of the reference sensor, that is, the binocular camera, obtained by SLAM or three-dimensional reconstruction. The two-dimensional lane line instance and the multi-frame long-range three-dimensional lane line instance are respectively 2D lane line instances and 3D lane line instances. The single-frame three-dimensional lane line spatial information can be a point cloud, and the multi-frame long-range three-dimensional lane line instance is referred to as a three-dimensional lane line instance. The vehicle odometer data is obtained by using methods such as SLAM. The odometer data can be obtained by binocular reconstruction method, or the lane line 3D information can be obtained by laser, and then the single-frame lane line point cloud is spliced ​​to obtain the three-dimensional lane line instance. In other words, all cameras need to detect 2D lane line instances. The 2D lane line instances detected by the reference camera (binocular) are used to restore the 3D lane line instances. The 2D lane line instances detected by the camera to be calibrated are used to construct a calibration error function with the 3D lane line instances.

[0063] It should be understood that by acquiring images captured by multiple cameras of a traveling vehicle, and performing single-frame lane line detection on the images to obtain a two-dimensional lane line instance, performing image depth analysis on the two-dimensional lane line instance to obtain single-frame three-dimensional lane line instance spatial information, obtaining a high-precision vehicle motion posture transformation trajectory captured by a binocular camera, performing three-dimensional reconstruction based on the single-frame three-dimensional lane line instance spatial information and the high-precision vehicle motion posture transformation trajectory to obtain a multi-frame long-range three-dimensional lane line instance in a binocular coordinate system, constructing a calibration mapping error between the multi-frame long-range three-dimensional lane line instance and the two-dimensional lane line instance of the camera to be calibrated, and optimizing the calibration mapping error to obtain the extrinsic rotation matrix from each camera to the vehicle coordinate system, and using three-dimensional lane lines to perform multi-camera extrinsic calibration, it is more lightweight, can use a pure vision configuration more flexibly, reduces the impact of time synchronization problems between different types of sensors, and effectively calibrates the extrinsic rotation matrix of multiple cameras relative to the vehicle coordinate system, thereby improving calibration accuracy and efficiency.

[0064] Optionally, image depth analysis is performed on the two-dimensional lane line instance to obtain single-frame three-dimensional lane line instance spatial information, including: acquiring feature points in the single-frame two-dimensional lane line instance, and performing feature matching on two single-frame two-dimensional lane line instances based on the feature points to obtain an instance matching pair; performing depth image analysis based on the instance matching pair to obtain pixel depth corresponding to the two-dimensional lane line instance in the single-frame image, wherein the depth image analysis includes laser projection alignment or binocular depth estimation; and restoring the single-frame three-dimensional lane line instance spatial information corresponding to the two-dimensional lane line instance based on the pixel depth.

[0065] In this embodiment, the spatial information of the single-frame three-dimensional lane line instance is spliced ​​according to the high-precision vehicle motion posture transformation trajectory to obtain a multi-frame long-range three-dimensional lane line instance. The SIFT (Scale-invariant feature transform) algorithm can be used to extract feature points in each camera image. When performing feature matching on each two single-frame two-dimensional lane line instances, the epipolar geometry can be calculated for each instance matching pair, the basic matrix or the essential matrix can be estimated, and the matching pair can be optimized and improved by the random sampling consensus algorithm. First, two frames of camera images can be initialized to perform the first bundle adjustment method, and then new camera images are added cyclically for new BA optimization, and finally until all camera images are added to the BA optimization, the visual point cloud map of the pose and scene of all camera images is obtained, that is, the 3D point coordinates of the world system corresponding to the feature points included in each camera image. In other words, the 3D lane line is provided by the high-precision vehicle motion pose information and the binocular 2D lane line instance under the binocular system, which has (x, y, z) or three-dimensional coordinate information.

[0066] The single-frame three-dimensional lane line instance spatial information is spliced ​​according to the high-precision vehicle motion posture transformation trajectory to obtain a multi-frame long-range three-dimensional lane line instance, including:

[0067] The coordinates of the jth 3D point in the i-th frame in the vehicle coordinate system of the current frame are preset to (x, y, z) ij , construct the pose transformation matrix [R|T] of the vehicle coordinate system A of the i-th frame relative to the starting reference system B, where R is a 3×3 rotation matrix and T is a 3×1 translation matrix. The corresponding expression is:

[0068] (x B ,y B ,z B ) T =T A→B (x A ,y A ,z A ) T +T A→B (1)

[0069] Among them, (x B ,y B ,z B ) represents the three-dimensional coordinates of the starting reference system B, z A represents the three-dimensional coordinates of the vehicle coordinate system A, R A→B represents the rotation transformation from the vehicle coordinate system A to the starting reference system B, T A→B Represents the translation transformation from the vehicle coordinate system A to the starting reference system B;

[0070] Performing a posture transformation on the three-dimensional point according to the posture transformation matrix to obtain the three-dimensional coordinates of the three-dimensional point in the starting reference system B;

[0071] A mapping error function between the two-dimensional lane line instance and the three-dimensional lane line instance is constructed according to the three-dimensional coordinates in the starting reference system B, and the mapping error function is optimized to obtain an extrinsic rotation matrix from each camera to the vehicle coordinate system, wherein the three-dimensional lane line instance includes a multi-frame long-range three-dimensional lane line instance.

[0072] It should be noted that, see Figure 2 and Figure 3 , constructing a mapping error function between the two-dimensional lane line instance and the three-dimensional lane line instance according to the three-dimensional coordinates in the starting reference system B, including:

[0073] The pixel position coordinate point set corresponding to the preset two-dimensional lane line instance is {(u0,v0),(u1,v1),(u2,v2)...,(u j ,v j ),...}, the corresponding two-dimensional lane line instance fitting equation is:

[0074] {u=a3*v 3 +a2*v 2 +a1*v+b,(v start ,v end)} (2)

[0075] Among them, v start and v end The v coordinates represent the starting point and the end point respectively, and a1, a2, a3 and b are constants;

[0076] The position coordinate point set of the vehicle coordinate system of the preset three-dimensional lane line instance {(x0,y0,z0),(x1,y1,z1),(x2,y2,z2)...,(x j ,y j ,z j ),...}, the fitting equation corresponding to the three-dimensional lane line instance is:

[0077] {y=c3*x 3 +c2*x 2 +c1*x+c0,z=d3*x 3 +d2*x 2 +d1*x+d0,(x start ,x end )} (3)

[0078] Among them, c3, c2, c1, c0, d3, d2, d1, d0 are constants, x start and x end are the x coordinates of the starting point and the end point respectively;

[0079] Then the three-dimensional coordinates of a point W in the vehicle coordinate system (x w ,y w ,z w ) is transformed to the camera coordinate system through the matrix [R|T]:

[0080]

[0081] Point W in the camera coordinate system is transformed to the pixel coordinate system through the camera projection model. The intrinsic distortion parameters are known, and the corresponding extrinsic rotation matrix expression is:

[0082]

[0083] Where, f is the focal length of the camera, f x , f y Represents the camera focal length in the x and y directions.

[0084] Specifically, see Figure 4 and Figure 5 , obtain the perception result of the camera sensing the projection of the two-dimensional lane line instance to the three-dimensional lane line instance, and construct a first point-line distance error function from the mapping point on the fitting surface of the perception result to the fitting curve equation of the corresponding three-dimensional lane line instance; construct a second point-line distance error function from the pixel point projected from the three-dimensional lane line instance on the camera image to the fitting curve equation of the corresponding two-dimensional lane line instance; optimize the mapping error function based on the first point-line distance error function and the second point-line distance error function to obtain the extrinsic parameter rotation matrix from each camera to the vehicle coordinate system. In other words, all cameras involved in the extrinsic parameter calibration process need to detect 2D lane line instances, and obtain the corresponding 2D lane line instances on each frame of the image through a deep learning algorithm combined with an image algorithm. Figure 4 In the figure, the solid circle represents the mapping point of the 2D lane line instance on the lane line fitting surface, the long solid line represents the 3D lane line instance equation, the arrow represents the projection or mapping, and the long dashed line represents the 3D lane line mapping equation on the xoy and xoz planes. The perception result can be obtained by sampling the spatial coordinates of the points on the lane line. Figure 5 In the figure, the solid circles represent the projection points of the 3D lane line instances on the image, and the long solid lines represent the equations of the 2D lane line instances.

[0085] Among them, a point-to-line distance error function is constructed from the mapping points of the camera-perceived 2D lane line instance projected onto the 3D lane line fitting surface to the corresponding 3D lane line instance fitting curve equation, and a point-to-line distance error function is constructed from the pixel points of the 3D lane line instance projected onto the camera image to the corresponding 2D lane line instance fitting curve equation. The mapping error of the 2D lane line instance projected onto the 3D lane line instance and the mapping error of the 3D lane line instance projected onto the 2D lane line instance can be obtained.

[0086] See also Figure 6 and Figure 7 The present invention also provides a vehicle multi-camera extrinsic parameter calibration system based on three-dimensional lane lines, comprising:

[0087] A lane line instance acquisition module is used to acquire images captured by multiple cameras of a vehicle, and perform single-frame lane line detection on the images to obtain two-dimensional lane line instances, wherein the two-dimensional lane line instances include single-frame two-dimensional lane line instances of a binocular camera and single-frame two-dimensional lane line instances of a camera to be calibrated;

[0088] A lane line instance detection module, used to perform image depth analysis on the two-dimensional lane line instance to obtain single-frame three-dimensional lane line instance spatial information;

[0089] A three-dimensional reconstruction module is used to obtain a high-precision vehicle motion posture transformation trajectory acquired by a binocular camera, and to perform three-dimensional reconstruction based on the single-frame three-dimensional lane line instance spatial information and the high-precision vehicle motion posture transformation trajectory to obtain a multi-frame long-range three-dimensional lane line instance in a binocular coordinate system;

[0090] A lane line detection module, used to determine a multi-frame long-range three-dimensional lane line instance in a binocular coordinate system according to a single-frame two-dimensional lane line instance of the binocular camera and the three-dimensional lane line spatial information;

[0091] The camera extrinsic calibration module is used to construct the calibration mapping error of the multi-frame long-range three-dimensional lane line instance and the two-dimensional lane line instance of the camera to be calibrated, and optimize the calibration mapping error to obtain the extrinsic rotation matrix of each camera to the vehicle coordinate system.

[0092] In this embodiment, 3D lane lines are used to calibrate multiple cameras, and a mapping error function is constructed between the corresponding 2D lane line instances and the 3D lane line instances to optimize the extrinsic parameter rotation matrix. 3D lane lines are used to optimize the extrinsic parameters of multiple cameras, which is more lightweight and can use only pure vision configuration. It is more flexible and reduces the impact of time synchronization problems between different types of sensors. It can effectively calibrate the extrinsic parameter rotation matrix of multiple cameras relative to the vehicle coordinate system.

[0093] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limiting, and thus other examples of the exemplary embodiments may have different values.

[0094] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0095] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A vehicle multi-camera extrinsic parameter calibration method based on three-dimensional lane lines, characterized in that: The following steps are involved: Acquire images captured by multiple cameras of a vehicle traveling, and perform single-frame lane line detection on the images to obtain two-dimensional lane line instances, wherein the two-dimensional lane line instances include single-frame two-dimensional lane line instances of a binocular camera and single-frame two-dimensional lane line instances of a camera to be calibrated; Performing image depth analysis on the two-dimensional lane line instance to obtain single-frame three-dimensional lane line instance spatial information; Acquire a high-precision vehicle motion posture transformation trajectory acquired by a binocular camera, and perform three-dimensional reconstruction based on the single-frame three-dimensional lane line instance spatial information and the high-precision vehicle motion posture transformation trajectory to obtain a multi-frame long-range three-dimensional lane line instance in a binocular coordinate system; Construct a calibration mapping error of the multi-frame long-range three-dimensional lane line instance and the two-dimensional lane line instance of the camera to be calibrated, and optimize the calibration mapping error to obtain an extrinsic rotation matrix from each camera to the vehicle coordinate system.

2. The vehicle multi-camera extrinsic parameter calibration method based on three-dimensional lane lines according to claim 1 is characterized in that: Acquire images captured by multiple cameras of a vehicle traveling, and perform single-frame lane line detection on the images to obtain a two-dimensional lane line instance, including: Obtain lane line images captured by each camera on which the vehicle is traveling and the position and posture of each camera; Single-frame lane line detection is performed on the lane line image according to the positions and postures of each camera to obtain a two-dimensional lane line instance.

3. The vehicle multi-camera extrinsic parameter calibration method based on three-dimensional lane lines according to claim 2 is characterized in that: Obtain high-precision vehicle motion pose transformation trajectory collected by the binocular camera, including: Calculate the binocular camera coordinate system corresponding to a single-frame 2D lane line instance based on the position and posture of each camera; A high-precision vehicle motion posture transformation trajectory captured by the binocular camera is determined according to the binocular camera coordinate system and the posture of each camera.

4. The vehicle multi-camera extrinsic parameter calibration method based on three-dimensional lane lines according to claim 3 is characterized in that: Performing image depth analysis on the two-dimensional lane line instance to obtain single-frame three-dimensional lane line instance spatial information includes: Acquire feature points in the single-frame two-dimensional lane line instance, and perform feature matching on two single-frame two-dimensional lane line instances according to the feature points to obtain an instance matching pair; Performing a depth image analysis on the instance matching pairs to obtain a pixel depth corresponding to a two-dimensional lane line instance in a single frame image, wherein the depth image analysis includes using laser projection alignment or binocular depth estimation; The single-frame three-dimensional lane line instance spatial information corresponding to the two-dimensional lane line instance is restored based on the pixel depth.

5. The vehicle multi-camera extrinsic parameter calibration method based on three-dimensional lane lines according to claim 4 is characterized in that: Also includes: The single-frame three-dimensional lane line instance spatial information is spliced ​​according to the high-precision vehicle motion posture transformation trajectory to obtain multi-frame long-range three-dimensional lane line instances.

6. The vehicle multi-camera extrinsic parameter calibration method based on three-dimensional lane lines according to claim 5 is characterized in that: The single-frame three-dimensional lane line instance spatial information is spliced ​​according to the high-precision vehicle motion posture transformation trajectory to obtain a multi-frame long-range three-dimensional lane line instance, including: The coordinates of the jth 3D point in the i-th frame in the vehicle coordinate system of the current frame are preset to (x, y, z) ij , construct the pose transformation matrix [R|T] of the vehicle coordinate system A of the i-th frame relative to the starting reference system B, where R is a 3×3 rotation matrix and T is a 3×1 translation matrix. The corresponding expression is: (x B ,y B ,z B ) T =R A→B (x A ,y A ,z A ) T +T A→B (1) Among them, (x B ,y B ,z B ) represents the three-dimensional coordinates of the starting reference system B, z A represents the three-dimensional coordinates of the vehicle coordinate system A, R A→B represents the rotation transformation from the vehicle coordinate system A to the starting reference system B, T A→B Represents the translation transformation from the vehicle coordinate system A to the starting reference system B; Performing a posture transformation on the three-dimensional point according to the posture transformation matrix to obtain the three-dimensional coordinates of the three-dimensional point in the starting reference system B; A mapping error function between the two-dimensional lane line instance and the three-dimensional lane line instance is constructed according to the three-dimensional coordinates in the starting reference system B, and the mapping error function is optimized to obtain an extrinsic rotation matrix from each camera to the vehicle coordinate system, wherein the three-dimensional lane line instance includes a multi-frame long-range three-dimensional lane line instance.

7. The vehicle multi-camera extrinsic parameter calibration method based on three-dimensional lane lines according to claim 6 is characterized in that: Constructing a mapping error function between the two-dimensional lane line instance and the three-dimensional lane line instance according to the three-dimensional coordinates in the starting reference system B, including: The pixel position coordinate point set corresponding to the preset two-dimensional lane line instance is {(u0,v0),(u1,v1),(u2,v2)...,(u j ,v j ),...}, the corresponding two-dimensional lane line instance fitting equation is: {u=a3*v 3 +a2*v 2 +a1*v+b,(v start ,v end )} (2) Among them, v start and v end The v coordinates represent the starting point and the end point respectively, and a1, a2, a3 and b are constants; The position coordinate point set of the vehicle coordinate system of the preset three-dimensional lane line instance {(x0,y0,z0),(x1,y1,z1),(x2,y2,z2)...,(x j ,y j ,z j ),...}, the fitting equation corresponding to the three-dimensional lane line instance is: {y=c3*x 3 +c2*x 2 +c1*x+c0,z=d3*x 3 +d2*x 2 +d1*x+d0,(x start ,x end )} (3) Among them, c3, c2, c1, c0, d3, d2, d1, d0 are constants, x start and x end are the x coordinates of the starting point and the end point respectively; Then the three-dimensional coordinates of a point W in the vehicle coordinate system (x w ,y w ,z w ) is transformed to the camera coordinate system through the matrix [R|T]: Among them, x c ,y c 、z c Respectively represent the coordinates of the camera coordinate system; Point W in the camera coordinate system is transformed to the pixel coordinate system through the camera projection model. The intrinsic distortion parameters are known, and the corresponding extrinsic rotation matrix expression is: Where, f is the focal length of the camera, f x 、f y Represents the camera focal length in the x and y directions.

8. The vehicle multi-camera extrinsic parameter calibration method based on three-dimensional lane lines according to claim 7 is characterized in that: Also includes: Obtain the perception result of the camera sensing the projection of the two-dimensional lane line instance onto the three-dimensional lane line instance, and construct a first point line distance error function from the mapping point on the fitting surface of the perception result to the fitting curve equation of the corresponding three-dimensional lane line instance; Construct a line distance error function from the pixel point projected from the three-dimensional lane line instance on the camera image to the second point of the fitting curve equation corresponding to the two-dimensional lane line instance; The mapping error function is optimized based on the first point-line distance error function and the second point-line distance error function to obtain an extrinsic rotation matrix from each camera to the vehicle coordinate system.

9. A vehicle multi-camera extrinsic calibration system based on three-dimensional lane lines, characterized in that: The vehicle multi-camera extrinsic parameter calibration method based on three-dimensional lane lines as described in any one of claims 1 to 8 comprises: A lane line instance acquisition module is used to acquire images captured by multiple cameras of a vehicle, and perform single-frame lane line detection on the images to obtain two-dimensional lane line instances, wherein the two-dimensional lane line instances include single-frame two-dimensional lane line instances of a binocular camera and single-frame two-dimensional lane line instances of a camera to be calibrated; A lane line instance detection module, used to perform image depth analysis on the two-dimensional lane line instance to obtain single-frame three-dimensional lane line instance spatial information; A three-dimensional reconstruction module is used to obtain a high-precision vehicle motion posture transformation trajectory acquired by a binocular camera, and to perform three-dimensional reconstruction based on the single-frame three-dimensional lane line instance spatial information and the high-precision vehicle motion posture transformation trajectory to obtain a multi-frame long-range three-dimensional lane line instance in a binocular coordinate system; The camera extrinsic calibration module is used to construct the calibration mapping error of the multi-frame long-range three-dimensional lane line instance and the two-dimensional lane line instance of the camera to be calibrated, and optimize the calibration mapping error to obtain the extrinsic rotation matrix of each camera to the vehicle coordinate system.

Images