Laser radar-camera fusion calibration method and system

Abstract

The invention discloses a laser radar-camera fusion calibration method and system, and belongs to the field of calibration. According to the method, after a target is shot through a laser radar and a camera, multiple pairs of corresponding three-dimensional point clouds and two-dimensional images are obtained; angular point coordinates in the phase unwrapped image are obtained through the phase unwrapped image and the two-dimensional real image, three-dimensional angular point coordinates in the three-dimensional point cloud are further obtained, the two-dimensional angular point coordinates and the three-dimensional angular point coordinates are input into an external parameter calibration program of the laser radar and the camera, a rigid matrix between two sensor coordinate systems is obtained, and calibration is completed. According to the method, a fast inversion method of angular points on a phase unwrapped graph based on a Gaussian elimination method is designed, a calculation strategy of laser radar point cloud-phase unwrapped graph-angular point detection-angular point three-dimensional coordinates is provided, after three-dimensional point cloud and two-dimensional images of a target are collected, data can be imported into a calibration program for automatic calculation, and the accuracy of calibration is improved. The method has the advantages of less human intervention, high automation degree, fast operation and high efficiency.

Description

Technical Field

[0001] The present invention belongs to the joint calibration of cameras and lidar in the field of calibration, and specifically relates to a calibration method and system for lidar-camera fusion. Background Art

[0002] In recent years, multi-sensor fusion detection and collaborative perception technologies based on lidar-camera have been increasingly widely used in the fields of autonomous driving, UAV remote sensing, and aerospace. Lidar can perform three-dimensional imaging of real scenes to obtain information such as three-dimensional point clouds and reflectivity of targets. Cameras can capture images to obtain rich texture details. Therefore, industrially, the three-dimensional point clouds collected by lidar and the two-dimensional textures collected by cameras are often combined to perform highly realistic three-dimensional reconstruction of test scenes and targets, and can improve the accuracy of solving the motion parameters of dynamic targets. The key step in realizing lidar-camera fusion is to align the sensor coordinate systems between the two, that is, the calibration of the spatial coordinate system. In essence, it is to accurately solve the rigid transformation matrix between sensors (including the rotation matrix R and the translation vector T).

[0003] Currently, the mainstream calibration methods are mainly divided into two types: with target and without target. Among them, the non-target method extracts features and corresponding homologous points by using some reference objects with obvious features such as straight lines, planes, cylinders, and spheres in the task scene, and then performs optimization and solution. However, this method is greatly affected by the surrounding environment and is difficult to popularize. The with-target method often uses some unconventional plane pattern designs, triangular pyramids, polyhedrons, etc. for calibration, and the generality of its hardware design and feature recognition is very limited. In the existing plane target (such as black and white checkerboard pattern) calibration methods, the extraction of the point cloud of the target still stays at the level of three-dimensional clustering and voxelization, which takes a long time to process; or manually screen the corner coordinates of the target after opening it with three-dimensional processing software, which is time-consuming, has a low degree of automation, and varies from person to person. Summary of the Invention

[0004] An object of the present invention is to overcome the above-mentioned disadvantages of the prior art and provide a calibration method and system for lidar-camera fusion to solve the problems of low efficiency, low degree of automation, and limited generality of the existing spatial coordinate system calibration methods.

[0005] To achieve the above object, the present invention adopts the following technical solutions: A calibration method for lidar-camera fusion includes the following steps: The lidar and the camera simultaneously take pictures of the target multiple times to obtain multiple pairs of corresponding three-dimensional point clouds and two-dimensional images; Detect the two-dimensional image by the corner detection method to obtain the two-dimensional corner coordinates of each corner in the two-dimensional image; Unwrap the three-dimensional point cloud in the circumferential phase according to the radar horizontal scanning order to obtain an unwrapped phase diagram; convert the unwrapped phase diagram into a two-dimensional real image, and obtain the two-dimensional real corner coordinates on the two-dimensional real image through the corner detection method; based on the two-dimensional real corner coordinates, obtain the corner coordinates in the unwrapped phase diagram through a mapping matrix, and the mapping matrix is obtained through the Gaussian elimination method based on the edge points of the two-dimensional real image and the unwrapped phase diagram; based on the corner coordinates in the unwrapped phase diagram, obtain the index value of the corner, and based on the index value, obtain the three-dimensional corner coordinates in the three-dimensional point cloud; Based on the two-dimensional corner coordinates and the three-dimensional corner coordinates, through the external parameter calibration method, obtain the rigid matrix between the coordinate system of the lidar and the coordinate system of the camera to complete the calibration.

[0006] A further improvement of the present invention lies in: Preferably, the target is in the form of a black and white checkerboard pattern, and the number of shootings is 20 to 30 times.

[0007] Preferably, after obtaining the two-dimensional coordinates of each corner, calibrate the internal parameters of the camera, and the internal parameters of the camera include the imaging parameters and distortion parameters of the camera.

[0008] Preferably, the specific process of the calibration is: obtain two-dimensional images in multiple poses, detect the checkerboard corners on each two-dimensional image frame by frame, use the nonlinear estimation and adjustment function to calculate and optimize the initial values of the internal parameters, obtain the imaging parameters and distortion parameters, and complete the calibration of the internal parameters of the camera.

[0009] Preferably, the method of converting the unwrapped phase diagram into a two-dimensional real image is: save the unwrapped phase diagram as a two-dimensional real image in a specified file format according to the form of the current frame.

[0010] Preferably, the specific process of obtaining the mapping matrix through the Gaussian elimination method based on the edge points of the two-dimensional real image and the unwrapped phase diagram is: (1) Obtain the four edge points of the two-dimensional real image and the four edge points of the unwrapped phase diagram, and form point pairs through the corresponding edge points; (2) Construct a Gaussian elimination matrix based on the point pairs; (3) Obtain the correspondence between the unwrapped phase diagram and the two-dimensional real image based on the Gaussian elimination matrix; (4) Obtain the corner coordinate calculation formula of the unwrapped phase diagram based on the correspondence.

[0011] Preferably, the process of obtaining the three-dimensional corner coordinates in the three-dimensional point cloud based on the index value is: based on the corner coordinates in the unwrapped phase diagram, perform a neighborhood search to obtain the index value of the corner; according to the index value, obtain the three-dimensional corner coordinates of the corresponding corner in the three-dimensional point cloud.

[0012] Preferably, based on the two-dimensional corner coordinates and three-dimensional corner coordinates, a rigid matrix between the coordinate system of the lidar and the coordinate system of the camera is obtained through an external parameter calibration method. The specific process of calibration is as follows: Input the two-dimensional corner coordinates and three-dimensional corner coordinates into the R|T solution model, convert the R|T solution model into a loss function, and after optimizing the loss function through a non-linear method, obtain the final rigid transformation matrix.

[0013] Preferably, the loss function is: (8) In the formula: E represents the reprojection error corresponding to each corner point, represents the two-dimensional corner coordinates on the two-dimensional image collected by the camera, represents the three-dimensional corner point projected onto the two-dimensional corner coordinates on the camera image according to the camera parameter matrix A and the rigid transformation matrix R|T. The initial value of R is the identity matrix, and the initial value of T is the zero matrix.

[0014] A calibration system for lidar-camera fusion includes: An acquisition unit, which is used for the lidar and the camera to simultaneously take pictures of the target multiple times to obtain multiple pairs of corresponding three-dimensional point clouds and two-dimensional images; A two-dimensional unit, which is used to detect the two-dimensional image by the corner detection method to obtain the two-dimensional corner coordinates of each corner point in the two-dimensional image; A three-dimensional unit, which is used to expand the three-dimensional point cloud in the circumferential phase according to the radar horizontal scanning order to obtain a phase-expanded diagram; transform the phase-expanded diagram into a two-dimensional real image, and obtain two-dimensional real corner coordinates on the two-dimensional real image by the corner detection method; based on the two-dimensional real corner coordinates, obtain the corner coordinates in the phase-expanded diagram through a mapping matrix, and the mapping matrix is obtained through the Gaussian elimination method based on the edge points of the two-dimensional real image and the phase-expanded diagram; based on the corner coordinates in the phase-expanded diagram, obtain the index value of the corner point, and based on the index value, obtain the three-dimensional corner coordinates in the three-dimensional point cloud; A calibration unit, which is used to obtain a rigid matrix between the coordinate system of the lidar and the coordinate system of the camera through an external parameter calibration method based on the two-dimensional corner coordinates and three-dimensional corner coordinates to complete the calibration.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention discloses a calibration method for lidar-camera fusion. First, the positions of the lidar and the camera are fixed, and a planar target is placed in their common field of view. Point cloud data and image data of the target points are collected at different spatial positions and postures. Then, the phase unwrapping method is used to reduce the dimension of the data collected by the lidar, and a point cloud phase unwrapping map containing the target and point reflectivity is obtained. Next, the existing open-source corner detection algorithm is used to detect corners in the point cloud phase unwrapping map and the camera image, and the respective corner detection results are obtained. The Gaussian elimination method is used to inversely search for the three-dimensional points of the detected corners on the point cloud phase unwrapping map, thereby obtaining the three-dimensional corner coordinates. Finally, the R|T matrix between the lidar and the camera can be solved using the three-dimensional corner coordinates and two-dimensional corner coordinates of the target corners to complete the calibration. The present invention first introduces the phase unwrapping method in the corner calibration process. By using the phase unwrapping method, the lidar three-dimensional point cloud is processed into a two-dimensional phase unwrapping map, and then the two-dimensional phase unwrapping map is transformed into a two-dimensional real image, enabling the use of common open-source corner detection algorithms to obtain detectable corners on the two-dimensional real image, thus facilitating the rapid extraction of corners on the two-dimensional image and effectively avoiding the disadvantages of three-dimensional point cloud clustering. The present invention designs a fast inversion method for corners on the phase unwrapping map based on the Gaussian elimination method, and proposes a calculation strategy for lidar point cloud-phase unwrapping map-corner detection-corner three-dimensional coordinates, providing a new solution idea for the automatic calibration problem of lidar in complex scenarios. The above process enables the present invention to achieve automatic and rapid calibration of the lidar-camera group using a conventional black-and-white checkerboard target. The hardware is simple, the algorithm is easy to implement. After collecting the three-dimensional point cloud and two-dimensional image of the target, the data can be imported into the calibration program for automatic calculation, with less human intervention, high automation, fast operation, and high efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram of the calibration scenario of the present invention; Figure 2 is a flowchart of the present invention; Figure 3 is an internal parameter calibration diagram of the present invention; Among them, figure (a) is the corner detection result of the first frame image; figure (b) is the horizontal internal parameter calibration diagram; Figure 4 is a diagram of lidar target point cloud, phase unwrapping map and corner detection result; Among them, figure (a) is the target point cloud; figure (b) is the point cloud phase unwrapping map; figure (c) is the real two-dimensional image and corner detection diagram; Figure 5 is a schematic diagram for solving the mapping relationship between the phase unwrapping map and its real two-dimensional image; Among them, (a) is the edge point of the real two-dimensional image; (b) is the edge map of the point cloud phase expansion; (c) is the reverse search map of the corner points on the phase expansion map; Figure 6 The texture of the camera image is projected onto 3D points to reconstruct the 3D scene. DETAILED DESCRIPTION

[0017] In the following, the terms "first", "second", "third", and "fourth" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first", "second", "third", and "fourth" may explicitly or implicitly include one or more of the features.

[0018] The co-shooting method provided in the embodiment of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0019] It should be noted that the terms "first", "second", etc. in the specification and drawings of the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] The first aspect of the present invention discloses a laser radar-camera fusion calibration method, such as Figure 1 The figure shows a schematic diagram of the calibration scene. A flat checkerboard target is placed in front of the lidar-camera system. The lidar and camera simultaneously capture the target data in a certain pose and take multiple point cloud-image pairs. After the 3D corner points and 2D corner points are extracted, the corner point extraction results are input into the least squares iterative algorithm to minimize the average reprojection error of the 3D point projection onto the 2D image. After the algorithm converges, a more accurate R|T matrix can be obtained.

[0021] See Figure 2 This is the flow chart of the method of the present invention, including the following steps: S1. The lidar and the camera simultaneously collect target data in multiple different poses.

[0022] At the same pose, a two-dimensional image is collected by the camera, and a three-dimensional point cloud image is obtained by lidar scanning, obtaining a two-dimensional image and a three-dimensional point cloud image; shooting simultaneously at multiple poses to obtain multiple corresponding two-dimensional images and three-dimensional point cloud images.

[0023] In the present invention, the target is designed according to the classic Zhang Zhengyou calibration method, and the pattern is a black and white checkerboard pattern. In a specific embodiment of the present invention, the black and white checkerboard is 9 rows × 11 columns, and the size of each square grid is 96 mm.

[0024] The principle to be followed during data collection is: within the field of view of the lidar and the camera, cover the common viewing area with as many poses as possible, which can effectively ensure the fusion calibration accuracy. Generally, the field of view of the lidar is larger than that of the camera. Therefore, on the premise that the optical axes of the lidar and the camera both point to the target, it should first be ensured that the target is within the imaging field of view of the camera, and then the target should also be within the imaging field of view of the lidar.

[0025] In some specific embodiments of the present invention, during collection, generally place the target within a distance range of 1.5 to 20 meters in front of the lidar-camera system. The reason for not being lower than 2 meters is that there is a near-measurement blind area in the lidar, and it can only image normally after being greater than 1.5 meters. The reason for being less than 20 meters is that the size of the checkerboard on the target is limited. If the distance is too far, the black and white features on the point cloud may not be obvious, affecting the corner extraction process. It should be understood that the above 1.5 to 20 meters is the detection size corresponding to the target of the present invention. If the size of the target changes, the corresponding detection distance decreases.

[0026] In some specific embodiments of the present invention, within the range of 1.5 to 20 meters, change the pose of the target multiple times to collect 20 to 30 sets of point cloud-image data.

[0027] During the collection process, each time of collection, the distances between the lidar and the camera and the target are always equal to ensure that the images collected by the two can be accurately calibrated. As a preferred method, the lidar and the camera are connected by a fixing device, while ensuring that the distances between the two and the target are always equal, the poses of the two corresponding to each image collection are similar.

[0028] S2. Use the target image collected by the camera to calibrate the internal parameters of the camera.

[0029] Multiple two-dimensional images collected by the camera are inspected through a corner detection method to obtain the two-dimensional corner coordinates of all corners on the two-dimensional images.

[0030] In some embodiments of the present invention, a total of 25 target images are collected, and the corner detection function built in the Matlab software is used to detect the corners on the target. The detection effect of the 25 images is good, as Figure 3 shown in (a).

[0031] Based on the collected two-dimensional corner coordinates, the internal parameters of the camera are calibrated. The calibration of the camera internal parameters refers to determining the imaging parameters and distortion parameters of the camera. Among them, the imaging parameters mainly include the focal length f and the principal point deviation ( x 0 ,y 0 ), and the distortion parameters mainly include the radial coefficient, the tangential coefficient, and the image plane distortion coefficient. The radial coefficient includes K 1 , K 2 , and K 3 , the tangential coefficient includes B 1 and B 2 , and the plane distortion coefficient includes E 1 and E 2 . Equation (1) is its expression, where △ x and △ y respectively represent the distortions in two directions. After obtaining the two-dimensional target images in multiple poses, the checkerboard corners on the images are detected frame by frame, and then the initial values of the internal parameters are calculated and optimized using the nonlinear estimation and adjustment function. Finally, accurate imaging and distortion parameters can be obtained to complete the calibration of the infrared camera internal parameters.

[0032] (1) where r represents the projection radius of the image point from the principal point in the x and y directions.

[0033] The obtained radial distortion coefficients B1 and B2 are -0.1357 and -2.2789 respectively, and the tangential coefficients K1, K2, K3 and the image plane distortion coefficients E1, E2 are all 0. As Figure 3 shown in (b), the overall calibration reprojection error is 0.25 pixel, with relatively high accuracy. The obtained camera internal parameter matrix A is: (2) S3. Process the three-dimensional point cloud of the lidar into a phase unwrapping diagram, and detect the target corners on it after reading.

[0034] Unfold the imaging point cloud of the lidar for a scene into a two-dimensional pixel map for data dimensionality reduction. To prevent occlusion in direct projection, the point cloud is unfolded in the circular phase according to the lidar horizontal scanning order. That is, for the three-dimensional point cloud set P(x, y, z, In), the representation of its two-dimensional image points is as follows: (3) In Equation (3), and respectively represent the azimuth angle and elevation angle when point P is scanned, which are given by lidar data; and respectively represent the average horizontal and vertical azimuth resolutions. and respectively represent the row coordinates and column coordinates of each three-dimensional point on the image after phase unfolding, with the unit of pixels. Similarly, after obtaining the two-dimensional pixel points, information such as distance depth and echo intensity In can be used to color the binary map.

[0035] The phase unfolding map is a cluster of data with two-dimensional coordinates and reflection intensities, which can be considered as a 2.5D image and cannot be directly processed. Therefore, it is necessary to output this map as a real two-dimensional image with a specified image format (such as png / bmp, etc.). This real two-dimensional image can be read and written in the computer. The specific conversion method is to save the information displayed in the phase unfolding map in the form of maintaining the current frame through the specified file format (i.e., the specified image format). By converting the phase unfolding map into a real two-dimensional image, it is convenient to subsequently obtain each corner point from the real two-dimensional image using the corner detection method.

[0036] As Figure 4 shows, the lidar target point cloud, phase unfolding map, and corner detection results are presented. The parameter information is , , the horizontal imaging field of view is (-60°, 60°), and the vertical imaging field of view is (-15°, 10°). Figure 4 In (b), the echo intensity information is used for color display, and the composition effect is good, effectively avoiding occlusion in the straight-line view, and the scene composition is relatively smooth and complete. By using the built-in corner detection function in Matlab, the corner points in the real two-dimensional image of the phase unfolding map can be detected. The effect is as Figure 4 (c) shows. All the corner points can be detected. Each yellow dot on the figure is the detected corner point result. Denote this corner point cluster as .

[0037] S4. According to the target corner points obtained from the phase unfolding map, reverse-search their three-dimensional coordinates.

[0038] In order to accurately obtain the three-dimensional coordinates of the corner points on the target, it is necessary to reverse-map the corner points on the phase-unwrapped real two-dimensional image to the original three-dimensional point cloud. Since there are certain pixel ratio changes in the real two-dimensional image, it is first necessary to determine the one-to-one mapping relationship between the phase-unwrapped image and its real two-dimensional image. The specific process is as follows: obtain the two-dimensional real corner point coordinates of each corner point from the two-dimensional real image through the corner point detection method, and transform the two-dimensional real corner point coordinates into the corner point coordinates in the phase-unwrapped image through the mapping matrix.

[0039] In the present invention, the Gaussian elimination method is adopted to solve the mapping matrix with the help of four pairs of data and the edge points of the image. As Figure 5 (a) The four edge points determined for the real two-dimensional image are shown above. , , , , Figure 5 (b) The four edge points determined for the phase-unwrapped image are shown above. , , , . According to the above two sets of point pairs, construct the Gaussian elimination matrix: , (4) Where M is..., N is..., , , and are the abscissas of the four edge points of the real two-dimensional image respectively, , , and are the ordinates of the four edge points of the real two-dimensional image respectively, , , and are the abscissas of the Figure 4 edge points of the phase-unwrapped image respectively, , , and are the ordinates of the Figure 4 edge points of the phase-unwrapped image respectively.

[0040] Then the correspondence between the phase-unwrapped image and its real two-dimensional image is solved as: (5) Then the corner point coordinates on the phase-unwrapped image can be calculated as: (6) As mentioned in the above process, the phase unwrapping diagram is a cluster of data with two-dimensional coordinates and reflection intensity, and each point has a corresponding index value. Therefore, The data index value of the real point can be found near each corner point in the corner point cluster through neighborhood search . Since there is a one-to-one correspondence between the three-dimensional point cloud of the lidar and the data of the phase unwrapping diagram, the original three-dimensional point cloud can be inversely indexed according to the index value . In the three-dimensional point cloud set, the three-dimensional corner point corresponding to the index value is found , and the three-dimensional corner point coordinates of the three-dimensional corner point are obtained, as shown in Figure 5 (c). All corner points obtain their original three-dimensional values according to this method , where represents the th target image collected. In the embodiment of the present invention ; represents the th corner point in the th target image. In the embodiment of the present invention .

[0041] In the embodiment of the present invention, the calculated . As shown in Table 1 are the coordinate values of the three-dimensional corner point extraction (the first 10 points, actually 60) of the first frame of lidar data.

[0042] Table 1 Three-dimensional corner point extraction (the first 10 points) of the first frame of lidar data

[0043] S5. Input the obtained target three-dimensional corner point - two-dimensional corner point into the R|T solution model, and obtain a more accurate result through a non-linear method.

[0044] According to the pinhole imaging model, the rigid transformation matrix between the lidar and the camera can be calibrated. As shown in Equation (7), the rigid transformation matrix includes the rotation matrix R and the translation vector T. (X L , Y L , Z L ) are the three-dimensional corner point coordinates of the lidar, (x, y) are the two-dimensional corner point coordinates of the camera, and s is the scale factor. The key step in the solution is to obtain the point pair relationship of the target corner points, that is, to accurately establish the mapping between the point cloud corner points and the image corner points, and obtain the cluster of corresponding point pairs for each frame, so as to calculate the initial value of R|T according to the overall constraint and further optimize.

[0045] (7) The idea of calibration is to optimize the average reprojection error of all three-dimensional corner points on the target projected onto the two-dimensional image, that is, to convert Equation (7) into a loss function, as follows: (8) In the formula, E represents the reprojection error corresponding to each corner point, that is, the loss function, represents the coordinates of the corner point on the two-dimensional image collected by the camera, represents the three-dimensional corner point is the coordinate of the corner point projected onto the camera image according to the camera parameter matrix A and the rigid transformation matrix R|T, The initial value of is the identity matrix, The initial value of is the zero matrix.

[0046] Substitute the above corresponding two-dimensional corner point coordinates and three-dimensional corner point coordinates into Equation (8), and after iterative solution and optimization through a non-linear algorithm (such as the least squares method), a rigid matrix is obtained.

[0047] In the embodiment of the present invention, a total of 25 frames of target point cloud and image data are collected, and 60 pairs of three-dimensional coordinates and two-dimensional pixel coordinates of corner points are extracted from each frame, as shown in Table 1. Input these 25 groups × 60 pairs of points into Equation (8), and use the Levenberg-Marquardt algorithm for solution and optimization, so as to obtain the rigid matrices R and T between the lidar and the camera as follows: , (9) The average reprojection error of the algorithm convergence is less than 1 pixel, which proves the accuracy of the calibration result. Finally, use the R|T matrix to verify the fusion effect between the two, as Figure 6 shown is the three-dimensional scene reconstruction map of the projection of the texture of the camera image onto three-dimensional points. Especially, the color of the target surface is very accurate, which proves that the present invention has a good fusion effect. And the average detection time of each frame of point cloud and image is 0.16 s, and the running time of the entire calibration algorithm is about 5 min. After collecting the target point cloud and image, the calibration process no longer requires manual intervention, so the calibration efficiency and automation degree are greatly improved.

[0048] The second aspect of the present invention discloses a calibration system for lidar-camera fusion, including: An acquisition unit for the lidar and the camera to simultaneously photograph the target multiple times to obtain multiple pairs of corresponding three-dimensional point clouds and two-dimensional images; A two-dimensional unit for detecting the two-dimensional image by the corner point detection method to obtain the two-dimensional corner point coordinates of each corner point in the two-dimensional image; A three-dimensional unit is used to perform circular phase unwrapping on a three-dimensional point cloud in the radar horizontal scanning order to obtain an unwrapped phase diagram; convert the unwrapped phase diagram into a two-dimensional real image, and obtain two-dimensional real corner coordinates on the two-dimensional real image through a corner detection method; based on the two-dimensional real corner coordinates, obtain the corner coordinates in the unwrapped phase diagram through a mapping matrix, and the mapping matrix is obtained by Gaussian elimination based on the edge points of the two-dimensional real image and the unwrapped phase diagram; based on the corner coordinates in the unwrapped phase diagram, obtain the index values of the real points near the corners, and based on the index values, obtain the three-dimensional corner coordinates in the three-dimensional point cloud. A calibration unit is used to obtain the rigid matrix between the coordinate system of the lidar and the coordinate system of the camera through an external parameter calibration method based on the two-dimensional corner coordinates and the three-dimensional corner coordinates to complete the calibration.

[0049] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A laser radar-camera fusion calibration method, characterized in that: The following steps are involved: The laser radar and camera take pictures of the target multiple times at the same time to obtain multiple corresponding 3D point clouds and 2D images; Detect the two-dimensional image by using the corner point detection method to obtain the two-dimensional corner point coordinates of each corner point in the two-dimensional image; The three-dimensional point cloud is unfolded in the circular phase according to the radar horizontal scanning sequence to obtain a phase unfolding image; The phase unwrapped image is transformed into a two-dimensional real image, and the two-dimensional real corner point coordinates are obtained on the two-dimensional real image by using a corner point detection method; Based on the two-dimensional real corner point coordinates, the corner point coordinates in the phase unfolding image are obtained through a mapping matrix, wherein the mapping matrix is ​​obtained through a Gaussian elimination method based on the edge points of the two-dimensional real image and the phase unfolding image; based on the corner point coordinates in the phase unfolding image, an index value of the corner point is obtained, and based on the index value, the three-dimensional corner point coordinates in the three-dimensional point cloud are obtained; Based on the two-dimensional corner point coordinates and the three-dimensional corner point coordinates, the rigid matrix between the coordinate system of the lidar and the coordinate system of the camera is obtained through the external parameter calibration method to complete the calibration.

2. The laser radar-camera fusion calibration method according to claim 1, characterized in that: The target is a black and white checkerboard pattern, and the number of shots is 20 to 30 times.

3. The laser radar-camera fusion calibration method according to claim 1, characterized in that: After obtaining the two-dimensional coordinates of each corner point, the camera intrinsic parameters are calibrated, and the camera intrinsic parameters include the imaging parameters and distortion parameters of the camera.

4. The laser radar-camera fusion calibration method according to claim 3, characterized in that: The specific process of the calibration is: obtaining two-dimensional images of multiple postures, detecting the checkerboard corner points on each two-dimensional image frame by frame, calculating and optimizing the initial value of the internal parameters using nonlinear estimation and adjustment functions, obtaining imaging parameters and distortion parameters, and completing the camera internal parameter calibration.

5. The laser radar-camera fusion calibration method according to claim 1, characterized in that: The method for converting the phase unwrapped image into a two-dimensional real image is: saving the phase unwrapped image in the form of the current image frame as a two-dimensional real image in a specified file format.

6. The laser radar-camera fusion calibration method according to claim 1, characterized in that: The mapping matrix is ​​based on the edge points of the two-dimensional real image and the phase expansion image, and the specific process of obtaining it through the Gaussian elimination method is: (1) Obtain four edge points of the two-dimensional real image and four edge points of the phase unwrapped image, and form point pairs through corresponding edge points; (2) Constructing Gaussian elimination matrix based on point pairs; (3) Obtain the correspondence between the phase unwrapped image and the two-dimensional real image based on the Gaussian elimination matrix; (4) Based on the corresponding relationship, a calculation formula for the corner point coordinates of the phase unwrapped image is obtained.

7. The laser radar-camera fusion calibration method according to claim 1, characterized in that: The process of obtaining the coordinates of the three-dimensional corner points in the three-dimensional point cloud is: based on the coordinates of the corner points in the phase unwrapped image, a neighborhood search is performed to obtain the index value of the corner points; According to the index value, the three-dimensional corner point coordinates of the corner point corresponding to the index value in the three-dimensional point cloud are obtained.

8. The laser radar-camera fusion calibration method according to claim 1, characterized in that: Based on the two-dimensional corner point coordinates and the three-dimensional corner point coordinates, the rigid matrix between the coordinate system of the laser radar and the coordinate system of the camera is obtained through the external parameter calibration method. The specific process of completing the calibration is: input the two-dimensional corner point coordinates and the three-dimensional corner point coordinates into the R|T solution model, convert the R|T solution model into a loss function, and after optimizing the loss function through a nonlinear method, obtain the final rigid transformation matrix.

9. The laser radar-camera fusion calibration method according to claim 8, characterized in that: The loss function is: （8） Where: E represents the reprojection error corresponding to each corner point, Represents the coordinates of the two-dimensional corner points on the two-dimensional image captured by the camera, Represents a 3D corner point According to the camera parameter matrix A and the rigid transformation matrix R|T projected onto the two-dimensional corner point coordinates on the camera image, the initial value of R is the unit matrix and the initial value of T is the 0 matrix.

10. A laser radar-camera fusion calibration system, characterized in that: include: The acquisition unit is used for the laser radar and camera to simultaneously shoot the target multiple times to obtain multiple corresponding 3D point clouds and 2D images; A two-dimensional unit, used to detect a two-dimensional image by a corner point detection method, and obtain the two-dimensional corner point coordinates of each corner point in the two-dimensional image; A three-dimensional unit is used to unfold the three-dimensional point cloud in a circular phase according to the radar horizontal scanning sequence to obtain a phase unfolding diagram; The phase unwrapped image is transformed into a two-dimensional real image, and the two-dimensional real corner point coordinates are obtained on the two-dimensional real image by using a corner point detection method; Based on the two-dimensional real corner point coordinates, the corner point coordinates in the phase unfolding image are obtained through a mapping matrix, wherein the mapping matrix is ​​obtained through a Gaussian elimination method based on the edge points of the two-dimensional real image and the phase unfolding image; based on the corner point coordinates in the phase unfolding image, the index values ​​of the real points near the corner points are obtained, and the three-dimensional corner point coordinates in the three-dimensional point cloud are obtained based on the index values; The calibration unit is used to obtain the rigid matrix between the coordinate system of the laser radar and the coordinate system of the camera through an external parameter calibration method based on the two-dimensional corner point coordinates and the three-dimensional corner point coordinates to complete the calibration.

Images