A camera and lidar automatic calibration method based on instance segmentation

Abstract

The application discloses a camera and laser radar automatic calibration method based on instance segmentation, which does not require a specific scene, and only needs sensor data collected when a vehicle drives on a road surface to complete automatic calibration, aims to simplify camera and laser radar external parameter calibration work in a self-driving vehicle, and when a relative position relationship of the camera and the laser radar is shaken or offset, online correction can be performed, so that the reliability of a self-driving fusion perception system is improved.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving, specifically relating to an automatic calibration method for cameras and lidar based on instance segmentation. Background Technology

[0002] With the rapid development of computer vision and AI technologies, autonomous driving is becoming a hot topic in the automotive industry. Autonomous driving will be the future trend of the automotive industry; the commercial development of autonomous vehicles will not only effectively solve traffic congestion and improve travel efficiency, but also reduce the incidence of traffic accidents.

[0003] In the perception component of autonomous vehicles, cameras and LiDAR are commonly used sensors. Cameras can acquire information such as color and texture, and have high resolution, but cannot obtain depth information of the target; LiDAR can acquire 3D information, but has low resolution and lacks color information. To achieve higher quality perception, multi-sensor fusion technology is often used, and the extrinsic parameter calibration between cameras and LiDAR is a crucial step in achieving accurate fusion.

[0004] Traditional manual calibration methods require the assistance of calibration boards and other equipment, which is not only time-consuming and labor-intensive, but also cannot correct errors in a timely manner when external parameters change due to severe vibrations or other reasons during the movement of autonomous vehicles. This leads to a decrease in sensor fusion effect or even failure.

[0005] Furthermore, object-free calibration methods based on features such as edges and planes in natural scenes have high requirements for acquiring these features and also depend on specific scenes, such as walls and buildings, thus limiting the calibration scenarios. Another motion-based method estimates calibration parameters by first estimating the sensor's own motion using inter-frame information collected by the sensor, and then solving for the extrinsic parameters between the sensors. However, this method works better on platforms such as robots because robotic platforms have greater degrees of freedom of movement, while autonomous vehicles cannot perform sufficient movements in a short time, thus the accuracy of this method is not high enough. Summary of the Invention

[0006] To address the technical problems existing in the background art, this invention aims to provide an automatic calibration method for cameras and LiDAR based on instance segmentation. By extracting and associating instances after instance segmentation, a consistent cost function is constructed, and the optimal extrinsic parameters are obtained by optimizing the cost function. This method does not require a specific scenario; it only needs sensor data collected while the vehicle is driving on the road to complete the automated calibration, thus ensuring the safety of autonomous vehicles.

[0007] To solve the technical problem, the technical solution of the present invention is as follows:

[0008] An automatic camera and LiDAR calibration method based on instance segmentation, the method comprising:

[0009] Preprocessing of images and point clouds yields several important instances;

[0010] Calculate the centroid coordinates of each important instance, and use a matching algorithm to obtain the correspondence between important instances in the image and point cloud, at the cost of the distance between centroids.

[0011] Construct a matching value map, and use the matching values ​​of important instances in the point cloud, combined with the distance between the corresponding centroids in the correspondence, to construct a consistency cost function;

[0012] By optimizing the consistency cost function, the extrinsic parameters of the camera and lidar are obtained, thus realizing the automatic calibration of the camera and lidar.

[0013] Furthermore, the preprocessing of the image and point cloud information specifically includes:

[0014] The acquired image and point cloud information are used to perform instance segmentation operation using a preset neural network model to obtain the first instance segmentation result. The first instance segmentation result is then filtered to obtain the second instance segmentation result. The second instance segmentation result is then filtered a second time to obtain multiple important instances, namely important image instances and important point cloud instances.

[0015] Furthermore, images and point cloud information are collected using cameras and lidar mounted on the autonomous vehicle, respectively. Based on the preset Panoptic-DeepLab model and DS-Net model, instance segmentation is performed on the collected images and point cloud information to obtain the first instance segmentation result.

[0016] The first instance segmentation result is filtered to select vehicle instances, which are the second instance segmentation results.

[0017] Based on the number of pixels in the second instance segmentation result, retain the k image vehicle instances and k point cloud vehicle instances with the most pixels, where k is adaptively adjusted according to the number of vehicles in the second instance segmentation result. The k image vehicle instances are important image instances, and the k point cloud vehicle instances are important point cloud instances.

[0018] Furthermore, the calculation of the centroid coordinates of each important instance specifically includes:

[0019] Calculate the coordinates of the point cloud projected onto the camera coordinate system based on the coordinates of the point cloud in the coordinate system of the lidar.

[0020] Based on the coordinates of the point cloud projected onto the camera coordinate system, the coordinates of the point cloud projected onto the image pixel coordinate system are calculated.

[0021] Based on the coordinates of the point cloud projected onto the image pixel coordinate system, calculate the average value of the coordinates of each important instance in the image pixel coordinate system, and obtain the centroid coordinates of each important instance projected onto the image pixel from the point cloud.

[0022] Based on the second instance segmentation result of the image, the important instances in the image are obtained, and the centroid coordinates of each important instance in the image are calculated.

[0023] Furthermore, the Hungarian matching algorithm is used to obtain the correspondence between important instances in the image and point cloud, specifically including:

[0024] Calculate the centroid coordinates of each significant instance projected onto the image pixels from the point cloud and the centroid distance value of the centroid coordinates of each significant instance in the image;

[0025] Using the distance values ​​of all centroids as costs, a cost matrix for the Hungarian matching algorithm is constructed. The Hungarian matching algorithm is then used to perform optimal matching, thereby obtaining the correspondence between important instances in the image and point cloud.

[0026] Furthermore, the construction of the matching value map specifically includes:

[0027] The image in the second instance segmentation result above is binarized, and the pixel values ​​of important instances in the image are set to 1, and the other pixels are set to 0, to obtain an initial matching value map composed of the matching values ​​of each pixel in the image.

[0028] Each important instance in the image is processed using a distance transformation algorithm to obtain the final matching value map.

[0029] Furthermore, the construction of the consistency cost function specifically includes:

[0030] After projecting the important instances of the point cloud onto the image pixel coordinate system, the matching value of each pixel is obtained according to the final matching value map. Then, the consistency cost function is constructed by combining the correspondence between the important instances in the image and the point cloud.

[0031] Furthermore, the optimization of the consistency cost function specifically includes:

[0032] By using the Rodriguez transform, the rotation matrix of the initial calibration extrinsic parameters is transformed into Euler angles, i.e., a 6-dimensional vector. Using this 6-dimensional vector as the parameter to be optimized, the consistency cost function is optimized using the differential evolution algorithm to obtain the extrinsic parameters of the camera and the lidar, thus realizing the automatic calibration of the camera and lidar.

[0033] Furthermore, the extrinsic parameters of the camera and lidar include: the optimal rotation vector and translation vector.

[0034] Furthermore, the initial calibration extrinsic parameter specifically refers to a preset initial coarse extrinsic parameter value.

[0035] Compared with the prior art, the advantages of the present invention are as follows:

[0036] It does not require specific scenarios, saving labor and time costs. It can complete automated calibration using only sensor data collected when the vehicle is driving on the road.

[0037] It has higher accuracy: the average rotation error of the final solved extrinsic parameters is within 0.5° and the average translation error is within 5cm. This is mainly due to the fact that instance segmentation can obtain information about each instance. By screening key instances, only those instances with larger targets and better segmentation results are retained to construct the cost function for optimization.

[0038] It exhibits higher robustness: multiple experiments in different scenarios revealed that the solution error was relatively small; the main reason is that the cost function to be optimized uses the matching value + centroid distance as a constraint, which has strong anti-interference ability.

[0039] This method can be applied to the calibration of other sensors, such as the extrinsic parameter calibration between cameras and between LiDAR sensors. Attached Figure Description

[0040] Figure 1 The main flowchart of an automatic camera and lidar calibration method based on instance segmentation according to the present invention;

[0041] Figure 2 Important image examples of an automatic camera and lidar calibration method based on instance segmentation according to the present invention;

[0042] Figure 3 The present invention provides a laser point cloud and camera image fusion diagram based on an automatic calibration method for cameras and lidar based on instance segmentation. Detailed Implementation

[0043] The specific implementation of the present invention is described below with reference to embodiments:

[0044] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0045] Furthermore, the terms such as "upper," "lower," "left," "right," "middle," and "one" used in this specification are merely for clarity of description and are not intended to limit the scope of the invention. Any changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention.

[0046] Example 1

[0047] An automatic camera and LiDAR calibration method based on instance segmentation, the method comprising:

[0048] Preprocessing of images and point clouds yields several important instances;

[0049] Calculate the centroid coordinates of each important instance, and use a matching algorithm to obtain the correspondence between important instances in the image and point cloud, at the cost of the distance between centroids.

[0050] Construct a matching value map, and use the matching values ​​of important instances in the point cloud, combined with the distance between the corresponding centroids in the correspondence, to construct a consistency cost function;

[0051] By optimizing the consistency cost function, the extrinsic parameters of the camera and lidar are obtained, thus realizing the automatic calibration of the camera and lidar.

[0052] Furthermore, the preprocessing of the image and point cloud information specifically includes:

[0053] The acquired image and point cloud information are used to perform instance segmentation operation using a preset neural network model to obtain the first instance segmentation result. The first instance segmentation result is then filtered to obtain the second instance segmentation result. The second instance segmentation result is then filtered a second time to obtain multiple important instances, namely important image instances and important point cloud instances.

[0054] Furthermore, images and point cloud information are collected using cameras and lidar mounted on the autonomous vehicle, respectively. Based on the preset Panoptic-DeepLab model and DS-Net model, instance segmentation is performed on the collected images and point cloud information to obtain the first instance segmentation result.

[0055] The first instance segmentation result is filtered to select vehicle instances, which are the second instance segmentation results.

[0056] Based on the number of pixels in the second instance segmentation result, the k image vehicle instances and k point cloud vehicle instances with the largest number of pixels are retained, where k is adaptively adjusted according to the number of vehicles in the second instance segmentation result. The k image vehicle instances are important image instances, and the k point cloud vehicle instances are important point cloud instances. Figure 2 As shown.

[0057] Furthermore, the calculation of the centroid coordinates of each important instance specifically includes:

[0058] Calculate the coordinates of the point cloud projected onto the camera coordinate system based on the coordinates of the point cloud in the coordinate system of the lidar.

[0059] Based on the coordinates of the point cloud projected onto the camera coordinate system, the coordinates of the point cloud projected onto the image pixel coordinate system are calculated.

[0060] Based on the coordinates of the point cloud projected onto the image pixel coordinate system, calculate the average value of the coordinates of each important instance in the image pixel coordinate system, and obtain the centroid coordinates of each important instance projected onto the image pixel from the point cloud.

[0061] Based on the second instance segmentation result of the image, the important instances in the image are obtained, and the centroid coordinates of each important instance in the image are calculated.

[0062] Furthermore, the Hungarian matching algorithm is used to obtain the correspondence between important instances in the image and point cloud, specifically including:

[0063] Calculate the centroid coordinates of each significant instance projected onto the image pixels from the point cloud and the centroid distance value of the centroid coordinates of each significant instance in the image;

[0064] Using the distance values ​​of all centroids as costs, a cost matrix for the Hungarian matching algorithm is constructed. The Hungarian matching algorithm is then used to perform optimal matching, thereby obtaining the correspondence between important instances in the image and point cloud.

[0065] Furthermore, the construction of the matching value map specifically includes:

[0066] The image in the second instance segmentation result above is binarized, and the pixel values ​​of important instances in the image are set to 1, and the other pixels are set to 0, to obtain an initial matching value map composed of the matching values ​​of each pixel in the image.

[0067] Each important instance in the image is processed using a distance transformation algorithm to obtain the final matching value map.

[0068] Furthermore, the construction of the consistency cost function specifically includes:

[0069] After projecting the important instances of the point cloud onto the image pixel coordinate system, the matching value of each pixel is obtained according to the final matching value map. Then, the consistency cost function is constructed by combining the correspondence between the important instances in the image and the point cloud.

[0070] Furthermore, the optimization of the consistency cost function specifically includes:

[0071] By using the Rodriguez transform, the rotation matrix of the initial calibration extrinsic parameters is transformed into Euler angles, i.e., a 6-dimensional vector. Using this 6-dimensional vector as the parameter to be optimized, the consistency cost function is optimized using the differential evolution algorithm to obtain the extrinsic parameters of the camera and the lidar, thus realizing the automatic calibration of the camera and lidar.

[0072] Furthermore, the extrinsic parameters of the camera and lidar include: the optimal rotation vector and translation vector.

[0073] Furthermore, the initial calibration extrinsic parameter specifically refers to a preset initial coarse extrinsic parameter value.

[0074] Example 2:

[0075] This invention provides an automatic calibration method for cameras and LiDAR based on instance segmentation. This method does not require a specific scene and only needs sensor data collected when the vehicle is driving on the road to complete the automatic calibration. The purpose is to simplify the calibration of extrinsic parameters of cameras and LiDAR in autonomous vehicles, and to perform online correction when the relative position relationship between the two changes or shifts, thereby improving the reliability of autonomous driving fusion perception system.

[0076] Instance segmentation was performed on the image and point cloud based on Panoptic-DeepLab and DS-Net respectively. In the segmentation results, vehicle instances were first selected, and then only the k instances with the most pixels in the image were retained. k was adaptively adjusted according to the number of vehicles in the image.

[0077] Based on the centroids of important instances in the projected point cloud and the centroids of important instances in the image, the Hungarian matching algorithm is used to match important instances in the point cloud and the image, with the distance between the centroids as the cost, to obtain the correspondence between point cloud and image instances.

[0078] Based on the segmented image, the image is binarized, and the pixel values ​​of important instances are set to 1, while others are set to 0. Then, a distance transform is applied to each important instance to obtain the final matching value of each pixel, thus forming a matching value map.

[0079] A consistency cost function is constructed based on the matching values ​​and centroid distances of key point cloud instances projected onto the image. The rotation matrix of the initial calibration extrinsic parameters is transformed into Euler angles and combined with the translation vector to form a 6-dimensional vector. Then, this 6-dimensional vector is used as the parameter to be optimized, and the consistency cost function is optimized using the differential evolution algorithm.

[0080] Example 3:

[0081] like Figure 1As shown, in step 1: the camera and lidar mounted on the autonomous vehicle acquire image and point cloud information respectively. Based on the Panoptic-DeepLab and DS-Net networks, instance segmentation is performed on the image and point cloud respectively. In the segmentation results, vehicle instances are first selected, and then only the k instances with the most pixels in the image are retained (k is adaptively adjusted according to the number of vehicles in the image). These k instances are important instances, including important image instances and important point cloud instances.

[0082] Step 2: Based on the preset initial coarse extrinsic parameter values, project the point cloud from the lidar coordinate system to the image pixel coordinate system, calculate the average value of the pixel position of each important instance in the image and the pixel position after the point cloud projection, and obtain the centroid coordinates of each important instance.

[0083] Step 3: Based on the centroids of the important instances in Step 2 after projection into the image and point cloud, the Hungarian matching algorithm is used to match the important instances in the point cloud and image, with the distance between the centroids as the cost.

[0084] Step 4: Binarize the image, set the pixel values ​​of important instances to 1 and the others to 0, and then apply distance transform (DT) to each important instance to obtain the final matching value of each pixel, thus forming a matching value map, i.e., the final matching value map.

[0085] Step 5: After projecting the important instances of the point cloud onto the image pixel coordinate system, obtain the matching value of each pixel based on the final matching value map, and then construct a consistency cost function by combining the correspondence between the important instances in the image and the point cloud; transform the rotation matrix of the initial calibration extrinsic parameters into Euler angles, i.e., a 6-dimensional vector, and use this 6-dimensional vector as the parameter to be optimized, and use the differential evolution algorithm to optimize the consistency cost function.

[0086] Furthermore, the specific process of image and point cloud processing in step 1 is as follows:

[0087] 1) Build the Panoptic-DeepLab and DS-Net networks using the PyTorch framework. Train the Panoptic-DeepLab dataset using the Cityscapes dataset and train the DS-Net dataset using the SemanticKITTI dataset to obtain the trained models.

[0088] 2) Obtain the original image and laser point cloud data, and input them into the trained Panoptic-DeepLab and DS-Net networks respectively to obtain the instance segmentation results.

[0089] 3) Each pixel and point cloud point is assigned a corresponding instance segmentation label. Based on the label values ​​of vehicle instances set in the model, only vehicle instances are retained. Among these vehicle instances, the top k image vehicle instances are selected from those with the most pixels to those with the fewest pixels, and the top k point cloud vehicle instances are also selected simultaneously.

[0090] 4) Method for selecting k: Calculate the percentage of pixels of each vehicle instance after image segmentation to the total number of pixels of all vehicle instances, and select the top 70% of vehicle instances from all vehicle pixels. The number of these vehicles is k.

[0091] Furthermore, the specific process for calculating the centroid coordinates of important instances in step 2 is as follows:

[0092] 1) Projecting point cloud onto image pixel coordinate system:

[0093]

[0094]

[0095] Wherein, equation (1) represents the projection of the point cloud from the lidar coordinate system to the camera coordinate system, and represents The coordinates of the i-th point cloud in the lidar coordinate system Let R represent the coordinates projected onto the camera coordinate system, R represent the rotation matrix, and t represent the translation vector; Equation (2) is the projection from the camera coordinate system to the image pixel coordinate system. κ represents the coordinates of the i-th point cloud projected into the image pixel coordinate system, and κ represents the camera's intrinsic parameters.

[0096] 2) The centroid coordinates of important instances after point cloud projection are:

[0097]

[0098] in, Let v represent the set of pixel coordinates projected from the v-th significant point cloud instance. This represents the centroid coordinates of the instance after projection.

[0099] 3) The centroid coordinates of important instances in the image are:

[0100]

[0101] in, The set representing the pixel coordinates of the v-th significant instance of the image. This represents the coordinates of the i-th pixel in the instance. This indicates the centroid coordinates of the instance in the image.

[0102] Furthermore, the specific process of the Hungarian matching algorithm in step 3 is as follows:

[0103] 1) Calculate the distance between the centroid of the significant instance in the image and the centroid of the projected point cloud:

[0104]

[0105] in, It is the centroid distance of the v-th instance.

[0106] 2) Using the centroid distance between all important instances as the cost, construct the cost matrix in the Hungarian algorithm, and use the Hungarian algorithm to solve for the optimal matching to obtain the correspondence between important instances in the image and point cloud.

[0107] Furthermore, the method for calculating the matching value of each pixel in the image in step 4 is as follows:

[0108] 1) Binarize the image, setting the pixel values ​​of important instances to 1 and others to 0, as shown in the following expression:

[0109]

[0110] Where K represents important instances, O represents other background information besides important instances, and m S This represents an initial matching value map composed of the matching values ​​of each pixel in the image.

[0111] 2) Apply distance transformation (DT) to important instances:

[0112]

[0113] Where α is set to 0.9 and γ is set to 0.6.

[0114] 3) Using equations (7) and (8), the final matching value for each pixel is:

[0115]

[0116] Furthermore, the construction and optimization process of the consistency cost function in step 5 is as follows:

[0117] 1) Use the Rodriguez formula to transform the rotation matrix to be optimized into a rotation vector, and then combine it with the translation vector to be optimized to write it in the form of a 6-dimensional vector:

[0118] T = (roll, pitch, yaw, t) x ,t y ,t z (10)

[0119] Where roll, pitch, and yaw represent roll angle, pitch angle, and yaw angle, respectively, and tx ,t y ,t z These represent the translation values ​​in the x, y, and z directions, respectively.

[0120] 2) Combining equations (1)-(10) above, the following cost function is constructed:

[0121]

[0122] Where r is the rotation vector and t is the translation vector.

[0123] 3) The differential evolution algorithm is used to optimize the parameters r and t in equation (11). The Geatpy genetic algorithm toolbox based on Python is called. When using the differential evolution algorithm, the soea_DE_targetToBest_1_bin_templet evolution template is used, the objective function is F(r,t), the upper and lower boundaries of the search for each dimension parameter are set according to the initial external parameter values, the population size is set to 20, and the maximum number of iterations is 30.

[0124]

[0125] in, It is the optimal rotation vector and translation vector obtained through optimization.

[0126] like Figure 3 As shown, the extrinsic parameters obtained according to the above calibration method are the result of fusing the laser point cloud and the camera image.

[0127] 1) Different instance segmentation methods can be used, such as Mask R-CNN for image instance segmentation and EfficientLPS for point cloud instance segmentation.

[0128] 2) In addition to the Hungarian algorithm, the KM algorithm and other algorithms can be used to match key instances.

[0129] In addition to differential evolution algorithm, other optimization algorithms for the cost function of this method include genetic algorithm (GA), ant colony optimization (ACO), and particle swarm optimization (PSO).

[0130] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

[0131] Many other changes and modifications can be made without departing from the concept and scope of this invention. It should be understood that this invention is not limited to the specific embodiments, and the scope of this invention is defined by the appended claims.

Claims

1. A camera and lidar automatic calibration method based on instance segmentation, characterized in that, The method comprises: preprocessing the image and the point cloud to obtain a plurality of important instances; calculating the centroid coordinates of each important instance, using a matching algorithm to obtain the correspondence of the important instances in the image and the point cloud, with the distance between the centroids as the cost; constructing a matching value map, using the matching values of the point cloud important instances, combining the distances between the corresponding centroids in the correspondence, and constructing a consistency cost function; optimizing the consistency cost function to obtain the extrinsic parameters of the camera and the lidar, i.e. realizing automatic calibration of the camera and the lidar; the preprocessing of the image and the point cloud information specifically comprises: performing instance segmentation on the collected image and point cloud information using a preset neural network model to obtain a first instance segmentation result, performing a first screening process on the first instance segmentation result to obtain a second instance segmentation result, and performing a second screening process on the second instance segmentation result to obtain a plurality of important instances, i.e. image important instances and point cloud important instances; the calculation of the centroid coordinates of each important instance specifically comprises: calculating the coordinates of the point cloud projected onto the camera coordinate system according to the coordinates of the point cloud in the coordinate system of the lidar; calculating the coordinates of the point cloud projected onto the image pixel coordinate system according to the coordinates of the point cloud projected onto the camera coordinate system; calculating the average value of the coordinates of each important instance in the image pixel coordinate system according to the coordinates of the point cloud projected onto the image pixel coordinate system to obtain the centroid coordinates of each important instance projected onto the image pixel by the point cloud; obtaining the important instances in the image according to the second instance segmentation result of the image, and calculating the centroid coordinates of each important instance in the image.

2. The camera and lidar automatic calibration method based on instance segmentation according to claim 1, characterized in that, The camera and the lidar mounted on the unmanned vehicle collect image and point cloud information respectively, and the collected image and point cloud information are segmented based on a preset Panoptic-DeepLab model and a DS-Net model to obtain a first instance segmentation result; screening the first instance segmentation result to screen out vehicle instances, which are the second instance segmentation result; retaining the k image vehicle instances with the most pixels and the k point cloud vehicle instances with the most pixels according to the number of pixels in the second instance segmentation result, wherein k is adaptively adjusted according to the number of vehicles in the second instance segmentation result, the k image vehicle instances are image important instances, and the k point cloud vehicle instances are point cloud important instances.

3. The camera and lidar automatic calibration method based on instance segmentation according to claim 1, characterized in that, The correspondence of the important instances in the image and the point cloud is obtained by using a Hungarian matching algorithm, specifically comprising: calculating the centroid distance values of the centroid coordinates of each important instance projected onto the image pixel by the point cloud and the centroid coordinates of each important instance in the image; constructing a cost matrix of the Hungarian matching algorithm using all centroid distance values as the cost value, and using the Hungarian matching algorithm to perform optimal matching to obtain the correspondence of the important instances in the image and the point cloud.

4. The camera and lidar automatic calibration method based on instance segmentation according to claim 1, characterized in that, The construction of the matching value map specifically comprises: performing binary processing on the image in the above-mentioned second instance segmentation result, setting the pixel values of the important instances in the image to 1 and setting other pixels to 0 to obtain an initial matching value map composed of the matching values of each pixel point in the image; The distance transform algorithm is used to process each important instance in the image to obtain a final matching value map.

5. The camera and lidar automatic calibration method based on instance segmentation according to claim 4, characterized in that, The consistency cost function specifically comprises: After projecting the point cloud important instances into the image pixel coordinate system, the matching value of each pixel point is obtained according to the final matching value map, and the consistency cost function is constructed in combination with the corresponding relationship of the important instances in the image and the point cloud.

6. The camera and lidar automatic calibration method based on instance segmentation according to claim 5, characterized in that, The optimization of the consistency cost function specifically comprises: The rotation matrix of the initial calibration external parameter is converted into the form of Euler angles, that is, a 6-dimensional vector, by using the Rodrigues transformation; the 6-dimensional vector is used as the parameter to be optimized; the differential evolution algorithm is used to optimize the consistency cost function to obtain the external parameter of the camera and the laser radar, that is, the automatic calibration of the camera and the laser radar is realized.

7. The camera and lidar automatic calibration method based on instance segmentation according to claim 1, characterized in that, The external parameter of the camera and the laser radar comprises an optimal rotation vector and a translation vector.

8. The camera and lidar automatic calibration method based on instance segmentation according to claim 6, characterized in that, The initial calibration external parameter is specifically a preset initial rough external parameter value.

Images