Methods, apparatus, equipment, and storage media for extrinsic parameter calibration based on cross-entropy.

Abstract

This invention discloses a method, apparatus, device, and storage medium for extrinsic parameter calibration based on cross-entropy, comprising: estimating the depth of camera images acquired by a camera using a visual depth estimation model to obtain a visual depth image; transforming laser point clouds acquired by a lidar from the lidar coordinate system to the camera coordinate system using initial extrinsic parameters; obtaining all image pixels corresponding to the lidar point clouds based on the lidar point clouds and the visual depth image in the camera coordinate system; obtaining a visual depth distribution histogram, a lidar depth distribution histogram, and a two-dimensional joint depth distribution histogram based on the visual depth value and lidar depth value of each image pixel; calculating the cross-entropy of each image pixel; accumulating the cross-entropy of each image pixel to obtain the cross-entropy sum of the current frame; and calibrating and optimizing the initial extrinsic parameters based on the cross-entropy sum. This invention enables rapid extrinsic parameter calibration between lidar and camera, is efficient, and does not depend on the external environment.

Description

Technical Field

[0001] This invention belongs to the field of camera and radar calibration technology, specifically relating to an external parameter calibration method, apparatus, device, and storage medium based on cross-entropy. Background Technology

[0002] With the rapid development of the field of autonomous driving, the challenges faced by vehicles in complex and ever-changing road environments are becoming increasingly significant. The driving environment on the road is highly dynamic, requiring vehicles to perceive the driving environment and detect obstacles such as roadblocks, vehicles, and other obstacles.

[0003] Localization and perception based on multi-sensor fusion have become an extremely important part of autonomous driving. Among these sensors, two of the most important types are LiDAR and cameras. If we need to fuse the information from these two sensors, that is, to unify the observations obtained by these two sensors into a single coordinate system, then we need to use their relative pose relationships.

[0004] However, calibrating the relative extrinsic parameters between LiDAR and the camera is a complex problem. This is because the two sensors observe different data: LiDAR is a ranging sensor, while the camera observes two-dimensional images. Existing calibration methods mainly include: calibration board-based methods, motion-based methods, feature matching-based methods, and deep learning-based methods. However, existing calibration methods still suffer from poor accuracy. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a method, apparatus, device, and storage medium for extrinsic parameter calibration based on cross-entropy.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] In a first aspect, this invention discloses an extrinsic parameter calibration method based on cross-entropy, comprising:

[0008] Step S1: Perform depth estimation on the camera images captured by the camera using a visual depth estimation model to obtain a visual depth image;

[0009] Step S2: Transform the laser point cloud acquired by the lidar from the laser coordinate system to the camera coordinate system using initial extrinsic parameters;

[0010] Step S3: Based on the laser point cloud and visual depth image in the camera coordinate system, obtain all image pixels corresponding to the laser point cloud. Each image pixel has a one-to-one corresponding visual depth value and laser depth value.

[0011] Step S4: Obtain a visual depth distribution histogram based on the visual depth value of each image pixel;

[0012] Step S5: Obtain a laser depth distribution histogram based on the laser depth value of each image pixel;

[0013] Step S6: Obtain a two-dimensional joint depth distribution histogram by combining the visual depth value and laser depth value of each image pixel;

[0014] Step S7: Calculate the cross-entropy of each image pixel based on the visual depth distribution histogram, the laser depth distribution histogram, and the two-dimensional joint depth distribution histogram;

[0015] Step S8: Accumulate the cross-entropy of each image pixel to obtain the sum of cross-entropy for this frame;

[0016] Step S9: Based on the cross-entropy sum, calibrate and optimize the initial extrinsic parameters.

[0017] Based on the above technical solution, the following improvements can be made:

[0018] As a preferred approach, the visual depth distribution histogram and the laser depth distribution histogram are obtained through the following steps:

[0019] Step A: Based on the visual depth value or laser depth value of each image pixel, find the corresponding maximum and minimum visual depth values, or the corresponding maximum and minimum laser depth values;

[0020] Step B: Based on the maximum and minimum visual depth values, or the maximum and minimum laser depth values, divide the corresponding depth value range into N depth groups on an average basis;

[0021] Step C: Traverse the visual depth value or laser depth value of each image pixel and classify it into the corresponding depth group;

[0022] Step D: Count the number of visual depth values ​​or laser depth values ​​assigned to each depth group to obtain a visual depth distribution histogram or a laser depth distribution histogram.

[0023] As a preferred embodiment, step S6 includes:

[0024] Step S6.1: Establish a two-dimensional depth distribution coordinate system, and define the two-dimensional point (x, y) for each image pixel.

[0025] Where: x is the visual depth value of an image pixel;

[0026] y represents the laser depth value of a pixel in the image;

[0027] Step S6.2: Based on the visual depth value of each two-dimensional point (x,y), obtain the group number idx of the visual depth group to which it belongs, according to the same labeling principle as the depth group number of the visual depth distribution histogram.

[0028] Step S6.3: Based on the laser depth value of each two-dimensional point (x,y), obtain the group number idy of the laser depth group to which it belongs, according to the same labeling principle as the depth group number of the laser depth distribution histogram.

[0029] Step S6.4: Count the number of image pixels assigned to each joint group number (idx, idy) to obtain a two-dimensional joint depth distribution histogram.

[0030] As a preferred embodiment, step S7 includes:

[0031] Step S7.1: Based on the visual depth value of the two-dimensional point (x,y), obtain the count value cnt_x of the depth group corresponding to group number idx on the visual depth distribution histogram, and obtain px by dividing cnt_x by the total number of laser points;

[0032] Step S7.2: Based on the laser depth value of the two-dimensional point (x,y), obtain the count value cnt_y of the depth group corresponding to group number idy on the laser depth distribution histogram, and obtain py by dividing cnt_y by the total number of laser points;

[0033] Step S7.3: Based on the visual depth value and laser depth value of the two-dimensional point (x,y), obtain the count value cnt_xy of the depth group with joint group number (idx,idy) on the two-dimensional joint depth distribution histogram, and obtain pxy by dividing cnt_xy by the total number of laser points;

[0034] Step S7.4: Obtain the cross-entropy val of the two-dimensional point (x,y) using the following formula;

[0035] val = pxy * log(pxy / (px * py));

[0036] Step S7.5: Repeat steps S7.1-S7.4 until the cross-entropy of each two-dimensional point is obtained.

[0037] Secondly, the present invention discloses an extrinsic parameter calibration device based on cross-entropy, comprising:

[0038] The depth estimation module is used to estimate the depth of camera images captured by the camera using a visual depth estimation model to obtain a visual depth image.

[0039] The point cloud conversion module is used to convert the laser point cloud acquired by the lidar from the laser coordinate system to the camera coordinate system using initial extrinsic parameters.

[0040] The pixel acquisition module is used to acquire all image pixels corresponding to the laser point cloud based on the laser point cloud and visual depth image in the camera coordinate system. Each image pixel has a one-to-one corresponding visual depth value and laser depth value.

[0041] The visual depth distribution acquisition module is used to obtain a visual depth distribution histogram based on the visual depth value of each image pixel.

[0042] The laser depth distribution acquisition module is used to obtain a laser depth distribution histogram based on the laser depth value of each image pixel.

[0043] The two-dimensional joint depth distribution acquisition module is used to obtain a two-dimensional joint depth distribution histogram by using the visual depth value and laser depth value of each image pixel.

[0044] The cross-entropy calculation module is used to calculate the cross-entropy of each image pixel based on the visual depth distribution histogram, the laser depth distribution histogram, and the two-dimensional joint depth distribution histogram.

[0045] The cross-entropy accumulation module is used to accumulate the cross-entropy of each image pixel to obtain the sum of the cross-entropy of the current frame;

[0046] The extrinsic parameter optimization module is used to calibrate and optimize the initial extrinsic parameters based on the cross-entropy sum.

[0047] As a preferred embodiment, both the visual depth distribution acquisition module and the laser depth distribution acquisition module include:

[0048] The depth extremum determination unit is used to find the corresponding maximum visual depth value, minimum visual depth value, or maximum laser depth value and minimum laser depth value based on the visual depth value or laser depth value of each image pixel.

[0049] The depth group determination unit is used to divide the corresponding depth value range into N depth groups on an average basis, based on the maximum and minimum visual depth values, or the maximum and minimum laser depth values.

[0050] The partitioning unit is used to traverse the visual depth value or laser depth value of each image pixel and divide it into the corresponding depth group;

[0051] The statistical unit is used to count the number of visual depth values ​​or laser depth values ​​assigned to each depth group, thereby obtaining a visual depth distribution histogram or a laser depth distribution histogram.

[0052] As a preferred embodiment, the two-dimensional joint depth distribution acquisition module includes:

[0053] The two-dimensional coordinate system establishment unit is used to establish a two-dimensional depth distribution coordinate system. Let each image pixel be a two-dimensional point (x, y), where x is the visual depth value of the image pixel.

[0054] y represents the laser depth value of a pixel in the image;

[0055] The first group number acquisition unit is used to obtain the group number idx of the visual depth group to which each two-dimensional point (x,y) belongs, based on the visual depth value and in accordance with the same labeling principle as the depth group number of the visual depth distribution histogram.

[0056] The second group number acquisition unit is used to obtain the group number idy of the laser depth group to which each two-dimensional point (x,y) belongs, based on the laser depth value and according to the same labeling principle as the group number of the depth group in the laser depth distribution histogram.

[0057] The joint group number acquisition unit is used to count the number of image pixels assigned to each joint group number (idx, idy) and obtain a two-dimensional joint depth distribution histogram.

[0058] As a preferred solution, the cross-entropy calculation module includes:

[0059] The first calculation unit is used to obtain the count value cnt_x of the depth group corresponding to group number idx on the visual depth distribution histogram based on the visual depth value of the two-dimensional point (x,y), and obtain px by dividing cnt_x by the total number of laser points;

[0060] The second calculation unit is used to obtain the count value cnt_y of the depth group corresponding to group number idy on the laser depth distribution histogram based on the laser depth value of the two-dimensional point (x,y), and obtain py by dividing cnt_y by the total number of laser points;

[0061] The third calculation unit is used to obtain the count value cnt_xy of the depth group with joint group number (idx,idy) on the two-dimensional joint depth distribution histogram based on the visual depth value and laser depth value of the two-dimensional point (x,y), and obtain pxy by dividing cnt_xy by the total number of laser points.

[0062] The cross-entropy calculation unit is used to obtain the cross-entropy val of a two-dimensional point (x,y) using the following formula;

[0063] val = pxy * log(pxy / (px * py));

[0064] The repeated execution unit is used to sequentially execute the methods in the first calculation unit, the second calculation unit, the third calculation unit, and the cross-entropy calculation unit until the cross-entropy of each two-dimensional point is obtained.

[0065] Thirdly, the present invention discloses a computing device, comprising:

[0066] One or more processors;

[0067] Memory;

[0068] And one or more programs, wherein the one or more programs are stored in memory and configured to be executed by one or more processors, and the one or more programs include instructions for any of the aforementioned cross-entropy-based extrinsic calibration methods.

[0069] Fourthly, the present invention discloses a storage medium storing one or more computer-readable programs, the one or more programs including instructions adapted to be loaded by a memory and executed by any of the above-described extrinsic parameter calibration methods based on cross-entropy.

[0070] This invention discloses a method, apparatus, device, and storage medium for extrinsic parameter calibration based on cross-entropy, which has the following beneficial effects:

[0071] First, the camera uses scaled depth estimation to align with the depth information of the LiDAR, and calculates cross-entropy based on the depth information as the calibration residual, thereby achieving rapid calibration of extrinsic parameters between the LiDAR and the camera, which is efficient and independent of the external environment.

[0072] Second, the larger the cross-entropy sum, the more accurate the initial extrinsic parameters are, and the more effectively the initial extrinsic parameters can be evaluated. Attached Figure Description

[0073] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 This is a flowchart of an external parameter calibration method provided in an embodiment of the present invention.

[0075] Figure 2 Example diagrams showing calibration results provided in embodiments of the present invention;

[0076] (a), (b), (c), and (d) are four example diagrams.

[0077] Figure 3 A block diagram of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0078] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0079] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0080] Using ordinal numbers such as “first,” “second,” “third,” etc. to describe ordinary objects merely indicates different instances of similar objects and is not intended to imply that the objects being described must have a given order in time, space, sequence, or any other way.

[0081] Furthermore, the expression "includes" is an "open-ended" expression, which means only that there is a corresponding component or step, and should not be interpreted as excluding additional components or steps.

[0082] To achieve the objectives of this invention, some embodiments of the extrinsic parameter calibration method based on cross-entropy, such as Figure 1 As shown, the external parameter calibration methods include:

[0083] Step S101: Perform depth estimation on the camera images captured by the camera using a visual depth estimation model to obtain a visual depth image;

[0084] Step S102: Transform the laser point cloud acquired by the lidar from the laser coordinate system to the camera coordinate system using initial extrinsic parameters;

[0085] Step S103: Based on the laser point cloud and visual depth image in the camera coordinate system, obtain all image pixels corresponding to the laser point cloud. Each image pixel has a one-to-one corresponding visual depth value and laser depth value.

[0086] Step S104: Obtain a visual depth distribution histogram based on the visual depth value of each image pixel;

[0087] Step S105: Obtain a laser depth distribution histogram based on the laser depth value of each image pixel;

[0088] Step S106: Obtain a two-dimensional joint depth distribution histogram by combining the visual depth value and laser depth value of each image pixel;

[0089] Step S107: Calculate the cross-entropy of each image pixel based on the visual depth distribution histogram, the laser depth distribution histogram, and the two-dimensional joint depth distribution histogram;

[0090] Step S108: Accumulate the cross-entropy of each image pixel to obtain the sum of cross-entropy for this frame;

[0091] Step S109: Based on the cross-entropy sum, calibrate and optimize the initial extrinsic parameters.

[0092] The various steps of this invention will now be described in detail.

[0093] Step S101: The camera image (e.g., monocular image) acquired by the camera is used to perform depth estimation through a visual depth estimation model to obtain a visual depth image (e.g., monocular depth map).

[0094] Step S102 transforms the real-time laser point cloud acquired by the lidar from the laser coordinate system to the camera coordinate system using initial extrinsic parameters, thus obtaining the real-time laser point cloud in the camera coordinate system.

[0095] The initial external parameters can be obtained using several methods in the existing technology, which is common knowledge in the field and will not be elaborated here.

[0096] Step S103 can be achieved by using the following method to obtain the image pixels corresponding to the laser point cloud.

[0097] The laser point cloud in the camera coordinate system is traversed and projected onto the visual depth image. The image pixel corresponding to each laser point cloud is obtained through linear interpolation. The visual depth value of each image pixel has been obtained through the visual depth estimation model in step S101, so each image pixel has a one-to-one corresponding visual depth value and laser depth value.

[0098] It is worth noting that in some embodiments, when processing the acquired laser point cloud, only the laser point cloud within the range of 0.2 meters to 40 meters is statistically analyzed. This is because the depth estimated by vision is too far and unreliable. The visually estimated depth value needs to be compared with the depth value of the laser point cloud, so the depth value of the laser point cloud must be set within a reasonable range of the visually estimated depth. The visually estimated depth value is also processed within a range. In this invention, the visually estimated depth value is set to be less than 300 because the model used is scaled, and the estimated depth has a scale but no real physical unit of length, so this threshold has no unit.

[0099] Furthermore, the visual depth distribution histogram and the laser depth distribution histogram are obtained through the following steps:

[0100] Step A: Based on the visual depth value or laser depth value of each image pixel, find the corresponding maximum and minimum visual depth values, or the corresponding maximum and minimum laser depth values;

[0101] Step B: Based on the maximum and minimum visual depth values, or the maximum and minimum laser depth values, divide the corresponding depth value range into N depth groups on an average basis;

[0102] Step C: Traverse the visual depth value or laser depth value of each image pixel and classify it into the corresponding depth group;

[0103] Step D: Count the number of visual depth values ​​or laser depth values ​​assigned to each depth group to obtain a visual depth distribution histogram or a laser depth distribution histogram.

[0104] By using a visual depth distribution histogram or a depth distribution histogram, we can determine the group number of the depth group corresponding to the visual depth value or laser depth value of each image pixel.

[0105] Specifically, step S104 includes:

[0106] Step S104.1: Based on the visual depth value of each image pixel, find the maximum visual depth value max_val1 and the minimum visual depth value min_val1;

[0107] Step S104.2: Based on the maximum visual depth max_val1 and the minimum visual depth min_val1, divide the visual depth range into N (e.g., 100) visual depth groups on an average basis;

[0108] The length of each visual depth group is bin_size1;

[0109] bin_size1=(max_val1-min_val1) / N;

[0110] For example, the range of the first visual depth group is [min_val1, min_val1 + bin_size1);

[0111] Step S104.3: Traverse the visual depth value of each image pixel and assign it to the corresponding visual depth group;

[0112] Step S104.4: Count the number of visual depth values ​​assigned to each visual depth group to obtain a visual depth distribution histogram.

[0113] Specifically, step S105 includes:

[0114] Step S105.1: Calculate the laser depth value for each laser point cloud, and find the maximum laser depth value max_val2 and the minimum laser depth value min_val2;

[0115] Step S105.2: Based on the maximum laser depth max_val2 and the minimum laser depth min_val2, divide the laser depth range into N (e.g., 100) laser depth groups on an average basis;

[0116] The length of each visual depth group is bin_size2;

[0117] bin_size2=(max_val2-min_val2) / N;

[0118] For example, the range of the first laser depth group is [min_val2, min_val2+bin_size2);

[0119] Step S105.3: Traverse the laser depth value of each laser point cloud and assign it to the corresponding laser depth group;

[0120] Step S105.4: Count the number of laser depth values ​​assigned to each laser depth group to obtain a laser depth distribution histogram.

[0121] Furthermore, step S106 includes:

[0122] Step S106.1: Establish a two-dimensional depth distribution coordinate system, and define the two-dimensional point (x, y) for each image pixel.

[0123] Where: x is the visual depth value of an image pixel;

[0124] y represents the laser depth value of a pixel in the image;

[0125] Step S106.2: Based on the visual depth value of each two-dimensional point (x,y), obtain the group number idx of the visual depth group to which it belongs, according to the same labeling principle as the depth group number of the visual depth distribution histogram.

[0126] Step S106.3: Based on the laser depth value of each two-dimensional point (x,y), obtain the group number idy of the laser depth group to which it belongs, according to the same labeling principle as the depth group number of the laser depth distribution histogram.

[0127] Step S106.4: Count the number of image pixels assigned to each joint group number (idx, idy) to obtain a two-dimensional joint depth distribution histogram.

[0128] Steps S106.2 and S106.3 are processed using a method similar to that used in steps S104 and S105.

[0129] Suppose that the two-dimensional point (x,y) has a visual depth value vis_depth and a laser depth value lidar_depth.

[0130] The minimum visual depth is min_val1, the maximum visual depth is max_val1, and there are a total of N visual depth groups.

[0131] The minimum laser depth is min_val2, the maximum laser depth is max_val2, and there are a total of N laser depth groups.

[0132] The group number idx of the visual depth group and the group number idy of the laser depth group can be obtained according to the following formula;

[0133] idx=std::floor((vis_depth-min_val1) / (max_val1-min_val1)*100);

[0134] idy=std::floor((lidar_depth-min_val2) / (max_val2-min_val2)*100);

[0135] Where: std::floor means rounding down to the nearest integer.

[0136] It is worth noting that the group numbers for the depth groups in the visual depth distribution histogram, the laser depth distribution histogram, and the two-dimensional joint depth distribution histogram are consistent.

[0137] Based on the visual depth distribution histogram, the depth distribution histogram, and the two-dimensional joint depth distribution histogram, we can determine the group number of the depth group corresponding to the visual depth value or laser depth value of each image pixel.

[0138] Furthermore, step S107 includes:

[0139] Step S107.1: Based on the visual depth value of the two-dimensional point (x,y), obtain the count value cnt_x of the depth group corresponding to group number idx on the visual depth distribution histogram, and obtain px by dividing cnt_x by the total number of laser points;

[0140] Step S107.2: Based on the laser depth value of the two-dimensional point (x,y), obtain the count value cnt_y of the depth group corresponding to group number idy on the laser depth distribution histogram, and obtain py by dividing cnt_y by the total number of laser points;

[0141] Step S107.3: Based on the visual depth value and laser depth value of the two-dimensional point (x,y), obtain the count value cnt_xy of the depth group with joint group number (idx,idy) on the two-dimensional joint depth distribution histogram, and obtain pxy by dividing cnt_xy by the total number of laser points;

[0142] Step S107.4: Obtain the cross-entropy val of the two-dimensional point (x,y) using the following formula;

[0143] val = pxy * log(pxy / (px * py));

[0144] Step S107.5: Repeat steps S107.1-S107.4 until the cross-entropy of each two-dimensional point is obtained.

[0145] After obtaining the cross-entropy of each two-dimensional point in step S107.5, step S108 accumulates them to obtain the cross-entropy sum under the initial extrinsic parameters. (Because this cross-entropy sum is obtained under the current initial extrinsic parameters, there is a one-to-one correspondence between the initial extrinsic parameters and the cross-entropy sum, that is, one extrinsic parameter corresponds to one cross-entropy sum).

[0146] Theoretically, the closer the extrinsic parameters are to the true value, the larger the sum of cross-entropy will be. Step S108: Accumulate the cross-entropy of each image pixel to obtain the sum of cross-entropy for this frame;

[0147] Further, step S109: Based on the cross-entropy sum, the initial extrinsic parameters are calibrated and optimized. Specifically, the optimization can be achieved by differentiating the initial extrinsic parameters using the cross-entropy sum and then performing least squares, thus optimizing the extrinsic parameters to obtain the optimal parameters. This part is common knowledge in the field and will not be elaborated further here. Experimental verification of this invention yields the following calibration results: Figure 2 As shown in (a), (b), (c), and (d), these images demonstrate the projection of laser point clouds onto a visual depth image.

[0148] By using cross-entropy calibration to optimize the external references and compare them with the true external references, the difference in angular accuracy is within 0.05 degrees and the difference in translation accuracy is within 1 cm.

[0149] This invention also discloses an embodiment of an extrinsic parameter calibration device based on cross-entropy, the extrinsic parameter calibration device comprising:

[0150] The depth estimation module is used to estimate the depth of camera images captured by the camera using a visual depth estimation model to obtain a visual depth image.

[0151] The point cloud conversion module is used to convert the laser point cloud acquired by the lidar from the laser coordinate system to the camera coordinate system using initial extrinsic parameters.

[0152] The pixel acquisition module is used to acquire all image pixels corresponding to the laser point cloud based on the laser point cloud and visual depth image in the camera coordinate system. Each image pixel has a one-to-one corresponding visual depth value and laser depth value.

[0153] The visual depth distribution acquisition module is used to obtain a visual depth distribution histogram based on the visual depth value of each image pixel.

[0154] The laser depth distribution acquisition module is used to obtain a laser depth distribution histogram based on the laser depth value of each image pixel.

[0155] The two-dimensional joint depth distribution acquisition module is used to obtain a two-dimensional joint depth distribution histogram by using the visual depth value and laser depth value of each image pixel.

[0156] The cross-entropy calculation module is used to calculate the cross-entropy of each image pixel based on the visual depth distribution histogram, the laser depth distribution histogram, and the two-dimensional joint depth distribution histogram.

[0157] The cross-entropy accumulation module is used to accumulate the cross-entropy of each image pixel to obtain the sum of the cross-entropy of the current frame;

[0158] The extrinsic parameter optimization module is used to calibrate and optimize the initial extrinsic parameters based on the cross-entropy sum.

[0159] Furthermore, both the visual depth distribution acquisition module and the laser depth distribution acquisition module include:

[0160] The depth extremum determination unit is used to find the corresponding maximum visual depth value, minimum visual depth value, or maximum laser depth value and minimum laser depth value based on the visual depth value or laser depth value of each image pixel.

[0161] The depth group determination unit is used to divide the corresponding depth value range into N depth groups on an average basis, based on the maximum and minimum visual depth values, or the maximum and minimum laser depth values.

[0162] The partitioning unit is used to traverse the visual depth value or laser depth value of each image pixel and divide it into the corresponding depth group;

[0163] The statistical unit is used to count the number of visual depth values ​​or laser depth values ​​assigned to each depth group, thereby obtaining a visual depth distribution histogram or a laser depth distribution histogram.

[0164] Furthermore, the two-dimensional joint depth distribution acquisition module includes:

[0165] The two-dimensional coordinate system establishment unit is used to establish a two-dimensional depth distribution coordinate system. Let each image pixel be a two-dimensional point (x, y).

[0166] Where: x is the visual depth value of an image pixel;

[0167] y represents the laser depth value of a pixel in the image;

[0168] The first group number acquisition unit is used to obtain the group number idx of the visual depth group to which each two-dimensional point (x,y) belongs, based on the visual depth value and in accordance with the same labeling principle as the depth group number of the visual depth distribution histogram.

[0169] The second group number acquisition unit is used to obtain the group number idy of the laser depth group to which each two-dimensional point (x,y) belongs, based on the laser depth value and according to the same labeling principle as the group number of the depth group in the laser depth distribution histogram.

[0170] The joint group number acquisition unit is used to count the number of image pixels assigned to each joint group number (idx, idy) and obtain a two-dimensional joint depth distribution histogram.

[0171] Furthermore, the cross-entropy calculation module includes:

[0172] The first calculation unit is used to obtain the count value cnt_x of the depth group corresponding to group number idx on the visual depth distribution histogram based on the visual depth value of the two-dimensional point (x,y), and obtain px by dividing cnt_x by the total number of laser points;

[0173] The second calculation unit is used to obtain the count value cnt_y of the depth group corresponding to group number idy on the laser depth distribution histogram based on the laser depth value of the two-dimensional point (x,y), and obtain py by dividing cnt_y by the total number of laser points;

[0174] The third calculation unit is used to obtain the count value cnt_xy of the depth group with joint group number (idx,idy) on the two-dimensional joint depth distribution histogram based on the visual depth value and laser depth value of the two-dimensional point (x,y), and obtain pxy by dividing cnt_xy by the total number of laser points.

[0175] The cross-entropy calculation unit is used to obtain the cross-entropy val of a two-dimensional point (x,y) using the following formula;

[0176] val = pxy * log(pxy / (px * py));

[0177] The repeated execution unit is used to sequentially execute the methods in the first calculation unit, the second calculation unit, the third calculation unit, and the cross-entropy calculation unit until the cross-entropy of each two-dimensional point is obtained.

[0178] Furthermore, it should be noted that the extrinsic parameter calibration device based on cross-entropy provided in the above embodiments is only illustrated by the division of the above functional modules when determining the dominant vertex. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the extrinsic parameter calibration device based on cross-entropy can be divided into different functional modules to complete all or part of the functions described above.

[0179] Furthermore, the extrinsic parameter calibration device based on cross-entropy provided in the above embodiments and the extrinsic parameter calibration method based on cross-entropy belong to the same concept. For details of their specific implementation process, please refer to the method embodiments, which will not be repeated here.

[0180] Furthermore, in some other embodiments, such as Figure 2 As shown, the present invention also discloses a computing device, comprising:

[0181] One or more processors 201;

[0182] Memory 202;

[0183] And one or more programs, wherein the one or more programs are stored in memory 202 and configured to be executed by one or more processors 201, the one or more programs including instructions for the cross-entropy-based extrinsic calibration method disclosed in the above embodiments.

[0184] Processor 201 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 201 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 201 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 201 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 201 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0185] The memory 202 may include one or more computer-readable storage media, which may be non-transitory. The memory 202 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 202 are used to store at least one instruction, which is executed by the processor 201 to implement the cross-entropy-based extrinsic parameter calibration method provided in the method embodiments of the present invention.

[0186] In addition, the computing device may optionally include: a peripheral device interface and at least one peripheral device. The processor 201, memory 202, and peripheral device interface can be connected via a bus or signal lines. Each peripheral device can be connected to the peripheral device interface via a bus, signal lines, or a circuit board. Illustratively, peripheral devices include, but are not limited to: radio frequency circuitry, a touchscreen display, audio circuitry, and a power supply.

[0187] Of course, the computing device may also include fewer or more components, and this embodiment does not limit this.

[0188] Furthermore, in some other embodiments, the present invention also discloses a storage medium storing one or more computer-readable programs, the one or more programs including instructions adapted to be loaded by memory and execute the cross-entropy-based extrinsic calibration method disclosed in the above embodiments.

[0189] This invention discloses a method, apparatus, device, and storage medium for extrinsic parameter calibration based on cross-entropy, which has the following beneficial effects:

[0190] First, the camera uses scaled depth estimation to align with the depth information of the LiDAR, and calculates cross-entropy based on the depth information as the calibration residual, thereby achieving rapid calibration of extrinsic parameters between the LiDAR and the camera, which is efficient and independent of the external environment.

[0191] Second, the larger the cross-entropy sum, the more accurate the initial extrinsic parameters are, and the more effectively the initial extrinsic parameters can be evaluated.

[0192] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the present invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope. All such changes and modifications fall within the scope of the present invention as claimed, which is defined by the appended claims and their equivalents.

Claims

1. An extrinsic parameter calibration method based on cross-entropy, characterized in that, include: Step S1: Perform depth estimation on the camera images captured by the camera using a visual depth estimation model to obtain a visual depth image; Step S2: Transform the laser point cloud acquired by the lidar from the laser coordinate system to the camera coordinate system using initial extrinsic parameters; Step S3: Based on the laser point cloud and visual depth image in the camera coordinate system, obtain all image pixels corresponding to the laser point cloud. Each image pixel has a one-to-one corresponding visual depth value and laser depth value. Step S4: Obtain a visual depth distribution histogram based on the visual depth value of each image pixel; Step S5: Obtain a laser depth distribution histogram based on the laser depth value of each image pixel; Step S6: Obtain a two-dimensional joint depth distribution histogram by combining the visual depth value and laser depth value of each image pixel; Step S7: Calculate the cross-entropy of each image pixel based on the visual depth distribution histogram, the laser depth distribution histogram, and the two-dimensional joint depth distribution histogram; Step S8: Accumulate the cross-entropy of each image pixel to obtain the sum of cross-entropy for this frame; Step S9: Based on the cross-entropy sum, calibrate and optimize the initial extrinsic parameters.

2. The external parameter calibration method according to claim 1, characterized in that, The visual depth distribution histogram and the laser depth distribution histogram are obtained through the following steps: Step A: Based on the visual depth value or laser depth value of each image pixel, find the corresponding maximum and minimum visual depth values, or the corresponding maximum and minimum laser depth values; Step B: Based on the maximum and minimum visual depth values, or the maximum and minimum laser depth values, divide the corresponding depth value range into N depth groups on an average basis; Step C: Traverse the visual depth value or laser depth value of each image pixel and classify it into the corresponding depth group; Step D: Count the number of visual depth values ​​or laser depth values ​​assigned to each depth group to obtain a visual depth distribution histogram or a laser depth distribution histogram.

3. The external parameter calibration method according to claim 2, characterized in that, Step S6 includes: Step S6.1: Establish a two-dimensional depth distribution coordinate system, and define the two-dimensional point (x, y) for each image pixel. Where: x is the visual depth value of an image pixel; y represents the laser depth value of a pixel in the image; Step S6.2: Based on the visual depth value of each two-dimensional point (x,y), obtain the group number idx of the visual depth group to which it belongs, according to the same labeling principle as the depth group number of the visual depth distribution histogram. Step S6.3: Based on the laser depth value of each two-dimensional point (x,y), obtain the group number idy of the laser depth group to which it belongs, according to the same labeling principle as the depth group number of the laser depth distribution histogram. Step S6.4: Count the number of image pixels assigned to each joint group number (idx, idy) to obtain a two-dimensional joint depth distribution histogram.

4. The external parameter calibration method according to claim 3, characterized in that, Step S7 includes: Step S7.1: Based on the visual depth value of the two-dimensional point (x,y), obtain the count value cnt_x of the depth group corresponding to group number idx on the visual depth distribution histogram, and obtain px by dividing cnt_x by the total number of laser points; Step S7.2: Based on the laser depth value of the two-dimensional point (x,y), obtain the count value cnt_y of the depth group corresponding to group number idy on the laser depth distribution histogram, and obtain py by dividing cnt_y by the total number of laser points; Step S7.3: Based on the visual depth value and laser depth value of the two-dimensional point (x,y), obtain the count value cnt_xy of the depth group with joint group number (idx,idy) on the two-dimensional joint depth distribution histogram, and obtain pxy by dividing cnt_xy by the total number of laser points; Step S7.4: Obtain the cross-entropy val of the two-dimensional point (x,y) using the following formula; val = pxy * log(pxy / (px * py)); Step S7.5: Repeat steps S7.1-S7.4 until the cross-entropy of each two-dimensional point is obtained.

5. An extrinsic parameter calibration device based on cross-entropy, characterized in that, include: The depth estimation module is used to estimate the depth of camera images captured by the camera using a visual depth estimation model to obtain a visual depth image. The point cloud conversion module is used to convert the laser point cloud acquired by the lidar from the laser coordinate system to the camera coordinate system using initial extrinsic parameters. The pixel acquisition module is used to acquire all image pixels corresponding to the laser point cloud based on the laser point cloud and visual depth image in the camera coordinate system. Each image pixel has a one-to-one corresponding visual depth value and laser depth value. The visual depth distribution acquisition module is used to obtain a visual depth distribution histogram based on the visual depth value of each image pixel. The laser depth distribution acquisition module is used to obtain a laser depth distribution histogram based on the laser depth value of each image pixel. The two-dimensional joint depth distribution acquisition module is used to obtain a two-dimensional joint depth distribution histogram by using the visual depth value and laser depth value of each image pixel. The cross-entropy calculation module is used to calculate the cross-entropy of each image pixel based on the visual depth distribution histogram, the laser depth distribution histogram, and the two-dimensional joint depth distribution histogram. The cross-entropy accumulation module is used to accumulate the cross-entropy of each image pixel to obtain the sum of the cross-entropy of the current frame; The extrinsic parameter optimization module is used to calibrate and optimize the initial extrinsic parameters based on the cross-entropy sum.

6. The external parameter calibration device according to claim 5, characterized in that, Both the visual depth distribution acquisition module and the laser depth distribution acquisition module include: The depth extremum determination unit is used to find the corresponding maximum visual depth value, minimum visual depth value, or maximum laser depth value and minimum laser depth value based on the visual depth value or laser depth value of each image pixel. The depth group determination unit is used to divide the corresponding depth value range into N depth groups on an average basis, based on the maximum and minimum visual depth values, or the maximum and minimum laser depth values. The partitioning unit is used to traverse the visual depth value or laser depth value of each image pixel and divide it into the corresponding depth group; The statistical unit is used to count the number of visual depth values ​​or laser depth values ​​assigned to each depth group, thereby obtaining a visual depth distribution histogram or a laser depth distribution histogram.

7. The external parameter calibration device according to claim 6, characterized in that, The two-dimensional joint depth distribution acquisition module includes: The two-dimensional coordinate system establishment unit is used to establish a two-dimensional depth distribution coordinate system. Let each image pixel be a two-dimensional point (x, y). Where: x is the visual depth value of an image pixel; y represents the laser depth value of a pixel in the image; The first group number acquisition unit is used to obtain the group number idx of the visual depth group to which each two-dimensional point (x,y) belongs, based on the visual depth value and in accordance with the same labeling principle as the depth group number of the visual depth distribution histogram. The second group number acquisition unit is used to obtain the group number idy of the laser depth group to which each two-dimensional point (x,y) belongs, based on the laser depth value and according to the same labeling principle as the group number of the depth group in the laser depth distribution histogram. The joint group number acquisition unit is used to count the number of image pixels assigned to each joint group number (idx, idy) and obtain a two-dimensional joint depth distribution histogram.

8. The external parameter calibration device according to claim 7, characterized in that, The cross-entropy calculation module includes: The first calculation unit is used to obtain the count value cnt_x of the depth group corresponding to group number idx on the visual depth distribution histogram based on the visual depth value of the two-dimensional point (x,y), and obtain px by dividing cnt_x by the total number of laser points; The second calculation unit is used to obtain the count value cnt_y of the depth group corresponding to group number idy on the laser depth distribution histogram based on the laser depth value of the two-dimensional point (x,y), and obtain py by dividing cnt_y by the total number of laser points; The third calculation unit is used to obtain the count value cnt_xy of the depth group with joint group number (idx,idy) on the two-dimensional joint depth distribution histogram based on the visual depth value and laser depth value of the two-dimensional point (x,y), and obtain pxy by dividing cnt_xy by the total number of laser points. The cross-entropy calculation unit is used to obtain the cross-entropy val of a two-dimensional point (x,y) using the following formula; val = pxy * log(pxy / (px * py)); The repeated execution unit is used to sequentially and repeatedly execute the methods in the first calculation unit, the second calculation unit, the third calculation unit, and the cross-entropy calculation unit until the cross-entropy of each two-dimensional point is obtained.

9. A computing device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions for the cross-entropy-based extrinsic calibration method as described in any of claims 1-4.

10. A storage medium, characterized in that, The storage medium stores one or more computer-readable programs, the programs including instructions adapted to be loaded by memory and executed as described in any of claims 1-4 above, based on the extrinsic parameter calibration method.

Images