External parameter calibration method and device based on cross entropy, equipment and storage medium

Abstract

The invention discloses an external parameter calibration method and device based on cross entropy, equipment and a storage medium, and the method comprises the steps: carrying out the depth estimation of a camera image collected by a camera through a visual depth estimation model, and obtaining a visual depth image; the laser point cloud collected by the laser radar is converted from a laser coordinate system to a camera coordinate system by adopting initial external parameters; obtaining all image pixel points corresponding to the laser point cloud based on the laser point cloud and the visual depth image under the camera coordinate system; respectively obtaining a visual depth distribution histogram, a laser depth distribution histogram and a two-dimensional joint depth distribution histogram based on the visual depth value and the laser depth value of each image pixel point; calculating the cross entropy of each image pixel point; accumulating the cross entropy of each image pixel point to obtain the sum of the cross entropy of the frame; and performing calibration optimization on the initial external parameters based on the cross entropy sum. The method can achieve the quick calibration of the external parameters between the laser radar and the camera, is efficient, and does not depend on an external environment scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of camera and radar calibration, and specifically relates to an external parameter calibration method, device, equipment and storage medium based on cross entropy. Background Art

[0002] With the rapid development of the field of autonomous driving, the challenges faced by vehicles in driving in complex and changing road environments have become increasingly significant. The vehicle driving environment on the road is highly dynamic, and the vehicle needs to perceive the driving environment and detect obstacles such as roadblocks, vehicles, and vehicles.

[0003] Positioning and perception based on multi-sensor fusion have become an extremely important part of autonomous driving. Among them, the two more important types of sensors are lidar and camera. If the information of these two sensors needs to be fused together, the fusion means unifying the observations obtained by these two sensors into one coordinate system, and the relative position relationship between them is needed here.

[0004] However, the relative external parameter calibration between LiDAR and camera is a relatively complex problem. Since the two sensors observe differently, LiDAR is a distance measurement sensor, and the camera observes two-dimensional images. Existing calibration methods mainly include: calibration plate-based methods, motion-based methods, feature matching-based methods, deep learning-based methods, etc. However, existing calibration methods still have the problem of poor accuracy. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a method, device, equipment and storage medium for extrinsic parameter calibration based on cross entropy.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] In a first aspect, the present invention discloses a method for calibrating an external parameter based on cross entropy, comprising:

[0008] Step S1: performing depth estimation on a camera image captured by a camera through a visual depth estimation model to obtain a visual depth image;

[0009] Step S2: converting the laser point cloud collected by the laser radar from the laser coordinate system to the camera coordinate system using the initial external parameters;

[0010] Step S3: Based on the laser point cloud and the visual depth image in the camera coordinate system, all image pixel points corresponding to the laser point cloud are obtained, and each image pixel point has a one-to-one corresponding visual depth value and laser depth value;

[0011] Step S4: obtaining a visual depth distribution histogram based on the visual depth value of each image pixel;

[0012] Step S5: obtaining a laser depth distribution histogram based on the laser depth value of each image pixel;

[0013] Step S6: The visual depth value and the laser depth value of each image pixel are used to obtain a two-dimensional joint depth distribution histogram;

[0014] Step S7: Calculate the cross entropy of each image pixel based on the visual depth distribution histogram, the laser depth distribution histogram and the two-dimensional joint depth distribution histogram;

[0015] Step S8: Accumulate the cross entropy of each image pixel to obtain the cross entropy sum of the current frame;

[0016] Step S9: Based on the cross entropy sum, calibrate and optimize the initial external parameters.

[0017] Based on the above technical solution, the following improvements can be made:

[0018] As a preferred solution, the visual depth distribution histogram and the laser depth distribution histogram are obtained respectively by the following steps, including:

[0019] Step A: based on the visual depth value or laser depth value of each image pixel, find the corresponding maximum visual depth value, minimum visual depth value, or the corresponding maximum laser depth value, minimum laser depth value;

[0020] Step B: Based on the maximum visual depth value and the minimum visual depth value, or the maximum laser depth value and the minimum laser depth value, the corresponding depth value range is evenly divided into N depth groups;

[0021] Step C: traverse the visual depth value or laser depth value of each image pixel and divide it into corresponding depth groups;

[0022] Step D: Count the number of visual depth values ​​or laser depth values ​​divided into each depth group to obtain a visual depth distribution histogram or a laser depth distribution histogram.

[0023] As a preferred solution, step S6 includes:

[0024] Step S6.1: Establish a two-dimensional depth distribution coordinate system, and set each image pixel as a two-dimensional point (x, y).

[0025] Where: x is the visual depth value of the image pixel;

[0026] y is the laser depth value of the image pixel;

[0027] Step S6.2: based on the visual depth value of each two-dimensional point (x, y), obtain the group number idx of the visual depth group to which it belongs according to the same labeling principle as the depth group number of the visual depth distribution histogram;

[0028] Step S6.3: Based on the laser depth value of each two-dimensional point (x, y), the group number idy of the laser depth group to which it belongs is obtained according to the same labeling principle as the depth group number of the laser depth distribution histogram;

[0029] Step S6.4: Count the number of image pixels divided into each joint group number (idx, idy) to obtain a two-dimensional joint depth distribution histogram.

[0030] As a preferred solution, step S7 includes:

[0031] Step S7.1: Based on the visual depth value of the two-dimensional point (x, y), obtain the count value cnt_x of the depth group corresponding to the group number idx on the visual depth distribution histogram, and obtain px by dividing cnt_x by the total number of laser points;

[0032] Step S7.2: Based on the laser depth value of the two-dimensional point (x, y), obtain the count value cnt_y of the depth group corresponding to the group number idy on the laser depth distribution histogram, and obtain py by dividing cnt_y by the total number of laser points;

[0033] Step S7.3: Based on the visual depth value and the laser depth value of the two-dimensional point (x, y), obtain the count value cnt_xy of the depth group with the joint group number (idx, idy) on the two-dimensional joint depth distribution histogram, and obtain pxy by dividing cnt_xy by the total number of laser points;

[0034] Step S7.4: Obtain the cross entropy val of the two-dimensional point (x, y) by the following formula;

[0035] val = pxy*log(pxy / (px*py));

[0036] Step S7.5: Repeat steps S7.1 to S7.4 until the cross entropy of each two-dimensional point is obtained.

[0037] In a second aspect, the present invention discloses an external parameter calibration device based on cross entropy, comprising:

[0038] A depth estimation module is used to perform depth estimation on the camera image captured by the camera through a visual depth estimation model to obtain a visual depth image;

[0039] The point cloud conversion module is used to convert the laser point cloud collected by the laser radar from the laser coordinate system to the camera coordinate system using the initial external parameters;

[0040] A pixel acquisition module is used to acquire all image pixels corresponding to the laser point cloud based on the laser point cloud and the visual depth image in the camera coordinate system, and each image pixel has a one-to-one corresponding visual depth value and laser depth value;

[0041] A visual depth distribution acquisition module is used to obtain a visual depth distribution histogram based on the visual depth value of each image pixel;

[0042] A laser depth distribution acquisition module is used to obtain a laser depth distribution histogram based on the laser depth value of each image pixel point;

[0043] A two-dimensional joint depth distribution acquisition module is used for the visual depth value and laser depth value of each image pixel to obtain a two-dimensional joint depth distribution histogram;

[0044] A cross entropy calculation module is used to calculate the cross entropy of each image pixel based on the visual depth distribution histogram, the laser depth distribution histogram and the two-dimensional joint depth distribution histogram;

[0045] The cross entropy accumulation module is used to accumulate the cross entropy of each image pixel to obtain the cross entropy sum of the current frame;

[0046] The external parameter optimization module is used to calibrate and optimize the initial external parameters based on the cross entropy sum.

[0047] As a preferred solution, the visual depth distribution acquisition module and the laser depth distribution acquisition module both include:

[0048] A depth extreme value determination unit, used to find the corresponding maximum visual depth value, minimum visual depth value, or the corresponding maximum laser depth value, minimum laser depth value based on the visual depth value or laser depth value of each image pixel point;

[0049] A depth group determination unit, configured to evenly divide a corresponding depth value range into N depth groups based on a maximum visual depth value and a minimum visual depth value, or a maximum laser depth value and a minimum laser depth value;

[0050] A division unit, used for traversing the visual depth value or laser depth value of each image pixel point and dividing it into a corresponding depth group;

[0051] The statistical unit is used to count the number of visual depth values ​​or laser depth values ​​divided into each depth group to obtain a visual depth distribution histogram or a laser depth distribution histogram.

[0052] As a preferred solution, the two-dimensional joint depth distribution acquisition module includes:

[0053] A two-dimensional coordinate system establishment unit is used to establish a two-dimensional depth distribution coordinate system, assuming that each image pixel is a two-dimensional point (x, y), where: x is the visual depth value of the image pixel;

[0054] y is the laser depth value of the image pixel;

[0055] A first group number obtaining unit, configured to obtain, based on the visual depth value of each two-dimensional point (x, y), a group number idx of the visual depth group to which it belongs according to the same labeling principle as the depth group number of the visual depth distribution histogram;

[0056] A second group number obtaining unit is used to obtain the group number idy of the laser depth group to which the laser depth value of each two-dimensional point (x, y) belongs based on the same numbering principle as the depth group number of the laser depth distribution histogram;

[0057] The joint group number acquisition unit is used to count the number of image pixels divided into each joint group number (idx, idy) to obtain a two-dimensional joint depth distribution histogram.

[0058] As a preferred solution, the cross entropy calculation module includes:

[0059] The first calculation unit is used to obtain the count value cnt_x of the depth group corresponding to the group number idx on the visual depth distribution histogram based on the visual depth value of the two-dimensional point (x, y), and obtain px by dividing cnt_x by the total number of laser points;

[0060] The second calculation unit is used to obtain the count value cnt_y of the depth group corresponding to the group number idy on the laser depth distribution histogram based on the laser depth value of the two-dimensional point (x, y), and obtain py by dividing cnt_y by the total number of laser points;

[0061] The third calculation unit is used to obtain the count value cnt_xy of the depth group with the joint group number (idx, idy) on the two-dimensional joint depth distribution histogram based on the visual depth value and the laser depth value of the two-dimensional point (x, y), and obtain pxy by dividing cnt_xy by the total number of laser points;

[0062] The cross entropy calculation unit is used to obtain the cross entropy val of the two-dimensional point (x, y) through the following formula;

[0063] val = pxy*log(pxy / (px*py));

[0064] The repeating execution unit is used to repeatedly execute the methods in the first calculation unit, the second calculation unit, the third calculation unit, and the cross entropy calculation unit in sequence until the cross entropy of each two-dimensional point is obtained.

[0065] In a third aspect, the present invention discloses a computing device, comprising:

[0066] one or more processors;

[0067] Memory;

[0068] And one or more programs, wherein the one or more programs are stored in a memory and configured to be executed by one or more processors, and the one or more programs include instructions for any of the above-mentioned cross-entropy-based external parameter calibration methods.

[0069] In a fourth aspect, the present invention discloses a storage medium storing one or more computer-readable programs, wherein the one or more programs include instructions suitable for being loaded by a memory and executing any of the above-mentioned cross-entropy-based external parameter calibration methods.

[0070] The present invention discloses a cross-entropy based external parameter calibration method, device, equipment and storage medium, which has the following beneficial effects:

[0071] First, the camera uses scaled depth estimation to align with the depth information of the lidar, and calculates the cross entropy based on the depth information as the calibration residual, realizing fast calibration of external parameters between the lidar and the camera, which is efficient and independent of the external environment scene.

[0072] Second, when the cross entropy sum is larger, it means that the initial external parameters are more accurate and the initial external parameters can be effectively evaluated. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0074] Figure 1 A flowchart of an external parameter calibration method provided in an embodiment of the present invention.

[0075] Figure 2 An example diagram showing the calibration results provided by an embodiment of the present invention;

[0076] (a), (b), (c) and (d) are four example images respectively.

[0077] Figure 3 A block diagram of a computing device provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0079] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] Using ordinal numbers “first,” “second,” “third,” etc. to describe common objects merely indicates that different instances of similar objects are involved and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

[0081] In addition, the expression of “comprising” an element is an “open” expression, which merely means that corresponding components or steps exist, and should not be interpreted as excluding additional components or steps.

[0082] In order to achieve the purpose of the present invention, in some embodiments of the external parameter calibration method based on cross entropy, such as Figure 1 As shown, the external parameter calibration method includes:

[0083] Step S101: performing depth estimation on a camera image captured by a camera through a visual depth estimation model to obtain a visual depth image;

[0084] Step S102: converting the laser point cloud collected by the laser radar from the laser coordinate system to the camera coordinate system using initial external parameters;

[0085] Step S103: based on the laser point cloud and the visual depth image in the camera coordinate system, all image pixel points corresponding to the laser point cloud are obtained, and each image pixel point has a one-to-one corresponding visual depth value and laser depth value;

[0086] Step S104: obtaining a visual depth distribution histogram based on the visual depth value of each image pixel;

[0087] Step S105: obtaining a laser depth distribution histogram based on the laser depth value of each image pixel;

[0088] Step S106: The visual depth value and the laser depth value of each image pixel are used to obtain a two-dimensional joint depth distribution histogram;

[0089] Step S107: Calculate the cross entropy of each image pixel based on the visual depth distribution histogram, the laser depth distribution histogram and the two-dimensional joint depth distribution histogram;

[0090] Step S108: Accumulate the cross entropy of each image pixel to obtain the cross entropy sum of the current frame;

[0091] Step S109: Based on the cross entropy sum, calibrate and optimize the initial external parameters.

[0092] Each step of the present invention is described in detail below.

[0093] Step S101 performs depth estimation on a camera image (eg, a monocular image) captured by a camera through a visual depth estimation model to obtain a visual depth image (eg, a monocular depth map).

[0094] In step S102, the real-time laser point cloud collected by the laser radar is converted from the laser coordinate system to the camera coordinate system using the initial external parameters to obtain the real-time laser point cloud in the camera coordinate system.

[0095] The initial external parameters can be obtained by using several methods in the prior art, which is common knowledge in the art and will not be elaborated here.

[0096] In step S103, the following method may be used to obtain image pixel points corresponding to the laser point cloud.

[0097] The laser point cloud in the camera coordinate system is traversed, and the laser point cloud is projected onto the visual depth image, and the image pixel corresponding to each laser point cloud is obtained by linear interpolation. The visual depth value of each image pixel has been obtained by the visual depth estimation model in step S101, so each image pixel has a one-to-one corresponding visual depth value and laser depth value.

[0098] It is worth noting that in some embodiments, when processing the collected laser point cloud, only the laser point cloud within the range of 0.2 meters to 40 meters is counted, because the visually estimated depth is too far to be credible. The visual depth value estimated by the visual method should be compared with the depth value of the laser point cloud, so the depth value of the laser point cloud should be set within a reasonable range of the visually estimated depth. The visual depth value of the visual estimation is also processed within a range. In the present invention, the visual depth value of the visual estimation is set to be less than 300, because the model used is scaled, and the estimated depth is scaled, and there is no real physical unit of length, so this threshold has no unit.

[0099] Further, the visual depth distribution histogram and the laser depth distribution histogram are obtained respectively by the following steps, including:

[0100] Step A: based on the visual depth value or laser depth value of each image pixel, find the corresponding maximum visual depth value, minimum visual depth value, or the corresponding maximum laser depth value, minimum laser depth value;

[0101] Step B: Based on the maximum visual depth value and the minimum visual depth value, or the maximum laser depth value and the minimum laser depth value, the corresponding depth value range is evenly divided into N depth groups;

[0102] Step C: traverse the visual depth value or laser depth value of each image pixel and divide it into corresponding depth groups;

[0103] Step D: Count the number of visual depth values ​​or laser depth values ​​divided into each depth group to obtain a visual depth distribution histogram or a laser depth distribution histogram.

[0104] Through the visual depth distribution histogram or the depth distribution histogram, the group number of the depth group corresponding to the visual depth value or the laser depth value of each image pixel can be known.

[0105] Specifically, step S104 includes:

[0106] Step S104.1: based on the visual depth value of each image pixel, find the maximum visual depth max_val1 and the minimum visual depth min_val1;

[0107] Step S104.2: Based on the maximum visual depth value max_val1 and the minimum visual depth value min_val1, the visual depth range is evenly divided into N (e.g., 100) visual depth groups;

[0108] The length of each visual depth group is bin_size1;

[0109] bin_size1=(max_val1-min_val1) / N;

[0110] For example, the range of the first visual depth group is [min_val1,min_val1+bin_size1);

[0111] Step S104.3: traverse the visual depth value of each image pixel and divide it into corresponding visual depth groups;

[0112] Step S104.4: Count the number of visual depth values ​​assigned to each visual depth group to obtain a visual depth distribution histogram.

[0113] Specifically, step S105 includes:

[0114] Step S105.1: Count the laser depth values ​​of each laser point cloud, and find the maximum laser depth max_val2 and the minimum laser depth min_val2;

[0115] Step S105.2: Based on the maximum laser depth value max_val2 and the minimum laser depth value min_val2, the laser depth range is evenly divided into N (e.g., 100) laser depth groups;

[0116] The length of each visual depth group is bin_size2;

[0117] bin_size2=(max_val2-min_val2) / N;

[0118] For example, the range of the first laser depth group is [min_val2,min_val2+bin_size2);

[0119] Step S105.3: traverse the laser depth values ​​of each laser point cloud and divide them into corresponding laser depth groups;

[0120] Step S105.4: Count the number of laser depth values ​​assigned to each laser depth group to obtain a laser depth distribution histogram.

[0121] Further, step S106 includes:

[0122] Step S106.1: Establish a two-dimensional depth distribution coordinate system, and set each image pixel as a two-dimensional point (x, y).

[0123] Where: x is the visual depth value of the image pixel;

[0124] y is the laser depth value of the image pixel;

[0125] Step S106.2: based on the visual depth value of each two-dimensional point (x, y), obtain the group number idx of the visual depth group to which it belongs according to the same labeling principle as the depth group number of the visual depth distribution histogram;

[0126] Step S106.3: Based on the laser depth value of each two-dimensional point (x, y), the group number idy of the laser depth group to which it belongs is obtained according to the same numbering principle as the depth group number of the laser depth distribution histogram;

[0127] Step S106.4: Count the number of image pixels divided into each joint group number (idx, idy) to obtain a two-dimensional joint depth distribution histogram.

[0128] The above steps S106.2 and S106.3 are processed using a method similar to that of steps S104 and S105.

[0129] Assume the visual depth value vis_depth and the laser depth value lidar_depth of the two-dimensional point (x, y).

[0130] The minimum visual depth value min_val1, the maximum visual depth value max_val1, and the total number of visual depth groups is N;

[0131] The minimum laser depth value is min_val2, the maximum laser depth value is max_val2, and there are N laser depth groups in total;

[0132] According to the following formula, the group number idx of the visual depth group and the group number idy of the laser depth group can be obtained;

[0133] idx=std::floor((vis_depth-min_val1) / (max_val1-min_val1)*100);

[0134] idy=std::floor((lidar_depth-min_val2) / (max_val2-min_val2)*100);

[0135] Among them: std::floor means rounding down to an integer.

[0136] It is worth noting that the group numbers of the depth groups of the visual depth distribution histogram, the laser depth distribution histogram, and the two-dimensional joint depth distribution histogram are unified.

[0137] According to the visual depth distribution histogram, the depth distribution histogram, and the two-dimensional joint depth distribution histogram, the group number of the depth group of each histogram corresponding to the visual depth value or laser depth value of each image pixel can be known.

[0138] Further, step S107 includes:

[0139] Step S107.1: Based on the visual depth value of the two-dimensional point (x, y), obtain the count value cnt_x of the depth group corresponding to the group number idx on the visual depth distribution histogram, and obtain px by dividing cnt_x by the total number of laser points;

[0140] Step S107.2: Based on the laser depth value of the two-dimensional point (x, y), obtain the count value cnt_y of the depth group corresponding to the group number idy on the laser depth distribution histogram, and obtain py by dividing cnt_y by the total number of laser points;

[0141] Step S107.3: Based on the visual depth value and the laser depth value of the two-dimensional point (x, y), obtain the count value cnt_xy of the depth group with the joint group number (idx, idy) on the two-dimensional joint depth distribution histogram, and obtain pxy by dividing cnt_xy by the total number of laser points;

[0142] Step S107.4: Obtain the cross entropy val of the two-dimensional point (x, y) by the following formula;

[0143] val = pxy*log(pxy / (px*py));

[0144] Step S107.5: Repeat steps S107.1 to S107.4 until the cross entropy of each two-dimensional point is obtained.

[0145] After the cross entropy of each two-dimensional point is obtained in step S107.5, step S108 accumulates it to obtain the cross entropy sum under the initial external parameter. (Because this cross entropy sum is obtained under the current initial external parameter, at this time, the initial external parameter and the cross entropy sum are also in a one-to-one correspondence, that is, one external parameter corresponds to one cross entropy sum).

[0146] Theoretically, the closer the external parameter is to the true value, the larger the cross entropy sum will be. Step S108: Accumulate the cross entropy of each image pixel to obtain the cross entropy sum of the current frame;

[0147] Further, step S109: based on the cross entropy sum, the initial external parameters are calibrated and optimized. The specific optimization can be: using the cross entropy sum to derive the initial external parameters, and then performing the least squares method, that is, the external parameters can be optimized to obtain the optimal external parameters. This part is common knowledge in the field and will not be described in detail here. The present invention is experimentally verified, and the calibration results are as follows Figure 2 As shown in (a), (b), (c), and (d), the laser point cloud is projected onto the visual depth image.

[0148] The extrinsic reference optimized by cross entropy calibration is compared with the true value extrinsic reference, and its angle accuracy difference is within 0.05 degrees, and the translation accuracy difference is within 1 cm.

[0149] The present invention also discloses an embodiment of an external parameter calibration device based on cross entropy, and the external parameter calibration device comprises:

[0150] A depth estimation module is used to perform depth estimation on the camera image captured by the camera through a visual depth estimation model to obtain a visual depth image;

[0151] The point cloud conversion module is used to convert the laser point cloud collected by the laser radar from the laser coordinate system to the camera coordinate system using the initial external parameters;

[0152] A pixel acquisition module is used to acquire all image pixels corresponding to the laser point cloud based on the laser point cloud and the visual depth image in the camera coordinate system, and each image pixel has a one-to-one corresponding visual depth value and laser depth value;

[0153] A visual depth distribution acquisition module is used to obtain a visual depth distribution histogram based on the visual depth value of each image pixel;

[0154] A laser depth distribution acquisition module is used to obtain a laser depth distribution histogram based on the laser depth value of each image pixel point;

[0155] A two-dimensional joint depth distribution acquisition module is used for the visual depth value and laser depth value of each image pixel to obtain a two-dimensional joint depth distribution histogram;

[0156] A cross entropy calculation module is used to calculate the cross entropy of each image pixel based on the visual depth distribution histogram, the laser depth distribution histogram and the two-dimensional joint depth distribution histogram;

[0157] The cross entropy accumulation module is used to accumulate the cross entropy of each image pixel to obtain the cross entropy sum of the current frame;

[0158] The external parameter optimization module is used to calibrate and optimize the initial external parameters based on the cross entropy sum.

[0159] Furthermore, the visual depth distribution acquisition module and the laser depth distribution acquisition module both include:

[0160] A depth extreme value determination unit, used to find the corresponding maximum visual depth value, minimum visual depth value, or the corresponding maximum laser depth value, minimum laser depth value based on the visual depth value or laser depth value of each image pixel point;

[0161] A depth group determination unit, configured to evenly divide a corresponding depth value range into N depth groups based on a maximum visual depth value and a minimum visual depth value, or a maximum laser depth value and a minimum laser depth value;

[0162] A division unit, used for traversing the visual depth value or laser depth value of each image pixel point and dividing it into a corresponding depth group;

[0163] The statistical unit is used to count the number of visual depth values ​​or laser depth values ​​divided into each depth group to obtain a visual depth distribution histogram or a laser depth distribution histogram.

[0164] Furthermore, the two-dimensional joint depth distribution acquisition module includes:

[0165] The two-dimensional coordinate system establishment unit is used to establish a two-dimensional depth distribution coordinate system. Let each image pixel be a two-dimensional point (x, y).

[0166] Where: x is the visual depth value of the image pixel;

[0167] y is the laser depth value of the image pixel;

[0168] A first group number obtaining unit, configured to obtain, based on the visual depth value of each two-dimensional point (x, y), a group number idx of the visual depth group to which it belongs according to the same labeling principle as the depth group number of the visual depth distribution histogram;

[0169] A second group number obtaining unit is used to obtain the group number idy of the laser depth group to which the laser depth value of each two-dimensional point (x, y) belongs based on the same numbering principle as the depth group number of the laser depth distribution histogram;

[0170] The joint group number acquisition unit is used to count the number of image pixels divided into each joint group number (idx, idy) to obtain a two-dimensional joint depth distribution histogram.

[0171] Furthermore, the cross entropy calculation module includes:

[0172] The first calculation unit is used to obtain the count value cnt_x of the depth group corresponding to the group number idx on the visual depth distribution histogram based on the visual depth value of the two-dimensional point (x, y), and obtain px by dividing cnt_x by the total number of laser points;

[0173] The second calculation unit is used to obtain the count value cnt_y of the depth group corresponding to the group number idy on the laser depth distribution histogram based on the laser depth value of the two-dimensional point (x, y), and obtain py by dividing cnt_y by the total number of laser points;

[0174] The third calculation unit is used to obtain the count value cnt_xy of the depth group with the joint group number (idx, idy) on the two-dimensional joint depth distribution histogram based on the visual depth value and the laser depth value of the two-dimensional point (x, y), and obtain pxy by dividing cnt_xy by the total number of laser points;

[0175] The cross entropy calculation unit is used to obtain the cross entropy val of the two-dimensional point (x, y) through the following formula;

[0176] val = pxy*log(pxy / (px*py));

[0177] The repeating execution unit is used to repeatedly execute the methods in the first calculation unit, the second calculation unit, the third calculation unit, and the cross entropy calculation unit in sequence until the cross entropy of each two-dimensional point is obtained.

[0178] Furthermore, it should be noted that: the cross-entropy-based external parameter calibration device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when determining the dominant vertex. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the cross-entropy-based external parameter calibration device is divided into different functional modules to complete all or part of the functions described above.

[0179] In addition, the cross-entropy-based external parameter calibration device provided in the above embodiment and the cross-entropy-based external parameter calibration method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0180] In addition, in some other embodiments, Figure 2 As shown, the present invention also discloses a computing device, including:

[0181] One or more processors 201;

[0182] Memory 202;

[0183] And one or more programs, wherein the one or more programs are stored in the memory 202 and are configured to be executed by one or more processors 201, and the one or more programs include instructions for the cross-entropy based external parameter calibration method disclosed in the above embodiment.

[0184] The processor 201 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 201 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 201 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 201 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 201 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0185] The memory 202 may include one or more computer-readable storage media, which may be non-transitory. The memory 202 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 202 is used to store at least one instruction, which is used to be executed by the processor 201 to implement the cross-entropy-based external parameter calibration method provided in the method embodiment of the present invention.

[0186] In addition, the computing device may optionally include: a peripheral device interface and at least one peripheral device. The processor 201, the memory 202 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0187] Of course, the computing device may also include fewer or more components, which is not limited in this embodiment.

[0188] In addition, in some other embodiments, the present invention also discloses a storage medium, which stores one or more computer-readable programs, and the one or more programs include instructions, and the instructions are suitable for being loaded by a memory and executing the cross-entropy-based external parameter calibration method disclosed in the above embodiment.

[0189] The present invention discloses a cross-entropy based external parameter calibration method, device, equipment and storage medium, which has the following beneficial effects:

[0190] First, the camera uses scaled depth estimation to align with the depth information of the lidar, and calculates the cross entropy based on the depth information as the calibration residual, realizing fast calibration of external parameters between the lidar and the camera, which is efficient and independent of the external environment scene.

[0191] Second, when the cross entropy sum is larger, it means that the initial external parameters are more accurate and the initial external parameters can be effectively evaluated.

[0192] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. The external parameter calibration method based on cross entropy is characterized by: include: Step S1: performing depth estimation on a camera image captured by a camera through a visual depth estimation model to obtain a visual depth image; Step S2: converting the laser point cloud collected by the laser radar from the laser coordinate system to the camera coordinate system using the initial external parameters; Step S3: Based on the laser point cloud and the visual depth image in the camera coordinate system, all image pixel points corresponding to the laser point cloud are obtained, and each image pixel point has a one-to-one corresponding visual depth value and laser depth value; Step S4: obtaining a visual depth distribution histogram based on the visual depth value of each image pixel; Step S5: obtaining a laser depth distribution histogram based on the laser depth value of each image pixel; Step S6: The visual depth value and the laser depth value of each image pixel are used to obtain a two-dimensional joint depth distribution histogram; Step S7: Calculate the cross entropy of each image pixel based on the visual depth distribution histogram, the laser depth distribution histogram and the two-dimensional joint depth distribution histogram; Step S8: Accumulate the cross entropy of each image pixel to obtain the cross entropy sum of the current frame; Step S9: Based on the cross entropy sum, calibrate and optimize the initial external parameters.

2. The external parameter calibration method according to claim 1, characterized in that: The visual depth distribution histogram and the laser depth distribution histogram are obtained respectively by the following steps, including: Step A: based on the visual depth value or laser depth value of each image pixel, find the corresponding maximum visual depth value, minimum visual depth value, or the corresponding maximum laser depth value, minimum laser depth value; Step B: Based on the maximum visual depth value and the minimum visual depth value, or the maximum laser depth value and the minimum laser depth value, the corresponding depth value range is evenly divided into N depth groups; Step C: traverse the visual depth value or laser depth value of each image pixel and divide it into corresponding depth groups; Step D: Count the number of visual depth values ​​or laser depth values ​​divided into each depth group to obtain a visual depth distribution histogram or a laser depth distribution histogram.

3. The external parameter calibration method according to claim 2, characterized in that: The step S6 comprises: Step S6.1: Establish a two-dimensional depth distribution coordinate system, and set each image pixel as a two-dimensional point (x, y). Where: x is the visual depth value of the image pixel; y is the laser depth value of the image pixel; Step S6.2: based on the visual depth value of each two-dimensional point (x, y), obtain the group number idx of the visual depth group to which it belongs according to the same labeling principle as the depth group number of the visual depth distribution histogram; Step S6.3: Based on the laser depth value of each two-dimensional point (x, y), the group number idy of the laser depth group to which it belongs is obtained according to the same labeling principle as the depth group number of the laser depth distribution histogram; Step S6.4: Count the number of image pixels divided into each joint group number (idx, idy) to obtain a two-dimensional joint depth distribution histogram.

4. The external parameter calibration method according to claim 3, characterized in that: The step S7 comprises: Step S7.1: Based on the visual depth value of the two-dimensional point (x, y), obtain the count value cnt_x of the depth group corresponding to the group number idx on the visual depth distribution histogram, and obtain px by dividing cnt_x by the total number of laser points; Step S7.2: Based on the laser depth value of the two-dimensional point (x, y), obtain the count value cnt_y of the depth group corresponding to the group number idy on the laser depth distribution histogram, and obtain py by dividing cnt_y by the total number of laser points; Step S7.3: Based on the visual depth value and the laser depth value of the two-dimensional point (x, y), obtain the count value cnt_xy of the depth group with the joint group number (idx, idy) on the two-dimensional joint depth distribution histogram, and obtain pxy by dividing cnt_xy by the total number of laser points; Step S7.4: Obtain the cross entropy val of the two-dimensional point (x, y) by the following formula; val = pxy*log(pxy / (px*py)); Step S7.5: Repeat steps S7.1 to S7.4 until the cross entropy of each two-dimensional point is obtained.

5. The external parameter calibration device based on cross entropy is characterized by: include: A depth estimation module is used to perform depth estimation on the camera image captured by the camera through a visual depth estimation model to obtain a visual depth image; The point cloud conversion module is used to convert the laser point cloud collected by the laser radar from the laser coordinate system to the camera coordinate system using the initial external parameters; A pixel acquisition module is used to acquire all image pixels corresponding to the laser point cloud based on the laser point cloud and the visual depth image in the camera coordinate system, and each image pixel has a one-to-one corresponding visual depth value and laser depth value; A visual depth distribution acquisition module is used to obtain a visual depth distribution histogram based on the visual depth value of each image pixel; A laser depth distribution acquisition module is used to obtain a laser depth distribution histogram based on the laser depth value of each image pixel point; A two-dimensional joint depth distribution acquisition module is used for the visual depth value and laser depth value of each image pixel to obtain a two-dimensional joint depth distribution histogram; A cross entropy calculation module is used to calculate the cross entropy of each image pixel based on the visual depth distribution histogram, the laser depth distribution histogram and the two-dimensional joint depth distribution histogram; The cross entropy accumulation module is used to accumulate the cross entropy of each image pixel to obtain the cross entropy sum of the current frame; The external parameter optimization module is used to calibrate and optimize the initial external parameters based on the cross entropy sum.

6. The external parameter calibration device according to claim 5, characterized in that: The visual depth distribution acquisition module and the laser depth distribution acquisition module both include: A depth extreme value determination unit, used to find the corresponding maximum visual depth value, minimum visual depth value, or the corresponding maximum laser depth value, minimum laser depth value based on the visual depth value or laser depth value of each image pixel point; A depth group determination unit, configured to evenly divide a corresponding depth value range into N depth groups based on a maximum visual depth value and a minimum visual depth value, or a maximum laser depth value and a minimum laser depth value; A division unit, used for traversing the visual depth value or laser depth value of each image pixel point and dividing it into a corresponding depth group; The statistical unit is used to count the number of visual depth values ​​or laser depth values ​​divided into each depth group to obtain a visual depth distribution histogram or a laser depth distribution histogram.

7. The external parameter calibration device according to claim 6, characterized in that: The two-dimensional joint depth distribution acquisition module includes: The two-dimensional coordinate system establishment unit is used to establish a two-dimensional depth distribution coordinate system. Let each image pixel be a two-dimensional point (x, y). Where: x is the visual depth value of the image pixel; y is the laser depth value of the image pixel; A first group number obtaining unit, configured to obtain, based on the visual depth value of each two-dimensional point (x, y), a group number idx of the visual depth group to which it belongs according to the same labeling principle as the depth group number of the visual depth distribution histogram; A second group number obtaining unit is used to obtain the group number idy of the laser depth group to which the laser depth value of each two-dimensional point (x, y) belongs based on the same numbering principle as the depth group number of the laser depth distribution histogram; The joint group number acquisition unit is used to count the number of image pixels divided into each joint group number (idx, idy) to obtain a two-dimensional joint depth distribution histogram.

8. The external parameter calibration device according to claim 7, characterized in that: The cross entropy calculation module includes: The first calculation unit is used to obtain the count value cnt_x of the depth group corresponding to the group number idx on the visual depth distribution histogram based on the visual depth value of the two-dimensional point (x, y), and obtain px by dividing cnt_x by the total number of laser points; The second calculation unit is used to obtain the count value cnt_y of the depth group corresponding to the group number idy on the laser depth distribution histogram based on the laser depth value of the two-dimensional point (x, y), and obtain py by dividing cnt_y by the total number of laser points; The third calculation unit is used to obtain the count value cnt_xy of the depth group with the joint group number (idx, idy) on the two-dimensional joint depth distribution histogram based on the visual depth value and the laser depth value of the two-dimensional point (x, y), and obtain pxy by dividing cnt_xy by the total number of laser points; The cross entropy calculation unit is used to obtain the cross entropy val of the two-dimensional point (x, y) through the following formula; val = pxy*log(pxy / (px*py)); The repeating execution unit is used to repeatedly execute the methods in the first calculation unit, the second calculation unit, the third calculation unit, and the cross entropy calculation unit in sequence until the cross entropy of each two-dimensional point is obtained.

9. A computing device, characterized in that include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and one or more of the programs include instructions for the cross-entropy based external parameter calibration method described in any one of claims 1-4.

10. A storage medium, characterized in that The storage medium stores one or more computer-readable programs, and the one or more programs include instructions, which are suitable for being loaded by the memory and executing the cross-entropy based external parameter calibration method described in any one of claims 1-4.

Images