Radar and camera extrinsic calibration method, electronic device and storage medium

Abstract

This application provides a method, electronic device, and storage medium for extrinsic parameter calibration of radar and camera. The method includes: projecting a 3D point cloud object detection box onto a 2D plane to obtain a corresponding 2D point cloud object detection box; matching the 2D image object detection box and the 2D point cloud object detection box to obtain at least one object matching pair; processing the corresponding 2D image of the object detection box in the object matching pair to obtain a transformed image; projecting the corresponding point cloud of the object detection box in the object matching pair onto the transformed image to obtain a projection result, and performing extrinsic parameter optimization to obtain an optimized result; and determining the target extrinsic parameters in response to the optimization result satisfying a preset optimization condition. This application extracts object information by performing 2D perception on the original image and 3D perception on the original point cloud, which is time-efficient and real-time. The extrinsic parameter optimization using at least one object matching pair effectively avoids local optimization.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a method for calibrating the external parameters of radar and cameras, electronic devices, and storage media. Background Technology

[0002] In the field of autonomous driving, mapping, localization, and perception detection all require raw data from multiple sensors, such as multiple cameras, LiDAR, Real-Time Kinematic Global Positioning System (RTKGPS), and Inertial Measurement Unit (IMU). Therefore, the fusion of data from multiple sensors is particularly important.

[0003] Among these parameters, the extrinsic parameters between the LiDAR and the camera are crucial. Calibration methods for these extrinsic parameters include factory calibration and online calibration. Online calibration can monitor and update the extrinsic parameters; however, some methods obtain these parameters through semantic segmentation of 2D images and 3D point clouds, which is time-consuming and may not yield the globally optimal solution. Summary of the Invention

[0004] This application provides a method for calibrating the external parameters of radar and cameras, an electronic device, and a storage medium to solve the above-mentioned problems.

[0005] The first aspect of this application provides a method for extrinsic parameter calibration of radar and camera, comprising: acquiring a current frame original image and a current frame original point cloud; acquiring a two-dimensional image object detection box from the current frame original image and a three-dimensional point cloud object detection box from the current frame original point cloud; projecting the three-dimensional point cloud object detection box onto a two-dimensional plane to obtain a corresponding two-dimensional point cloud object detection box; matching the two-dimensional image object detection box and the two-dimensional point cloud object detection box to obtain at least one object matching pair; processing the two-dimensional image corresponding to the two-dimensional image object detection box in the at least one object matching pair to obtain a transformed image; projecting the point cloud corresponding to the two-dimensional point cloud object detection box in the at least one object matching pair onto the transformed image to obtain the projection result of the point cloud in the transformed image; performing extrinsic parameter optimization operation based on the projection result to obtain an optimization result; and determining the target extrinsic parameters in response to the optimization result satisfying a preset optimization condition.

[0006] In some embodiments, projecting the three-dimensional point cloud object detection box onto a two-dimensional plane to obtain a corresponding two-dimensional point cloud object detection box includes: projecting the three-dimensional point cloud object detection box onto the coordinate system where the two-dimensional image object detection box is located to obtain a projection box; obtaining the vertices at the maximum and minimum values ​​of the projection box in the horizontal axis direction and the vertices at the maximum and minimum values ​​in the vertical axis direction of the coordinate system, thereby obtaining the two-dimensional point cloud object detection box.

[0007] In some embodiments, matching the two-dimensional image object detection box and the two-dimensional point cloud object detection box to obtain at least one object matching pair includes: using a preset matching algorithm to calculate a preset semantic distance between the two-dimensional image object detection box and the two-dimensional point cloud object detection box to obtain a distance value matrix; and performing a matching operation on the distance value matrix to obtain at least one object matching pair.

[0008] In some embodiments, calculating the preset semantic distance between the two-dimensional image object detection box and the two-dimensional point cloud object detection box includes: calculating the intersection-union ratio between the two-dimensional image object detection box and the two-dimensional point cloud object detection box.

[0009] In some embodiments, calculating the preset semantic distance between the two-dimensional image object detection box and the two-dimensional point cloud object detection box includes: calculating the intersection-union ratio (IU / U) between the two-dimensional image object detection box and the two-dimensional point cloud object detection box, and calculating the area ratio between the two-dimensional image object detection box and the two-dimensional point cloud object detection box; wherein, when calculating the area ratio, the object detection box with the larger area is used as the denominator, and the object detection box with the smaller area is used as the numerator; the calculation operation is performed using a preset semantic distance formula, wherein the preset semantic distance formula is:

[0010] D = t * IoU + (1 - t) * d area

[0011] Where t is the weight, t∈[0,1], IoU is the intersection-union ratio, and d area This represents the area ratio.

[0012] In some embodiments, processing the two-dimensional image corresponding to the object detection box in the at least one object matching pair to obtain a transformed image includes: binarizing the two-dimensional image corresponding to the object detection box in the at least one object matching pair to obtain a binarized image; and performing inverse distance transform calculation on the binarized image to obtain the transformed image.

[0013] In some embodiments, the projection result includes the pixel value of a pixel in the transformed image corresponding to each point in the point cloud; the step of performing extrinsic parameter optimization based on the projection result to obtain the optimization result includes: performing optimization calculation using a preset cost function to obtain the optimization result; wherein, the preset cost function is:

[0014]

[0015] Where N is the number of points in the point cloud, and the pixel value of the i-th pixel is p. i .

[0016] In some embodiments, the extrinsic parameter calibration method for radar and camera further includes: in response to the optimization result not meeting the preset optimization conditions, performing a search calculation based on the preset extrinsic parameter range and according to the preset step size using the preset cost function to obtain the calculation result; and in response to the calculation result meeting the preset search conditions, determining the target extrinsic parameters.

[0017] The second aspect of this application provides an electronic device including a memory and a processor coupled to each other, the processor being configured to execute program instructions stored in the memory to implement the radar and camera extrinsic parameter calibration method of the first aspect or the model evaluation method of the second aspect.

[0018] A third aspect of this application provides a non-volatile computer-readable storage medium for storing program instructions, which, when executed by a processor, are used to implement the radar and camera extrinsic parameter calibration method of the first aspect or the model evaluation method of the second aspect.

[0019] The above-mentioned scheme obtains two-dimensional object detection boxes from the original image of the current frame and three-dimensional point cloud object detection boxes from the original point cloud of the current frame. The three-dimensional point cloud object detection boxes are projected onto a two-dimensional plane to obtain corresponding two-dimensional point cloud object detection boxes. The two-dimensional image object detection boxes and the two-dimensional point cloud object detection boxes are matched to obtain at least one object matching pair. The corresponding two-dimensional images of the two-dimensional image object detection boxes in the at least one object matching pair are processed to obtain a transformed image. The corresponding point clouds of the two-dimensional point cloud object detection boxes in the at least one object matching pair are projected onto the transformed image to obtain a projection result. Extrinsic parameter optimization is performed based on the projection result to obtain an optimization result. The target extrinsic parameters are determined in response to the optimization result satisfying preset optimization conditions. The scheme in this application extracts object information by performing two-dimensional perception on the original image and three-dimensional perception on the original point cloud. This method is time-efficient and has strong real-time performance. Furthermore, by performing extrinsic parameter optimization through at least one object matching pair, it can effectively avoid local optimization.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0022] Figure 1 This is a flowchart illustrating the external parameter calibration method for radar and camera in an embodiment of this application;

[0023] Figure 2 This is a schematic diagram of the structure of the electronic device in the embodiments of this application;

[0024] Figure 3 This is a schematic diagram of the structure of a non-volatile computer-readable storage medium in an embodiment of this application. Detailed Implementation

[0025] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the application. Similarly, the following embodiments are only some, not all, embodiments of the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.

[0026] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0027] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, "many" in this document means two or more. Additionally, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. Furthermore, the terms "first," "second," and "third" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0028] As mentioned above, the extrinsic parameters between the LiDAR and the camera are crucial. Calibration methods for these extrinsic parameters include factory calibration and online calibration. Online calibration can monitor and update the extrinsic parameters; however, some methods obtain these parameters through semantic segmentation of 2D images and 3D point clouds, which is time-consuming and may not yield the globally optimal solution.

[0029] Therefore, this application provides a method for calibrating the external parameters of radar and cameras, an electronic device, and a storage medium to solve the above problems.

[0030] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for calibrating the extrinsic parameters of radar and camera in an embodiment of this application. It should be noted that if substantially the same result is obtained, the method of this application is not necessarily identical. Figure 1 The illustrated process sequence is limited. This method can be applied to electronic devices with computing or other functions. The electronic device can execute this method by receiving information collected by a sensor device. The electronic device in this application can be a server or a system consisting of a server and terminal devices working together. Furthermore, the aforementioned server can be hardware or software, without specific limitations.

[0031] Electronic devices can execute this method by receiving information collected by sensor devices. These sensor devices can be image sensors, radar sensors, inertial measurement units (IMUs), or global navigation satellite systems (GNSS) equipped in autonomous vehicles. During the autonomous vehicle's operation, the sensor devices perceive relevant data about the autonomous vehicle and the dynamic scene of the real physical world surrounding the vehicle. The scene may include vehicles, pedestrians, buildings, etc. In some possible implementations, the radar and camera extrinsic parameter calibration method of this application embodiment can be implemented by a processor calling computer-readable instructions stored in memory. Extrinsic parameters refer to the relative relationship between the coordinates of two sensor devices, generally six degrees of freedom, namely three degrees of relative translation and three degrees of relative rotation. Figure 1 As shown, the external parameter calibration method for radar and camera includes the following steps:

[0032] S1. Obtain the original image and the original point cloud of the current frame.

[0033] The sensors include radar sensors and image sensors. Image sensors can be used to acquire image data to obtain raw images, while radar sensors can be used to acquire point cloud data to obtain raw point clouds. For example, image sensors and radar sensors can be mounted on a mobile device. This mobile device can be an automated mobile device, such as a robot or an autonomous vehicle.

[0034] In some embodiments, the image sensor may be a camera, and the radar sensor may be a lidar, such as a mechanical lidar, a semi-solid-state lidar, or a solid-state lidar. In one embodiment, the radar may be any radar device that provides point cloud data and is used in autonomous driving to meet perception accuracy requirements.

[0035] In one application scenario, an autonomous vehicle is driving on a road. A camera installed on the autonomous vehicle collects at least one frame of image data within a preset time period to obtain at least one original image. Radar is used to collect at least one frame of point cloud data within a preset time period to obtain at least one original point cloud.

[0036] Acquire at least one original image frame and at least one original point cloud frame within a preset time period, wherein the acquisition time of the at least one original image frame and the at least one original point cloud frame corresponds. It is understood that the preset time period includes the current time and a preset number of times preceding the current time, such as three times preceding the current time, or other achievable numbers, which can be set according to actual usage requirements and are not specifically limited. For example, during the operation of an autonomous vehicle, the camera acquires image data in real time, and the radar acquires point cloud data in real time, with the acquisition times of the image data and point cloud data corresponding.

[0037] For example, the camera acquires one frame of image data at time t1, the radar acquires one frame of point cloud data at time t1, the camera acquires one frame of image data at time t2, the radar acquires one frame of point cloud data at time t2, the camera acquires one frame of image data at time t3, and the radar acquires one frame of point cloud data at time t3; the current time is t3, so the original image and the original point cloud of the current frame are obtained, that is, the original image and the original point cloud at time t3 are obtained.

[0038] S2. Obtain the two-dimensional object detection box from the original image of the current frame, and obtain the three-dimensional point cloud object detection box from the original point cloud of the current frame.

[0039] Image object detection box, that is, a box that encloses the image information of an object; point cloud object detection box, that is, a box that encloses the point cloud information of an object; the object can be a vehicle, street light, pedestrian, or other objects that can be realized, without specific limitations.

[0040] After obtaining the original image and the original point cloud of the current frame, a detection operation is performed on the original image to obtain 2D object detection boxes; and a detection operation is performed on the original point cloud to obtain 3D point cloud object detection boxes. The specific method of detection operation can be selected according to the usage requirements and is not specifically limited. For example, the corresponding 2D image object detection boxes and 3D point cloud object detection boxes can be extracted manually, and the number of 2D image object detection boxes and 3D point cloud object detection boxes can be one or more.

[0041] S3. Project the 3D point cloud object detection box onto the 2D plane to obtain the corresponding 2D point cloud object detection box.

[0042] After obtaining the 3D point cloud object detection box, projecting the 3D point cloud object detection box onto a 2D plane will yield the corresponding 2D point cloud object detection box.

[0043] S4. Match the object detection boxes in the 2D image and the object detection boxes in the 2D point cloud to obtain at least one object matching pair.

[0044] Matching the object detection boxes of the two-dimensional point cloud obtained by projection with the object detection boxes of the two-dimensional image can yield at least one object matching pair.

[0045] Understandably, the number of 3D point cloud object detection boxes can be one or more, resulting in the number of corresponding 2D point cloud object detection boxes being one or more, and the number of 2D image object detection boxes can also be one or more. Thus, matching one or more 2D image object detection boxes with one or more 2D point cloud object detection boxes can yield at least one object matching pair.

[0046] S5. Process the corresponding two-dimensional image of the object detection box in at least one object matching pair to obtain a transformed image.

[0047] Each of the at least one object matching pairs includes a two-dimensional image object detection box. The two-dimensional images corresponding to the two-dimensional image object detection boxes in the at least one object matching pair are processed to obtain the transformed image in each of the at least one object matching pair.

[0048] S6. Project the corresponding point cloud of the two-dimensional point cloud object detection box in at least one object matching pair onto the transformed image to obtain the projection result of the point cloud in the transformed image.

[0049] Each object matching pair in at least one object matching pair includes a two-dimensional point cloud object detection box. The point cloud corresponding to the two-dimensional point cloud object detection box in each object matching pair is projected onto the corresponding transformed image in the object matching pair to obtain the projection result of the point cloud in the transformed image.

[0050] S7. Based on the projection results, perform extrinsic parameter optimization to obtain the optimization results.

[0051] By performing extrinsic parameter optimization operations on the projection results obtained from point cloud projection, the optimized results can be obtained.

[0052] S8. In response to the optimization results meeting the preset optimization conditions, determine the target extrinsic parameters.

[0053] The optimized result is compared with the preset optimization conditions. If the optimized result meets the preset optimization conditions, the corresponding extrinsic parameter of the optimized result is determined as the target extrinsic parameter. The preset optimization conditions can be set according to actual usage requirements and are not specifically limited.

[0054] The above-mentioned scheme obtains two-dimensional object detection boxes from the original image of the current frame and three-dimensional point cloud object detection boxes from the original point cloud of the current frame. The three-dimensional point cloud object detection boxes are projected onto a two-dimensional plane to obtain corresponding two-dimensional point cloud object detection boxes. The two-dimensional image object detection boxes and the two-dimensional point cloud object detection boxes are matched to obtain at least one object matching pair. The corresponding two-dimensional images of the two-dimensional image object detection boxes in the at least one object matching pair are processed to obtain a transformed image. The corresponding point clouds of the two-dimensional point cloud object detection boxes in the at least one object matching pair are projected onto the transformed image to obtain a projection result. Extrinsic parameter optimization is performed based on the projection result to obtain an optimization result. The target extrinsic parameters are determined in response to the optimization result satisfying preset optimization conditions. The scheme in this application extracts object information by performing two-dimensional perception on the original image and three-dimensional perception on the original point cloud. This method is time-efficient and has strong real-time performance. Furthermore, by performing extrinsic parameter optimization through at least one object matching pair, it can effectively avoid local optimization.

[0055] As described above, a two-dimensional image object detection box is obtained from the original image of the current frame, and a three-dimensional point cloud object detection box is obtained from the original point cloud of the current frame. In one embodiment of this application, projecting the three-dimensional point cloud object detection box onto a two-dimensional plane to obtain a corresponding two-dimensional point cloud object detection box includes: projecting the three-dimensional point cloud object detection box onto the coordinate system where the two-dimensional image object detection box is located to obtain a projection box; obtaining the vertices at the maximum and minimum values ​​along the horizontal axis and the vertical axis of the projection box in the coordinate system, thereby obtaining the two-dimensional point cloud object detection box.

[0056] Understandably, a 3D point cloud object detection box consists of eight vertices. After projecting the 3D point cloud object detection box onto the coordinate system of the 2D image object detection box, the eight vertices may overlap. Therefore, the number of vertices in the resulting projection box may be four, five, or other numbers. We obtain the vertices at the maximum and minimum values ​​along the horizontal axis of the projection box in the coordinate system, and also obtain the vertices at the maximum and minimum values ​​along the vertical axis of the projection box. Based on these four vertices, we obtain the 2D point cloud object detection box.

[0057] As described above, the 3D point cloud object detection box is projected onto a 2D plane to obtain the corresponding 2D point cloud object detection box. In one embodiment of this application, matching the 2D image object detection box and the 2D point cloud object detection box to obtain at least one object matching pair includes: using a preset matching algorithm to calculate a preset semantic distance between the 2D image object detection box and the 2D point cloud object detection box to obtain a distance value matrix; and performing a matching operation on the distance value matrix to obtain at least one object matching pair.

[0058] By using a preset matching algorithm, a preset semantic distance is calculated between the object detection boxes in the 2D image and the object detection boxes in the 2D point cloud, resulting in a distance value matrix. A matching operation is then performed on the distance value matrix to obtain at least one object matching pair.

[0059] For example, the preset matching algorithm could be the Hungarian matching algorithm. Given m bounding boxes in the 2D image and n bounding boxes in the 2D point cloud, the Hungarian matching algorithm is used to calculate the preset semantic distances between each pair of the m bounding boxes in the 2D image and the n bounding boxes in the 2D point cloud. The resulting distance values ​​form an m×n distance value matrix. The Hungarian matching algorithm is then used to perform a matching operation on this distance value matrix to obtain at least one object matching pair.

[0060] In other embodiments, the preset matching algorithm can also be other feasible matching algorithms, which can be selected according to actual usage requirements, without specific limitations.

[0061] As described above, a preset semantic distance between the object detection boxes in the 2D image and the object detection boxes in the 2D point cloud is calculated using a preset matching algorithm to obtain a distance value matrix. In one embodiment of this application, calculating the preset semantic distance between the object detection boxes in the 2D image and the object detection boxes in the 2D point cloud includes: calculating the intersection-union ratio (IUGR) between the object detection boxes in the 2D image and the object detection boxes in the 2D point cloud.

[0062] Understandably, the Intersection of Union (LOU) ratio between the object detection boxes in the 2D image and the object detection boxes in the 2D point cloud is calculated to achieve the calculation of the preset semantic distance between the two-dimensional object detection boxes and the 2D point cloud object detection boxes. The LLOU ratio is represented as the ratio of the intersection to the union of the two-dimensional object detection boxes.

[0063] For example, the intersection-union ratio (IoU) between the object detection box A in the 2D image and the object detection box B in the 2D point cloud is (A∩B) / (A∪B).

[0064] As described above, a preset semantic distance between the object detection boxes in the 2D image and the object detection boxes in the 2D point cloud is calculated using a preset matching algorithm to obtain a distance value matrix. In one embodiment of this application, calculating the preset semantic distance between the object detection boxes in the 2D image and the object detection boxes in the 2D point cloud includes: calculating the intersection-union ratio (IUU) between the object detection boxes in the 2D image and the object detection boxes in the 2D point cloud, and calculating the area ratio between the object detection boxes in the 2D image and the object detection boxes in the 2D point cloud; wherein, when calculating the area ratio, the object detection box with the larger area is the denominator, and the object detection box with the smaller area is the numerator; the calculation operation is performed using a preset semantic distance formula, wherein the preset semantic distance formula is:

[0065] D = t * IoU + (1 - t) * d area

[0066] Where t is the weight, t∈[0,1], IoU is the intersection-union ratio, and d area This represents the area ratio.

[0067] Understandably, a preset semantic distance formula is used to calculate the preset semantic distance between the object detection boxes in the 2D image and the 2D point cloud. This preset semantic distance formula represents a weighted calculation of the intersection-union ratio (IU) and the area ratio. The IU is represented as the ratio of the intersection to the union of the two object detection boxes. The area ratio is represented as the ratio of the smaller area of ​​one object detection box to the larger area of ​​the other.

[0068] For example, calculate the area values ​​d1 and d2 of the object detection boxes in the 2D image and the 2D point cloud respectively, and compare the two area values ​​d1 and d2. Since the area value d2 is smaller than the area value d1, use the area value d2 as the numerator and the area value d1 as the denominator to obtain the area ratio d between the object detection boxes in the 2D image and the 2D point cloud. area =d2 / d1.

[0069] In one embodiment of this application, the two-dimensional image corresponding to the object detection box in at least one object matching pair is processed to obtain a transformed image, including: binarizing the two-dimensional image corresponding to the object detection box in at least one object matching pair to obtain a binarized image; and performing inverse distance transform calculation on the binarized image to obtain the transformed image.

[0070] For each two-dimensional image object detection box in at least one object matching pair, the corresponding two-dimensional image of the two-dimensional image object detection box is binarized to obtain a corresponding binarized image. The inverse distance transform is then performed on the obtained binarized image to obtain the corresponding transformed image.

[0071] As described above, the point cloud corresponding to the object detection box in at least one object matching pair is projected onto the transformed image to obtain the projection result of the point cloud in the transformed image. In one embodiment of this application, the projection result includes the pixel value of a pixel in the transformed image corresponding to each point in the point cloud; based on the projection result, an extrinsic parameter optimization operation is performed to obtain the optimization result, including: performing optimization calculation using a preset cost function to obtain the optimization result; wherein, the preset cost function is:

[0072]

[0073] Where N is the number of points in the point cloud, and the pixel value of the i-th pixel is p. i .

[0074] Understandably, the projection result obtained by projecting the corresponding point cloud of the object detection box in the 2D point cloud of the object matching pair onto the transformed image is that each point in the point cloud corresponds to the pixel value of a pixel in the transformed image. Optimization calculation is performed using a preset cost function, that is, the pixel values ​​corresponding to each point in the point cloud are summed. The optimization calculation can be nonlinear and is not specifically limited.

[0075] For example, if the number of points N in the point cloud is 3, in the projection result, the 3 points correspond to the pixel values ​​of 3 pixels. The pixel value of the first pixel is p1, the pixel value of the second pixel is p2, and the pixel value of the third pixel is p3, thus optimizing the result e = p1 + p2 + p3.

[0076] In response to the optimization result satisfying the preset optimization conditions, the target extrinsic parameters are determined. The preset optimization conditions can be that the optimization result e = 0, indicating that the object information in the 2D point cloud object detection box completely matches the object information in the 2D image object detection box. The corresponding extrinsic parameters are then determined as the target extrinsic parameters, thereby achieving extrinsic parameter calibration.

[0077] In other embodiments, the preset optimization condition can also be other achievable situations, such as the optimization result e being less than a preset value. The preset value can be selected according to actual usage requirements and is not specifically limited.

[0078] In one embodiment of this application, the extrinsic parameter calibration method for radar and camera further includes: in response to the optimization result not meeting the preset optimization conditions, performing a search calculation based on the preset extrinsic parameter range and according to the preset step size using a preset cost function to obtain the calculation result; and in response to the calculation result meeting the preset search conditions, determining the target extrinsic parameters.

[0079] If the optimization result does not meet the preset optimization conditions, which can be interpreted as optimization failure, then based on the preset extrinsic parameter range, according to the preset step size, the preset cost function is used to perform search calculation to obtain the calculation result.

[0080] For example, the preset extrinsic parameter range can be -2° to 2°, and the step size can be 0.1°. The preset cost function is used to perform search calculations to obtain the calculated value corresponding to each step size within the range of -2° to 2°, thereby obtaining the calculation result.

[0081] If any calculated value in the calculation result satisfies a preset search condition, then the extrinsic parameter corresponding to that calculated value is determined as the target extrinsic parameter. The preset search condition can be that the calculated value in the calculation result is less than a preset value. The preset value can be selected according to actual usage requirements and is not specifically limited.

[0082] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0083] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device in an embodiment of this application. The electronic device 200 includes a memory 201 and a processor 202 coupled to each other. The processor 202 executes program instructions stored in the memory 201 to implement the steps in the above-described embodiments of the radar and camera extrinsic parameter calibration method. In a specific implementation scenario, the electronic device 200 may include, but is not limited to, a microcomputer or a server.

[0084] Specifically, processor 202 controls itself and memory 201 to implement the steps in the radar and camera extrinsic parameter calibration method embodiments described above. Processor 202 can also be called a CPU (Central Processing Unit), and may be an integrated circuit chip with signal processing capabilities. Processor 202 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 202 can be implemented using integrated circuit chips.

[0085] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a non-volatile computer-readable storage medium in an embodiment of this application. The computer-readable storage medium 300 is used to store program instructions 301, which, when executed by the processor 202, are used to implement the steps in the above-described embodiment of the radar and camera extrinsic parameter calibration method.

[0086] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed methods and related devices can be implemented in other ways. For example, the related device implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication disconnection shown or discussed may be indirect coupling or communication disconnection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0088] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0089] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for calibrating extrinsic parameters of a radar and a camera, characterized in that, include: Obtain the original image and original point cloud of the current frame; Two-dimensional object detection boxes are obtained from the original image of the current frame, and three-dimensional point cloud object detection boxes are obtained from the original point cloud of the current frame. The three-dimensional point cloud object detection box is projected onto a two-dimensional plane to obtain the corresponding two-dimensional point cloud object detection box. The two-dimensional image object detection box and the two-dimensional point cloud object detection box are matched to obtain at least one object matching pair; The corresponding two-dimensional image of the object detection box in the at least one object matching pair is processed to obtain a transformed image; The point cloud corresponding to the two-dimensional point cloud object detection box in the at least one object matching pair is projected onto the transformed image to obtain the projection result of the point cloud in the transformed image. Based on the projection results, extrinsic parameter optimization is performed to obtain the optimization results; In response to the optimization result satisfying the preset optimization conditions, the target extrinsic parameters are determined; The step of processing the two-dimensional image corresponding to the object detection box in the at least one object matching pair to obtain a transformed image includes: The corresponding two-dimensional image of the object detection box in the at least one object matching pair is binarized to obtain a binarized image; Perform inverse distance transform calculation on the binarized image to obtain the transformed image; The projection result includes the pixel value of a pixel in the transformed image corresponding to each point in the point cloud; The step of performing extrinsic parameter optimization based on the projection results to obtain the optimization results includes: The optimization result is obtained by performing optimization calculations using a preset cost function; wherein the preset cost function is: Where N is the number of points in the point cloud, and the pixel value of the i-th pixel is pi.

2. The method of claim 1, wherein, The step of projecting the 3D point cloud object detection box onto a 2D plane to obtain a corresponding 2D point cloud object detection box includes: The 3D point cloud object detection box is projected onto the coordinate system of the 2D image object detection box to obtain the projection box; The vertex at the maximum and minimum values ​​along the horizontal axis and the vertex at the minimum and maximum values ​​along the vertical axis of the projection frame in the coordinate system are obtained to obtain the two-dimensional point cloud object detection frame.

3. The method of claim 1, wherein, The step of matching the object detection bounding box in the two-dimensional image and the object detection bounding box in the two-dimensional point cloud to obtain at least one object matching pair includes: Using a preset matching algorithm, a preset semantic distance is calculated between the two-dimensional image object detection box and the two-dimensional point cloud object detection box to obtain a distance value matrix; A matching operation is performed on the distance value matrix to obtain at least one object matching pair.

4. The method of claim 3, wherein, The calculation of the preset semantic distance between the two-dimensional image object detection box and the two-dimensional point cloud object detection box includes: Calculate the intersection-union ratio (IUGR) between the object detection bounding box in the 2D image and the object detection bounding box in the 2D point cloud.

5. The method of claim 3, wherein, The calculation of the preset semantic distance between the two-dimensional image object detection box and the two-dimensional point cloud object detection box includes: Calculate the intersection-union ratio between the object detection boxes in the two-dimensional image and the object detection boxes in the two-dimensional point cloud, and calculate the area ratio between the object detection boxes in the two-dimensional image and the object detection boxes in the two-dimensional point cloud; wherein, when calculating the area ratio, the object detection box with the larger area is the denominator, and the object detection box with the smaller area is the numerator. The calculation is performed using a preset semantic distance formula, wherein the preset semantic distance formula is: D = t IoU + (1 - t) d area Where t is the weight, t∈[0,1], IoU is the intersection-union ratio, and darea is the area ratio.

6. The method according to any one of claims 1 to 5, characterized in that, The external parameter calibration method for the radar and camera further includes: In response to the optimization result not meeting the preset optimization conditions, a search calculation is performed using the preset cost function based on the preset extrinsic parameter range and according to the preset step size to obtain the calculation result; In response to the calculation result satisfying the preset search conditions, the target extrinsic parameters are determined.

7. An electronic device, comprising: It includes a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the extrinsic parameter calibration method for radar and camera as described in any one of claims 1-6.

8. A non-transitory computer readable storage medium, comprising: The computer-readable storage medium is used to store program instructions, which, when executed by a processor, are used to implement the external parameter calibration method for radar and camera as described in any one of claims 1-6.

Images