Radar camera calibration method, system, equipment, medium and product

Abstract

The invention relates to the technical field of camera calibration, and discloses a radar camera calibration method, system, device, medium and product, and the method comprises the steps: inputting color image data obtained by a camera into a pre-trained neural network, outputting a depth image, carrying out the projection of point cloud data obtained by a laser radar, and carrying out the depth assignment, thereby achieving the calibration of a radar camera. The method comprises the steps of obtaining a point cloud depth image consistent with a depth image in size, fusing and splicing the depth image and the point cloud depth image, respectively inputting a depth fusion image into two full-connection neural network layers to obtain a translation vector and a rotation vector, determining an external parameter matrix of a to-be-calibrated camera according to the translation vector and the rotation vector, and calibrating the to-be-calibrated camera according to the external parameter matrix. According to the method, a traditional calibration tool is replaced, the operation convenience and flexibility of calibration are improved, the calibration efficiency is improved, and the application range is expanded.

Description

Technical Field

[0001] The present invention relates to the technical field of camera calibration, and particularly to a radar-camera calibration method, system, device, medium, and product. Background Art

[0002] For multi-sensor devices, determining the position and attitude relationship between different sensors is the basis for fusing multi-sensor data. The need for calibrating the position and attitude of radar and camera devices has also emerged along with various multi-sensor devices incorporating radar and cameras. It is difficult to use traditional methods to automatically calculate corresponding points for attitude calculation for different types of sensors. Therefore, in traditional methods, some auxiliary means such as calibration plates are generally required, and manual annotation is combined to determine the corresponding points of different sensor data to calculate the position and attitude relationship, but these methods are cumbersome in data acquisition and replication operations.

[0003] In multi-sensor devices, especially in the combined application of radar and cameras, accurately determining the position and attitude relationship between different sensors is a key step in realizing data fusion. However, traditional calibration methods have many deficiencies, which are specifically manifested in the following aspects:

[0004] Complex data acquisition: To obtain sufficient information to calculate the relative position and attitude between sensors, it is usually necessary to use special calibration plates or other auxiliary tools to collect data. This not only increases the time cost in the experimental preparation stage but also makes the whole process more complex.

[0005] Complicated calibration operation: The process of manually annotating corresponding points is very time-consuming and error-prone. For non-professionals, it is difficult to ensure that each measurement can meet the high-precision requirements. In addition, when environmental conditions change (such as light intensity change), the calibration work may need to be redone, further increasing the workload.

[0006] Poor flexibility: Methods based on physical markers often work effectively only under specific scenarios or conditions. Once outside the preset range, their accuracy will be greatly reduced. Therefore, such methods lack sufficient adaptability and generality.

[0007] Low efficiency: Due to the above-mentioned various limitations, the overall efficiency of traditional methods is low and it is difficult to meet the requirements for speed in modern industrial automated production lines. Especially in the case of large-scale deployment, this problem of low efficiency is particularly prominent.

[0008] In summary, although some existing solutions can solve the calibration problem between radar and cameras to a certain extent, they generally have problems such as inconvenient operation, poor flexibility, low efficiency, and limited application scope. Summary of the Invention

[0009] In view of this, the present invention provides a radar camera calibration method, system, device, medium and product, which solves the technical problems of inconvenient operation, poor flexibility, low efficiency and limited application range of traditional calibration methods.

[0010] In the first aspect of the present invention, a radar camera calibration method is provided, including:

[0011] Obtain color image data of the target space, and obtain point cloud data of the target space;

[0012] Input the color image data into a pre-trained neural network to output a depth image;

[0013] Perform point cloud projection of the point cloud data onto a two-dimensional image plane to obtain a point cloud projection image, and perform depth assignment on the point cloud projection image to obtain a point cloud depth image with the same size as the depth image;

[0014] Fuse and splice the depth image and the point cloud depth image to obtain a depth fusion image;

[0015] Input the depth fusion image into two fully connected neural network layers respectively to obtain a translation vector and a rotation vector;

[0016] Determine the external parameter matrix of the camera to be calibrated according to the translation vector and the rotation vector, and calibrate the camera to be calibrated through the external parameter matrix.

[0017] Preferably, the performing point cloud projection of the point cloud data onto a two-dimensional image plane to obtain a point cloud projection image, and performing depth assignment on the point cloud projection image to obtain a point cloud depth image with the same size as the depth image includes:

[0018] Determine an estimated external parameter matrix according to the relative position relationship between the lidar and the camera to be calibrated;

[0019] Project the point cloud data onto the two-dimensional image plane according to the estimated external parameter matrix and a preset internal parameter matrix to obtain the point cloud projection image;

[0020] Assign depth values to each pixel point in the point cloud projection image to form a point cloud depth image with the same size as the depth image.

[0021] Preferably, the method further includes:

[0022] Back-project the point cloud projection image according to the external parameter matrix and a preset internal parameter matrix to obtain back-projected point cloud data;

[0023] Determine a depth loss value according to the depth difference between the point cloud data and the back-projected point cloud data;

[0024] Determine whether the depth loss value is greater than a preset depth loss threshold;

[0025] If it is determined that the depth loss value is greater than the preset depth loss threshold, then re - go to the step of projecting the point cloud data onto a two - dimensional image plane to obtain a point cloud projection image, and performing depth assignment on the point cloud projection image to obtain a point cloud depth image with the same size as the depth image, until the depth loss value is not greater than the preset depth loss threshold, and output the external parameter matrix.

[0026] Preferably, the method further includes:

[0027] Back - project the point cloud projection image according to the external parameter matrix and a preset internal parameter matrix to obtain back - projected point cloud data;

[0028] Determine a point cloud loss value according to the chamfer distance between the point cloud data and the back - projected point cloud data;

[0029] Determine whether the point cloud loss value is greater than a preset point cloud loss threshold;

[0030] If it is determined that the point cloud loss value is greater than the preset point cloud loss threshold, then re - go to the step of projecting the point cloud data onto a two - dimensional image plane to obtain a point cloud projection image, and performing depth assignment on the point cloud projection image to obtain a point cloud depth image with the same size as the depth image, until the point cloud loss value is not greater than the preset point cloud loss threshold, and output the external parameter matrix.

[0031] Preferably, the method further includes:

[0032] Perform depth reset on the point cloud depth image, and the depth reset includes noise elimination and error correction.

[0033] Preferably, before the step of inputting the depth fusion image into two fully - connected neural network layers to obtain a translation vector and a rotation vector, it further includes:

[0034] Perform normalization processing on the depth fusion image.

[0035] In a second aspect, the present invention provides a radar - camera calibration system, including:

[0036] A data acquisition module, configured to acquire color image data of a target space and acquire point cloud data of the target space;

[0037] An image output module, configured to input the color image data into a pre - trained neural network and output a depth image;

[0038] A depth image module, configured to project the point cloud data onto a two-dimensional image plane to obtain a point cloud projection image, and perform depth assignment on the point cloud projection image to obtain a point cloud depth image having the same size as the depth image;

[0039] A fusion and stitching module, configured to fuse and stitch the depth image and the point cloud depth image to obtain a depth fusion image;

[0040] A vector determination module, configured to input the depth fusion image into two fully connected neural network layers respectively to obtain a translation vector and a rotation vector;

[0041] A camera calibration module, configured to determine an external parameter matrix of the camera to be calibrated according to the translation vector and the rotation vector, and calibrate the camera to be calibrated through the external parameter matrix.

[0042] In a third aspect, the present invention provides an electronic device, which includes a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of the radar camera calibration method as described in the first aspect.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed, the steps of the radar camera calibration method as described in the first aspect are implemented.

[0044] In a fifth aspect, the present invention provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the steps of the radar camera calibration method as described in the first aspect.

[0045] As can be seen from the above technical solutions, the present invention inputs the color image data acquired by a camera into a pre-trained neural network to output a depth image, projects the point cloud data acquired by a lidar, and performs depth assignment to obtain a point cloud depth image having the same size as the depth image. The depth image and the point cloud depth image are fused and stitched, and the depth fusion image is input into two fully connected neural network layers respectively to obtain a translation vector and a rotation vector. According to the translation vector and the rotation vector, an external parameter matrix of the camera to be calibrated is determined, so as to calibrate the camera to be calibrated, replacing traditional calibration tools, improving the operation convenience and flexibility of calibration, as well as improving the calibration efficiency and expanding the applicable range. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0047] Figure 1 The application environment of a radar camera calibration method provided by an embodiment of the present invention;

[0048] Figure 2 The flowchart of a radar camera calibration method provided by an embodiment of the present invention;

[0049] Figure 3 The logic block diagram of a radar camera calibration method provided by an embodiment of the present invention;

[0050] Figure 4 The structural schematic diagram of a radar camera calibration system provided by an embodiment of the present invention;

[0051] Figure 5 The structural schematic diagram of an electronic device provided by an embodiment of the present invention. Specific embodiments

[0052] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0053] The radar camera calibration method provided by the embodiments of the present application can be applied to, for example Figure 1In the application environment shown. Among them, the terminal 101 communicates with the server 102 through the network. The data storage system can store the data that the server 102 needs to process. The data storage system can be integrated on the server 102, or can be placed on the cloud or other network servers. The terminal 101 or the server 102 acquires the color image data of the target space, and acquires the point cloud data of the target space; inputs the color image data into a pre-trained neural network to output a depth image; projects the point cloud data onto a two-dimensional image plane to obtain a point cloud projection image, and performs depth assignment on the point cloud projection image to obtain a point cloud depth image with the same size as the depth image; fuses and stitches the depth image and the point cloud depth image to obtain a depth fusion image; inputs the depth fusion image into two fully connected neural network layers respectively to obtain a translation vector and a rotation vector; determines the external parameter matrix of the camera to be calibrated according to the translation vector and the rotation vector, and calibrates the camera to be calibrated through the external parameter matrix.

[0054] The terminal 101 can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, etc.

[0055] The server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0056] Such as Figure 2 As shown, an embodiment of the present application provides a radar camera calibration method. Taking the method applied to the terminal 101 or the server 102 in Figure 1 as an example, it includes the following steps S1 to step S6. Among them:

[0057] Step S1, acquire the color image data of the target space, and acquire the point cloud data of the target space.

[0058] Among them, the color image data of the target space is captured by a camera, and the color image data includes the texture and color information of the target space.

[0059] The point cloud data of the target space is acquired by a lidar.

[0060] Step S2, input the color image data into a pre-trained neural network to output a depth image.

[0061] Among them, a suitable neural network architecture (such as CNN, ResNet, PointNet, etc.) can be selected to train historical training samples, where the historical training samples include color image data and depth images, so as to extract the depth features of the color image data and output a depth image, which contains the distance information from each pixel point in the target space to the camera. The generation of the depth image is achieved by performing depth estimation on each pixel point in the color image data, and the accuracy of the depth estimation depends on the training effect of the neural network and the richness of the dataset.

[0062] Step S3: Project the point cloud data onto a two-dimensional image plane to obtain a point cloud projection image, and perform depth assignment on the point cloud projection image to obtain a point cloud depth image with the same size as the depth image.

[0063] Among them, in the process of processing the point cloud data, first, the point cloud data in these three-dimensional spaces needs to be mapped onto a two-dimensional image plane through a specific projection method to generate a point cloud projection image. This process involves complex geometric transformations and calculations, aiming to convert the point cloud information in the three-dimensional space into the pixel point representation on the two-dimensional image. After the projection is completed, the next step is to perform depth assignment on each pixel point in the projected two-dimensional image. The depth assignment is determined according to the distance from each point in the point cloud data to the observation point, so as to ensure that each pixel point carries the corresponding depth information. Finally, through this series of processes, a point cloud depth image with exactly the same size as the original depth image can be obtained. This image not only contains the two-dimensional projection information of the point cloud but also contains the depth information of each point, providing important data support for subsequent image processing and analysis.

[0064] Step S4: Fuse and splice the depth image and the point cloud depth image to obtain a depth fusion image.

[0065] Among them, by splicing different modalities of images, such as RGB images and depth images, and depth images from different network branches, on the channel dimension, the color information and spatial information can be effectively fused. This fusion operation forms a multi-channel input data, such as 3840-dimensional feature data, which is then provided to the subsequent processing layer. Such a processing method helps to enhance the robustness of feature expression, thereby improving the performance and accuracy of the entire system.

[0066] Step S5: Input the depth fusion image into two fully connected neural network layers respectively to obtain a translation vector and a rotation vector.

[0067] Among them, the translation vector represents the position of the camera in the world coordinate system. The rotation vector represents the orientation of the camera coordinate system relative to the world coordinate system. At the same time, it is converted into a rotation matrix according to the rotation vector (such as through the Rodriguez formula).

[0068] Exemplarily, the 3840-dimensional features enter two different fully connected neural network layers in two paths respectively, and the corresponding translation vector r(t1, t2, t3) and rotation vector r(r1, r2, r3) are obtained through inference.

[0069] Step S6: Determine the external parameter matrix of the camera to be calibrated according to the translation vector and the rotation vector, and calibrate the camera to be calibrated through the external parameter matrix.

[0070] Among them, it is converted into a rotation matrix according to the rotation vector, and the rotation matrix and the translation vector are combined to form the external parameter matrix of the camera to be calibrated. The external parameter matrix is:

[0071]

[0072] In the formula, T is the external parameter matrix, t is the translation vector, and R is the rotation matrix.

[0073] It should be noted that in the embodiment of the present application, the color image data acquired by the camera is input into a pre-trained neural network to output a depth image, and the point cloud data acquired by the lidar is projected and depth-assigned to obtain a point cloud depth image with the same size as the depth image. The depth image and the point cloud depth image are fused and stitched, and the depth fusion image is input into two fully connected neural network layers respectively to obtain a translation vector and a rotation vector. According to the translation vector and the rotation vector, the external parameter matrix of the camera to be calibrated is determined, so as to calibrate the camera to be calibrated, replacing the traditional calibration tool, improving the operation convenience and flexibility of calibration, as well as improving the calibration efficiency and expanding the applicable range.

[0074] In some embodiments, the point cloud data is projected onto a two-dimensional image plane to obtain a point cloud projection image, and the point cloud projection image is depth-assigned to obtain a point cloud depth image with the same size as the depth image, including:

[0075] Step S301: Determine the estimated external parameter matrix according to the relative position relationship between the lidar and the camera to be calibrated.

[0076] Among them, in order to estimate the external parameter matrix between the camera and the lidar, the relative position and direction between the two sensors can be clarified, so as to determine the estimated external parameter matrix. This estimated external parameter matrix provides an initial conversion relationship for subsequent point cloud projection and depth assignment.

[0077] Step S302: Project the point cloud data onto a two-dimensional image plane according to the estimated external parameter matrix and the preset internal parameter matrix to obtain a point cloud projection image.

[0078] Among them, the preset internal parameter matrix can be pre-calibrated by other technical means. According to the described estimated external parameter matrix and the preset internal parameter matrix, perform a point cloud projection operation on the point cloud data, so as to map and project the point cloud data in these three-dimensional spaces onto the two-dimensional image plane to obtain a point cloud projection image.

[0079] Step S303: Assign depth values to each pixel point in the point cloud projection image to form a point cloud depth image with the same size as the depth image.

[0080] Among them, in the process of processing the point cloud projection image, a corresponding depth value is assigned to each pixel point in the image. In this way, a point cloud depth image with exactly the same size as the original depth image is constructed.

[0081] In some embodiments, the method further includes:

[0082] Step S701: Back-project the point cloud projection image according to the external parameter matrix and the preset internal parameter matrix to obtain back-projected point cloud data;

[0083] Step S702: Determine the depth loss value according to the depth difference between the point cloud data and the back-projected point cloud data;

[0084] Step S703: Determine whether the depth loss value is greater than the preset depth loss threshold;

[0085] Step S704: If it is determined that the depth loss value is greater than the preset depth loss threshold, then re-turn to the step of projecting the point cloud data onto the two-dimensional image plane to obtain a point cloud projection image, and performing depth assignment on the point cloud projection image to obtain a point cloud depth image with the same size as the depth image, until the depth loss value is not greater than the preset depth loss threshold, and output the external parameter matrix.

[0086] Among them, through the external parameter matrix, the back-projected point cloud is obtained by back-projecting the known camera internal parameter matrix, and the depth loss function is used to compare the original point cloud with the back-projected point cloud to obtain the corresponding depth loss value. Among them, the depth loss function is:

[0087]

[0088] In the formula, is the depth loss value, is the number of point clouds, is the depth of the nth back-projected point cloud, is the depth of the nth original point cloud.

[0089] In some embodiments, the method further includes:

[0090] Step S711: Back-project the point cloud projection image according to the external parameter matrix and the preset internal parameter matrix to obtain back-projected point cloud data;

[0091] Step S712: Determine the point cloud loss value according to the chamfer distance between the point cloud data and the back-projected point cloud data;

[0092] Step S713: Determine whether the point cloud loss value is greater than a preset point cloud loss threshold;

[0093] Step S714: If it is determined that the point cloud loss value is greater than the preset point cloud loss threshold, then re-turn to the step of projecting the point cloud data onto the two-dimensional image plane to obtain a point cloud projection image, and performing depth assignment on the point cloud projection image to obtain a point cloud depth image with the same size as the depth image, until the point cloud loss value is not greater than the preset point cloud loss threshold, and output the external parameter matrix.

[0094] Among them, the chamfer distance between the point cloud data and the back-projected point cloud data is determined by a point cloud chamfer distance loss function, where the point cloud chamfer distance loss function is:

[0095]

[0096] In the formula, is the point cloud chamfer distance, is the original point cloud data, is the back-projected point cloud data, is the original point cloud serial number, is the back-projected point cloud serial number.

[0097] In some embodiments, the method further includes:

[0098] Performing depth reset on the point cloud depth image, and the depth reset includes noise elimination and error correction.

[0099] Among them, the depth residual is predicted by a neural network to perform end-to-end optimization of errors.

[0100] In some embodiments, before the depth fusion image is respectively input into two fully connected neural network layers to obtain a translation vector and a rotation vector, it further includes:

[0101] Performing normalization processing on the depth fusion image.

[0102] Among them, the depth fusion image is standardized (with a mean of 0 and a variance of 1) to accelerate training and improve the generalization ability of the model.

[0103] The following provides an exemplary description of the radar camera calibration method for a clearer elaboration of the radar camera calibration method.

[0104] In this example, as Figure 3 shown, before the start of training, the camera internal parameter matrix can be pre-calibrated through other technical means. At the same time, based on the specified relationship between the actual positions of the camera and the radar and the camera coordinates and radar coordinates, an estimated external parameter matrix can be obtained.

[0105] Initialize parameters;

[0106] Both the pre-processing and post-processing of the data rely on the internal and external parameter matrices. Since the randomly initialized external parameter matrix may cause the point cloud to be completely unable to be projected onto the image, it is necessary to estimate an external parameter matrix and read it into the program during the initialization stage;

[0107] Feed in the data and perform inference calculations;

[0108] Calculate the loss value;

[0109] In the loss calculation stage, through inference calculations, a new external parameter matrix has been obtained. Therefore, both the projection calculation of the point cloud and the back-projection of the depth image are based on the new external parameter matrix to calculate the network loss;

[0110] Update the network parameters and the external parameter matrix;

[0111] Update the network parameters through backpropagation and overwrite the previous external parameter matrix with the new external parameter matrix;

[0112] The training converges;

[0113] Judge the convergence situation of the model. If it has not converged, re-feed in the data and perform inference calculations.

[0114] Parameter calibration:

[0115] After training the model, the parameters of the new device can be calibrated through the above network, and complex manual data annotation is not required.

[0116] Use the device to collect a set of data, including images and point clouds, and input them into the above-trained model network.

[0117] Feed in the data:

[0118] Send the image into the residual network, and obtain the depth estimation feature map through the residual network;

[0119] Send the point cloud data into the point-to-point depth map node, and obtain a depth map with the same size as the input image through point cloud projection;

[0120] The two depth feature maps pass through the connection layer to obtain a spliced feature map;

[0121] The spliced feature map undergoes batch normalization calculation to obtain 3,840 - dimensional features;

[0122] The 3,840 - dimensional features enter two different fully - connected neural network layers in two paths respectively, and the corresponding translation vector r(t1, t2, t3) and rotation vector r(r1, r2, r3) are deduced;

[0123] The rotation vector can be used to calculate the rotation matrix R, and after combination, the external parameter matrix can be obtained as:

[0124] 。

[0125] Based on the same inventive concept, the embodiment of the present application also provides a radar - camera calibration system for implementing the radar - camera calibration method involved above.

[0126] The implementation solution provided by this system to solve the problem is similar to the implementation solution recorded in the above - mentioned method. Therefore, the specific limitations in one or more embodiments of the radar - camera calibration system provided below can refer to the limitations on the radar - camera calibration method in the above text, and will not be repeated here.

[0127] As Figure 4 shown, the embodiment of the present application provides a radar - camera calibration system, including:

[0128] A data acquisition module 100, configured to acquire color image data of the target space and acquire point - cloud data of the target space;

[0129] An image output module 200, configured to input the color image data into a pre - trained neural network and output a depth image;

[0130] A depth - image module 300, configured to project the point - cloud data onto a two - dimensional image plane to obtain a point - cloud projection image, and perform depth assignment on the point - cloud projection image to obtain a point - cloud depth image with the same size as the depth image;

[0131] A fusion and splicing module 400, configured to fuse and splice the depth image and the point - cloud depth image to obtain a depth - fused image;

[0132] A vector determination module 500, configured to input the depth - fused image into two fully - connected neural network layers respectively to obtain a translation vector and a rotation vector;

[0133] A camera calibration module 600, configured to determine the external parameter matrix of the camera to be calibrated according to the translation vector and the rotation vector, and calibrate the camera to be calibrated through the external parameter matrix.

[0134] In some embodiments, the depth - image module 300 is configured to:

[0135] Determine an estimated extrinsic parameter matrix based on the relative positional relationship between the lidar and the camera to be calibrated;

[0136] Project the point cloud data onto a two-dimensional image plane according to the estimated extrinsic parameter matrix and the preset intrinsic parameter matrix to obtain a point cloud projection image;

[0137] Assign depth values to each pixel point in the point cloud projection image to form a point cloud depth image with the same size as the depth image.

[0138] In some embodiments, the system further includes: a first loss calculation module, configured to:

[0139] Back-project the point cloud projection image according to the extrinsic parameter matrix and the preset intrinsic parameter matrix to obtain back-projected point cloud data;

[0140] Determine a depth loss value based on the depth difference between the point cloud data and the back-projected point cloud data;

[0141] Judge whether the depth loss value is greater than a preset depth loss threshold;

[0142] If it is judged that the depth loss value is greater than the preset depth loss threshold, then re-turn to the step of projecting the point cloud data onto a two-dimensional image plane to obtain a point cloud projection image, and performing depth assignment on the point cloud projection image to obtain a point cloud depth image with the same size as the depth image, until the depth loss value is not greater than the preset depth loss threshold, and output the extrinsic parameter matrix.

[0143] In some embodiments, the system further includes: a second loss calculation module, configured to:

[0144] Back-project the point cloud projection image according to the extrinsic parameter matrix and the preset intrinsic parameter matrix to obtain back-projected point cloud data;

[0145] Determine a point cloud loss value based on the chamfer distance between the point cloud data and the back-projected point cloud data;

[0146] Judge whether the point cloud loss value is greater than a preset point cloud loss threshold;

[0147] If it is judged that the point cloud loss value is greater than the preset point cloud loss threshold, then re-turn to the step of projecting the point cloud data onto a two-dimensional image plane to obtain a point cloud projection image, and performing depth assignment on the point cloud projection image to obtain a point cloud depth image with the same size as the depth image, until the point cloud loss value is not greater than the preset point cloud loss threshold, and output the extrinsic parameter matrix.

[0148] In some embodiments, the system further includes: a depth reset module, configured to perform depth reset on the point cloud depth image, and the depth reset includes noise elimination and error correction.

[0149] In some embodiments, the system further includes: a normalization module for normalizing the depth fusion image.

[0150] As Figure 5 shown, an embodiment of the present application provides an electronic device. The electronic device 10 includes a memory 20 and a processor 30. When a computer program stored in the memory 20 is executed by the processor 30, the processor 30 is caused to execute the steps of the radar-camera calibration method in the above embodiments.

[0151] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed, the steps of the radar-camera calibration method in the above embodiments are implemented.

[0152] An embodiment of the present application provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer is caused to execute the steps of the radar-camera calibration method in the above embodiments.

[0153] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described system, electronic device, computer storage medium, and computer program product can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0154] It should be noted that the terms "including" and "having" in the specification and claims of the present invention and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0155] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are displayed in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless clearly stated herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0156] In several embodiments provided by the present invention, it should be understood that the disclosed systems, electronic devices, computer storage media, computer program products, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0157] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0159] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. And the aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (English full name: Read-Only Memory, English abbreviation: ROM), random access memories (English full name: Random Access Memory, English abbreviation: RAM), magnetic disks, or optical discs that can store program codes.

[0160] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A radar camera calibration method, characterized in that: include: Acquiring color image data of a target space, and acquiring point cloud data of the target space; Inputting the color image data into a pre-trained neural network and outputting a depth image; Performing point cloud projection on the point cloud data to a two-dimensional image plane to obtain a point cloud projection image, and performing depth assignment on the point cloud projection image to obtain a point cloud depth image having a size consistent with the depth image; Fusing and splicing the depth image and the point cloud depth image to obtain a depth fused image; Inputting the deep fusion image into two fully connected neural network layers respectively to obtain a translation vector and a rotation vector; An extrinsic parameter matrix of the camera to be calibrated is determined according to the translation vector and the rotation vector, and the camera to be calibrated is calibrated by using the extrinsic parameter matrix.

2. The radar camera calibration method according to claim 1, characterized in that: The step of performing point cloud projection on the point cloud data to a two-dimensional image plane to obtain a point cloud projection image, and performing depth assignment on the point cloud projection image to obtain a point cloud depth image having the same size as the depth image, includes: Determine an estimated extrinsic parameter matrix according to the relative position relationship between the laser radar and the camera to be calibrated; Performing point cloud projection on the point cloud data onto the two-dimensional image plane according to the estimated external parameter matrix and the preset internal parameter matrix to obtain the point cloud projection image; A depth value is assigned to each pixel in the point cloud projection image to form a point cloud depth image having a size consistent with the depth image.

3. The radar camera calibration method according to claim 1, characterized in that: Also includes: Back-projecting the point cloud projection image according to the external parameter matrix and a preset internal parameter matrix to obtain back-projected point cloud data; Determining a depth loss value according to a depth difference between the point cloud data and the back-projected point cloud data; Determining whether the depth loss value is greater than a preset depth loss threshold; If it is determined that the depth loss value is greater than the preset depth loss threshold, the process returns to the step of projecting the point cloud data onto a two-dimensional image plane to obtain a point cloud projection image, and assigning a depth to the point cloud projection image to obtain a point cloud depth image that is consistent with the size of the depth image, until the depth loss value is no greater than the preset depth loss threshold, and the external parameter matrix is ​​output.

4. The radar camera calibration method according to claim 1, characterized in that: Also includes: Back-projecting the point cloud projection image according to the external parameter matrix and a preset internal parameter matrix to obtain back-projected point cloud data; Determining a point cloud loss value according to a chamfer distance between the point cloud data and the back-projected point cloud data; Determine whether the point cloud loss value is greater than a preset point cloud loss threshold; If it is determined that the point cloud loss value is greater than the preset point cloud loss threshold, the process returns to the step of projecting the point cloud data onto a two-dimensional image plane to obtain a point cloud projection image, and assigning a depth to the point cloud projection image to obtain a point cloud depth image that is consistent with the size of the depth image, until the point cloud loss value is no greater than the preset point cloud loss threshold, and the external parameter matrix is ​​output.

5. The radar camera calibration method according to claim 1, characterized in that: Also includes: The point cloud depth image is depth reset, and the depth reset includes noise elimination and error correction.

6. The radar camera calibration method according to claim 1, characterized in that: The deep fusion image is input into two fully connected neural network layers respectively to obtain a translation vector and a rotation vector, and the above also includes: The depth fusion image is normalized.

7. A radar camera calibration system, characterized in that: include: A data acquisition module, used to acquire color image data of a target space and to acquire point cloud data of the target space; An image output module, used to input the color image data into a pre-trained neural network and output a depth image; A depth image module is used to perform point cloud projection on the point cloud data to a two-dimensional image plane to obtain a point cloud projection image, and to perform depth assignment on the point cloud projection image to obtain a point cloud depth image having a size consistent with that of the depth image; A fusion and stitching module, used to fuse and stitch the depth image and the point cloud depth image to obtain a depth fusion image; A vector determination module, used for inputting the deep fusion image into two fully connected neural network layers respectively to obtain a translation vector and a rotation vector; The camera calibration module is used to determine the extrinsic parameter matrix of the camera to be calibrated according to the translation vector and the rotation vector, and calibrate the camera to be calibrated by using the extrinsic parameter matrix.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the radar camera calibration method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the steps of the radar camera calibration method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions, wherein when the program instructions are executed by a computer, the computer is caused to perform the steps of the radar camera calibration method according to any one of claims 1 to 6.

Images