Joint calibration method and device for camera and 4D millimeter wave radar

Abstract

The invention provides a joint calibration method and device for a camera and a 4D millimeter wave radar, and the method comprises the steps: obtaining at least six groups of calibration data through placing a calibration plate at different positions of a common view field of the 4D radar and the camera, and each group of calibration data comprises the RGB image of the camera and the point cloud data of the 4D radar; respectively acquiring three-dimensional coordinate points of the feature points of the calibration plate in a radar coordinate system and two-dimensional coordinate points of the feature points of the calibration plate in a camera pixel coordinate system from each group of calibration data; and constructing a transformation equation by using the corresponding relationship between the three-dimensional coordinate points of the feature points in the radar coordinate system and the two-dimensional coordinate points of the camera pixel coordinate system, and solving through the equation to obtain calibration parameters of the 4D radar and the camera. The calibration precision and the algorithm robustness of the 4D millimeter wave radar and the camera are improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation technology, and in particular to a joint calibration method and device for a camera and a 4D millimeter-wave radar. Background Art

[0002] Installing infrared cameras and 4D radars on roadside lighthouses can be used to collect RGB images and point cloud data at intersections, and then fuse them for target detection. Camera-4D radar fusion can achieve high-resolution all-weather and long-distance environmental perception. Camera-4D radar joint calibration is an upstream task for target tracking and positioning, 3D environment reconstruction, obstacle detection, etc. The accuracy of the results directly affects the effect of multi-sensor data fusion and collaborative processing.

[0003] Currently, there is no effective solution to the problem of joint calibration of infrared cameras and 4D radars. Summary of the invention

[0004] In view of this, the present application provides a joint calibration method and device for a camera and a 4D millimeter-wave radar to solve the above technical problems.

[0005] In a first aspect, an embodiment of the present application provides a joint calibration method of a camera and a 4D millimeter wave radar, including:

[0006] By placing the calibration plate at different positions of the common field of view of the 4D radar and the camera, at least six sets of calibration data are obtained, each set of calibration data includes the point cloud data of the 4D radar and the RGB image of the camera;

[0007] Obtain the three-dimensional coordinate points of the feature points of the calibration plate in the radar coordinate system and the two-dimensional coordinate points in the camera pixel coordinate system from each set of calibration data;

[0008] The corresponding relationship between the three-dimensional coordinates of the feature points in the radar coordinate system and the two-dimensional coordinates of the camera pixel coordinate system is used to construct the transformation equation, and the calibration parameters of the 4D radar and camera are obtained by solving the equation.

[0009] Furthermore, the side of the calibration plate facing the 4D radar and the camera is a checkerboard, and a corner reflector is arranged at the center point of the other side of the calibration plate, and the imaging point of the corner reflector is a feature point.

[0010] Furthermore, obtaining the three-dimensional coordinates of the feature points of the calibration plate in the radar coordinate system includes:

[0011] Get the reflection intensity of each point in the point cloud data of 4D radar;

[0012] The three-dimensional coordinate point of the point with the largest reflection intensity is taken as the three-dimensional coordinate point of the feature point in the radar coordinate system.

[0013] Furthermore, the two-dimensional coordinates of the feature points of the calibration plate in the camera pixel coordinate system are obtained, including:

[0014] A three-dimensional reference object coordinate system is established based on the calibration plate plane, and the calibration plate plane is set as the plane where z=0 of the reference object coordinate system;

[0015] According to the two-dimensional coordinates of multiple corner points of the chessboard on the calibration plate plane and the two-dimensional coordinates in the camera pixel coordinate system, a conversion matrix between the reference object coordinate system and the camera pixel coordinate system is established;

[0016] According to the three-dimensional coordinates of the corner reflector in the reference object coordinate system and the transformation matrix between the reference object coordinate system and the camera pixel coordinate system, the two-dimensional coordinates of the corner reflector in the camera pixel coordinate system are obtained, thereby obtaining the two-dimensional coordinates of the feature points of the calibration plate in the camera pixel coordinate system.

[0017] Furthermore, the correspondence between the three-dimensional coordinates of the feature points in the radar coordinate system and the two-dimensional coordinates of the camera pixel coordinate system is used to construct a transformation equation, and the calibration parameters of the 4D radar and camera are obtained by solving the equation; including:

[0018] Establish the transformation equation of the feature point's three-dimensional coordinate point in the radar coordinate system and the two-dimensional coordinate point in the camera pixel coordinate system:

[0019]

[0020] Among them, (X R，i , Y R，i , Z R，i ) is the three-dimensional coordinate point of the feature point obtained in the i-th group of calibration data in the radar coordinate system, (x i ,y i ) is the normalized two-dimensional coordinate point of the feature point obtained in the i-th group of calibration data in the camera coordinate system, s i is the scale value of the feature point obtained in the i-th group of calibration data, (u i , v i ) is the two-dimensional coordinate point of the feature point obtained in the i-th group of calibration data in the camera pixel coordinate system; t1, ···t 12 are 12 calibration parameters; K is the camera intrinsic parameter matrix;

[0021] The 12 calibration parameters are solved using the least squares method.

[0022] Furthermore, the method further includes: detecting whether the 4D radar and the camera need to be calibrated.

[0023] Furthermore, it is necessary to detect whether the 4D radar and camera need to be calibrated; including:

[0024] Step S1: Step S1: Acquire an observation data and M reference data; the observation data includes a pair of 4D radar point cloud and RGB image; the reference data includes a pair of 4D radar point cloud and RGB image;

[0025] Step S2: Extract feature points from the RGB image of the observed data and the RGB image of the mth reference data using an image feature extraction algorithm, and generate corresponding feature descriptors; use the feature descriptors to match the key points of the RGB image of the observed data and the RGB image of the mth reference data to obtain possible feature point matching relationship pairs; based on all feature point matching relationship pairs, calculate the essential matrix between the RGB image of the mth reference data and the RGB image of the observed data The first rotation matrix between the RGB image of the mth reference data and the RGB image of the observed data is obtained by decomposing the essential matrix through singular value decomposition.

[0026] Step S3: According to known calibration parameters, the 4D radar point cloud in the observation data is converted to the camera coordinate system to obtain observation pose data, and the 4D radar point cloud in the M reference data is converted to the camera coordinate system to obtain M reference pose data;

[0027] Step S4: Use the iterative nearest neighbor algorithm to obtain the second rotation matrix from the mth reference pose data to the observed pose data

[0028] Step S5: Calculate the yaw angle deviation according to the first rotation matrix and the second rotation matrix, and calculate the deviation matrix R:

[0029]

[0030] in, is the first rotation matrix The transpose of

[0031] Calculate the yaw angle deviation e according to the deviation matrix R ψ，i , pitch angle deviation e θ，m and roll angle deviation e φ，m :

[0032]

[0033] Among them, R 11 , R 21 , R 31 , R 32 and R 33 are the components of the deviation matrix R;

[0034] Step S6: Count the number of data in which the yaw angle deviation, pitch angle deviation and roll angle deviation are greater than the set threshold value among the M reference data. If the number is greater than [·] indicates rounding down, which means the 4D radar and camera need to be recalibrated.

[0035] In a second aspect, an embodiment of the present application provides a joint calibration device for a camera and a 4D millimeter wave radar, including:

[0036] An acquisition unit, used for acquiring at least six sets of calibration data by placing calibration plates at different positions of a common field of view of the 4D radar and the camera, each set of calibration data including point cloud data of the 4D radar and an RGB image of the camera;

[0037] A first processing unit is used to obtain three-dimensional coordinate points of feature points of the calibration plate in the radar coordinate system and two-dimensional coordinate points in the camera pixel coordinate system from each set of calibration data;

[0038] The second processing unit is used to construct a transformation equation by using the correspondence between the three-dimensional coordinate points of the feature points in the radar coordinate system and the two-dimensional coordinate points in the camera pixel coordinate system, and obtain the calibration parameters of the 4D radar and the camera by solving the equation.

[0039] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the embodiment of the present application when executing the computer program.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions implement the method of the embodiment of the present application when executed by a processor.

[0041] This application improves the calibration accuracy and algorithm robustness of 4D millimeter-wave radar and camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 is a flow chart of a joint calibration method of a camera and a 4D millimeter-wave radar provided in an embodiment of the present application;

[0044] Figure 2 is a schematic diagram of a calibration plate provided in an embodiment of the present application;

[0045] Figure 3 is a schematic diagram of a corner reflector provided in an embodiment of the present application;

[0046] Figure 4 These are two template matching prototype images used in detecting chessboard corner points provided in the embodiments of the present application;

[0047] Figure 5 is a functional structure diagram of a joint calibration device for a camera and a 4D millimeter-wave radar provided in an embodiment of the present application;

[0048] Figure 6 It is a functional structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0050] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0051] First, the design concept of the embodiments of the present application is briefly introduced.

[0052] At present, the roadside perception system perceives the environment through the fusion of cameras and lidar. Based on 3D lidar, 4D lidar can capture more elevation information of the target and realize imaging capabilities in the form of high-resolution point clouds. As a distance sensor, radar can measure Doppler velocity and radar cross-sectional area (RCS), which is expected to better detect and classify targets. In addition, 4D lidar also has the advantages of long detection distance (up to 300m), all-weather operation, low power consumption and low cost. The single camera detection method has the defects of not adapting to rapid changes in light, physical distortion of the lens and poor perception of bad weather. Therefore, 4D lidar can be a good supplement to visual single-modal detection.

[0053] In order to realize the fusion of radar data and camera images, the joint calibration between the camera and 4D radar must be realized. First, the calibration plate is placed in the common visual range of the camera and 4D radar. The camera and 4D radar respectively collect the two-dimensional image coordinates and three-dimensional point cloud coordinates corresponding to the vertices of the corner reflector, and construct a set of equations to solve the joint calibration matrix.

[0054] Compared with the prior art, this application has the following beneficial effects:

[0055] 1. The calibration plate is simple and easy to make and can be manufactured manually;

[0056] 2. It can accurately determine whether the camera and 4D radar need to be calibrated. The determination method is simple and convenient, which can reduce useless recalibration work;

[0057] 3. In solving the equation group of calibration parameters, the nonlinear least squares optimization method is used to improve the calibration accuracy and algorithm robustness.

[0058] After introducing the application scenarios and design concepts of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0059] like Figure 1 As shown, the embodiment of the present application provides a joint calibration method of a camera and a 4D millimeter wave radar, including:

[0060] Step 101: obtaining at least six sets of calibration data by placing calibration plates at different positions of a common field of view of the 4D radar and the camera, each set of calibration data including point cloud data of the 4D radar and an RGB image of the camera;

[0061] Figure 2 The combined calibration plate required for the implementation of the embodiment of the present application is composed of a rectangular foam plate covered with a checkerboard pattern and a triangular corner reflector. Since the foam is mostly made of air (98%), it is almost invisible to the radar, and its flat shape can help the camera achieve precise positioning. The selection of the chessboard size is a comprehensive result of considering the number of corner points of the calibration plate and the long-distance detection capability.

[0062] Figure 3 A schematic diagram of the structure of a corner reflector provided in an embodiment of the present application. The corner reflector is composed of three mutually orthogonal planar triangular glass plates, which are fixed on the back of a calibration plate, and a metal ball is fixed in the space formed by the calibration plate.

[0063] Exemplarily, the camera and 4D radar to be calibrated are fixed on the vehicle body or other equipment at the same viewing angle. During the entire calibration process, it is necessary to ensure that the sensor position can observe a clear and obvious calibration plate. That is, the camera should be able to clearly observe the entire two-dimensional chessboard on the calibration plate; the calibration plate should fall within the common visual range of the camera and the 4D radar. In this embodiment, the visible light camera is modeled as a pinhole camera with radial and tangential distortion. Select a suitable calibration position and move the calibration plate by left and right translation, height adjustment, and distance movement.

[0064] Step 102: Obtain the three-dimensional coordinate points of the feature points of the calibration plate in the radar coordinate system and the two-dimensional coordinate points in the camera pixel coordinate system from each set of calibration data;

[0065] In the processing of camera RGB images, we first detect the checkerboard corners in the reference image. In order to obtain good positioning accuracy even in the case of cluttered background and blur artifacts, we use a detection algorithm based on template matching and design two different n×n corner prototypes to calculate a corner likelihood at each pixel in the image: one for axis-aligned corners (such as Figure 4 left), a corner point for a 45° rotation (as shown Figure 4 shown on the right).

[0066] The obtained corner prototype can detect corners in the range distortion caused by perspective transformation. In order to generate a list of candidate corners, non-maximum suppression is used for filtering, and the candidate corners are verified by gradient statistics within a local n×n pixel neighborhood.

[0067] The specific approach is as follows: Calculate the weighted direction histogram based on the Sobel filter response and use mean shift to find two dominant modes α1 and α2. Construct the expected gradient intensity template based on the edge direction. Perform a normalized cross-correlation between the template and the gradient intensity and perform a dot product with the corner likelihood value to obtain the corner score, and set a threshold to filter the candidate corners.

[0068] Since the calibration is based on the corner position at the sub-pixel level, the corner position and edge direction also need to be refined.

[0069] The checkerboard is grown by minimizing the energy function, thereby effectively locating the checkerboard corners in the image. Based on the known target configuration information, the homography matrix is ​​used to restore the transformation relationship from the checkerboard plane to the camera pixel coordinate system.

[0070] Specifically, obtaining the three-dimensional coordinates of the feature points of the calibration plate in the radar coordinate system includes:

[0071] Get the reflection intensity of each point in the point cloud data of 4D radar;

[0072] The three-dimensional coordinate point of the point with the largest reflection intensity is taken as the three-dimensional coordinate point of the feature point in the radar coordinate system.

[0073] Specifically, obtaining the two-dimensional coordinates of the feature points of the calibration plate in the camera pixel coordinate system includes:

[0074] A three-dimensional reference object coordinate system is established based on the calibration plate plane, and the calibration plate plane is set as the plane where z=0 of the reference object coordinate system;

[0075] According to the two-dimensional coordinates of multiple corner points of the chessboard on the calibration plate plane and the two-dimensional coordinates in the camera pixel coordinate system, a conversion matrix between the reference object coordinate system and the camera pixel coordinate system is established;

[0076] According to the three-dimensional coordinates of the corner reflector in the reference object coordinate system and the transformation matrix between the reference object coordinate system and the camera pixel coordinate system, the two-dimensional coordinates of the corner reflector in the camera pixel coordinate system are obtained, thereby obtaining the two-dimensional coordinates of the feature points of the calibration plate in the camera pixel coordinate system.

[0077] Step 103: construct a transformation equation using the correspondence between the three-dimensional coordinate points of the feature points in the radar coordinate system and the two-dimensional coordinate points in the camera pixel coordinate system, and obtain the calibration parameters of the 4D radar and the camera by solving the equation.

[0078] Specifically, use O R -X R Y R Z R Represents the 4D radar coordinate system, O C -X C Y C Z C represents the camera pixel coordinate system. R∈SO(3) and Respectively represent the rotation matrix and translation vector between the two coordinate systems.

[0079] The transformation relationship between the camera pixel coordinate system and the 4D radar coordinate system is as follows:

[0080]

[0081] The Kalibr toolbox is used to calculate the intrinsic parameters and distortion parameters of the visible light camera, and the subsequent process will be performed on the corrected image. Among them, the distortion model and pinhole model are:

[0082] r 2 =x 2 +y 2

[0083]

[0084] u=f x x′+cx

[0085] v=f y y′+c y

[0086] Where r is the distance from the projection point of the point light source to the center of the image; (x, y) is the normalized coordinate data of the projection point of the point light source in the pinhole camera pixel coordinate system; (x′, y′) is the actual coordinate data of the projection point of the point light source; f x is the resolution in the x direction, f y is the resolution in the y direction; (c x , c y ) is the principal point of the pinhole camera, that is, the image center; k1, k2, k3, k4, k5, k6, p1, and p2 are all distortion coefficients of the pinhole camera.

[0087] After obtaining at least six sets of point pairs, multiple sets of linear equations can be constructed based on the direct linear transformation method (DLT) using the two-dimensional pixel coordinates corresponding to the corner reflector in each frame and the three-dimensional target radar point coordinates. The multiple sets of linear equations are solved to obtain the initial value of the joint calibration between the two sensors.

[0088] The specific formula is as follows:

[0089]

[0090] Among them, (X R，i , Y R，i , Z R，i ) is the three-dimensional coordinate point of the feature point obtained in the i-th group of calibration data in the radar coordinate system, (x i ,y i ) is the normalized two-dimensional coordinate point of the feature point obtained in the i-th group of calibration data in the camera coordinate system, s i is the scale value of the feature point obtained in the i-th group of calibration data, (u i , v i ) is the two-dimensional coordinate point of the feature point obtained in the i-th group of calibration data in the camera pixel coordinate system; t1,…t 12 are 12 calibration parameters; K is the camera intrinsic parameter matrix;

[0091] Eliminate s i Then get the information about t i The linear equations are:

[0092]

[0093] Among them, each feature point provides two linear constraints on the calibration parameters.

[0094] For N feature points, 2N linear equations about calibration parameters can be obtained. Since there are 12 calibration parameters in total, at least 6 pairs of feature points are required to achieve linear solution of all calibration parameters. The matrix composed of all solved variables is the joint calibration parameters between 4D radar and camera, including transformation matrix and translation vector.

[0095] The LM (Levenberg-Marquardt) optimization algorithm is a classic algorithm that has been verified many times and can effectively solve nonlinear least squares functions. This algorithm integrates the advantages of the Gauss-Newton method and the gradient descent method. By controlling the damping factor variable, that is, adjusting the step size of each algorithm iteration, it effectively improves the iterative optimization efficiency, realizes the least squares optimization of the objective function, and finds the best rotation and translation matrix.

[0096] The direct linear transformation method is used to obtain the initial value of the joint calibration matrix between the camera and the 4D radar. When more point pairs are obtained, the corresponding 3D target radar points are projected onto the image plane, the reprojection error is constructed, and the LM method is used for nonlinear optimization.

[0097] For N calibration points, the solved joint calibration initial value can be optimized by minimizing the objective function.

[0098]

[0099] Among them, (x d，i ,y d，i ) is the coordinate of the feature point in the ith calibration data after reprojection of the point cloud to the normalized plane of the image; (u i , v i ) is the actual pixel coordinate corresponding to the feature point in the i-th calibration data. x is the resolution of the camera in the x direction, a y is the resolution of the camera in the y direction, and (u0, v0) is the principal point of the pinhole camera.

[0100] The LM algorithm integrates the advantages of the steepest descent method and the Gauss-Newton method. k To make adjustments. The step length d of the Gauss-Newton method to solve the problem k It can be expressed as a function:

[0101]

[0102] in, is F(x k )About x k The derivative (Jacobian matrix) of .

[0103] Gauss-Newton method requires The matrix is ​​a full row rank matrix, so it cannot be used The LM algorithm improves the Gauss-Newton method by introducing a positive definite matrix uI and adding it to On the matrix, make it a positive definite matrix. At this time, the iteration step length of the LM algorithm is:

[0104]

[0105] Where: I is the unit matrix; u is the LM parameter. When u→0, d k Tends to Gauss-Newton law, when u→∞, d k tends to be gradient descent.

[0106] The joint calibration matrix obtained by the direct linear transformation method is used as the initial value of the function and an iterative optimization solution is performed.

[0107] The detailed steps for implementing the LM algorithm are as follows:

[0108] a) Input the initial value x0, initialize the parameter u = 0.01, and set the precision ε = 1×10 -8 ;

[0109] b) Calculate F(x0) and The value of

[0110] c) Calculate d k , calculate x k+1 =x k +d k ;

[0111] d) If F(x k+1 )<F(x k ), and ||d k ||<ε, the iteration stops and the optimized joint calibration result is output; otherwise, u k+1 =u k / 2, go to step b) and continue the cycle;

[0112] e) If F(x k+1 )≥F(x k ),u k+1 =2u k , recalculate d k The value of returns to step d) and continues the loop until the iteration termination condition is met, then the loop stops.

[0113] Specifically, the correspondence between the three-dimensional coordinates of the feature points in the radar coordinate system and the two-dimensional coordinates of the camera pixel coordinate system is used to construct a transformation equation, and the calibration parameters of the 4D radar and camera are obtained by solving the equation; including:

[0114] Establish the transformation equation of the feature point's three-dimensional coordinate point in the radar coordinate system and the two-dimensional coordinate point in the camera pixel coordinate system:

[0115]

[0116] The 12 calibration parameters are solved using the least squares method.

[0117] In general, due to the constraints of external parameters, if the relative posture of the camera and 4D radar with calibrated external parameters changes at different times, the coordinate transformation of the two sensors should be performed through the known external parameters to convert the posture of the two sensors to the same coordinate system. If the external parameters of the two sensors have not changed, the posture changes of the two sensors should remain consistent; however, if the posture changes of the two sensors are quite different (reaching the preset difference degree), it means that the external parameters of the two sensors have changed. In this embodiment, according to the projection principle, the deviation in the rotation direction will be more sensitive than the translation direction, and a very small rotation deviation will cause a large alignment error; while the alignment error caused by the translation change to a slightly distant target is small, and the deviation in the rotation direction is more common. Therefore, the rotation component deviation of the image and point cloud transformation matrix can be used as the judgment condition of the sensor calibration state: the transformation matrix of each frame of reference data to the observed data is estimated by image registration and point cloud registration respectively, and the rotation component deviation of each group of data transformation matrix is ​​solved, and the current calibration state is judged according to the rotation component deviation.

[0118] Based on the above design concept, the method further includes: detecting whether the 4D radar and the camera need to be calibrated, specifically including:

[0119] Step S1: Acquire an observation data and M reference data; the observation data includes a pair of 4D radar point cloud and RGB image; the reference data includes a pair of 4D radar point cloud and RGB image;

[0120] Among them, only the matching result of one frame of reference data and the observed data can be used to determine whether the sensor posture changes; selecting multiple frames of reference data can effectively reduce the judgment error. Therefore, the number of reference data frames is M frames.

[0121] Step S2: Extract feature points from the RGB image of the observed data and the RGB image of the mth reference data using an image feature extraction algorithm, and generate corresponding feature descriptors; use the feature descriptors to match the key points of the RGB image of the observed data and the RGB image of the mth reference data to obtain possible feature point matching relationship pairs; based on all feature point matching relationship pairs, calculate the essential matrix between the RGB image of the mth reference data and the RGB image of the observed data The first rotation matrix between the RGB image of the mth reference data and the RGB image of the observed data is obtained by decomposing the essential matrix through singular value decomposition.

[0122] Step S3: According to known calibration parameters, the 4D radar point cloud in the observation data is converted to the camera coordinate system to obtain observation pose data, and the 4D radar point cloud in the M reference data is converted to the camera coordinate system to obtain M reference pose data;

[0123] Step S4: Use the iterative nearest neighbor algorithm to obtain the second rotation matrix from the mth reference pose data to the observed pose data

[0124] Step S5: Calculate the yaw angle deviation according to the first rotation matrix and the second rotation matrix, and calculate the deviation matrix R:

[0125]

[0126] in, is the first rotation matrix The transpose of

[0127] Calculate the yaw angle deviation e according to the deviation matrix R ψ，i , pitch angle deviation e θ，m and roll angle deviation e φ，m :

[0128]

[0129] Among them, R 11 , R 21 , R 31 , R 32 and R 33 are the components of the deviation matrix R;

[0130] Step S6: Count the number of data in which the yaw angle deviation, pitch angle deviation and roll angle deviation are greater than the set threshold value among the M reference data. If the number is greater than [·] indicates rounding down, which means the 4D radar and camera need to be recalibrated.

[0131] Based on the above embodiments, the present application provides a joint calibration device for a camera and a 4D millimeter wave radar. Figure 5 As shown, the joint calibration device 200 of the camera and 4D millimeter wave radar provided in the embodiment of the present application at least includes:

[0132] An acquisition unit 201 is used to acquire at least six sets of calibration data by placing calibration plates at different positions of a common field of view of the 4D radar and the camera, each set of calibration data including point cloud data of the 4D radar and an RGB image of the camera;

[0133] The first processing unit 202 is used to obtain the three-dimensional coordinate points of the feature points of the calibration plate in the radar coordinate system and the two-dimensional coordinate points in the camera pixel coordinate system from each set of calibration data;

[0134] The second processing unit 203 is used to construct a transformation equation by using the correspondence between the three-dimensional coordinate points of the feature points in the radar coordinate system and the two-dimensional coordinate points in the camera pixel coordinate system, and obtain the calibration parameters of the 4D radar and the camera by solving the equation.

[0135] It should be noted that the principle of solving the technical problem by the joint calibration device 200 of the camera and 4D millimeter-wave radar provided in the embodiment of the present application is similar to the method provided in the embodiment of the present application. Therefore, the implementation of the joint calibration device 200 of the camera and 4D millimeter-wave radar provided in the embodiment of the present application can refer to the implementation of the method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0136] Based on the above embodiments, the present application also provides an electronic device, referring to Figure 6 As shown, the electronic device 300 provided in the embodiment of the present application includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the joint calibration method of the camera and 4D millimeter-wave radar provided in the embodiment of the present application is implemented.

[0137] The electronic device 300 provided in the embodiment of the present application may further include a bus 303 connecting different components (including the processor 301 and the memory 302). The bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0138] The memory 302 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022 , and may further include a read-only memory (ROM) 3023 .

[0139] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, the program modules 3024 including but not limited to: an operating subsystem, one or more applications, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0140] The electronic device 300 may also communicate with one or more external devices 304 (e.g., keyboards, remote controls, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 300 (e.g., mobile phones, computers, etc.), and / or communicate with any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 305. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. Figure 6 As shown, the network adapter 306 communicates with other modules of the electronic device 300 via the bus 303. It should be understood that although Figure 6 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0141] It should be noted that Figure 6 The electronic device 300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0142] The embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by the processor, the joint calibration method of the camera and 4D millimeter-wave radar provided in the embodiment of the present application is implemented. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the joint calibration method of the camera and 4D millimeter-wave radar provided in the embodiment of the present application by executing the built-in or installed executable program.

[0143] The joint calibration method of the camera and 4D millimeter-wave radar provided in the embodiment of the present application can also be implemented as a program product, which includes a program code. When the program product can be run on the electronic device 300, the program code is used to enable the electronic device 300 to execute the joint calibration method of the camera and 4D millimeter-wave radar provided in the embodiment of the present application.

[0144] The program product provided in the embodiments of the present application may adopt any combination of one or more readable media, wherein the readable medium may be a readable signal medium or a readable storage medium, and the readable storage medium may be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0145] The program product provided in the embodiment of the present application may adopt a CD-ROM and include program code, and may also be run on a computing device. However, the program product provided in the embodiment of the present application is not limited thereto. In the embodiment of the present application, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0146] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0147] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.

Claims

1. A joint calibration method for a camera and a 4D millimeter-wave radar, characterized in that: include: By placing the calibration plate at different positions of the common field of view of the 4D radar and the camera, at least six sets of calibration data are obtained, each set of calibration data includes the point cloud data of the 4D radar and the RGB image of the camera; Obtain the three-dimensional coordinate points of the feature points of the calibration plate in the radar coordinate system and the two-dimensional coordinate points in the camera pixel coordinate system from each set of calibration data; The corresponding relationship between the three-dimensional coordinates of the feature points in the radar coordinate system and the two-dimensional coordinates of the camera pixel coordinate system is used to construct the transformation equation, and the calibration parameters of the 4D radar and camera are obtained by solving the equation.

2. The method according to claim 1, characterized in that The side of the calibration plate facing the 4D radar and the camera is a chessboard, and a corner reflector is arranged at the center point of the other side of the calibration plate, and the imaging point of the corner reflector is a feature point.

3. The method according to claim 2, characterized in that Get the three-dimensional coordinates of the feature points of the calibration plate in the radar coordinate system, including: Get the reflection intensity of each point in the point cloud data of 4D radar; The three-dimensional coordinate point of the point with the largest reflection intensity is taken as the three-dimensional coordinate point of the feature point in the radar coordinate system.

4. The method according to claim 2, characterized in that: Get the 2D coordinates of the feature points of the calibration plate in the camera pixel coordinate system, including: A three-dimensional reference object coordinate system is established based on the calibration plate plane, and the calibration plate plane is set as the plane where z=0 of the reference object coordinate system; According to the two-dimensional coordinates of multiple corner points of the chessboard on the calibration plate plane and the two-dimensional coordinates in the camera pixel coordinate system, a conversion matrix between the reference object coordinate system and the camera pixel coordinate system is established; According to the three-dimensional coordinates of the corner reflector in the reference object coordinate system and the transformation matrix between the reference object coordinate system and the camera pixel coordinate system, the two-dimensional coordinates of the corner reflector in the camera pixel coordinate system are obtained, thereby obtaining the two-dimensional coordinates of the feature points of the calibration plate in the camera pixel coordinate system.

5. The method according to claim 4, characterized in that Using the correspondence between the three-dimensional coordinates of the feature points in the radar coordinate system and the two-dimensional coordinates of the camera pixel coordinate system, a transformation equation is constructed, and the calibration parameters of the 4D radar and camera are obtained by solving the equation; including: Establish the transformation equation of the feature point's three-dimensional coordinate point in the radar coordinate system and the two-dimensional coordinate point in the camera pixel coordinate system: Among them, (X R,i ,Y R,i ,Z R,i ) is the three-dimensional coordinate point of the feature point obtained in the i-th group of calibration data in the radar coordinate system, (x i ,y i ) is the normalized two-dimensional coordinate point of the feature point obtained in the i-th group of calibration data in the camera coordinate system, s i is the scale value of the feature point obtained in the i-th group of calibration data, (u i ,v i ) is the two-dimensional coordinate point of the feature point obtained in the i-th group of calibration data in the camera pixel coordinate system; t1,…t 12 are 12 calibration parameters; K is the camera intrinsic parameter matrix; The 12 calibration parameters are solved using the least squares method.

6. The method according to claim 1, characterized in that The method further includes: detecting whether the 4D radar and the camera need to be calibrated.

7. The method according to claim 6, characterized in that Check whether 4D radar and camera need calibration; including: Step S1: Acquire an observation data and M reference data; the observation data includes a pair of 4D radar point cloud and RGB image; the reference data includes a pair of 4D radar point cloud and RGB image; Step S2: Extract feature points from the RGB image of the observed data and the RGB image of the mth reference data using an image feature extraction algorithm, and generate corresponding feature descriptors; use the feature descriptors to match the key points of the RGB image of the observed data and the RGB image of the mth reference data to obtain possible feature point matching relationship pairs; based on all feature point matching relationship pairs, calculate the essential matrix between the RGB image of the mth reference data and the RGB image of the observed data The first rotation matrix between the RGB image of the mth reference data and the RGB image of the observed data is obtained by decomposing the essential matrix through singular value decomposition. 1≤m≤M; Step S3: According to known calibration parameters, the 4D radar point cloud in the observation data is converted to the camera coordinate system to obtain observation pose data, and the 4D radar point cloud in the M reference data is converted to the camera coordinate system to obtain M reference pose data; Step S4: Use the iterative nearest neighbor algorithm to obtain the second rotation matrix from the mth reference pose data to the observed pose data Step S5: Calculate the yaw angle deviation according to the first rotation matrix and the second rotation matrix, and calculate the deviation matrix R: in, is the first rotation matrix The transpose of Calculate the yaw angle deviation e according to the deviation matrix R ψ,i , pitch angle deviation e θ,m and roll angle deviation e φ,m : Among them, R 11 , R 21 , R 31 , R 32 and R 33 are the components of the deviation matrix R; Step S6: Count the number of data in which the yaw angle deviation, pitch angle deviation and roll angle deviation are greater than the set threshold value among the m reference data. If the number is greater than [·] indicates rounding down, which means the 4D radar and camera need to be recalibrated.

8. A joint calibration device for a camera and a 4D millimeter-wave radar, characterized in that: include: An acquisition unit, used for acquiring at least six sets of calibration data by placing calibration plates at different positions of a common field of view of the 4D radar and the camera, each set of calibration data including point cloud data of the 4D radar and an RGB image of the camera; A first processing unit is used to obtain three-dimensional coordinate points of feature points of the calibration plate in the radar coordinate system and two-dimensional coordinate points in the camera pixel coordinate system from each set of calibration data; The second processing unit is used to construct a transformation equation by using the correspondence between the three-dimensional coordinate points of the feature points in the radar coordinate system and the two-dimensional coordinate points in the camera pixel coordinate system, and obtain the calibration parameters of the 4D radar and the camera by solving the equation.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images