4D millimeter wave radar and camera external parameter calibration method
By acquiring and calculating the external parameters of 4D mmWave radar and the camera, using the reprojection verification and calibration parameter fine-tuning module, the problem of coordinate association difficulty in external parameter calibration of 4D mmWave radar is solved, and high-precision external parameter calibration and visualization are achieved Assisted, suitable for external parameter calibration of 4D mmWave radar and camera.
Patent Information
- Application Number
- CN202411168114.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-03
AI Technical Summary
The existing camera and radar external parameter calibration methods cannot be applied to 4D millimeter wave radar because its point cloud density is insufficient, effective plane information cannot be formed, and the spatial position distribution does not have geometric patterns, so it cannot match lidar.
By collecting camera pictures and millimeter-wave radar point cloud data, calculate the external parameters of 4D millimeter-wave radar and camera, use the reprojection verification and calibration parameter fine-tuning module to adjust the external parameters until the accuracy requirements are met, and the effective 4D millimeter-wave radar and camera are achieved. External parameter calibration.
The coordinate points of the pixel coordinate system and the millimeter wave radar coordinate system in the same scene were successfully obtained, which solved the problem of difficulty in correlation between the coordinates in the camera's external parameter calibration, realized visual assisted calibration, and improved calibration accuracy.
Smart Images

Figure CN119068062A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of millimeter wave radars, and in particular to a 4D millimeter wave radar and camera extrinsic parameter calibration method. Background Art
[0002] With the development of 4D millimeter-wave radar technology, perception solutions based on 4D millimeter-wave radar are emerging one after another, such as human fall detection and dynamic target recognition in the field of assisted driving. Compared with traditional 3D millimeter-wave radar, 4D millimeter-wave radar has increased the perception capability in the vertical direction, which greatly improves the number of point clouds and imaging area.
[0003] The current camera and radar extrinsic calibration scheme is mainly aimed at the joint calibration of cameras and lidars, such as the invention patent CN114758005A. Compared with 4D millimeter-wave radar, the point cloud of lidar is denser. The method of joint calibration is generally to extract the edge information and normal vector information of the calibration plate in the scene, and construct the relationship between the lidar point cloud and the point cloud in the camera coordinate system, so as to obtain the transformation matrix between the two coordinate systems. However, for 4D millimeter-wave radar, the density of the point cloud is not enough to form effective plane information, and the spatial position of the point cloud of the 4D millimeter-wave radar cannot be arranged in a regular manner like the lidar, and is limited by the surface reflectivity of the object, multipath interference, etc. For 4D millimeter-wave radar, the spatial distribution of the returned point cloud does not have a strong geometric distribution law. Therefore, the existing method will not be applicable to the extrinsic calibration of 4D millimeter-wave radar to camera.
[0004] Based on this, the present invention designs a 4D millimeter-wave radar and camera extrinsic parameter calibration method to solve the above problems. Summary of the invention
[0005] In view of the above-mentioned shortcomings of the prior art, the present invention provides a 4D millimeter-wave radar and camera extrinsic parameter calibration method.
[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions:
[0007] A 4D millimeter wave radar and camera extrinsic parameter calibration method, comprising the following steps:
[0008] 1. The coordinate acquisition module collects camera images and millimeter wave radar point cloud data;
[0009] 2. The external parameter calculation module calculates the corresponding external parameters of the 4D millimeter wave radar to the camera.
[0010] 3. The reprojection verification and calibration parameter fine-tuning module calculates the reprojection error and determines whether the reprojection error meets the accuracy requirements. If not, the external parameters are fine-tuned until the accuracy requirements are met.
[0011] Furthermore, the specific steps of step one are:
[0012] (1) Place the device to be calibrated in the selected area X and the radar corner reflector in the common view area A, so that the radar corner reflector can be seen in the image display of the visualization system, and the only high-intensity point can be seen in the point cloud display, and it can be determined that it corresponds to the radar corner reflector;
[0013] (2) Run the acquisition script to save the camera image and millimeter-wave radar point cloud data at this time;
[0014] (3) Adjust the position of the radar corner reflector and repeat steps (1) to (2).
[0015] Furthermore, the specific steps of step 2 are:
[0016] (1) Analyze the camera image and point cloud data obtained by the coordinate acquisition module to find the pixel coordinates of the radar corner reflector in the dedistorted image and the corresponding point cloud coordinates in the 4D millimeter wave radar;
[0017] (2) Use the 4D millimeter wave radar to camera external parameter calibration script to read the camera internal parameters, the pixel coordinates of the radar corner reflectors in the image arranged in sequence, and the point cloud coordinates of the radar corner reflectors in the 4D millimeter wave radar in the same order, call the external parameter calculation function, and obtain the corresponding 4D millimeter wave radar to camera external parameters.
[0018] Furthermore, the specific steps of step three are:
[0019] (1) Taking the external parameters from the 4D millimeter wave radar to the camera, the internal parameters of the camera, and the point cloud collected from the original 4D millimeter wave radar as input, the pixel coordinates after reprojection are calculated;
[0020] (2) Calculate the average error between the pixel coordinates after reprojection and the pixel coordinates of the radar corner reflector on the original acquired distorted image to obtain the reprojection error;
[0021] (3) Determine whether the reprojection error meets the accuracy requirements. If not, fine-tune the translation and rotation in the extrinsic parameters to obtain the fine-tuned extrinsic parameters and repeat the above steps until the accuracy requirements are met.
[0022] Furthermore, the coordinate acquisition module includes:
[0023] A device to be calibrated includes two sensors: a 4D millimeter wave radar and a camera, and the 4D millimeter wave radar and the camera have a common view area A;
[0024] A visualization system that displays the camera image and 4D mmWave point cloud in real time;
[0025] A data acquisition system that can simultaneously collect images and 4D millimeter wave radar point cloud information through scripts;
[0026] A highly reflective, light-proof radar corner reflector.
[0027] Furthermore, the visualization system includes two display windows, which are used to display the camera image and the 4D millimeter wave point cloud in real time respectively.
[0028] Furthermore, the calculation steps of the external parameter calculation module are as follows:
[0029] (1) Determine whether the collected scene has been analyzed. If not, analyze the next set of collected camera images and point cloud data, obtain the points of the radar corner reflector from the point cloud, dedistort the image, obtain the pixel coordinates of the corresponding radar corner reflector on the dedistorted image, and then obtain a set of 4D millimeter-wave point cloud coordinates and pixel coordinates of the radar corner reflector; if yes, obtain multiple sets of 4D millimeter-wave point cloud coordinates and pixel coordinates of the radar corner reflector;
[0030] (2) The external parameters are calculated by using the camera internal parameters, pixel coordinates and 4D millimeter wave point cloud coordinates.
[0031] Compared with the prior art, the present invention has the following beneficial effects:
[0032] The present invention aims at the sensor characteristics of 4D millimeter wave radar, and can obtain a set of coordinate points of the pixel coordinate system and the millimeter wave radar coordinate system of the same scene, and realize external parameter calibration based on the collected multiple sets of coordinates. It solves the problem of difficult coordinate association in the external parameter calibration of 4D millimeter wave radar and camera.
[0033] The present invention realizes visualization of the extrinsic parameter calibration process of 4D millimeter-wave radar and camera, effectively assists the extrinsic parameter calibration, and is conducive to intuitively selecting better calibration scenes and calibration data. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0035] Figure 1 It is a schematic diagram of the collection scene of the present invention;
[0036] Figure 2 It is a schematic structural diagram of the radar corner reflector of the present invention;
[0037] Figure 3A schematic diagram of the light reflection path of the radar corner reflector of the present invention;
[0038] Figure 4 It is a flow chart of the coordinate acquisition module of the present invention;
[0039] Figure 5 It is a flow chart of the external parameter calculation module of the present invention;
[0040] Figure 6 It is a flow chart of the reprojection verification and calibration parameter fine-tuning module of the present invention. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0042] Embodiment 1
[0043] In some embodiments, please refer to the attached instructions. Figure 1-3 and 6, a 4D millimeter wave radar and camera extrinsic calibration system, comprising a coordinate acquisition module, an extrinsic parameter calculation module, and a reprojection verification and calibration parameter fine-tuning module;
[0044] Among them, the coordinate acquisition module includes:
[0045] A device to be calibrated includes two sensors: a 4D millimeter-wave radar and a camera. The 4D millimeter-wave radar and the camera have a common viewing area A, that is, the FOV (field of view) has an overlapping area;
[0046] Figure 1 The middle shaded area is the common viewing area A of the camera and 4D millimeter wave radar. The coordinate collection required for external parameter calibration needs to be performed in the shaded area.
[0047] A visualization system, including two display windows, one for real-time display of camera images and the other for 4D millimeter-wave point cloud. The color of the millimeter-wave radar point cloud indicates the corresponding point cloud signal strength.
[0048] A data acquisition system that can simultaneously collect images and 4D millimeter wave radar point cloud information through scripts;
[0049] A highly reflective, light-proof radar corner reflector that can be adjusted in height using a tripod or fixed bracket based on the height of the sensor to be calibrated;
[0050] like Figure 2 The radar corner reflector is a solid angle formed by three mutually perpendicular metal plates, that is, the S1, S2, and S3 planes are perpendicular to each other.
[0051] like Figure 3 No matter from which direction the light is directed to the radar corner reflector, the radar corner reflector can reflect the incident light back in the opposite direction, so that the point cloud returned by the 4D millimeter-wave radar on the radar corner reflector has the characteristic of high intensity.
[0052] A relatively open and flat area X, the range of which is selected as needed according to the detection distance of the camera and the 4D millimeter-wave radar.
[0053] The acquisition steps of the coordinate acquisition module are as follows:
[0054] (1) Place the device to be calibrated in the selected area X and the radar corner reflector in the common view area A, so that the radar corner reflector can be seen in the image display of the visualization system, and the only high-intensity point can be seen in the point cloud display, and it can be determined that it corresponds to the radar corner reflector;
[0055] (2) Run the acquisition script to save the camera image and millimeter-wave radar point cloud data at this time;
[0056] (3) Adjust the position of the radar corner reflector and repeat steps (1) to (2) to ensure that the position of the radar corner reflector is distributed over different distances and angle ranges.
[0057] Among them, the calculation steps of the external parameter calculation module are as follows:
[0058] (1) Analyze the camera image and point cloud data obtained by the coordinate acquisition module to find the pixel coordinates of the radar corner reflector in the dedistorted image and the corresponding point cloud coordinates (radar coordinates) in the 4D millimeter wave radar;
[0059] (2) Use the 4D millimeter wave radar to camera external parameter calibration script to read the camera internal parameters, the pixel coordinates of the radar corner reflectors in the image arranged in sequence, and the point cloud coordinates of the radar corner reflectors in the 4D millimeter wave radar in the same order, call the external parameter calculation function, and obtain the corresponding 4D millimeter wave radar to camera external parameters.
[0060] The main calculation process is as follows:
[0061] _, rvec, tvec, _=cv2.solvePnPRansac(radar_points, pixel_points, camera_matrix, None)
[0062] mR, _ = cv2.Rodrigues(rvec)
[0063] radar2camera=np.concatenate([mR, tvec], axis=1)
[0064] radar2camera=np.concatenate([result,np.array([0,0,0,1]).reshape(1,4)],axis=0)
[0065] Among them, radar_points, pixel_points, and camera_matrix are the point cloud coordinates of the 4D millimeter-wave radar, the pixel coordinates of the radar corner reflector on the image after distortion processing, and the camera internal parameters. solvePnPRansac is an OpenCV library function used to obtain the rotation vector rvec and translation vector tvec of the coordinate system transformation between the millimeter-wave radar and the camera. Rodrigues is an OpenCV library function used to convert the rotation vector into a rotation matrix. radar2camera is the final external parameter matrix of the 4D millimeter-wave radar to the camera, which is a 4x4 matrix composed of a rotation matrix and a translation vector.
[0066] The working steps of the reprojection verification and calibration parameter fine-tuning module are as follows:
[0067] (1) Taking the external parameters from the 4D millimeter wave radar to the camera, the internal parameters of the camera, and the point cloud collected from the original 4D millimeter wave radar as input, the pixel coordinates after reprojection are calculated;
[0068] The calculation formula is as follows:
[0069]
[0070] Among them, K is the camera internal parameter, R and T are the external parameters of the 4D millimeter wave radar to the camera, and X radar , Y radar, Z radar is the point cloud of the original 4D millimeter wave radar collected, u, v are the pixel coordinates to be calculated, Z c It is an intermediate variable in the calculation process.
[0071] (2) Calculate the average error between the pixel coordinates after reprojection and the pixel coordinates of the radar corner reflector on the original acquired distorted image to obtain the reprojection error;
[0072] (3) Determine whether the reprojection error meets the accuracy requirements. If not, fine-tune the translation and rotation in the external parameters. The translation fine-tuning amount can be 5.0 cm, 2.5 cm, 1.0 cm, and the rotation fine-tuning amount can be 1.0 degree, 0.5 degree, 0.1 degree. You can also customize the fine-tuning amount according to the actual situation. After obtaining the fine-tuned external parameters, repeat the above steps until the accuracy requirements are met.
[0073] The core of the reprojection verification and calibration parameter fine-tuning module is the reprojection error calculation. Reprojection refers to the process of back-projecting the coordinates of the 4D millimeter-wave radar to the pixel space of the image through the external parameters of the 4D millimeter-wave radar to the camera and the internal parameters of the camera. The average error between the calculated pixel coordinates and the pixel coordinates of the radar corner reflector on the original acquired distorted image is calculated, which is the reprojection error.
[0074] The present invention aims at the sensor characteristics of 4D millimeter-wave radar and solves the problem of difficulty in associating coordinates in extrinsic parameter calibration of 4D millimeter-wave radar and camera by associating pixel coordinates in camera images and point cloud coordinates in 4D millimeter-wave radar.
[0075] The present invention realizes visualization of the extrinsic parameter calibration process of 4D millimeter-wave radar and camera, effectively assists the extrinsic parameter calibration, and is conducive to intuitively selecting better calibration scenes and calibration data.
[0076] Embodiment 2
[0077] In some embodiments, a 4D millimeter wave radar and camera extrinsic calibration method includes the following steps:
[0078] 1. The coordinate acquisition module collects camera images and millimeter wave radar point cloud data. The specific steps are:
[0079] (1) Place the device to be calibrated in the selected area X and the radar corner reflector in the common view area A, so that the radar corner reflector can be seen in the image display of the visualization system, and the only high-intensity point can be seen in the point cloud display, and it can be determined that it corresponds to the radar corner reflector;
[0080] (2) Run the acquisition script to save the camera image and millimeter-wave radar point cloud data at this time;
[0081] (3) Adjust the position of the radar corner reflector and repeat steps (1) to (2) to ensure that the position of the radar corner reflector is distributed over different distances and angle ranges.
[0082] Second, the external parameter calculation module calculates the corresponding external parameters of the 4D millimeter wave radar to the camera. The specific steps are:
[0083] (1) Analyze the camera image and point cloud data obtained by the coordinate acquisition module to find the pixel coordinates of the radar corner reflector in the dedistorted image and the corresponding point cloud coordinates (radar coordinates) in the 4D millimeter wave radar;
[0084] (2) Use the 4D millimeter wave radar to camera external parameter calibration script to read the camera internal parameters, the pixel coordinates of the radar corner reflectors in the image arranged in sequence, and the point cloud coordinates of the radar corner reflectors in the 4D millimeter wave radar in the same order, call the external parameter calculation function, and obtain the corresponding 4D millimeter wave radar to camera external parameters.
[0085] 3. Reprojection verification and calibration parameter fine-tuning module calculates the reprojection error and determines whether the reprojection error meets the accuracy requirements. If not, fine-tune the external parameters until the accuracy requirements are met. The specific steps are:
[0086] (1) Taking the external parameters from the 4D millimeter wave radar to the camera, the internal parameters of the camera, and the point cloud collected from the original 4D millimeter wave radar as input, the pixel coordinates after reprojection are calculated;
[0087] (2) Calculate the average error between the pixel coordinates after reprojection and the pixel coordinates of the radar corner reflector on the original acquired distorted image to obtain the reprojection error;
[0088] (3) Determine whether the reprojection error meets the accuracy requirements. If not, fine-tune the translation and rotation in the extrinsic parameters to obtain the fine-tuned extrinsic parameters and repeat the above steps until the accuracy requirements are met.
[0089] The present invention aims at the sensor characteristics of 4D millimeter wave radar, and can obtain a set of coordinate points of the pixel coordinate system and the millimeter wave radar coordinate system of the same scene, and realize external parameter calibration based on the collected multiple sets of coordinates. It solves the problem of difficult coordinate association in the external parameter calibration of 4D millimeter wave radar and camera.
[0090] The present invention realizes visualization of the extrinsic parameter calibration process of 4D millimeter-wave radar and camera, effectively assists the extrinsic parameter calibration, and is conducive to intuitively selecting better calibration scenes and calibration data.
[0091] Embodiment 3
[0092] In some embodiments, Figure 4 As shown, as a preferred embodiment of the present invention, the acquisition steps of the coordinate acquisition module are as follows:
[0093] (1) Select a data collection scene, place the equipment to be collected, place the equipment to be calibrated in the selected area X, and place the radar corner reflector in the common view area A, so that the radar corner reflector can be seen in the image display of the visualization system, and the only high-intensity point can be seen in the point cloud display, and it can be determined that it corresponds to the radar corner reflector;
[0094] (2) Start the device, open the visualization system and data acquisition script, and ensure that the hardware and software environment are fully prepared;
[0095] (3) Set the position of the radar corner reflector and confirm the point cloud of the image and 4D millimeter-wave radar in the visualization to ensure that the point with the strongest point cloud intensity is on the radar corner reflector;
[0096] (4) Run the acquisition script to save the camera image and millimeter-wave radar point cloud data at this time;
[0097] (5) Adjust the position of the radar corner reflector and repeat steps (3) to (4) to try to ensure that the position of the radar corner reflector is distributed in different distances and angle ranges. The process ends when the position, angle, and number of the collected radar corner reflectors meet the requirements.
[0098] Embodiment 4
[0099] In some embodiments, Figure 5 As shown, as a preferred embodiment of the present invention, the calculation steps of the external parameter calculation module are as follows:
[0100] (1) Determine whether the collected scene has been analyzed. If not, analyze the next set of collected camera images and point cloud data, obtain the points of the radar corner reflectors from the point cloud, dedistort the images, obtain the pixel coordinates of the corresponding radar corner reflectors on the dedistorted images, and then obtain a set of 4D millimeter-wave radar coordinates and pixel coordinates of the radar corner reflectors; if yes, obtain multiple sets of 4D millimeter-wave radar coordinates and pixel coordinates of the radar corner reflectors;
[0101] (2) The external parameters are calculated by using the camera internal parameters, pixel coordinates and 4D millimeter wave radar coordinates.
[0102] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A 4D millimeter wave radar and camera extrinsic parameter calibration method, characterized in that: The following steps are involved:
1. The coordinate acquisition module collects camera images and millimeter wave radar point cloud data; Second, the external parameter calculation module calculates the external parameters corresponding to the 4D millimeter wave radar to the camera; 3. The reprojection verification and calibration parameter fine-tuning module calculates the reprojection error and determines whether the reprojection error meets the accuracy requirements. If not, the external parameters are fine-tuned until the accuracy requirements are met.
2. The 4D millimeter wave radar and camera extrinsic parameter calibration method according to claim 1, characterized in that: Step 1 The specific steps are: (1) Place the device to be calibrated in area X and the radar corner reflector in the common view area A, ensuring that the radar corner reflector can be seen in the image display of the visualization system, and that a unique high-intensity point can be seen in the point cloud display, and that it can be determined that it corresponds to the radar corner reflector; (2) Run the acquisition script to save the camera image and millimeter-wave radar point cloud data at this time; (3) Adjust the position of the radar corner reflector and repeat steps (1) to (2).
3. The 4D millimeter wave radar and camera extrinsic parameter calibration method according to claim 2, characterized in that: Step 2 The specific steps are: (1) Analyze the camera image and point cloud data obtained by the coordinate acquisition module to find the pixel coordinates of the radar corner reflector in the dedistorted image and the corresponding point cloud coordinates in the 4D millimeter wave radar; (2) Use the 4D millimeter wave radar to camera external parameter calibration script to read the camera internal parameters, the pixel coordinates of the radar corner reflectors in the image arranged in sequence, and the point cloud coordinates of the radar corner reflectors in the 4D millimeter wave radar in the same order, call the external parameter calculation function, and obtain the corresponding 4D millimeter wave radar to camera external parameters.
4. The 4D millimeter wave radar and camera extrinsic parameter calibration method according to claim 3, characterized in that: Step 3 The specific steps are: (1) Taking the external parameters from the 4D millimeter wave radar to the camera, the internal parameters of the camera, and the point cloud collected from the original 4D millimeter wave radar as input, the pixel coordinates after reprojection are calculated; (2) Calculate the average error between the pixel coordinates after reprojection and the pixel coordinates of the radar corner reflector on the original acquired distorted image to obtain the reprojection error; (3) Determine whether the reprojection error meets the accuracy requirements. If not, fine-tune the translation and rotation in the extrinsic parameters to obtain the fine-tuned extrinsic parameters and repeat the above steps until the accuracy requirements are met.
5. The 4D millimeter wave radar and camera extrinsic parameter calibration method according to claim 4, characterized in that: The coordinate acquisition module includes: A device to be calibrated includes two sensors: a 4D millimeter wave radar and a camera, and the 4D millimeter wave radar and the camera have a common view area A; A visualization system that displays the camera image and 4D mmWave point cloud in real time; A data acquisition system that can simultaneously collect images and 4D millimeter wave radar point cloud information through scripts; A highly reflective, light-proof radar corner reflector.
6. The 4D millimeter wave radar and camera extrinsic parameter calibration method according to claim 5, characterized in that: The visualization system includes two display windows, which are used to display the camera image and the 4D millimeter wave point cloud in real time respectively.
7. The 4D millimeter wave radar and camera extrinsic parameter calibration method according to claim 6, characterized in that: The calculation steps of the external parameter calculation module are as follows: (1) Determine whether the collected scene has been analyzed. If not, analyze the next set of collected camera images and point cloud data, obtain the points of the radar corner reflector from the point cloud, dedistort the image, obtain the pixel coordinates of the corresponding radar corner reflector on the dedistorted image, and then obtain a set of 4D millimeter-wave point cloud coordinates and pixel coordinates of the radar corner reflector; if yes, obtain multiple sets of 4D millimeter-wave point cloud coordinates and pixel coordinates of the radar corner reflector; (2) The external parameters are calculated by using the camera internal parameters, pixel coordinates and 4D millimeter wave point cloud coordinates.
Citation Information
Patent Citations
Laser radar and camera external parameter calibration method and device
CN114758005A
Cited By
High-precision millimeter wave radar-camera calibration method and system based on angle reflection
CN120235960A
