A method for calibrating a low-resolution millimeter wave radar and an optical camera

Abstract

This invention discloses a calibration method for low-resolution millimeter-wave radar and optical cameras. The method fuses single-frame radar point cloud information acquired by the millimeter-wave radar with pose information acquired by a pose sensor to output a point cloud map containing calibration board position information. Point cloud filtering and clustering algorithms are used to process the point cloud map to extract the millimeter-wave radar point cloud of the calibration board. A target segmentation algorithm is used to segment the two-dimensional edge feature point position information of the calibration board in the original image information. A three-dimensional model of the calibration board is constructed based on its structural priors and registered with the millimeter-wave radar point cloud blocks to obtain the fitted three-dimensional edge feature point position information of the calibration board. This fitted model is then matched with the segmented two-dimensional edge feature point position information to obtain multiple sets of two-dimensional and three-dimensional matched feature point pairs. These multiple sets of two-dimensional and three-dimensional matched feature point pairs are used as input to the PnP algorithm to calculate the extrinsic parameter matrix used for millimeter-wave radar and camera calibration.

Description

Technical Field

[0001] This invention relates to the field of extrinsic parameter calibration technology for multi-sensor fusion, and specifically to a calibration method for low-resolution millimeter-wave radar and optical cameras. Background Technology

[0002] 3D environmental perception has always been crucial for tasks such as autonomous vehicle localization, path planning and navigation, and 3D reconstruction. Sensors such as LiDAR, millimeter-wave radar, and optical cameras play important roles in 3D environmental perception. Compared to radar, optical cameras have received widespread attention due to their ability to acquire dense RGB information of a scene; however, neither monocular nor binocular cameras can obtain accurate 3D information of a scene. Therefore, in practical applications, they are often fused with other ranging sensors, such as LiDAR, to complete tasks such as scale-based 3D reconstruction, real-time localization, and mapping. In recent years, millimeter-wave radar has attracted significant attention from scholars both domestically and internationally due to its ability to operate in special environments such as heavy fog and heavy rain, resulting in numerous research findings. However, currently available millimeter-wave radar still has many drawbacks compared to LiDAR, such as sparse point cloud data in a single frame, low accuracy, susceptibility to interference, and numerous noise points. This makes spatial calibration of millimeter-wave radar with other sensors more difficult. Multi-sensor spatial calibration is often a key step in completing multi-source information fusion perception algorithms.

[0003] Calibration of millimeter-wave radar and cameras typically involves matching the radar-measured target velocity, range, and horizontal azimuth information with images or image streams. Examples include methods using highly reflective calibration objects and adding geometric constraints to the calibration equations to achieve 360° registration between the millimeter-wave radar and camera; and methods using the radar-measured target velocity dimension information and finding corresponding targets in image sequences to calibrate the millimeter-wave radar and camera. However, these calibration methods are primarily used for 3D reconstruction of large outdoor environments. The millimeter-wave radars used in these methods have high resolution, and the calibration schemes lack universality. In practical engineering, due to cost constraints, most applications use millimeter-wave radar with sparser data, making the aforementioned calibration methods unsuitable for spatial calibration of millimeter-wave radar and optical cameras. Summary of the Invention

[0004] The purpose of this invention is to provide a calibration method for low-resolution millimeter-wave radar and optical cameras. The calibration results have high accuracy and are effective and robust in relatively open environments and environments with a lot of clutter, and have a wide range of applications.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A calibration method for low-resolution millimeter-wave radar and optical camera, comprising the following steps:

[0007] S1. Set up the calibration board and calibration acquisition equipment, which includes a camera, millimeter-wave radar, and pose sensor;

[0008] S2. The camera acquires the original image information of the calibration board, the millimeter-wave radar acquires the single-frame radar point cloud information of the calibration board, and the pose sensor acquires the pose information of the mobile platform.

[0009] S3. Fuse single-frame radar point cloud information with pose information to output a point cloud map containing calibration board position information.

[0010] S4. The point cloud map is processed using point cloud filtering and clustering algorithms to extract the millimeter-wave radar point cloud blocks of the calibration board.

[0011] S5. Use the target segmentation algorithm to segment the two-dimensional edge feature point location information of the calibration board in the original image information;

[0012] S6. Construct a three-dimensional model of the calibration board based on its structural priors, and register it with the millimeter-wave radar point cloud block to obtain the fitted three-dimensional edge feature point position information of the calibration board. Then match it with the two-dimensional edge feature point position information in step S5 to obtain multiple sets of two-dimensional and three-dimensional matching feature point pairs.

[0013] S7. Use multiple sets of two-dimensional and three-dimensional matching feature point pairs as input to the PnP algorithm to calculate the extrinsic parameter matrix used for millimeter-wave radar and camera calibration.

[0014] As a preferred embodiment, step S3 includes the following steps:

[0015] S3.1 Delineate the effective area in the radar point cloud information and filter out low-precision points in a single frame;

[0016] S3.2 Filter out millimeter-wave radar noise based on preset CFAR threshold;

[0017] S3.3. Based on the pose information corresponding to each frame of radar point cloud information, unify the radar point cloud information to the global coordinate system.

[0018] S3.4 Use a probability model to determine the probability of a voxel being occupied when given a measurement value to determine whether a map point should be used;

[0019] S3.5. Based on the map points used in step S3.4, complete the millimeter-wave radar mapping and output a point cloud map containing calibration board location information.

[0020] As a preferred embodiment, step S3.3, which unifies the radar point cloud information to a global coordinate system based on the pose information corresponding to each frame of radar point cloud information, uses the following calculation formula:

[0021]

[0022] in, This represents the coordinates of a single-frame radar point cloud in the global coordinate system obtained by millimeter-wave radar detection at time t. This represents the coordinates of a single frame of radar point cloud detected by millimeter-wave radar in the local coordinate system at time t. Let represent the 3D rotation matrix from the lower global coordinate system to the camera coordinate system at time t. Let t represent the three-dimensional translation vector from the global coordinate system to the camera coordinate system at time t.

[0023] As a preferred option, the probability model used in step S3.4 is:

[0024]

[0025] in Let P(n) be the probability that node n is occupied given the measurement value before time t, and let P(n) be the prior probability. Let n be the probability that node n is occupied at time t-1 given a measurement value. This represents the probability that node n is occupied at time t given a measurement value. This represents the radar point cloud coordinates in the global coordinate system for all frames detected by the millimeter-wave radar before time t.

[0026] As a preferred embodiment, step S4 includes the following steps:

[0027] S4.1. Downsample the point cloud map and use a point cloud filtering algorithm to filter out outliers;

[0028] S4.2. Use the Euclidean clustering algorithm to cluster different parts of the point cloud map;

[0029] S4.3. Based on the edge and surface features of each point cloud class, extract the millimeter-wave radar point cloud blocks of the calibration board.

[0030] As a preferred option, step S1 specifically involves:

[0031] Two reflectors at a 90° angle are used as calibration plates, and millimeter-wave radar and camera are fixedly installed on a mobile platform.

[0032] As a preferred option, the three-dimensional model described in step S6 is constructed based on the actual size and included angle of the calibration plate.

[0033] As a preferred option, in step S6, the iterative nearest point algorithm is used to match the three-dimensional model of the calibration board with the millimeter-wave radar point cloud block.

[0034] As a preferred option, the pose sensor uses a wheeled odometer.

[0035] As a preferred embodiment, step S7 includes the following steps:

[0036] S7.1, Optimization issues:

[0037]

[0038] S7.2. Transform the solution of the extrinsic parameter matrix into a nonlinear least squares problem:

[0039]

[0040] S7.3 Solve for the calibration matrix using the LM optimization method;

[0041] Among them, z c Let M be the scale factor, M be the intrinsic parameter matrix of the camera, R be the 3D rotation matrix from the millimeter-wave radar coordinate system to the camera coordinate system, T be the 3D translation vector between the millimeter-wave radar coordinate system and the camera coordinate system, and p' be the scale factor. P P represents the estimated coordinates of two-dimensional edge feature points. r To determine the coordinates of the three-dimensional edge feature points of the calibration board, p r These are the coordinates of two-dimensional edge feature points.

[0042] The beneficial effects of this invention are:

[0043] 1. This invention fuses single-frame radar point cloud information acquired by millimeter-wave radar with pose information acquired by a pose sensor to output a point cloud map containing calibration board position information. Point cloud filtering and clustering algorithms are used to process the point cloud map to extract the millimeter-wave radar point cloud of the calibration board. A target segmentation algorithm is used to segment the two-dimensional edge feature point position information of the calibration board in the original image information. A three-dimensional model of the calibration board is constructed based on its structural priors and registered with the millimeter-wave radar point cloud blocks to obtain the fitted three-dimensional edge feature point position information of the calibration board. This fitted model is then matched with the segmented two-dimensional edge feature point position information to obtain multiple sets of two-dimensional and three-dimensional matched feature point pairs. These multiple sets of two-dimensional and three-dimensional matched feature point pairs are used as input to the PnP algorithm to calculate the extrinsic parameter matrix used for millimeter-wave radar and camera calibration. The extrinsic parameter matrix obtained using the algorithm described in this invention achieves high accuracy in millimeter-wave radar and optical camera calibration tasks, and is effective and robust in relatively open environments and environments with a lot of clutter, making it suitable for a wide range of applications.

[0044] 2. This invention is the first to use a calibration board to complete the calibration task of low-resolution millimeter-wave radar and optical camera. The calibration environment and calibration acquisition equipment are simple and easy to operate. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart of a calibration method for a low-resolution millimeter-wave radar and optical camera as described in this invention.

[0047] Figure 2 This is a diagram showing the fixed installation location of the millimeter-wave radar and optical camera on a mobile platform.

[0048] Figure 3 This is a data comparison chart showing the accuracy quantification results of the reprojection effect obtained by manually extracting feature points of the calibration board and by extracting feature points of the calibration board using the algorithm described in this invention in the calibration environment and scenario one, respectively.

[0049] Figure 4 The image shows the precision quantification results of the reprojection effect obtained in two experiments, Scene 1 and Scene 2, using the algorithm described in this invention. Detailed Implementation

[0050] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0051] Reference Figure 1 This embodiment provides a calibration method for low-resolution millimeter-wave radar and optical camera, including the following steps:

[0052] S1. Set up the calibration board and calibration acquisition equipment, which includes a camera, millimeter-wave radar, and pose sensor;

[0053] S2. The camera acquires the original image information of the calibration board, the millimeter-wave radar acquires the single-frame radar point cloud information of the calibration board, and the pose sensor acquires the pose information of the mobile platform.

[0054] S3. Fuse single-frame radar point cloud information with pose information to output a point cloud map containing calibration board position information.

[0055] S4. The point cloud map is processed using point cloud filtering and clustering algorithms to extract the millimeter-wave radar point cloud blocks of the calibration board.

[0056] S5. Use the target segmentation algorithm to segment the two-dimensional edge feature point location information of the calibration board in the original image information;

[0057] S6. Construct a three-dimensional model of the calibration board based on its structural priors, and register it with the millimeter-wave radar point cloud block to obtain the fitted three-dimensional edge feature point position information of the calibration board. Then match it with the two-dimensional edge feature point position information in step S5 to obtain multiple sets of two-dimensional and three-dimensional matching feature point pairs.

[0058] S7. Use multiple sets of two-dimensional and three-dimensional matching feature point pairs as input to the PnP algorithm to calculate the extrinsic parameter matrix used for millimeter-wave radar and camera calibration.

[0059] The extrinsic parameter matrix obtained using the algorithm described in this invention achieves high accuracy in calibration tasks for millimeter-wave radar and optical cameras. It is effective and robust in both open and cluttered environments, making it suitable for a wide range of applications.

[0060] Specifically:

[0061] Step S3 includes the following steps:

[0062] S3.1 Delineate the effective area in the radar point cloud information and filter out low-precision points in a single frame;

[0063] S3.2 Filter out millimeter-wave radar noise based on preset CFAR threshold;

[0064] S3.3. Based on the pose information corresponding to each frame of radar point cloud information, unify the radar point cloud information to the global coordinate system.

[0065] S3.4 Use a probability model to determine the probability of a voxel being occupied when given a measurement value to determine whether a map point should be used;

[0066] S3.5. Based on the map points used in step S3.4, complete the millimeter-wave radar mapping and output a point cloud map containing calibration board location information.

[0067] In step S3.3, the radar point cloud information is unified to the global coordinate system based on the pose information corresponding to each frame of radar point cloud information. The calculation formula used is as follows:

[0068]

[0069] in, This represents the coordinates of a single-frame radar point cloud in the global coordinate system obtained by millimeter-wave radar detection at time t. This represents the coordinates of a single frame of radar point cloud detected by millimeter-wave radar in the local coordinate system at time t. Let represent the 3D rotation matrix from the lower global coordinate system to the camera coordinate system at time t. Let t represent the three-dimensional translation vector from the global coordinate system to the camera coordinate system at time t.

[0070] The probability model used in step S3.4 is:

[0071]

[0072] in Let P(n) be the probability that node n is occupied given the measurement value before time t, and let P(n) be the prior probability. Let n be the probability that node n is occupied at time t-1 given a measurement value. This represents the probability that node n is occupied at time t given a measurement value. This represents the radar point cloud coordinates in the global coordinate system for all frames detected by the millimeter-wave radar before time t.

[0073] Step S4 includes the following steps:

[0074] S4.1. Downsample the point cloud map and use a point cloud filtering algorithm to filter out outliers;

[0075] S4.2. Use the Euclidean clustering algorithm to cluster different parts of the point cloud map;

[0076] S4.3. Based on the edge and surface features of each point cloud class, extract the millimeter-wave radar point cloud blocks of the calibration board.

[0077] Step S1 is as follows:

[0078] Two reflectors at a 90° angle are used as calibration plates. A millimeter-wave radar and camera are fixedly mounted on a mobile platform. The optical camera of this invention uses the left camera of a ZED2 binocular camera as a monocular camera, with a resolution of 672×376, a horizontal field of view of 110°, and a maximum vertical field of view of 70°. The camera's intrinsic parameters default to the official data from the ZED2. The millimeter-wave radar uses the TI IWR1443, with a maximum horizontal field of view of 120°, a maximum pitch angle of 30°, a preferred range resolution of 4cm, a horizontal angular resolution of 15°, and a vertical angular resolution of 60°. The mobile robot platform used is the Turtlebot2. See [link to documentation] for the fixed mounting positions of the millimeter-wave radar and camera. Figure 2.

[0079] The pose sensor mentioned in step S2 is a wheeled odometer. The pose information of the mobile platform collected by the pose sensor is the position and orientation of the mobile platform, and the changes in pose information include both translation and rotation.

[0080] The three-dimensional model described in step S6 is constructed based on the actual size and included angle of the calibration plate.

[0081] In step S6, the iterative nearest point algorithm is used to match the 3D model of the calibration board with the millimeter-wave radar point cloud blocks.

[0082] Step S7 includes the following steps:

[0083] S7.1, Optimization issues:

[0084]

[0085] S7.2. Transform the solution of the extrinsic parameter matrix into a nonlinear least squares problem:

[0086]

[0087] S7.3 Solve for the calibration matrix using the LM optimization method;

[0088] Among them, z c Let M be the scale factor, M be the intrinsic parameter matrix of the camera, R be the 3D rotation matrix from the millimeter-wave radar coordinate system to the camera coordinate system, T be the 3D translation vector between the millimeter-wave radar coordinate system and the camera coordinate system, and p' be the scale factor. P P represents the estimated coordinates of two-dimensional edge feature points. r To determine the coordinates of the three-dimensional edge feature points of the calibration board, p r These are the coordinates of two-dimensional edge feature points.

[0089] The advantages of the calibration method described in this embodiment are illustrated through specific experiments below:

[0090] The calibration environment in the experiment was as follows: in an empty room, there were only two reflectors at a 90° angle to each other, which served as calibration plates; Scenario 1 was an empty room with other interfering objects in addition to the two calibration plates; Scenario 2 was a room with more clutter than Scenario 1.

[0091] Six 3D feature points were extracted from the point cloud block of the calibration board using both manual extraction and the algorithm described in this invention. In the calibration environment and Scene 1, the position information of the 3D feature points extracted manually and using the algorithm described in this invention were matched with the segmented 2D feature point position information. The six sets of matched 2D and 3D feature point pairs obtained using the two methods were used as input to the calibration algorithm to obtain the calibration results. For a comparison of the accuracy quantization results of the reprojection effect obtained using the two different methods in the constructed calibration environment and Scene 1, please refer to [link to relevant documentation]. Figure 3 .

[0092] according to Figure 3 The experimental results show that the method of manually extracting feature points achieves better reprojection results in the calibration environment, but it obtains an incorrect local optimum, resulting in poor projection of the point cloud onto the target pixel region in Scene 1. In contrast, the algorithm described in this invention, which extracts feature points, achieves high reprojection accuracy in both the calibration environment and Scene 1, and in Scene 1, the reprojection effect is significantly better than that of the method using manual feature point extraction.

[0093] Using the extrinsic parameter matrix obtained through the algorithm described in this invention, two experiments were conducted in scene one and scene two, respectively. The accuracy quantification results of the reprojection effect are shown in [reference]. Figure 4 .according to Figure 4 The experimental results shown demonstrate that the extrinsic parameter matrix obtained using the algorithm described in this invention is effective and robust in both open and cluttered environments, with an average reprojection verification accuracy of around 0.7.

[0094] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope of the present invention.

Claims

1. A calibration method for a low-resolution millimeter-wave radar and an optical camera, characterized in that, Including the following steps: S1. Set up the calibration board and calibration acquisition equipment, which includes a camera, millimeter-wave radar, and pose sensor; S2. The camera acquires the original image information of the calibration board, the millimeter-wave radar acquires the single-frame radar point cloud information of the calibration board, and the pose sensor acquires the pose information of the mobile platform. S3. Fuse single-frame radar point cloud information with pose information to output a point cloud map containing calibration board position information. S4. The point cloud map is processed using point cloud filtering and clustering algorithms to extract the millimeter-wave radar point cloud blocks of the calibration board. S5. Use the target segmentation algorithm to segment the two-dimensional edge feature point location information of the calibration board in the original image information; S6. Construct a three-dimensional model of the calibration board based on its structural priors, and register it with the millimeter-wave radar point cloud block to obtain the fitted three-dimensional edge feature point position information of the calibration board. Then match it with the two-dimensional edge feature point position information in step S5 to obtain multiple sets of two-dimensional and three-dimensional matching feature point pairs. S7. Use multiple sets of two-dimensional and three-dimensional matching feature point pairs as input to the PnP algorithm to calculate the extrinsic parameter matrix used for millimeter-wave radar and camera calibration. Step S3 includes the following steps: S3.1 Delineate the effective area in the radar point cloud information and filter out low-precision points in a single frame; S3.2 Filter out millimeter-wave radar noise based on preset CFAR threshold; S3.

3. Based on the pose information corresponding to each frame of radar point cloud information, unify the radar point cloud information to the global coordinate system. S3.4 Use a probability model to determine the probability of a voxel being occupied when given a measurement value to determine whether a map point should be used; S3.

5. Based on the map points used in step S3.4, complete the millimeter-wave radar mapping and output a point cloud map containing calibration board location information. In step S3.3, the radar point cloud information is unified to the global coordinate system based on the pose information corresponding to each frame of radar point cloud information. The calculation formula used is as follows: ， in, express The coordinates of a single-frame radar point cloud in the global coordinate system obtained by millimeter-wave radar detection at a given moment. express The coordinates of a single-frame radar point cloud in the local coordinate system obtained by millimeter-wave radar detection at a given moment. express The 3D rotation matrix from the local global coordinate system to the camera coordinate system at any given time. express The 3D translation vector from the global coordinate system to the camera coordinate system at time t.

2. The calibration method for a low-resolution millimeter-wave radar and optical camera according to claim 1, characterized in that, The probability model used in step S3.4 is: ， in for When the measured value is given before the time, the node The probability of being occupied. For prior probability, for When the measured value is given at any time, the node The probability of being occupied. express When the measured value is given at any time, the node The probability of being occupied. express The radar point cloud coordinates of all frames detected by the millimeter-wave radar in the global coordinate system before the specified time.

3. The calibration method for a low-resolution millimeter-wave radar and optical camera according to claim 1, characterized in that, Step S4 includes the following steps: S4.

1. Downsample the point cloud map and use a point cloud filtering algorithm to filter out outliers; S4.

2. Use the Euclidean clustering algorithm to cluster different parts of the point cloud map; S4.

3. Based on the edge and surface features of each point cloud class, extract the millimeter-wave radar point cloud blocks of the calibration board.

4. The calibration method for a low-resolution millimeter-wave radar and optical camera according to claim 1, characterized in that, Step S1 is as follows: Two reflectors at a 90° angle are used as calibration plates, and millimeter-wave radar and camera are fixedly installed on a mobile platform.

5. The calibration method for a low-resolution millimeter-wave radar and optical camera according to claim 4, characterized in that, The three-dimensional model described in step S6 is constructed based on the actual size and included angle of the calibration plate.

6. The calibration method for a low-resolution millimeter-wave radar and optical camera according to claim 5, characterized in that, In step S6, the iterative nearest point algorithm is used to match the 3D model of the calibration board with the millimeter-wave radar point cloud blocks.

7. The calibration method for a low-resolution millimeter-wave radar and optical camera according to claim 1, characterized in that, The pose sensor uses a wheel-type odometer.

8. The calibration method for a low-resolution millimeter-wave radar and optical camera according to claim 1, characterized in that, Step S7 includes the following steps: S7.1, Construction Optimization Problem: ， S7.

2. Transform the solution of the extrinsic parameter matrix into a nonlinear least squares problem: ， S7.3 Solve for the calibration matrix using the LM optimization method; in, As a scale factor, This is the intrinsic parameter matrix of the camera. This is a three-dimensional rotation matrix from the millimeter-wave radar coordinate system to the camera coordinate system. Let be the three-dimensional translation vector between the millimeter-wave radar coordinate system and the camera coordinate system. These are the estimated coordinates of two-dimensional edge feature points. To determine the coordinates of the three-dimensional edge feature points of the calibration plate, These are the coordinates of two-dimensional edge feature points.

Images