An online automated joint calibration method for radar-visual fusion without special calibration objects

By using azimuth angle and angle change rate in the lightning fusion to construct the radar-camera corresponding point set and perform automated joint calibration, the interpretability and real-time problems of the targetless calibration method in the existing technology are solved, and ordinary monitoring is achieved Accurate thunder vision fusion and digital twin effects in the scene.

CN118311513BActive Publication Date: 2024-12-03SHANGHAI LIMING RUIDA ELECTRONIC TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410397055.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-03
Publication Date
2024-12-03
Estimated Expiration
2044-04-03

AI Technical Summary

Technical Problem

The existing targetless calibration method has poor interpretability, is sensitive to device jitter and cannot achieve real-time online updates in the lightning vision fusion, resulting in large errors in matching points between the radar and camera coordinate systems, and cannot be used in ordinary monitoring scenarios. Achieve accurate Leivis fusion automation joint calibration.

Method used

By obtaining the azimuth angle and angular change rate of millimeter wave radar and camera, a radar-camera corresponding point set that conforms to the principle of cone uniqueness is constructed, the initial external parameter matrix is ​​calculated, and coarse calibration and fine calibration is performed through image projection and data matching. The Hungarian algorithm performs data matching to realize automated joint calibration without external parameters, ensuring the reliability and real-time update of calibration results.

Benefits of technology

It realizes automatic joint calibration of Leishi fusion without external parameters, avoids misupdated updates caused by equipment jitter, ensures the accuracy and real-timeness of calibration results, and is suitable for Leishi fusion and digital twin effects in congested scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118311513B_ABST
    Figure CN118311513B_ABST
Patent Text Reader

Abstract

The present invention discloses an online radar-vision fusion automatic joint calibration method without special calibration objects, which relates to the field of intelligent transportation and is used to solve the problem of radar-vision fusion automatic calibration. The method mainly includes: rough calibration without any external parameter preset input, finding point correspondence through common characteristic angles and angle change rates in radar coordinate system and pixel coordinate system; rough calibration completion judgment, using the matching rate of the number of matches after rough calibration to the number of radar detection targets as the judgment basis; updating the external parameter matrix, saving a frame of data, and using the updated external parameter matrix to calculate the number of matches and the reprojection error in the radar coordinate system as the judgment criteria for subsequent steps; using the rough positioning or the previous round of precise positioning external parameter matrix to find the point corresponding to the precise calibration in a larger range; precise calibration update judgment, and compare with the previous round of precise calibration results to ensure that the effect of each update is better. The fully automated joint calibration of radar and vision is realized and can be updated online in real time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of intelligent transportation, and in particular to an online radar-visual fusion automatic joint calibration method without special calibration objects. Background Art

[0002] Intelligent roadside perception equipment for vehicle-road collaboration can perform holographic perception of traffic participants in real time. It can not only provide traffic conditions outside the perception range of single vehicles, but also provide a digital twin perspective of real-time traffic scenes for smart cities, thereby realizing refined traffic management and control. Intelligent roadside perception equipment has developed from single-sensor perception (such as millimeter-wave radar, lidar, camera, etc.) to multi-sensor fusion perception. Fusion perception can simultaneously play the advantages of each sensor and make up for the disadvantages of each sensor. Among them, the fusion of millimeter-wave radar and video has gradually become the mainstream development trend of intelligent roadside perception equipment due to its strong complementarity and low price.

[0003] Since the detection results of millimeter-wave radar and camera are in two different coordinate systems, the prerequisite for radar-vision fusion is to realize the conversion between coordinate systems and unify the detection results into the same coordinate system, that is, joint calibration technology or spatial synchronization technology. Joint calibration technology is divided into target and non-target methods. The target calibration method needs to rely on special objects, such as corner reflectors and chessboards. The pixel coordinates of special calibration objects are manually marked to form a corresponding point set with the coordinates of special targets detected by radar, and then the transformation matrix is ​​solved. There are also RTK devices used to find the corresponding point set of camera-global coordinates and the corresponding point set of radar-global coordinates, and the camera-global coordinate transformation matrix and radar-global coordinate transformation matrix are respectively calculated. There are also radar data that estimates its own speed in the geodetic coordinate system and radar coordinate system to solve the radar-geod coordinate transformation matrix, and the camera-geod coordinate transformation matrix is ​​solved by the camera's own attitude estimation. The target calibration method relies on special calibration objects and scenes, and when the device changes the transformation matrix due to vibration, temperature change, wear and other factors, it cannot be calibrated in time. The target-free calibration method can overcome the defects of the target calibration method, so it has gradually become a hot research topic. Target-free calibration uses neural networks to train radar RAD (Range-Azimuth-Doppler) graphs and images separately to obtain unified classification results, and find radar-camera corresponding point sets based on types. This method must be calibrated in a scene with a unique category. There are also neural networks that directly train radar targets to image frames, but this method requires separate training for each set of equipment, which is less practical. There is also a method of presetting the transformation matrix, finding the corresponding point set inside the neural network, and directly training the transformation matrix error. With the development of video detection technology, video 3D detection has become possible, and the bird's-eye view generated by it only requires simple processing to be directly aligned with the radar coordinate system.

[0004] The target-free calibration method needs to find the common features of radar and camera to construct the corresponding point set. However, millimeter-wave radar sensors cannot provide detailed structural information, so they all rely on neural networks to learn the deeper common features between them. However, the uninterpretability of neural network feature extraction and the jitter of millimeter-wave radar data will lead to mismatching of corresponding point sets, resulting in large deviations in the solution of the PnP problem. Therefore, there is an urgent need for an automatic joint calibration method of radar-vision fusion that is interpretable, suitable for ordinary monitoring scenarios, does not require any preset parameters, and can be updated online in real time. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides an online radar-visual fusion automatic joint calibration method without special calibration objects, comprising the following steps:

[0006] S1: Acquire millimeter wave radar detection data and camera detection data, and calculate the azimuth angle and angle change rate of the millimeter wave radar detection data and the camera detection data;

[0007] S2: confirming radar-camera corresponding points that meet the unique principle of the cone of vision according to the azimuth angle and the angle change rate, constructing a radar-camera corresponding point set that meets the set configuration requirements according to the millimeter-wave radar detection data and the camera detection data under multiple frames, obtaining the intrinsic parameter matrix of the millimeter-wave radar and the camera, obtaining the camera distortion coefficient, and calculating the initial external parameter matrix according to the radar-camera corresponding point set, the intrinsic parameter matrix and the camera distortion coefficient;

[0008] S3: Acquire the last frame of radar detection data and the last frame of camera detection data, and perform a rough calibration verification on the initial external parameter matrix according to the last frame of radar detection data and the last frame of camera detection data. After the verification passes, use the initial external parameter matrix as the rough calibration matrix and execute S4. If the verification fails, repeat S1 to S3.

[0009] S4: performing image projection on the radar detection data using the coarse calibration matrix to obtain projection image data, performing data matching on the radar detection data and the projection image data, performing reliability screening and maximum distribution screening on the radar detection data and the projection image data according to the result of the data matching, constructing a final corresponding point set according to the data that pass the reliability screening and the maximum distribution screening, and calculating a precise calibration external parameter matrix according to the final corresponding point set, the intrinsic parameter matrix and the camera distortion coefficient;

[0010] S5: performing image projection on the last frame of radar detection data in S3 using the precise calibration external parameter matrix to obtain precise calibration verification image data, and performing data matching on the precise calibration verification camera data and the last frame of camera detection data using the Hungarian algorithm to obtain the number of precise calibration matches;

[0011] S6: Perform precision calibration verification on the precision calibration extrinsic parameter matrix according to the precision calibration matching number. If the verification is passed, use the precision calibration extrinsic parameter matrix as an update matrix and repeat S4 to S6. If the verification fails, repeat steps S4 to S6.

[0012] Specifically, the millimeter wave radar detection data is [x, y, 0] T The calculation method of the azimuth and angle change rate of the millimeter wave radar detection data is as follows:

[0013]

[0014] in, represents the azimuth of the millimeter-wave radar, represents the angle change rate of the millimeter-wave radar, k represents the time, h represents the radar installation height, T represents the radar frame interval, and w represents the set weight.

[0015] Specifically, the camera detection data is [u, v] T , the calculation method of the azimuth and angle change rate of the camera detection data is as follows:

[0016]

[0017] in, represents the azimuth angle of the camera, represents the angle change rate of the camera, f is the focal length of the camera, cu and cv are the camera intrinsic parameters.

[0018] Specifically, the method for confirming the radar-camera corresponding points that meet the unique principle of the cone of vision in S2 is as follows:

[0019] S201: Filtering out data corresponding to the only azimuth angle that exists within a set angle range of ±Θ from the azimuth angles of each frame of millimeter-wave radar detection data and camera detection data;

[0020] S202: Filtering out data whose speed of the millimeter-wave radar detection data is greater than a set threshold from the data filtered in S201;

[0021] S203: Use a formula to traverse the remaining data after S201 and S202 to obtain the radar-camera corresponding points. The formula is:

[0022]

[0023] Among them, θ thres ,ω thres is the threshold for determining whether the data corresponds to the same target. is the angle between the normal direction of the millimeter-wave radar coordinates and the normal direction of the camera.

[0024] Specifically, the configuration requirement in S2 is that the distance between the radar-camera corresponding points in the radar-camera corresponding point set is less than a set distance threshold.

[0025] Specifically, the reliability screening in S4 is as follows: in a continuous set period, the target IDs of the same group of the millimeter-wave radar detection data and the camera detection data can be successfully matched, and the target speed of the millimeter-wave radar detection data is greater than the set speed threshold.

[0026] Specifically, the maximum distribution screening in S4 is as follows: the distance between the millimeter-wave radar detection data is greater than a set distance threshold, and there are no three collinear millimeter-wave radar detection data.

[0027] Specifically, the rough calibration verification in S3 specifically includes:

[0028] The initial external parameter matrix is ​​used to perform image projection on the last frame of radar detection data to obtain coarse calibration verification camera data, and the coarse calibration verification camera data and the last frame of camera detection data are matched using the Hungarian algorithm to obtain the number of coarse calibration matches. When the matching rate of the number of coarse calibration matches is greater than or equal to a preset coarse calibration completion threshold, the initial external parameter matrix is ​​used as the coarse calibration matrix and S4 is executed. When the matching rate is less than the preset coarse calibration completion threshold, S1 to S3 are repeatedly executed.

[0029] Specifically, the precise calibration verification specifically includes:

[0030] The coarse calibration matrix and the fine calibration matrix are used to perform radar projection on the last frame of camera detection data in S3 to obtain fine calibration benchmark data and fine calibration verification data. The reprojection error benchmark value and the reprojection error verification value of the fine calibration benchmark data and the fine calibration verification data are calculated according to the last frame of camera detection data. When the number of coarse calibration matches is less than the number of fine calibration matches and the reprojection error benchmark value is greater than the reprojection error verification value, the fine calibration extrinsic parameter matrix is ​​used as the update matrix to execute S4 to S6. When the number of coarse calibration matches is greater than or equal to the number of fine calibration matches and the reprojection error benchmark value is less than or equal to the reprojection error verification value, steps S4 to S6 are repeated.

[0031] The beneficial effects of the present invention are:

[0032] (1) Without any preset external parameter input, point correspondence is found through the common characteristic angles and angle change rates in the radar coordinate system and the pixel coordinate system. The initial external parameter matrix obtained by coarse calibration provides the possibility for finding point correspondence in a larger range in subsequent fine calibration, thus realizing fully automated calibration.

[0033] (2) Use the same data to compare the results of the previous round of calibration with the results of the current round of precision calibration. On the basis of realizing the function of real-time online update, it is ensured that the effect of each update is better and there will be no erroneous update due to device jitter.

[0034] (3) The reprojection error in the radar coordinate system is used as the basis for judging the final effect, avoiding the error caused by reprojection in the pixel coordinate system. It can be accurately calibrated in congested scenes and provide guarantees for the continuity of the subsequent radar-visual fusion trajectory and the effect of the digital twin. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0036] Figure 1 Schematic diagram of the method of the present invention DETAILED DESCRIPTION

[0037] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention are clearly and completely described below in combination with the accompanying drawings and preferred embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Reference Figure 1 , which is an online radar-visual fusion automatic joint calibration method without special calibration objects disclosed by the present invention, step S1, rough calibration: In this embodiment, a total of three coordinate systems are involved, namely, the pixel coordinate system of the image, the camera coordinate system, and the c x c y c z c and radar coordinate system o r x r y r z r The target detected by the millimeter wave radar can be expressed as [x, y, 0] in the radar coordinate system. T , the target detected by the camera can be expressed as [u,v] in the pixel coordinate system. TThe transformation from radar coordinate system to pixel coordinate system can be divided into the transformation from radar coordinate system to camera coordinate system, the transformation from camera coordinate system to imaging plane, and the transformation from imaging plane to pixel coordinate system.

[0039]

[0040] The latter two conversions involve the camera's internal parameters f u 、f v 、c u and c v The relevant information can be obtained in advance through the checkerboard calibration plate and the Camera Calibrator tool in Matlab. Select the focal length of the camera f = 12mm and the pixel size of the camera sensor d pixel = 2 μm, and the camera resolution is 3840 × 2160. Use a 100 mm × 100 mm checkerboard optical calibration plate to take 60 pictures at various positions on the camera screen and send them to the Camera Calibrator tool in Matlab to get f u =5983.6, f v =5983.2, c u =1996.3 and c v =1162.5. The conversion from radar coordinate system to camera coordinate system is a rigid body rotation problem, which involves external parameter matrices, including rotation matrix R and translation matrix T. Through at least 4 sets of corresponding points of the same target in radar coordinate system and pixel coordinate system, the rotation matrix R and translation matrix T can be solved by solvePnP of opencv.

[0041] In the radar coordinate system, the target detected by the radar is [x, y, 0] T Calculate the azimuth angles of all targets in k frames, and use the frame interval to calculate the angle change rate of all targets in k frames.

[0042]

[0043] For example, the installation height of the integrated laser vision device is h=6m, the frame interval is T=100ms, the weights are w=0.6, and w'=0.4.

[0044] In the pixel coordinate system, the target [u,v] detected by the camera is T Calculate the azimuth angles of all targets in k frames, and use the frame interval to calculate the angle change rate of all targets in k frames.

[0045]

[0046] If f = 12 mm, d pixel =2μm, c u =1996.3,c v=1162.5, frame interval T=100ms, weight w=0.6, w'=0.4.

[0047] Find the target speed greater than in the radar data set obtained by k frames And there is only one target data within ±1°, so n targets form a data set. Find data with only one target within the range of ±1° in the camera data set obtained by k frames, and obtain a data set consisting of m targets Compare the two data sets using the following rules.

[0048]

[0049] like θ thres =0.5°,ω thres = 0.2° / s. Find several radar-camera corresponding points in k frames to form a data set {(x ki ,y ki ,u kj ,v kj )|1≤i≤n,1≤j≤m}, and compare the data set with the final corresponding point set {(x i ,y i ,u i ,v i )|1≤i≤p} According to the Euclidean distance of the radar coordinates of any two corresponding points greater than 20m, multiple cycles are repeated until the number of p reaches the requirement of the solvePnP function of opencv.

[0050] Step S2, rough calibration completion judgment: The rotation matrix R and translation matrix T obtained by the solvePnP function of opencv in step S101 are used to convert the last frame of rough calibration data from camera to radar and from radar to camera according to formula (1). The Hungarian algorithm is used to match the camera's detection data with the pixel coordinates after radar conversion in the pixel coordinate system to obtain the number of matches. The number of targets detected by the radar in this frame N r The ratio is greater than R thres =0.8, then go to step S103, otherwise go back to step S101.

[0051] Step S3, updating the external parameter matrix.

[0052] Save the current frame data, and use the updated external parameter matrices R and T to convert the saved data from camera to radar and from radar to camera according to formula (1). Use the Hungarian algorithm in the pixel coordinate system to match the camera's own detection data with the pixel coordinates converted by the radar to obtain the matching relationship and the number of matches. The matching relationship is used to calculate the reprojection error between the camera detection target projected into the radar coordinate system and the matched radar detection target.

[0053]

[0054] And will match the number and reprojection error As the judgment condition of step S105.

[0055] Step S4, fine calibration: Use the external parameter matrix obtained in step S102 or step S105 to project the radar data onto the image, and use the Hungarian algorithm to perform camera-radar matching in the pixel coordinate system. cycle =25 cycles can be matched, that is, the target ID detected and tracked by the same group of radar and camera can be matched successfully for 25 consecutive cycles, and the coordinates of the target corresponding to the radar ID and camera ID in the radar coordinate system and the pixel coordinate system in the 25th cycle are extracted, and the target speed detected by the radar is greater than Form a candidate corresponding point set. Add the candidate corresponding point set to the final corresponding point set {(x i ,y i ,u i ,v i )|1≤i≤p}, the optimal distribution is screened for the final corresponding point set. First, the target with the previous serial number is used as the reference point, and the points with a distance between points less than 20m, or a distance in the x direction less than 1m and a distance in the y direction less than 50m are deleted. Then the target with the previous serial number is used as the reference point, such as point P i , and other points P j The straight line P i P j , if point P k On the straight line P i P j After the above optimal distribution screening is completed, if the number of p meets the requirements of the solvePnP function of opencv, the final corresponding point set is sent to the solvePnP function of opencv to solve the rotation matrix R and translation matrix T of the external parameter matrix.

[0056] Step S5, accurate calibration update judgment: The rotation matrix R and translation matrix T obtained by accurate calibration are used to perform camera-to-radar and radar-to-camera coordinate conversion on the data saved in step S103 using formula (1). The Hungarian algorithm is used to match the camera's own detection data with the pixel coordinates converted by the radar in the pixel coordinate system to obtain the matching relationship and matching number. The matching relationship is used to calculate the reprojection error between the camera detection target projected into the radar coordinate system and the matched radar detection target. Compare the number of matches generated in step S103 and reprojection error If satisfied and Then go to step S103, otherwise go to step S4.

[0057] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. An online radar-visual fusion automated joint calibration method without special calibration objects. It is characterized in that The following steps are involved: S1: Obtain millimeter wave radar detection data and camera detection data, and calculate the azimuth and angle change rate of the millimeter wave radar detection data and the camera detection data, where the millimeter wave radar detection data is [x, y, 0] T The calculation method of the azimuth and angle change rate of the millimeter wave radar detection data is as follows: in, represents the azimuth of the millimeter-wave radar at time k, represents the angle change rate of the millimeter-wave radar at time k, h is the radar installation height, T is the radar frame interval, and w is the set weight; The camera detection data is [u,v] T , the calculation method of the azimuth and angle change rate of the camera detection data is as follows: in, represents the azimuth of the camera at time k, represents the angle change rate of the camera at time k, f is the focal length of the camera, c u and c v is the camera internal reference; S2: confirming radar-camera corresponding points that meet the unique principle of the cone of vision according to the azimuth angle and the angle change rate, constructing a radar-camera corresponding point set that meets the set configuration requirements according to the millimeter-wave radar detection data and the camera detection data under multiple frames, obtaining the intrinsic parameter matrix of the millimeter-wave radar and the camera, obtaining the camera distortion coefficient, and calculating the initial external parameter matrix according to the radar-camera corresponding point set, the intrinsic parameter matrix and the camera distortion coefficient; S3: Acquire the last frame of radar detection data and the last frame of camera detection data, and perform a rough calibration verification on the initial external parameter matrix according to the last frame of radar detection data and the last frame of camera detection data. After the verification passes, use the initial external parameter matrix as the rough calibration matrix and execute S4. If the verification fails, repeat S1 to S3. S4: performing image projection on the radar detection data using the coarse calibration matrix to obtain projection image data, performing data matching on the radar detection data and the projection image data, performing reliability screening and maximum distribution screening on the radar detection data and the projection image data according to the result of the data matching, constructing a final corresponding point set according to the data that pass the reliability screening and the maximum distribution screening, and calculating a precise calibration external parameter matrix according to the final corresponding point set, the intrinsic parameter matrix and the camera distortion coefficient; S5: performing image projection on the last frame of radar detection data in S3 using the precise calibration external parameter matrix to obtain precise calibration verification image data, and performing data matching on the precise calibration verification camera data and the last frame of camera detection data using the Hungarian algorithm to obtain the number of precise calibration matches; S6: Perform precision calibration verification on the precision calibration extrinsic parameter matrix according to the precision calibration matching number. If the verification is passed, use the precision calibration extrinsic parameter matrix as an update matrix and repeat S4 to S6. If the verification fails, repeat steps S4 to S6.

2. The method according to claim 1, It is characterized in that The method for confirming the radar-camera corresponding points that meet the unique principle of the viewing cone in S2 is specifically: S201: Filtering out data corresponding to the only azimuth angle that exists within a set angle range of ±Θ from the azimuth angles of each frame of millimeter-wave radar detection data and camera detection data; S202: Filtering out data whose speed of the millimeter-wave radar detection data is greater than a set threshold from the data filtered in S201; S203: Use a formula to traverse the remaining data after S201 and S202 to obtain the radar-camera corresponding points. The formula is: Among them, θ thres ,ω thres is the threshold for determining whether the data corresponds to the same target. is the angle between the normal direction of the millimeter-wave radar coordinates and the normal direction of the camera.

3. The method according to claim 1, It is characterized in that The configuration requirement in S2 is that the distance between the radar-camera corresponding points in the radar-camera corresponding point set is less than a set distance threshold.

4. The method according to claim 1, It is characterized in that The reliability screening in S4 is specifically as follows: in a continuous set period, the target IDs of the same group of the millimeter-wave radar detection data and the camera detection data can be successfully matched, and the target speed of the millimeter-wave radar detection data is greater than the set speed threshold.

5. The method according to claim 1, It is characterized in that The maximum distribution screening in S4 is specifically as follows: the distance between the millimeter-wave radar detection data is greater than a set distance threshold, and there are no three collinear millimeter-wave radar detection data.

6. The method according to claim 1, It is characterized in that The rough calibration verification in S3 specifically includes: The initial external parameter matrix is ​​used to perform image projection on the last frame of radar detection data to obtain coarse calibration verification camera data, and the coarse calibration verification camera data and the last frame of camera detection data are matched using the Hungarian algorithm to obtain the number of coarse calibration matches. When the matching rate of the number of coarse calibration matches is greater than or equal to a preset coarse calibration completion threshold, the initial external parameter matrix is ​​used as the coarse calibration matrix and S4 is executed. When the matching rate is less than the preset coarse calibration completion threshold, S1 to S3 are repeatedly executed.

7. The method according to claim 1, It is characterized in that The precise calibration verification specifically includes: The coarse calibration matrix and the fine calibration extrinsic parameter matrix are used to perform radar projection on the last frame of camera detection data in S3 to obtain fine calibration benchmark data and fine calibration verification data. The reprojection error benchmark value and the reprojection error verification value of the fine calibration benchmark data and the fine calibration verification data are calculated according to the last frame of camera detection data. When the number of coarse calibration matches is less than the number of fine calibration matches and the reprojection error benchmark value is greater than the reprojection error verification value, the fine calibration extrinsic parameter matrix is ​​used as the update matrix to execute S4 to S6. When the number of coarse calibration matches is greater than or equal to the number of fine calibration matches and the reprojection error benchmark value is less than or equal to the reprojection error verification value, steps S4 to S6 are repeated.