A radar and vision detection online joint calibration method

Abstract

This invention discloses an online joint calibration method for radar and vision detection, comprising the following steps: S1, detecting vehicle targets in video images and constructing 3D bounding boxes, obtaining target trajectory information, and extracting the first and second vanishing points; S2, mapping image pixels to the world coordinate system based on the camera's extrinsic parameter matrix to the world coordinate system; S3, finding matching points between millimeter-wave radar targets and visual targets using elliptic correlation gates and KNN algorithms based on target data in the world coordinate system and millimeter-wave radar target data; S4, optimizing camera extrinsic parameters relative to the radar using the PnP principle and minimizing reprojection error based on the matching points between the millimeter-wave radar targets and visual targets; This method can automatically determine the transformation relationship between the image coordinate system and the millimeter-wave radar coordinate system, and the millimeter-wave radar and camera can be deployed at any position above the road. This method only utilizes vehicle targets on the road and does not rely on lane lines, road signs, or other reference objects for calibration.

Description

Technical Field

[0001] This invention relates to the field of multi-source heterogeneous data fusion sensing, specifically to an online joint calibration method for radar and visual detection. Background Technology

[0002] To address the needs of traffic and security sectors for target and environmental perception, the use of millimeter-wave radar, vision sensors, and information fusion technologies to achieve multi-target detection, tracking, and identification has become a mainstream perception solution. Temporal and spatial synchronization of data is a prerequisite for associating millimeter-wave radar targets with visual targets; the accuracy of joint calibration directly affects the accuracy of fused perception and the digital reconstruction effect of targets and the environment. Currently, timing technology is becoming increasingly mature, and time synchronization can meet the requirements of low cost and high precision. However, existing joint calibration methods for millimeter-wave radar and vision suffer from problems such as weak universality and a lack of mature and convenient automatic calibration methods.

[0003] For traffic and security monitoring, to achieve better coverage and clarity, installation and debugging are typically carried out in the field. This affects the intrinsic and extrinsic parameters of millimeter-wave radar and cameras. However, camera calibration methods based on checkerboard patterns or other calibration objects are difficult to adapt to such outdoor engineering applications. In practical applications, the need for automatic calibration parameter updates and a combined automatic calibration method for millimeter-wave radar and vision is becoming increasingly urgent to address changes in camera intrinsic parameters and installation platform vibrations. Summary of the Invention

[0004] The purpose of this invention is to solve the problems of the current millimeter-wave radar and vision joint calibration methods being not very universal and lacking mature and simple automatic calibration methods.

[0005] To achieve the above objectives, this invention proposes an online joint calibration method for radar and visual detection. It utilizes the vehicle target's geometric dimensions and motion trajectory to achieve automatic camera calibration, addressing the problem of strong dependence on calibration objects in traffic and security monitoring. Furthermore, it employs data association based on the nearest neighbor method to match millimeter-wave radar targets with visual targets. Finally, it utilizes the PnP principle and minimizing reprojection error to obtain the optimal solution for the camera's intrinsic and extrinsic parameters, solving the joint calibration problem of millimeter-wave radar and visual detection under asynchronous sampling conditions. The method specifically includes the following steps:

[0006] S1. Detect vehicle targets in video images and construct 3D bounding boxes; obtain target trajectory information and extract the first vanishing point; extract the second vanishing point using diamond-shaped space cumulative voting.

[0007] S2. Calculate the initial value of the camera focal length based on the first and second vanishing points, and then map the image pixels to the world coordinate system according to the camera's external parameter matrix to the world coordinate system.

[0008] S3. Based on step S2, transform the coordinates of the vehicle target in the camera to the world coordinate system to obtain world coordinate system target data; use millimeter-wave radar to sample the vehicle target to obtain millimeter-wave radar target data of the vehicle target; based on the world coordinate system target data and millimeter-wave radar target data, use elliptic correlation gate and KNN algorithm to find the matching point between the millimeter-wave radar target and the visual target.

[0009] S4. Based on the matching points between millimeter-wave radar targets and visual targets, the PnP principle and minimizing reprojection error are used to optimize the camera's extrinsic parameters relative to the radar.

[0010] Step S1 includes the following steps:

[0011] S11. After obtaining the video image stream through the camera, the YOLO neural network is used to detect vehicle targets in the video images and determine the vehicle target category;

[0012] S12. The DeepSORT tracking algorithm is used to obtain the target trajectory information of the vehicle target in the video image. The target trajectory information includes reference points and the bounding box sequence of the target in the image. The center of the bottom of the vehicle bounding box is extracted as the reference point. Based on the bounding box sequence of the vehicle target and the selected reference point, the reference point sequence of the target is obtained.

[0013] S13. Construct the 3D bounding box of the vehicle target;

[0014] S14. Extract the first vanishing point based on the reference point sequence of the vehicle target;

[0015] S15. Extract the second vanishing point based on the gradient and eigenvalue decomposition of the image pixels.

[0016] The vehicle target categories include cars, vans, box trucks, trucks, and buses; each vehicle target category corresponds to different length, width, and height dimensions of the 3D bounding box.

[0017] Specifically, step S14 involves: using the vehicle target reference point sequence, employing least squares fitting to obtain the straight line direction along the road travel direction, transforming the straight line formed by the vehicle reference point sequence into a rhombus space, and performing cumulative voting to obtain the maximum point in the rhombus space as the first vanishing point u.

[0018] Step S15 further includes the following steps:

[0019] S151. Calculate the gradient of each pixel in the video image, take 75% of the maximum gradient value as the threshold, and extract the pixels in the image whose gradient is greater than the threshold as candidate points.

[0020] S152. In a 9×9 neighborhood, the candidate point with the largest gradient value is taken as the seed point. The neighborhood matrix of the seed point is decomposed using eigenvalue decomposition. The eigenvector corresponding to the larger value among the eigenvalues ​​obtained after decomposition is the direction of the edge.

[0021] S153. Remove edges that are in the same direction as the first vanishing point. Convert the remaining edges and seed points to a rhombus space and perform cumulative voting. The maximum point in the resulting rhombus space is the second vanishing point v.

[0022] Step S2 further includes the following steps:

[0023] S21. Calculate the initial value of the camera focal length based on the first and second vanishing points.

[0024] S22. Map the image pixels to the world coordinate system using three-dimensional spatial coordinates P, where... p = [p x p y f], where p x p y λ represents the x and y coordinates of the pixel, f is the camera focal length, and λ is the scale parameter. Where w = u × v.

[0025] Step S4 further includes the following steps:

[0026] S41. Select 5 target reference points from the video image, and ensure that no 3 of the target reference points are collinear;

[0027] S42. Select 3 points P. i =[P i .x w P i .y w P i .z w ] T i = 1, 2, 3;

[0028] S43. Establish a reference coordinate system with P1 as the origin, P2 on the positive X-axis, and P3 in the first quadrant of the XOY plane. Define the transformation matrices from the camera coordinate system and radar coordinate system to the established reference coordinate system as M. c and M w Therefore, the extrinsic parameter matrix is ​​E = M. c -1 M w ;

[0029] S44. Calculate the transformation matrix M from the radar coordinate system to the reference coordinate system. w =T rx Try T rz T t , among which, T t It is a transformation that turns point P1 into the origin, in T t Under the transformation, points P2 and P3 are transformed into Q2 and Q3; T rz This transformation converts Q2 and Q3 into R2 and R3, where the transformed R2 must satisfy R2.y w =0 and R2.x w >0; T ry This is a transformation that converts R2 and R3 into S2 and S3, where the transformed S2 must satisfy S2.z. w =0;T rx This is a transformation that rotates S3 around the X-axis, such that S3.z w =0;

[0030] S45. Calculate the transformation matrix M from the camera coordinate system to the reference coordinate system. c M c The calculation process and M w The same applies, only the input points need to be changed to the camera coordinate system;

[0031] S46. Calculate and verify the candidate values ​​of the extrinsic parameter matrix, calculate the rotation angle based on the candidate values ​​of the extrinsic parameter matrix, and select the candidate values ​​of the extrinsic parameter matrix that meet the engineering rationality as the extrinsic parameter matrix.

[0032] S47. Select three more different points P. i Repeat steps S44-S46 to obtain multiple extrinsic parameter matrices, and calculate their average value as the final solution.

[0033] Among them, T rx T ry T rz T t They are respectively

[0034]

[0035]

[0036]

[0037]

[0038] The selection logic for candidate values ​​of the extrinsic parameter matrix in step S46 is as follows:

[0039] ①If |E candidate (3, 1) ≤ 1|, then Otherwise, determine E candidate To prevent the growth of new roots, they should be discarded;

[0040] ② If r 21 E candidate (2, 1) < 0, then The value is the opposite of the number.

[0041] ③ If r 32 E candidate (3, 2) < 0, then If sinω < 0, then ω = -arccos(cosω);

[0042] Where ω and The value should be appropriate to the installation deviation. If it exceeds the normal range or does not conform to the engineering rationality, the solution of the extrinsic parameter matrix is ​​considered invalid.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] In practical engineering applications, after debugging the millimeter-wave radar and camera field of view, this method can automatically determine the transformation relationship between the image coordinate system and the millimeter-wave radar coordinate system.

[0045] Millimeter-wave radar and cameras can be deployed anywhere above the road, and online calibration only uses vehicle targets on the road, thus achieving calibration without relying on reference objects such as lane lines and road signs. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating an online joint calibration method for radar and visual inspection according to the present invention. Detailed Implementation

[0047] The technical solutions, structural features, achieved objectives, and effects of the present invention will be described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0048] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only the expressly listed elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0049] This embodiment discloses an online joint calibration method for radar and visual detection, such as... Figure 1 As shown, it includes the following steps:

[0050] S1. Use the YOLO neural network to detect vehicle targets in video images and construct 3D bounding boxes. Use DeepSORT to obtain target trajectory information and extract the first vanishing point. Use diamond space cumulative voting to extract the second vanishing point. The specific steps include:

[0051] S11. After obtaining the video image stream through the camera, the YOLO neural network is used to detect vehicle targets in the video images and determine the vehicle target category. The vehicle target category includes cars, vans, box trucks, trucks, and buses.

[0052] S12. The DeepSORT tracking algorithm is used to obtain the target trajectory information of the vehicle target in the video image. The target trajectory information includes reference points and the bounding box sequence of the target in the image.

[0053] Extract the center of the bottom of the vehicle bounding box as the reference point, and obtain the reference point sequence of the target based on the bounding box sequence of the vehicle target and the selected reference point position;

[0054] S13. Construct the 3D bounding box of the vehicle target, where each vehicle target category corresponds to a different length, width and height dimension of the 3D bounding box;

[0055] S14. Based on the reference point sequence of the vehicle target, extract the first vanishing point, specifically as follows:

[0056] Using the vehicle target reference point sequence, the straight line direction along the road travel direction is fitted by least squares. The straight line formed by the vehicle reference point sequence is transformed into a rhombus space, and cumulative voting is performed. The maximum point in the rhombus space is the first vanishing point u.

[0057] The process of accumulating votes to obtain the maxima in the rhombus space refers to using a cascaded Hough transform to extract the first vanishing point. This method is a standard practice in the field, so it will only be briefly described here: The cascaded Hough transform describes a "line-point-line" transformation process. Three collinear points and a straight line in an image become three straight lines and a point after the Hough transform. In particular, the point at infinity in the image becomes a straight line parallel to the coordinate axes after the transformation. By constructing two mappings S and T, four mappings SS, ST, TS, and TT are combined to map the points in the image space to the rhombus space. The straight lines in the image are transformed into polylines in the rhombus space. Finally, since the vanishing point is the intersection of all parallel straight lines in the same direction and has the largest weight, after rasterizing the rhombus space, voting can be performed on the polylines, and the maxima in the rhombus space are the vanishing points in the image space.

[0058] S15. Extract the second vanishing point, which includes the following steps:

[0059] S151. Calculate the gradient of each pixel in the video image, take 75% of the maximum gradient value as a threshold, and extract the pixels in the image whose gradient is greater than the threshold as candidate points.

[0060] S152. In the 9×9 neighborhood of each pixel in the video image, the candidate point with the largest gradient value is selected as the seed point. The neighborhood matrix of the seed point is decomposed using eigenvalue decomposition. The horizontal and vertical gradient values ​​of the 81 pixels form an 81×2 matrix. After decomposition, two eigenvalues ​​are obtained. The eigenvector corresponding to the larger value is taken as the direction of the edge.

[0061] S153. Remove the edges that are in the same direction as the first vanishing point. Convert the remaining edges and seed points after removal to a rhombus space and perform cumulative voting. The maximum point in the resulting rhombus space is the second vanishing point v.

[0062] S2. Assuming the optical center of the camera is at the center of the image and the focal lengths are equal in the two-dimensional direction, calculate the initial value of the camera focal length based on the first and second vanishing points, and map the image pixels to the world coordinate system according to the external parameter matrix of the camera to the world coordinate system. The specific steps include:

[0063] S21. Calculate the initial value of the camera focal length based on the first and second vanishing points.

[0064] S22. Map the image pixels to the world coordinate system;

[0065] Calculate the third vanishing point w = u × v, and thus the unit vector in the vertical direction of the road surface plane is... Therefore, the coordinates of any pixel in the image are p = [p x p y f] can be converted into three-dimensional spatial coordinates. Where, p x p y , where are the horizontal and vertical coordinates of the pixel, respectively, and f is the camera focal length; the scale parameter λ is used to convert the three-dimensional coordinate scale unit to the unit "meter". The scale parameter λ is calculated using the horizontal edge endpoints of the 3D bounding box and the actual width of the vehicle, and the image pixels are mapped to the world coordinate system through the three-dimensional spatial coordinates P.

[0066] S3. Based on the camera-to-world coordinate system extrinsic parameter matrix from step S2, transform the coordinates of the vehicle target in the camera to the world coordinate system to obtain world coordinate system target data; use millimeter-wave radar to sample the vehicle target to obtain millimeter-wave radar target data of the vehicle target; based on the world coordinate system target data and millimeter-wave radar target data, use the elliptic correlation gate and KNN (K-Nearest Neighbor) algorithm to find the matching point between the millimeter-wave radar target and the visual target (i.e., the world coordinate system target); it should be noted that the time difference between the sampling of the millimeter-wave radar and the camera should be considered.

[0067] S4. Based on the matching points between millimeter-wave radar targets and visual targets, the PnP (Perspective-n-Point) principle and minimizing reprojection error are used to optimize the camera's extrinsic parameters relative to the radar.

[0068] The PnP principle refers to the fact that, in the absence of lens distortion, rotation and translation transformations in space do not change the distance between two points. Step S4 further includes:

[0069] S41. Select 5 target reference points from the video image, stipulating that any 3 of the target reference points are not collinear;

[0070] S42. Select 3 points P. i =[P i .x w P i .y w P i .z w ] T i = 1, 2, 3;

[0071] S43. Establish a reference coordinate system with P1 as the origin, P2 on the positive X-axis, and P3 in the first quadrant of the XOY plane. Define the transformation matrices from the camera coordinate system and radar coordinate system to the established reference coordinate system as M. c and M w Therefore, the extrinsic parameter matrix is ​​E = M. c -1 M w ;

[0072] S44. Calculate the transformation matrix M from the radar coordinate system to the reference coordinate system. w ;

[0073] M w =T rx T ry T rz T t

[0074] In the above formula, T tIt is a transformation that turns point P1 into the origin, in T t Under the transformation, points P2 and P3 are transformed into Q2 and Q3; T rz This transformation converts Q2 and Q3 into R2 and R3, where the transformed R2 must satisfy R2.y w =0 and R2.x w >0; T ry This is a transformation that converts R2 and R3 into S2 and S3, where the transformed S2 must satisfy S2.z. w =0;T rx This is a transformation that rotates S3 around the X-axis, such that S3.z w =0, specifically, T rx T ry T rz T t They are respectively

[0075]

[0076]

[0077]

[0078]

[0079] After the above transformation matrix M w This allows the radar coordinate system to be transformed to the reference coordinate system.

[0080] S45. Calculate the transformation matrix M from the camera coordinate system to the reference coordinate system. c ;

[0081] Transformation matrix M c The calculation process and M w The same applies; only the input point P needs to be changed. i Change to camera coordinate system (i.e., based on step S2, change the input point P) i M can be calculated by mapping from the pixel coordinate system to the world coordinate system. c ;

[0082] S46. Calculate and verify candidate values ​​of the extrinsic parameter matrix, calculate the rotation angle based on the candidate values ​​of the extrinsic parameter matrix (extrinsic parameter matrix refers to the linear transformation in three-dimensional space, including translation and rotation transformation, and the rotation angle is the angle of rotation transformation), and select the candidate values ​​of the extrinsic parameter matrix that meet the engineering rationality as the extrinsic parameter matrix.

[0083] Based on the transformation matrix M obtained in steps S421 and S422 w and M c The candidate values ​​E of the extrinsic parameter matrix are obtained. candidate =M c- 1 M w ; Calculate its rotation angle:

[0084] ①If |E candidate (3, 1) ≤ 1|, then Otherwise, determine E candidate To prevent the growth of new roots, they should be discarded;

[0085] ② r 21 = If r 21 E candidate (2,1)<0, then The value is the opposite of the number.

[0086] ③ if

[0087] r 32 E candidate (3,2)<0, then if

[0088] sinω<0, then ω=-arccos(cosω);

[0089] Where ω and The value of should be appropriate to the installation deviation. If it exceeds the normal range or does not conform to the engineering rationality, the solution of the extrinsic parameter matrix is ​​considered invalid.

[0090] S47. Select three more different points P. i Repeat steps S44-S46 to obtain multiple extrinsic parameter matrices. Calculate their average value as the final solution, which is the extrinsic parameter matrix of the camera relative to the radar.

[0091] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A method for online joint calibration of radar and visual inspection, characterized in that, Includes the following steps: S1. Detect vehicle targets in video images and construct 3D bounding boxes; acquire target trajectory information and extract the first vanishing point; extract the second vanishing point using a diamond space cumulative voting method; wherein, the 3D bounding box construction uses the center of the bottom of the extracted vehicle bounding box as the reference point, and obtains the target's reference point sequence based on the vehicle target's bounding box sequence and the selected reference point; the method for extracting the second vanishing point includes: removing edges with the same direction as the first vanishing point, converting the remaining edges and seed points after removal to a diamond space, and performing cumulative voting, with the maximum point in the resulting diamond space being the second vanishing point. S2. Calculate the initial value of the camera focal length based on the first and second vanishing points, and then map the image pixels to the world coordinate system according to the camera's extrinsic parameter matrix to the world coordinate system. S3. Based on step S2, transform the coordinates of the vehicle target in the camera to the world coordinate system to obtain the target data in the world coordinate system. Use millimeter-wave radar to sample the vehicle target to obtain the millimeter-wave radar target data. Based on the world coordinate system target data and the millimeter-wave radar target data, use the elliptic correlation gate and KNN algorithm to find the matching point between the millimeter-wave radar target and the visual target. S4. Based on the matching point between the millimeter-wave radar target and the visual target, use the PnP principle and the most... Minimize reprojection error and optimize camera extrinsic parameters relative to radar; wherein, step S1 includes: decomposing the gradient and eigenvalues ​​of image pixels, and then obtaining the second vanishing point through the second vanishing point extraction method; the decomposition based on the gradient and eigenvalues ​​of image pixels includes the following steps: calculating the gradient of each pixel in the video image, taking 75% of the maximum gradient value as a threshold, and extracting pixels in the image whose gradient is greater than the threshold as candidate points; in a 9×9 neighborhood, taking the candidate point with the largest gradient value as the seed point, and using eigenvalue decomposition to decompose the neighborhood matrix of the seed point, wherein the eigenvector corresponding to the larger value of the eigenvalues ​​obtained after decomposition is the direction of the edge.

2. The online joint calibration method for radar and visual detection as described in claim 1, characterized in that, Before step S1, which decomposes the image pixels based on gradients and feature values, and then obtains the second vanishing point using the second vanishing point extraction method, the following steps are also included: S11, after obtaining the video image stream through the camera, using a YOLO neural network to detect vehicle targets in the video image and determine the vehicle target category; S12, using the DeepSORT tracking algorithm to obtain the target trajectory information of the vehicle targets in the video image, the target trajectory information including reference points and the bounding box sequence of the targets in the image; S13, constructing the 3D bounding box of the vehicle target; S14, extracting the first vanishing point based on the reference point sequence of the vehicle target.

3. The online joint calibration method for radar and visual inspection as described in claim 2, characterized in that, The vehicle target categories include cars, vans, box trucks, trucks, and buses; each vehicle target category corresponds to different length, width, and height dimensions of the 3D bounding box.

4. The online joint calibration method for radar and visual inspection as described in claim 2, characterized in that, Step S14 specifically involves: using the vehicle target reference point sequence, employing least squares fitting to determine the straight line direction along the road travel direction; transforming the straight line formed by the vehicle reference point sequence into a rhombus space; and performing cumulative voting; the maximum point in the resulting rhombus space is the first vanishing point. .

5. The online joint calibration method for radar and visual detection as described in claim 4, characterized in that, Step S2 further includes the following steps: S21, calculating the initial value of the camera focal length based on the first vanishing point and the second vanishing point. S22, through three-dimensional spatial coordinates Map image pixels to the world coordinate system, where , ,in, These are the horizontal and vertical coordinates of the pixel, respectively. For camera focal length, For scale parameters, ,in .

6. The online joint calibration method for radar and visual detection as described in claim 5, characterized in that, Step S4 further includes the following steps: S41, selecting 5 target reference points from the video image, wherein any 3 of the target reference points are not collinear; S42, selecting 3 of the points. , S43, with Origin On the positive half-axis of the X-axis, A reference coordinate system is established in the first quadrant of the XOY plane, and the transformation matrices from the camera coordinate system and the radar coordinate system to the established reference coordinate system are defined as follows: and Therefore, the extrinsic parameter matrix is S44. Calculate the transformation matrix from the radar coordinate system to the reference coordinate system. ,in, It is the point Transformation to the origin, in Under the transformation, , Point transformation , ; It is , Transform into , The transformation, where the transformed Must meet and ; It is , Transform into , The transformation, where the transformed Must meet ; It is The transformation of rotation about the X-axis makes S45. Calculate the transformation matrix from the camera coordinate system to the reference coordinate system. , The calculation process and The same applies; only the input point needs to be changed. Switch to camera coordinate system; S46, calculate and verify candidate values ​​of the extrinsic parameter matrix, calculate the rotation angle based on the candidate values, and select the candidate values ​​of the extrinsic parameter matrix that meet the engineering rationality as the extrinsic parameter matrix; S47, select three other different points. Repeat steps S44-S46 to obtain multiple extrinsic parameter matrices, and calculate their average value as the final solution.

7. The online joint calibration method for radar and visual inspection as described in claim 6, characterized in that, The selection logic for candidate values ​​of the extrinsic parameter matrix in step S46 is as follows: ① If ,but Otherwise, it will be judged. To prevent the growth of new roots, they should be discarded; ② , , , ,if ,but The value is the opposite of the number. ; ③ , , ,if ,but , ;if ,but ; in, and The value should be appropriate to the installation deviation. If it exceeds the normal range or does not conform to the engineering rationality, the solution of the extrinsic parameter matrix is ​​considered invalid.

Images