Target-free integrated camera millimeter wave radar automatic calibration method based on trajectory association

Abstract

The invention provides an automatic calibration method for a target-free integrated camera millimeter wave radar based on trajectory association, and belongs to the technical field of sensing data fusion. According to the invention, camera and millimeter wave radar data are collected and initial external parameter estimation is carried out, then clue sensing trajectory similarity (CATS) is used to match target trajectories captured by the millimeter wave radar and the camera, and a random sampling consensus algorithm (DBN-RANSAC) combined with a dynamic Bayesian network is used to reduce interference of outliers and realize coarse calibration. And then a Levenberg-Marquardt (LM) algorithm is adopted to carry out iterative optimization on a coarse calibration result, and an optimal external parameter matrix is obtained. According to the invention, external parameter calibration between the millimeter wave radar and the camera sensor can be automatically completed without depending on any specific calibration equipment. The method disclosed by the invention also has good performance under a low-light condition, and shows a wide development prospect and engineering application potential.

Description

Technical Field

[0001] The invention belongs to the technical field of sensor data fusion, and in particular relates to an automatic calibration method of a target-free integrated camera millimeter wave radar based on trajectory association. Background Art

[0002] With the advancement of sensor technology, sensor fusion technology has become increasingly important in the fields of autonomous driving and intelligent transportation. The accuracy of sensor fusion depends largely on accurate external parameter calibration. Cameras and millimeter-wave radars are two commonly used sensors that complement each other in data: cameras provide rich visual information, and millimeter-wave radars can provide accurate position and speed information and work stably under various weather conditions. Through automatic calibration of the integrated camera-millimeter-wave radar, the consistency of sensor data in space and time is ensured, human errors are reduced, calibration efficiency is improved, and it can adapt to environmental changes, providing stable and accurate perception data for sensor fusion.

[0003] Existing automatic calibration technologies are mainly divided into two types: with target and without target. The targeted calibration method relies on specially designed targets, such as corner reflectors or plates based on conductive materials. These targets can ensure stable detection of the radar. In addition, this process relies on a specific environment and requires manual intervention to place markers during use, which increases labor costs. In contrast, targetless calibration does not rely on specific targets, but usually relies on additional modules or prior knowledge to associate sensor data. It requires a large amount of labeled data to train supervised learning models, and its performance is limited in the absence of sufficient visual information. Summary of the invention

[0004] The present invention is conducted to solve the problems existing in the existing targetless automatic calibration technology, and aims to provide a targetless integrated camera millimeter wave radar automatic calibration method based on trajectory association.

[0005] The present invention provides a method for automatic calibration of a non-target integrated camera millimeter wave radar based on trajectory association, which has the following characteristics: S10, collecting data of the camera and the millimeter wave radar; S20, estimating and setting initial external parameters according to the installation height and angle of the camera and the millimeter wave radar; H 0; S30, obtain the camera 2D trajectory obtained by detecting the target with the camera, according to the initial external parameters H 0. Transform the radar 3D trajectory detected by the millimeter wave radar into the two-dimensional image plane to obtain the radar 2D trajectory; S40. Use the CATS algorithm to associate the camera 2D trajectory with the radar 2D trajectory to obtain an associated trajectory. id S50, the points in the camera 2D trajectory are recorded as 2D camera data points, and the points in the radar 3D trajectory corresponding to the radar 2D trajectory are recorded as 3D radar data points. idThe correct correspondence between the camera 2D trajectory and the radar 2D trajectory is obtained by processing outliers through the RANSAC algorithm combined with the dynamic Bayesian network to complete the rough calibration of the 2D camera data points and the 3D radar data points; S60, the Levenberg-Marquardt algorithm is used to perform a fine calibration calculation on the result of the rough calibration in step S50 to obtain the final optimal extrinsic parameters.

[0006] The target-free integrated camera millimeter-wave radar automatic calibration method based on trajectory association provided by the present invention may also have the following feature: wherein, in step S20, the positions of the camera and the millimeter-wave radar are fixed, and the translation vector between the two is not considered and is set to a zero vector.

[0007] In the target-free integrated camera millimeter-wave radar automatic calibration method based on trajectory association provided by the present invention, it can also have the following characteristics: wherein step S30 includes the following sub-steps: S31, using the YOLO v3 algorithm to perform target detection on the image collected by the camera; S32, using the Bytetrack algorithm to track the target detected in step S31 to obtain the camera 2D trajectory; S33, according to the following formula, converting the radar 3D trajectory into the radar 2D trajectory, the calculation formula is: ,in,( u , v ) is the radar 2D trajectory, ( X r , Y r , Z r ) is the radar 3D trajectory, Z c is the depth information, Z c Indicates the distance from the target vehicle point to the camera, u is the horizontal coordinate of the radar 2D trajectory point, v is the ordinate of the radar 2D trajectory point, X r is the horizontal coordinate of the radar 3D trajectory point, Y r is the ordinate of the radar 3D trajectory point, Z r is the vertical coordinate of the radar 3D trajectory point, K is the camera internal parameter, ( R | t ) is a 3×4 matrix with 6 degrees of freedom, and ( R | t ) is defined as the initial value required for calibration H The number 0 indicates that the matrix is ​​represented in the form of a homogeneous matrix.

[0008] The target-free integrated camera millimeter-wave radar automatic calibration method based on trajectory association provided by the present invention may also have the following features: wherein, in step S31 to step S32, a detection frame is generated for each detected target, and a Kalman filter is used to predict the position of the target trajectory at the next moment, which is recorded as a prediction frame, and the intersection-over-union ratio of the detection frame and the prediction frame is calculated, and the Hungarian algorithm is used to match the high-score detection frame in the current frame with the prediction frame of the trajectory of the previous frame to update the trajectory information.

[0009] The target-free integrated camera millimeter-wave radar automatic calibration method based on trajectory association provided by the present invention may also have the following characteristics: wherein, in step S40, the clue score of each data point in the camera 2D trajectory relative to the radar 2D trajectory is calculated, thereby obtaining a similarity measure between the camera 2D trajectory and the radar 2D trajectory, and for any camera 2D trajectory or radar 2D trajectory, there is a corresponding radar 2D trajectory or camera 2D trajectory with the highest clue score.

[0010] The target-free integrated camera millimeter wave radar automatic calibration method based on trajectory association provided by the present invention may also have the following features: The CATS algorithm in step S40 calculates the similarity between the camera 2D trajectory and the radar 2D trajectory according to the following steps: S41, use the following spatial attenuation function to measure the spatial proximity of the data points of the camera 2D trajectory and the radar 2D trajectory: ; S42, considering the time threshold, calculate the clue score of each data point in the camera 2D trajectory relative to the radar 2D trajectory, or the clue score of each data point in the radar 2D trajectory relative to the camera 2D trajectory: ; S43, using the CATS algorithm to obtain the similarity measure between the camera 2D trajectory and the radar 2D trajectory: , In steps S41 to S43, p m,e and p j,k They are data points from two different trajectories, the camera 2D trajectory and the radar 2D trajectory. T m and T j They are p m,e and p j,k The camera 2D track or radar 2D track where it is located, | Tm | is a data point p m,e The number of t m,e It's a track T m Medium data point p m,e time frame, t j,k It's a track T j Medium data point p j,k time frame, t is the time threshold, e is the spatial threshold, f ε ( p m,e , p j,k ) is a spatial attenuation function with a value between 0 and 1, f ε ( p m,e , p j,k )along with p m,e and p j,k dist represents the Euclidean distance between two points.

[0011] The targetless integrated camera millimeter wave radar automatic calibration method based on trajectory association provided by the present invention may also have the following characteristics: wherein the spatial threshold e Less than 18 units in length, time threshold t The corresponding time difference is less than 2 frames.

[0012] In the target-free integrated camera millimeter-wave radar automatic calibration method based on trajectory association provided by the present invention, it can also have the following characteristics: wherein step S50 includes the following sub-steps: S51, given an initial data set S, the initial data set S is the associated trajectory in step S40 id ; S52, assigning an initial inlier probability to each data point and setting it to 0.5 to indicate the uncertainty of whether it is an inlier or an outlier; S53, in the data set S, using the RANSAC algorithm to select at least 6 pairs of matching 2D camera data points and 3D radar data points to estimate the model parameters of the digital model of the data set S H; S54, calculate the consistency probability of each data point with the current digital model, so as to determine the current inlier probability; S55, use the dynamic Bayesian network to iterate the belief score of each data point, the belief score is the inlier probability; S56, the number of data points detected as inliers, if the number of inliers increases, update the digital model and its model parameters H , and repeat steps S54 to S56. If the number of internal points does not increase, return to and repeat steps S53 to S56 until the number of iterations in step S55 reaches a predetermined value and / or the number of internal points does not increase after multiple iterations, and then output the optimal digital model.

[0013] The target-free integrated camera millimeter wave radar automatic calibration method based on trajectory association provided by the present invention may also have the following features: Among them, in step S53, the model parameters H The derivation process is as follows: S53-1, the 2D camera data points in the initial data set S are known { p c,i， i =1,2,…, n} and its corresponding 3D radar data point { p r,i， i =1,2,…, n}, we can get the external parameter matrix ( R | t ), and expand to get formula (1): ; S53-2, expand equation (1) to obtain equation (2): ; S53-3, in elimination formula (2) l We get formula (3): ; S53-4, Formula (4) is derived from Formula (3): ; S53-5, due to the x There are 12 dimensions in total. A pair of 2D camera data points and their corresponding 3D radar data points can provide two equations, so at least six pairs of matching points are needed to solve them. n When ≥6, the equation is generated: Axe =0, because in practical applications x is not equal to 0, so the goal is to make Axe As close to 0 as possible, it actually turns into a least squares problem to solve x , the calculation formula is: , in || x ||=1 A Perform singular value decomposition and obtain the matrix A The right singular value vector corresponding to the minimum singular value is x An approximate solution of , where r It is the three-dimensional radar coordinate system that coincides with the world coordinate system. c is the two-dimensional image coordinate system, t 11 ~ t 34 for( R | t ) is the parameter to be found in the matrix, t 11 ~ t 34 Used to transform points in the world coordinate system into the camera coordinate system. u is the horizontal coordinate of the image plane, u N is the image plane N The horizontal coordinate of the point, v is the ordinate of the image plane, v N is the image plane N The vertical coordinate of the point, X r is the horizontal coordinate of the radar point, Y r is the ordinate of the radar point, Z r is the vertical coordinate of the radar point, l is the scale factor representing the depth information, T represents the transpose of a vector, P N rT Indicates the radar coordinate system N The transpose of the coordinate vector of the point, N is the number of points.

[0014] The target-free integrated camera millimeter-wave radar automatic calibration method based on trajectory association provided by the present invention may also have the following features: wherein step S60 includes the following sub-steps: S61, the precision calibration process is regarded as a nonlinear least squares optimization problem: , f ( x ) is the residual function; S62, will f ( x )Taylor expansion, taking the first-order linear term to approximate formula (5): ,in, J is the Jacobian matrix, whose columns are the objective function f Partial derivatives with respect to each variable; S63, find the partial derivative of equation (5) and make it equal to 0 to obtain equation (6): ; S64, the final iterative formula obtained from formula (6) is: , in, x k is the variable value of the current iteration, g k is the gradient vector, E As the identity matrix, we get the optimal x That is, the optimal rotation matrix R is obtained; S65, the result of the precise calibration is output as the required external parameter matrix.

[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention uses clue-aware trajectory similarity to match the target trajectory captured by the millimeter-wave radar and the camera, combines the random sampling consistency algorithm of the dynamic Bayesian network to reduce the interference of outliers, and uses the Levenberg-Marquardt algorithm to iteratively optimize the results of the rough calibration, and finally outputs the external parameter results.

[0016] (2) The DBN-RANSAC-LM algorithm of the present invention introduces an optimization measure for removing noise points during the calibration process, which improves the robustness of the calibration process and reduces the adverse effects of noise points on the calibration accuracy.

[0017] (3) The present invention can automatically complete the external parameter calibration between the millimeter-wave radar and the camera sensor without relying on any specific calibration equipment. The external points are optimized through a dynamic Bayesian network and a random sampling consistency algorithm, thereby improving the overall robustness of the system and reducing the negative impact of noise on the calibration accuracy.

[0018] (4) The present invention also has good performance under low light conditions, showing broad development prospects and engineering application potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a working schematic diagram of a target-free integrated camera millimeter-wave radar automatic calibration method based on trajectory association in an embodiment of the present invention.

[0020] Figure 2 It is an operation flow chart of an automatic calibration method of a target-free integrated camera millimeter-wave radar based on trajectory association in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and the accompanying drawings specifically illustrate the automatic calibration method of the targetless integrated camera millimeter wave radar based on trajectory association of the present invention.

[0022] <Example> This embodiment provides a method for automatic calibration of a target-free integrated camera millimeter wave radar based on trajectory association, comprising the following steps: S10 collects data from cameras and millimeter-wave radar.

[0023] S20 estimates and sets the initial external parameters based on the installation height and angle of the camera and millimeter wave radar H 0.

[0024] The camera and millimeter wave radar are fixed in position, so the translation vector between them is not considered and is set to zero vector. x , z , y Sequential rotation, according to the rotation formula R = R y R z R x Calculate the initial external parameters H 0.

[0025] S30, trajectory acquisition, includes the following sub-steps S31 to S33: S31, uses the YOLO v3 algorithm to perform target detection on the images captured by the camera.

[0026] S32, using the Bytetrack algorithm to track the target detected in step S31 to obtain a 2D trajectory of the camera.

[0027] In the above steps S31~S32, a detection frame is generated for each detected target, and the Kalman filter is used to predict the position of the target trajectory at the next moment, which is recorded as the prediction frame. The intersection and union ratio of the detection frame and the prediction frame is calculated, and the Hungarian algorithm is used to match the high-score detection frame in the current frame with the prediction frame of the trajectory of the previous frame to update the trajectory information.

[0028] S33, convert the radar 3D trajectory into a radar 2D trajectory (a point trajectory in the image plane) according to the following formula: .

[0029] In the above formula, ( u , v ) is the radar 2D trajectory, (X r , Y r , Z r ) is the radar 3D trajectory, Z c is the depth information, Z c Indicates the distance from the target vehicle point to the camera, u is the horizontal coordinate of the radar 2D trajectory point, v is the ordinate of the radar 2D trajectory point, X r is the horizontal coordinate of the radar 3D trajectory point, Y r is the ordinate of the radar 3D trajectory point, Z r is the vertical coordinate of the radar 3D trajectory point, K is the camera internal parameter, ( R | t ) is a 3×4 matrix with 6 degrees of freedom, and ( R | t ) is defined as the initial value required for calibration H The number 1 indicates that the matrix is ​​represented in the form of a homogeneous matrix.

[0030] S40, using the CATS algorithm to associate the camera 2D trajectory with the radar 2D trajectory to obtain an associated trajectory id The CATS algorithm calculates the similarity between the camera 2D trajectory and the radar 2D trajectory according to the following steps S41 to S43: S41, spatial attenuation function: The following spatial attenuation function is used to measure the spatial proximity of the data points of the camera 2D trajectory and the radar 2D trajectory: .

[0031] In the above formula, p m,e and p j,k They are data points from two different trajectories, the camera 2D trajectory and the radar 2D trajectory. f ε ( p m,e , p j,k ) is a spatial attenuation function with a value between 0 and 1, f ε ( p m,e , p j,k )along with p m,eand p j,k The closer the two points are, the larger the distance between them will be. e is the space threshold (set to less than 18 units in length).

[0032] This function is used to measure the spatial proximity of two data points. If the distance between the two points is less than a given spatial threshold e , the function returns a value between 0 and 1, indicating their spatial similarity; if the distance exceeds the set threshold, it returns 0.

[0033] S42, clue score: Considering the time threshold, calculate the clue score of each data point in the camera 2D trajectory relative to the radar 2D trajectory, or the clue score of each data point in the radar 2D trajectory relative to the camera 2D trajectory, the formula is as follows: .

[0034] In the above formula, T m and T j They are p m,e and p j,k The camera 2D track or radar 2D track, t m,e It's a track T m Medium data point p m,e time frame, t j,k It's a track T j Medium data point p j,k time frame, t is the time threshold (the corresponding time difference is set to less than 2 frames).

[0035] This score reflects the proximity of the data point to the midpoint of the reference trajectory in space and time. The calculation of the clue score takes into account the temporal threshold t , that is, two data points are considered to be clued only when they are also close enough in time.

[0036] S43, cue-aware trajectory similarity, obtains the similarity measure between the camera 2D trajectory and the radar 2D trajectory, and the formula is as follows: .

[0037] In the above formula, | T m | is a data point pm,e The number of, for any camera trajectory T m , you can find the highest scoring trajectory T j Correspondingly, vice versa.

[0038] In order to reduce the interference of noise points on matching, the correct corresponding relationship is found through the following steps S50-S60.

[0039] S50, using the DBN-RANSAC algorithm to roughly calibrate the 2D camera data points and the 3D radar data points to calculate external parameters, process outliers, and retain a good set of internal points, including the following sub-steps S51 to S56: S51, associated point trace set: given an initial data set S, the initial data set S is the associated trajectory in step S40 id The set of corresponding 2D camera data points and 3D radar data points in .

[0040] S52, Initialize belief scores: assign initial inlier probabilities to each data point , and set it to 0.5 to indicate the uncertainty of whether it is an interior point or an exterior point.

[0041] S53, weighted sampling to select data points and estimate model parameters: In the data set S, use the RANSAC algorithm to select at least 6 pairs of matching 2D camera data points and 3D radar data points to estimate the model parameters of the digital model of the data set S H .

[0042] In this step, the model parameters of the digital model H The derivation process is as follows: Steps S53-1 to S53-5: S53-1, the 2D camera data points in the initial data set S are known { p c,i， i =1,2,…, n} and its corresponding 3D radar data point { p r,i， i =1,2,…, n}, we can get the external parameter matrix ( R | t ), and expand to get formula (1): .

[0043] In the above formula (1), r It is the three-dimensional radar coordinate system that coincides with the world coordinate system. c is the two-dimensional image coordinate system, l is the scale factor representing the depth information,u is the horizontal coordinate of the image plane, v is the ordinate of the image plane, T represents the transpose of a vector, t 11 ~ t 34 for( R | t ) is the parameter to be found in the matrix, t 11 ~ t 34 Used to transform points in the world coordinate system into the camera coordinate system. X r is the horizontal coordinate of the radar point, Y r is the ordinate of the radar point, Z r is the vertical coordinate of the radar point.

[0044] S53-2, expand equation (1) to obtain equation (2): .

[0045] S53-3, in elimination formula (2) l We get formula (3): .

[0046] S53-4, Formula (4) is derived from Formula (3): .

[0047] In the above formula (4), u N is the image plane N The horizontal coordinate of the point, v N is the image plane N The vertical coordinate of the point, P N rT Indicates the radar coordinate system N The transpose of the coordinate vector of the point, N is the number of points.

[0048] S53-5, due to the x There are 12 dimensions in total. A pair of 2D camera data points and their corresponding 3D radar data points can provide two equations, so at least six pairs of matching points are needed to solve them. n When ≥6, the equation is generated: Axe =0.

[0049] Because in practical applications x is not equal to 0, so the goal is to make Axe As close to 0 as possible, it actually turns into a least squares problem to solve x , the calculation formula is: .

[0050] In || x ||=1 A Perform singular value decomposition and obtain the matrix A The right singular value vector corresponding to the minimum singular value is x The approximate solution of .

[0051] S54, calculate the consistency probability of each data point with the current digital model , thereby determining the current interior point probability.

[0052] S55, Update belief score: Use dynamic Bayesian network to update the belief score of each data point Perform update iterations, where the belief score is the probability that the data point is classified as an internal point. α is a normalization factor that ensures that the sum of the probabilities is 1.

[0053] is a recursive function that recursively combines all previous classifications and evidence to calculate the current belief score. represents the probability of a data point being classified as an inlier in the current iteration given the classification and evidence of the previous iteration, j This allows the model to consider the latest evidence and previous classification results in each iteration.

[0054] definition Indicated in k In the iteration, given the classification of the previous iteration , data points n The probability of being classified as an inlier or outlier; Indicated in k In iterations, given the evidence of the current iteration and the classification of the previous iteration , data points n is the probability of an interior point.

[0055] S56, check the number of inliers: the number of data points detected as inliers.

[0056] If the number of internal points increases, update the digital model and its model parameters H , so that the new model can more accurately identify the inliers, and repeat steps S54 to S56; If the number of inliers does not increase, return to and repeat steps S53 to S56 until the number of iterations in step S55 reaches a predetermined value and / or the number of inliers does not increase after multiple iterations, indicating that the model converges and the optimal digital model is output.

[0057] After the above steps S51 to S56 are processed, the transformation matrix parameters after rough calibration can be obtained.

[0058] S60, using the Levenberg-Marquardt algorithm (LM algorithm) to perform fine calibration calculation on the result of the rough calibration in step S50. The key of the LM algorithm is that it can automatically adjust parameters, adapt to different optimization stages, and avoid unstable factors in numerical calculations. Specifically, it includes the following sub-steps S61 to S65: S61, the precision calibration process is regarded as a nonlinear least squares optimization problem: , f ( x ) is the residual function.

[0059] S62, will f ( x )Taylor expansion, taking the first-order linear term to approximate formula (5): .

[0060] In the above formula (5), J is the Jacobian matrix, whose columns are the objective function f Partial derivatives with respect to each variable.

[0061] S63, find the partial derivative of equation (5) and make it equal to 0 to obtain equation (6): .

[0062] S64, the final iterative formula obtained from formula (6) is: .

[0063] in, x k is the variable value of the current iteration, g k is the gradient vector, E As the identity matrix, we get the optimal x That is, the optimal rotation matrix R is obtained.

[0064] S65, outputting the result of the precise calibration as the required external parameter matrix.

[0065] Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for automatic calibration of target-free integrated camera millimeter wave radar based on trajectory association, characterized in that: The following steps are involved: S10, collects data from cameras and millimeter-wave radar; S20, according to the installation height and angle of the camera and the millimeter wave radar, estimate and set the initial external parameters H 0; S30, obtaining the camera 2D trajectory obtained by detecting the target with the camera, and H 0. Transforming the radar 3D trajectory detected by the millimeter wave radar into a two-dimensional image plane to obtain a radar 2D trajectory; S40, using a CATS algorithm to associate the camera 2D trajectory with the radar 2D trajectory to obtain an associated trajectory id ; S50, recording points in the camera 2D trajectory as 2D camera data points, and recording points in the radar 3D trajectory corresponding to the radar 2D trajectory as 3D radar data points, According to the associated trajectory id The correct correspondence between the camera 2D trajectory and the radar 2D trajectory is obtained by processing outliers through the RANSAC algorithm combined with a dynamic Bayesian network to complete the rough calibration of the 2D camera data points and the 3D radar data points; S60, using the Levenberg-Marquardt algorithm to perform fine calibration calculation on the result of the rough calibration in step S50 to obtain the final optimal external parameters.

2. The method for automatic calibration of target-free integrated camera millimeter-wave radar based on trajectory association according to claim 1 is characterized in that: in, In step S20, the positions of the camera and the millimeter wave radar are fixed, and the translation vector between the two is not considered and is set to a zero vector.

3. The method for automatic calibration of target-free integrated camera millimeter-wave radar based on trajectory association according to claim 1 is characterized in that: in, Step S30 includes the following sub-steps: S31, performing target detection on the image captured by the camera using the YOLO v3 algorithm; S32, using the Bytetrack algorithm to track the target detected in step S31 to obtain the camera 2D trajectory; S33, converting the radar 3D trajectory into the radar 2D trajectory according to the following formula, the calculation formula is: ， in,( u , v ) is the radar 2D trajectory, ( X r , Y r , Z r ) is the radar 3D trajectory, Z c is the depth information, Z c Indicates the distance from the target vehicle point to the camera, u is the horizontal coordinate of the radar 2D trajectory point, v is the ordinate of the radar 2D trajectory point, X r is the horizontal coordinate of the radar 3D trajectory point, Y r is the ordinate of the radar 3D trajectory point, Z r is the vertical coordinate of the radar 3D trajectory point, K is the camera internal parameter, ( R | t ) is a 3×4 matrix with 6 degrees of freedom, and ( R | t ) is defined as the initial value required for calibration H The number 1 indicates that the matrix is ​​represented in the form of a homogeneous matrix.

4. The method for automatic calibration of target-free integrated camera millimeter-wave radar based on trajectory association according to claim 3 is characterized in that: in, In step S31 to step S32, a detection frame is generated for each detected target, and the Kalman filter is used to predict the position of the target trajectory at the next moment, which is recorded as the prediction frame. The intersection and union ratio of the detection frame and the prediction frame is calculated, and the Hungarian algorithm is used to match the high-score detection frame in the current frame with the prediction frame of the trajectory of the previous frame to update the trajectory information.

5. The method for automatic calibration of target-free integrated camera millimeter-wave radar based on trajectory association according to claim 3 is characterized in that: in, In step S40, a clue score of each data point in the camera 2D trajectory relative to the radar 2D trajectory is calculated, thereby obtaining a similarity measure between the camera 2D trajectory and the radar 2D trajectory. For any of the camera 2D tracks or the radar 2D tracks, there is a corresponding radar 2D track or the camera 2D track with the highest clue score.

6. The method for automatic calibration of target-free integrated camera millimeter-wave radar based on trajectory association according to claim 5 is characterized in that: in, The CATS algorithm in step S40 calculates the similarity between the camera 2D trajectory and the radar 2D trajectory according to the following steps: S41, using the following spatial attenuation function to measure the spatial proximity of the data points of the camera 2D trajectory and the radar 2D trajectory: ； S42, considering the time threshold, calculating the clue score of each data point in the camera 2D trajectory relative to the radar 2D trajectory, or the clue score of each data point in the radar 2D trajectory relative to the camera 2D trajectory: ； S43, using the CATS algorithm to obtain a similarity measure between the camera 2D trajectory and the radar 2D trajectory: ， In steps S41 to S43, p m,e and p j,k are data points from two different tracks in the camera 2D track and the radar 2D track, respectively. T m and T j They are p m,e and p j,k The camera 2D trajectory or the radar 2D trajectory, | T m | is a data point p m,e The number of t m,e It's a track T m Medium data point p m,e time frame, t j,k It's a track T j Medium data point p j,k time frame, τ is the time threshold, ε is the spatial threshold, f ε ( p m,e , p j,k ) is a spatial attenuation function with a value between 0 and 1, f ε ( p m,e , p j,k )along with p m,e and p j,k dist represents the Euclidean distance between two points.

7. The method for automatic calibration of target-free integrated camera millimeter-wave radar based on trajectory association according to claim 6 is characterized in that: in, Spatial Threshold ε Less than 18 units in length, time threshold τ The corresponding time difference is less than 2 frames.

8. The method for automatic calibration of target-free integrated camera millimeter-wave radar based on trajectory association according to claim 6 is characterized by: in, Step S50 includes the following sub-steps: S51, given an initial data set S, the initial data set S is the associated trajectory in step S40 id A set of the corresponding 2D camera data points and the 3D radar data points; S52, assigning an initial inlier probability to each data point and setting it to 0.5 to indicate the uncertainty of whether it is an inlier or an outlier; S53, in the data set S, using the RANSAC algorithm to select at least 6 pairs of matching 2D camera data points and 3D radar data points to estimate model parameters of the digital model of the data set S H ; S54, calculating the consistency probability between each data point and the current digital model, thereby determining the current interior point probability; S55, using a dynamic Bayesian network to iteratively update the belief score of each of the data points, the belief score being the inlier probability; S56, the number of the data points detected as inliers, If the number of interior points increases, the digital model and its model parameters are updated H , and repeat steps S54 to S56, If the number of inliers does not increase, the process returns to and repeats steps S53 to S56 until the number of iterations in step S55 reaches a predetermined value and / or the number of inliers does not increase after multiple iterations, and then outputs the optimal digital model.

9. The method for automatic calibration of target-free integrated camera millimeter-wave radar based on trajectory association according to claim 8, characterized in that: in, In step S53, the model parameters H The derivation process is as follows: S53-1, the 2D camera data points in the initial data set S are known { p c,i ，i =1,2,…, n } and the corresponding 3D radar data point { p r,i ，i =1,2,…, n }, we can get the external parameter matrix ( R | t ), and expand to get formula (1): ； S53-2, expand equation (1) to obtain equation (2): ； S53-3, in elimination formula (2) λ We get formula (3): ； S53-4, Formula (4) is derived from Formula (3): ； S53-5, due to the x There are 12 dimensions in total. A pair of the 2D camera data point and its corresponding 3D radar data point can provide two equations, so at least six pairs of matching points are needed to solve. n When ≥6, the equation is generated: Ax =0, Because in practical applications x is not equal to 0, so the goal is to make Ax As close to 0 as possible, it actually turns into a least squares problem to solve x , the calculation formula is: ， In || x Under the constraint of ||=1, A Perform singular value decomposition and obtain the matrix A The right singular value vector corresponding to the minimum singular value is x The approximate solution of in, r It is the three-dimensional radar coordinate system that coincides with the world coordinate system. c is the two-dimensional image coordinate system, t 11 ~ t 34 for( R | t ) is the parameter to be found in the matrix, t 11 ~ t 34 Used to transform points in the world coordinate system into the camera coordinate system. u is the horizontal coordinate of the image plane, u N is the image plane N The horizontal coordinate of the point, v is the ordinate of the image plane, v N is the image plane N The vertical coordinate of a point, X r is the horizontal coordinate of the radar point, Y r is the ordinate of the radar point, Z r is the vertical coordinate of the radar point, λ is the scale factor representing the depth information, T represents the transpose of a vector, P N rT Indicates the radar coordinate system N The transpose of the coordinate vector of the point, N is the number of points.

10. The method for automatic calibration of target-free integrated camera millimeter-wave radar based on trajectory association according to claim 9, Features: Wherein, step S60 includes the following sub-steps: S61, the precise calibration process is regarded as a nonlinear least squares optimization problem: , f ( x ) is the residual function; S62, will f ( x )Taylor expansion, taking the first-order linear term to approximate formula (5): ， in, J is the Jacobian matrix, whose columns are the objective function f Partial derivatives with respect to each variable; S63, find the partial derivative of equation (5) and make it equal to 0 to obtain equation (6): ； S64, the final iterative formula obtained from formula (6) is: ， in, x k is the variable value of the current iteration, g k is the gradient vector, E is the identity matrix, Get the best x That is, the optimal rotation matrix R is obtained; S65, outputting the result of the precise calibration as the required external parameter matrix.

Images