Automatic calibration method and system based on millimeter wave radar and camera

Abstract

The application relates to a sensor fusion technology, and relates to an automatic calibration method and system based on a millimeter wave radar and a camera. The method comprises the following steps: obtaining N radar detection targets through the radar, obtaining M image detection targets through the camera, and respectively performing feature extraction to obtain corresponding feature vectors; according to the matching probability and the Euclidean distance between the feature vectors of the radar detection targets and the image detection targets, the corresponding relationship between the feature vectors of the radar detection targets and the image detection targets is obtained as a matching correlation result between the radar detection targets and the image detection targets; and according to the matching correlation result, the external parameters between the camera reference coordinate system and the radar reference coordinate system are estimated. The automatic calibration of the millimeter wave radar and the camera is converted into an optimal transmission problem to estimate the external parameters, and a specific reference object is not needed; when the relative positions of the radar and the camera change, the external parameters can be automatically adjusted.

Description

TECHNICAL FIELD

[0001] The present application relates to sensor fusion technology, in particular to an automatic calibration method and system based on millimeter wave radar and camera. BACKGROUND

[0002] One of the prerequisites for sensor fusion is the spatial alignment of different sensor data, that is, the calibration between different sensors; for example, for radar and camera, the coordinate transformation relationship between the radar reference coordinate system and the camera reference coordinate system needs to be obtained. The coordinate transformation relationship is usually described by extrinsic and intrinsic parameters. The intrinsic parameter is determined by the physical characteristics of the camera and is generally obtained by Zhang Zhengyou calibration method. The extrinsic parameter describes the relative position relationship between sensors, and the calibration is generally completed by setting a reference object and obtaining the target points of the radar reference coordinate system and the camera reference coordinate system. Since the intrinsic parameter of the camera describes the inherent physical characteristics of the camera, the calibration problem between sensors mainly lies in estimating the relative position relationship between the camera and the radar, i.e., the extrinsic parameter.

[0003] The PnP (Perspective-n-Point) problem refers to the problem of solving the pose of the camera reference coordinate system relative to the world coordinate system through n 3D points to 2D point pairs. The PnP algorithm describes how to estimate the extrinsic parameter when the coordinates of n 3D points (relative to the world coordinate system) and the pixel coordinates corresponding to these points are known. The relationship between the radar coordinate p and the pixel coordinate q is as follows:

[0004] q=K[RT]p (1)

[0005] where R and T are extrinsic parameters, which are rotation matrix and translation vector respectively; K is the intrinsic parameter of the camera (i.e., the camera).

[0006] Most of the existing calibration methods need a reference object (such as a corner reflector) or a rough estimate of the extrinsic parameter, in order to obtain the point pairs in the radar-image, and then use PnP algorithms such as DLT (Direct Linear Transform) to solve the extrinsic parameter. This kind of calibration method either needs a reference object or needs a rough estimate of the extrinsic parameter, which requires a lot of manpower and material resources and takes a long time in the actual deployment process. Moreover, when the relative position of the radar sensor and the camera sensor changes, these methods cannot automatically adjust the extrinsic parameter. In addition, the scheme requiring a rough estimate of the extrinsic parameter is highly dependent on the accuracy of the initial extrinsic parameter. When the initial extrinsic parameter value is not accurate enough, the solution to the PnP problem will fall into local optimum, and therefore the robustness is poor. SUMMARY

[0007] In order to solve the problems existing in the prior art, the application provides an automatic calibration method and system based on a millimeter wave radar and a camera, which converts automatic calibration of the millimeter wave radar and the camera into solving an optimal transmission problem to estimate external parameters without specific reference objects and rough external parameter priors; and when the relative positions of the radar and the camera change, the external parameters can be automatically adjusted.

[0008] The automatic calibration method based on the millimeter wave radar and the camera provided in the embodiments of the application includes the following steps:

[0009] N radar detection targets are obtained by the radar, and M image detection targets are obtained by the camera, feature vectors of the radar detection targets and the image detection targets are extracted respectively, and the corresponding feature vectors are obtained;

[0010] According to the matching probability and the Euclidean distance between the feature vectors of the radar detection targets and the feature vectors of the image detection targets, a corresponding relationship between the feature vectors of the radar detection targets and the feature vectors of the image detection targets is obtained as a matching association result between the radar detection targets and the image detection targets;

[0011] According to the matching association result between the radar detection targets and the image detection targets, external parameters between a camera reference coordinate system and a radar reference coordinate system are estimated, and automatic calibration between the millimeter wave radar and the camera is completed.

[0012] Preferably, the problem of obtaining the corresponding relationship between the radar detection targets and the image detection targets is converted into an optimal transmission problem, and the corresponding relationship is obtained by solving the optimal transmission problem.

[0013] Preferably, obtaining the corresponding relationship between the feature vectors of the radar detection targets and the feature vectors of the image detection targets includes the following steps:

[0014] r and c are defined as the probability distribution of the feature vectors of the radar detection targets and the probability distribution of the feature vectors of the image detection targets respectively;

[0015] The Euclidean distance between the feature vectors of the radar detection targets and the feature vectors of the image detection targets is defined as a transition cost matrix L;

[0016] According to the probability distribution of the feature vectors of the radar detection targets and the probability distribution of the feature vectors of the image detection targets, a matching probability matrix P is designed, wherein an element in the i-th row and the j-th column of the matching probability matrix P represents the matching probability of the feature vector of the i-th radar target and the feature vector of the j-th image target;

[0017] The matching probability matrix P is represented as a function of the transition cost matrix L;

[0018] Screening the first K maximum values of the matching probability matrix, to obtain K groups of matching association results between the radar detection targets and the image detection targets.

[0019] The automatic calibration system based on the millimeter wave radar and the camera provided in the embodiment of the application comprises modules.

[0020] The data acquisition module is configured to obtain N radar detection targets by the radar and M image detection targets by the camera, and perform feature extraction on the detected radar detection targets and image detection targets respectively to obtain corresponding feature vectors.

[0021] The target matching association module is configured to obtain the corresponding relationship between the feature vectors of the radar detection targets and the feature vectors of the image detection targets according to the matching probability and the Euclidean distance between the feature vectors of the radar detection targets and the feature vectors of the image detection targets, as the matching association result between the radar detection targets and the image detection targets.

[0022] The automatic calibration module is configured to estimate the external parameters between the camera reference coordinate system and the radar reference coordinate system according to the matching association result between the radar detection targets and the image detection targets, and complete the automatic calibration between the millimeter wave radar and the camera.

[0023] From the above technical solutions, it can be known that the application does not need a reference object, extracts the spatial geometric features of the target points, and uses the Sinkhorn algorithm to associate the target detection results from the millimeter wave radar and the image; compared with the prior art, the application achieves the following technical effects:

[0024] 1. The camera and the radar are used for target detection, the spatial geometric feature vectors of each point are extracted from the image pixel points and the millimeter wave radar point cloud by the feature extraction network, the Sinkhorn algorithm is used to estimate the corresponding relationship of the targets from different sensors, and thus the association result of the millimeter wave radar detection target-image detection target is obtained. Thus, the automatic calibration of the millimeter wave radar and the camera can be converted into solving the optimal transmission PnP problem, and thus the external parameters are estimated.

[0025] 2. The application realizes spatial calibration at the data level, does not need a specific reference object and a rough external parameter priori in the process of estimating the external parameters, the process can be automated, and the robustness is high; and when the relative positions of the radar and the camera change, the external parameters can be automatically adjusted. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is the flowchart of the millimeter wave radar-camera automatic calibration in the embodiment of the application;

[0027] Figure 2is a flowchart for obtaining the correspondence between the feature vector of the radar detection target and the feature vector of the image detection target in the embodiment of the present application.

[0028] Figure 3 is a schematic diagram of matching the probability matrix in the embodiment of the present application. DETAILED DESCRIPTION

[0029] The present application first obtains the radar detection target and the image detection target through the target detection method of the millimeter wave radar and the camera; the feature extraction network is used to obtain the corresponding feature vector of each target. Then the optimal transmission problem between the two groups of feature vectors is solved to obtain the matching probability matrix, and the PnP algorithm is used to solve the external parameter between the camera reference coordinate system and the radar reference coordinate system. The present application does not use specific reference objects such as corner reflectors, and the process can be automated, thereby realizing the automatic calibration of the millimeter wave radar and the camera.

[0030] In order to better understand the technical scheme of the present application, the present application will be further described in conjunction with the drawings and embodiments. However, the embodiments of the present application are not limited thereto, and all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0031] Embodiment 1

[0032] The present embodiment is an automatic calibration method based on millimeter wave radar and camera, as shown in Figure 1 , comprising the following steps:

[0033] S1, obtaining N radar detection targets through the radar and M image detection targets through the camera, respectively extracting the features of the detected radar detection targets and image detection targets to obtain the corresponding feature vectors.

[0034] This step obtains the radar echo containing the target through the millimeter wave radar, and extracts the position of each target from it; obtains the image containing the target through the camera, and extracts the position of each target from it. The detection targets of the millimeter wave radar and the image are respectively extracted, and each target corresponds to a feature vector that encodes its geometric features.

[0035] Suppose that in a certain data frame, the number of targets detected by the radar is N, which is a 3D coordinate point; the number of targets detected by the camera is M, which is a 2D coordinate point. The detected 3D coordinate points and 2D coordinate points are input into the feature extraction network to obtain the spatial geometric feature vector (referred to as feature vector) of each point. Among them, the feature vector of the radar detection target is Nxd, and the feature vector of the image detection target is Mxd.

[0036] S2, according to the matching probability, the Euclidean distance between the feature vectors of the radar detection target and the feature vectors of the image detection target, the correspondence between the feature vectors of the radar detection target and the feature vectors of the image detection target is obtained as the matching association result between the radar detection target and the image detection target.

[0037] The embodiment converts the problem of obtaining the correspondence between the radar detection target and the image detection target into an optimal transport problem, and obtains the correspondence by solving the optimal transport problem. The Sinkhorn algorithm is an algorithm for solving the optimal transport problem, and can give an approximate solution to the problem. The solution is in the form of a matching probability matrix P with a dimension of MxN.

[0038] The embodiment uses the Sinkhorn algorithm to realize the correspondence between the feature vectors of the radar detection target and the feature vectors of the image detection target, as shown in Figure 2 , which includes the following steps:

[0039] S21, define r, c as the probability distribution of the feature vectors of the radar detection target and the probability distribution of the feature vectors of the image detection target, and λ as a hyperparameter.

[0040] S22, define the Euclidean distance between the feature vectors of the radar detection target and the feature vectors of the image detection target as a transfer cost matrix L, and the dimension of the transfer cost matrix is MxN. In the embodiment, the transfer cost matrix has the following formula:

[0041] L(i,j) = ||f ri -f cj ||2 (2)

[0042] where f ri is the feature vector of the i-th radar detection target, f cj is the feature vector of the j-th image detection target, and L(i,j) is the transfer cost matrix.

[0043] In the embodiment, the Euclidean distances between the feature vectors from the millimeter wave radar and the camera are calculated two by two to obtain an MxN-dimensional cost matrix.

[0044] S23, according to the probability distribution of the feature vectors of the radar detection target and the probability distribution of the feature vectors of the image detection target, a matching probability matrix is designed, wherein the element in the i-th row and the j-th column of the matching probability matrix represents the matching probability of the feature vector of the i-th radar target and the feature vector of the j-th image target. The designed matching probability matrix is as shown in Figure 3 .

[0045] The Sinkhorn algorithm is used in this step, and it is assumed that the radar detected targets and the image detected targets are subject to uniform distribution, so that a matching probability matrix of M*N dimensions can be designed.

[0046] In S24, the matching probability matrix P is represented as a function of the transition cost matrix L, and the function is as follows:

[0047] P=e -λL (3)

[0048] It is determined whether the matching probability matrix P converges, and if not, the probability distribution r of the feature vector of the radar detected target is used to normalize the rows of the matching probability matrix P, so that the sum of the rows is equal to the sum of the probability distribution r of the feature vector of the radar detected target, and the probability distribution c of the feature vector of the image detected target is used to normalize the columns of the matching probability matrix P, so that the sum of the columns is equal to the sum of the probability distribution c of the feature vector of the image detected target; until the matching probability matrix P converges, and the final matching probability matrix P is obtained.

[0049] In S25, the top K maximum values of the matching probability matrix are screened out, and K groups of matching association results between the radar detected targets and the image detected targets are obtained.

[0050] The Top-K algorithm is an algorithm for screening out the top K maximum values, and in this embodiment, the Top-K algorithm is used to screen out the K groups of most possible matching association results from the matching probability matrix as the input of the PnP algorithm.

[0051] In S3, the matching association results between the radar detected targets and the image detected targets are used as the input of the PnP algorithm, and the external parameters R and T between the camera reference coordinate system and the radar reference coordinate system are estimated by the PnP algorithm, so as to complete the automatic calibration between the millimeter wave radar and the camera.

[0052] The PnP algorithm is an algorithm for estimating the external parameters R and T by the combination of the given 3D coordinate points and 2D coordinate points. For n 3D-2D coordinate point pairs, 2n equation groups can be obtained from the relationship (1) between the radar coordinates p and the pixel coordinates q, and the external parameters R and T have 6 degrees of freedom in total, so that generally more than 3 coordinate points are needed to estimate the external parameters.

[0053] In S4, the internal parameters of the camera are obtained by the Zhang Zhengyou calibration method.

[0054] The application obtains two groups of feature vectors by feature extraction on detected targets from millimeter wave radar and images, obtains a matching probability matrix by solving an optimal transmission problem between the two groups of feature vectors, and then solves the external parameters between the camera reference coordinate system and the radar reference coordinate system by using a PnP algorithm, so as to complete the automatic calibration between the millimeter wave radar and the camera. Compared with the traditional calibration method, the application does not need specific reference objects such as corner reflectors and can realize the automation of the process. Sensor calibration is the premise of sensor fusion, so the application can be widely applied in the calibration of millimeter wave radars and cameras.

[0055] Embodiment 2

[0056] Based on the same inventive concept as embodiment 1, the present embodiment is an automatic calibration system based on millimeter wave radar and camera, comprising:

[0057] A data acquisition module is configured to obtain N radar detection targets by radar and M image detection targets by camera, and perform feature extraction on the detected radar detection targets and image detection targets respectively to obtain corresponding feature vectors;

[0058] A target matching and association module is configured to obtain a corresponding relationship between the feature vectors of the radar detection targets and the feature vectors of the image detection targets according to the matching probability and the Euclidean distance between the feature vectors of the radar detection targets and the feature vectors of the image detection targets, as a matching and association result between the radar detection targets and the image detection targets;

[0059] An automatic calibration module is configured to estimate the external parameters between the camera reference coordinate system and the radar reference coordinate system according to the matching and association result between the radar detection targets and the image detection targets, and complete the automatic calibration between the millimeter wave radar and the camera.

[0060] In the present embodiment, the target matching and association module converts the problem of obtaining the corresponding relationship between the radar detection targets and the image detection targets into an optimal transmission problem, and obtains the corresponding relationship by solving the optimal transmission problem; the process of obtaining the corresponding relationship comprises:

[0061] Defining r and c as the probability distribution of the feature vectors of the radar detection targets and the probability distribution of the feature vectors of the image detection targets respectively;

[0062] Defining the Euclidean distance between the feature vectors of the radar detection targets and the feature vectors of the image detection targets as a transition cost matrix L;

[0063] According to the probability distribution of the feature vectors of the radar detected targets and the probability distribution of the feature vectors of the image detected targets, a matching probability matrix is designed, wherein an element in the ith row and the jth column of the matching probability matrix represents a matching probability of the feature vector of the ith radar detected target and the feature vector of the jth image detected target.

[0064] The matching probability matrix P is represented as a function of the transition cost matrix L.

[0065] The first K maximum values of the matching probability matrix are screened out, and K groups of matching association results between the radar detected targets and the image detected targets are obtained.

[0066] The modules of the embodiment are intended to realize the steps of the embodiment 1, and the detailed implementation process can be referred to the embodiment 1.

[0067] The above embodiments of the present application are merely examples for clearly illustrating the present application, but not intended to limit the modes of implementation of the present application. Any modification, equivalent replacement and improvement made on the basis of the above description for those skilled in the art should be included in the protection scope of the claims of the present application.

Claims

1. An automatic calibration method based on millimeter-wave radar and camera, characterized in that, Including the following steps: N radar-detected targets are obtained through radar, and M image-detected targets are obtained through camera. Feature extraction is performed on the detected radar-detected targets and image-detected targets respectively to obtain the corresponding feature vectors. Based on the matching probability and Euclidean distance between the feature vectors of radar-detected targets and image-detected targets, the correspondence between the feature vectors of radar-detected targets and image-detected targets is obtained, which serves as the matching association result between radar-detected targets and image-detected targets. Based on the matching and correlation results between radar-detected targets and image-detected targets, the external parameters between the camera reference coordinate system and the radar reference coordinate system are estimated to complete the automatic calibration between the millimeter-wave radar and the camera. The problem of finding the correspondence between radar-detected targets and image-detected targets is transformed into an optimal transmission problem, and the correspondence is obtained by solving the optimal transmission problem. To determine the correspondence between the feature vectors of radar-detected targets and the feature vectors of image-detected targets, the following steps are included: Let r and c be the probability distributions of the feature vectors of the target detected by radar and the feature vectors of the target detected by image, respectively. The Euclidean distance between the feature vectors of radar-detected targets and the feature vectors of image-detected targets is defined as the transfer cost matrix L; Based on the probability distribution of the feature vectors of radar-detected targets and the probability distribution of the feature vectors of image-detected targets, a matching probability matrix is ​​designed, where the element in the i-th row and j-th column of the matching probability matrix represents the matching probability of the feature vector of the i-th radar target and the feature vector of the j-th image target. The matching probability matrix P is represented as a function of the transition cost matrix L; By selecting the top K maximum values ​​of the matching probability matrix, we obtain K sets of matching association results between radar-detected targets and image-detected targets. Before selecting the first K maximum values ​​of the matching probability matrix, it is determined whether the matching probability matrix P has converged. If not, the rows of the matching probability matrix P are normalized using the probability distribution r of the feature vectors of the radar-detected targets, so that the sum of the rows is equal to the sum of the probability distributions r of the feature vectors of the radar-detected targets. The columns of the matching probability matrix P are normalized using the probability distribution c of the feature vectors of the image-detected targets, so that the sum of the columns is equal to the sum of the probability distributions c of the feature vectors of the image-detected targets. This process continues until the matching probability matrix P converges, yielding the final matching probability matrix P.

2. The automatic calibration method according to claim 1, characterized in that, The functional expression for the transfer cost matrix L is: P=e -λL Where λ is a hyperparameter.

3. The automatic calibration method according to claim 1, characterized in that, The formula for the transfer cost matrix is: L(i,j)=||f ri -f cj ||2 Where f ri Let f be the feature vector of the i-th radar-detected target. cj Let L(i,j) be the feature vector of the j-th image detection target, and L(i,j) be the transition cost matrix.

4. The automatic calibration method according to claim 1, characterized in that, The automatic calibration method further includes the following steps: The internal parameters of the camera were obtained using the Zhang Zhengyou calibration method.

5. The automatic calibration method according to claim 1, characterized in that, Assuming that both radar-detected targets and image-detected targets follow a uniform distribution, the Sinkhorn algorithm is used to design an M×N dimensional matching probability matrix.

6. An automatic calibration system based on millimeter-wave radar and camera, implemented using the automatic calibration method described in any one of claims 1-5, characterized in that, Includes modules: The data acquisition module is used to acquire N radar-detected targets and M image-detected targets from the camera, and to extract features from the detected radar-detected targets and image-detected targets to obtain the corresponding feature vectors. The target matching and association module is used to calculate the correspondence between the feature vectors of radar-detected targets and the feature vectors of image-detected targets based on the matching probability and Euclidean distance between the feature vectors of radar-detected targets and the feature vectors of image-detected targets, and use this as the matching and association result between radar-detected targets and image-detected targets. The automatic calibration module is used to estimate the external parameters between the camera reference coordinate system and the radar reference coordinate system based on the matching and correlation results between the radar-detected target and the image-detected target, and to complete the automatic calibration between the millimeter-wave radar and the camera.

Images