A millimeter-wave radar and video joint calibration method, system and terminal device

Abstract

The present invention discloses a method, system and terminal device for joint calibration of millimeter-wave radar and video. The method includes obtaining a video stream to be jointly calibrated, millimeter-wave radar data at four corner points of a rectangular area in the video stream acquisition section to be jointly calibrated, at near points and far points of traffic targets in each lane, and pixel coordinates of the center points of traffic targets in each lane at near points and far points, calculating a projection transformation matrix and a scaling matrix, and performing joint calibration of millimeter-wave radar data and video data based on the projection transformation matrix and the scaling matrix; by using millimeter-wave radar data at four corner points of a rectangular area in the video stream acquisition section to be jointly calibrated, at near points and far points of traffic targets in each lane, and pixel coordinates of the center points of traffic targets in each lane at near points and far points, and calculating a projection transformation matrix and a scaling matrix, joint calibration of millimeter-wave radar data and video data can be achieved. At the same time, the calculation steps are simple, and the processing efficiency can be greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of position calibration, and particularly to a method, system and terminal device for jointly calibrating a millimeter-wave radar and a video. Background Art

[0002] With the rapid development of multi-sensor fusion technology, the method for jointly calibrating the positions between multi-sensors has received extensive attention and research. Especially in the fields of visual navigation of intelligent robots and intelligent driving of vehicles, the fusion perception algorithm between a radar and a camera can provide a more reliable, accurate and stable environmental perception ability, which is convenient for the subsequent realization of intelligent decision-making and intelligent driving of robots and vehicles. And joint calibration is the prerequisite guarantee for realizing the fusion perception algorithm. The purpose of joint calibration is to achieve data matching between multi-sensors.

[0003] In the published invention patent with the application number 201910624873.2, a method for jointly calibrating a millimeter-wave radar and a camera based on the LM algorithm is disclosed, which can effectively realize the joint calibration of the millimeter-wave radar and the camera and ensure a certain accuracy rate. However, although the existing methods for jointly calibrating a millimeter-wave radar and a camera can realize the joint calibration of the millimeter-wave radar and the camera and have a certain accuracy rate, the calculation process is relatively complex, resulting in low processing efficiency. Summary of the Invention

[0004] In view of this, the present invention provides a method, system and terminal device for jointly calibrating a millimeter-wave radar and a video, which can solve the defect of low processing efficiency existing in the prior art.

[0005] The technical solution of the present invention is implemented as follows:

[0006] A method for jointly calibrating a millimeter-wave radar and a video specifically includes:

[0007] Obtain a video stream to be jointly calibrated;

[0008] Collect millimeter-wave radar data at the four corner points of the rectangular area of the video stream acquisition section to be jointly calibrated, at the near point and the far point of each lane traffic target, and pixel coordinates of the center point of each lane traffic target at the near point and the far point;

[0009] Calculate a projection transformation matrix according to the four corner points of the section rectangular area;

[0010] Calculate a scaling matrix according to the millimeter-wave radar data of each lane traffic target at the near point and the far point, and the pixel coordinates of the center point of each lane traffic target at the near point and the far point;

[0011] Based on the projection transformation matrix and the scaling matrix, the millimeter-wave radar data and the video data are combined to achieve the joint calibration of the millimeter-wave radar and the video.

[0012] As a further optional solution of the millimeter-wave radar and video joint calibration method, calculating the projection transformation matrix based on the four corner points of the road section rectangular area specifically includes:

[0013] Determine the picture size of the road section rectangular area according to the four corner points of the road section rectangular area;

[0014] Compare the picture size of the video stream to be jointly calibrated with the picture size of the road section rectangular area to obtain the projection transformation matrix.

[0015] As a further optional solution of the millimeter-wave radar and video joint calibration method, calculating the scaling matrix based on the millimeter-wave radar data of each lane traffic target at the near point and the far point, and the pixel coordinates of the center point of each lane traffic target at the near point and the far point specifically includes:

[0016] Perform projection transformation on the pixel coordinates of the center point of each lane traffic target at the near point and the far point to obtain the projection coordinates of the center point of each lane traffic target at the near point and the far point;

[0017] Compare the projection coordinates of the center point of each lane traffic target at the near point and the far point with the millimeter-wave radar data of each lane traffic target at the near point and the far point to obtain the scaling matrix.

[0018] As a further optional solution of the millimeter-wave radar and video joint calibration method, the algorithm used for comparing the projection coordinates of the center point of each lane traffic target at the near point and the far point with the millimeter-wave radar data of each lane traffic target at the near point and the far point is the least squares method.

[0019] A millimeter-wave radar and video joint calibration system, the system includes:

[0020] An acquisition module, used to acquire the video stream to be jointly calibrated;

[0021] A collection module, used to collect the four corner points of the road section rectangular area in the video stream to be jointly calibrated, the millimeter-wave radar data of each lane traffic target at the near point and the far point, and the pixel coordinates of the center point of each lane traffic target at the near point and the far point;

[0022] A first calculation module, used to calculate the projection transformation matrix according to the four corner points of the road section rectangular area;

[0023] A second calculation module, configured to calculate a scaling matrix based on millimeter-wave radar data of each lane traffic target at the near point and the far point, and pixel coordinates of the center point of each lane traffic target at the near point and the far point;

[0024] A joint calibration module, configured to perform joint calibration of millimeter-wave radar data and video data based on the projection transformation matrix and the scaling matrix.

[0025] As a further optional solution of the millimeter-wave radar and video joint calibration system, the first calculation module includes:

[0026] A screen size determination module, configured to determine the screen size of the road section rectangular area based on the four corner points of the road section rectangular area;

[0027] A first comparison module, configured to compare the screen size of the video stream to be jointly calibrated with the screen size of the road section rectangular area to obtain a projection transformation matrix.

[0028] As a further optional solution of the millimeter-wave radar and video joint calibration system, the second calculation module includes:

[0029] A transformation module, configured to perform projection transformation on the pixel coordinates of the center point of each lane traffic target at the near point and the far point to obtain the projection coordinates of the center point of each lane traffic target at the near point and the far point;

[0030] A second comparison module, configured to compare the projection coordinates of the center point of each lane traffic target at the near point and the far point with the millimeter-wave radar data of each lane traffic target at the near point and the far point to obtain a scaling matrix.

[0031] As a further optional solution of the millimeter-wave radar and video joint calibration system, the algorithm adopted by the second comparison module is the least squares method.

[0032] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any one of the above millimeter-wave radar and video joint calibration methods are implemented.

[0033] The beneficial effects of the present invention are as follows: By collecting the four corner points of the road section rectangular area in the video stream to be jointly calibrated, the millimeter-wave radar data of each lane traffic target at the near point and the far point, and the pixel coordinates of the center point of each lane traffic target at the near point and the far point, and calculating the projection transformation matrix and the scaling matrix, the joint calibration of millimeter-wave radar data and video data can be effectively achieved. At the same time, the calculation steps are simple, and the processing efficiency can be greatly improved. Description of the Drawings

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0035] Figure 1 It is a schematic flow chart of a method for joint calibration of a millimeter-wave radar and a video according to the present invention;

[0036] Figure 2 It is a schematic diagram of the composition of a system for joint calibration of a millimeter-wave radar and a video according to the present invention. Specific embodiments

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0038] Refer to Figure 1-2 , a method for joint calibration of a millimeter-wave radar and a video, specifically including:

[0039] Obtain the video stream to be jointly calibrated;

[0040] At the four corner points of the rectangular area of the video stream acquisition section to be jointly calibrated, the millimeter-wave radar data of each lane traffic target at the near point and the far point, and the pixel coordinates of the center point of each lane traffic target at the near point and the far point;

[0041] Calculate the projection transformation matrix according to the four corner points of the said road section rectangular area;

[0042] Calculate the scaling matrix according to the millimeter-wave radar data of each lane traffic target at the near point and the far point, and the pixel coordinates of the center point of each lane traffic target at the near point and the far point;

[0043] Perform the joint of the millimeter-wave radar data and the video data according to the projection transformation matrix and the scaling matrix, so as to realize the joint calibration of the millimeter-wave radar and the video.

[0044] In this embodiment, by using the millimeter-wave radar data at the four corner points of the rectangular area of the video stream acquisition section to be jointly calibrated, the millimeter-wave radar data at the near point and far point of each lane traffic target, and the pixel coordinates of the center point of each lane traffic target at the near point and far point, and calculating the projection transformation matrix and the scaling matrix, the joint calibration of the millimeter-wave radar data and the video data can be effectively achieved. At the same time, the calculation steps are simple, which can greatly improve the processing efficiency.

[0045] It should be noted that the four corner points of the rectangular area of the section are respectively the upper left corner point, the upper right corner point, the lower left corner point, and the lower right corner point of the video stream to be jointly calibrated. In addition, the projection transformation matrix is used to enable the mutual conversion between pixel coordinates and projection coordinates, and the scaling matrix is used to enable the mutual conversion between millimeter-wave radar coordinates and projection coordinates, so as to indirectly achieve the mutual conversion between pixel coordinates and millimeter-wave radar coordinates, and then achieve the combination of millimeter-wave radar data and video data.

[0046] Preferably, calculating the projection transformation matrix according to the four corner points of the rectangular area of the section specifically includes:

[0047] Determine the picture size of the rectangular area of the section according to the four corner points of the rectangular area of the section;

[0048] Compare the picture size of the video stream to be jointly calibrated with the picture size of the rectangular area of the section to obtain the projection transformation matrix.

[0049] In this embodiment, by first determining the picture size of the rectangular area of the section, and then performing matching projection on the picture size of the video stream to be jointly calibrated according to the picture size of the rectangular area of the section, the projection transformation matrix can be accurately obtained, and the accuracy of obtaining the projection transformation matrix can be improved.

[0050] Preferably, calculating the scaling matrix according to the millimeter-wave radar data at the near point and far point of each lane traffic target, and the pixel coordinates of the center point of each lane traffic target at the near point and far point specifically includes:

[0051] Perform projection transformation on the pixel coordinates of the center point of each lane traffic target at the near point and far point to obtain the projection coordinates of the center point of each lane traffic target at the near point and far point;

[0052] Compare the projection coordinates of the center point of each lane traffic target at the near point and far point with the millimeter-wave radar data at the near point and far point of each lane traffic target to obtain the scaling matrix.

[0053] In this embodiment, the pixel coordinates of the center points of each lane traffic target at the near point and the far point are subjected to projection transformation through a projection transformation matrix to obtain corresponding projection coordinates, and then the projection coordinates are compared with the corresponding millimeter-wave radar data, so that a scaling matrix can be accurately obtained, and the accuracy of obtaining the scaling matrix is improved.

[0054] Preferably, the algorithm used to compare the projection coordinates of the center points of each lane traffic target at the near point and the far point with the millimeter-wave radar data of each lane traffic target at the near point and the far point is the least squares method.

[0055] Embodiment 1:

[0056] Four points of the rectangular area of the section are collected on the video, and are respectively denoted as A, B, C, and D in the order of upper left, upper right, lower left, and lower right. The projection transformation matrices between the polygon ABCD and the rectangle A'B'C'D' are calculated as M1 and M2 respectively;

[0057] The millimeter-wave radar data at two positions, the near point and the far point, are collected in each lane, and the pixel coordinates of the center points of each lane traffic target at the near point and the far point on the video are collected;

[0058] First, the pixel coordinates of the car are converted into projection coordinates through projection transformation, and the scaling matrix T1 for converting the projection coordinates into millimeter-wave radar coordinates and the scaling matrix T2 for converting the millimeter-wave radar coordinates into projection coordinates are calculated respectively by the least squares method;

[0059] The process of converting the pixel coordinates of the target center point into millimeter-wave radar coordinates: First, the pixel coordinates are converted into projection coordinates through the projection transformation M1, and the converted millimeter-wave radar coordinates = projection coordinates * scaling matrix T1;

[0060] The process of converting the millimeter-wave radar coordinates into the pixel coordinates of the target center point: First, calculate the projection coordinates = millimeter-wave radar coordinates * scaling matrix T2, and the projection coordinates are converted into pixel coordinates through the projection transformation matrix M2.

[0061] A millimeter-wave radar and video joint calibration system, the system includes:

[0062] An acquisition module, configured to acquire a video stream to be jointly calibrated;

[0063] A collection module, configured to collect four corner points of the rectangular area of the section to be jointly calibrated in the video stream, the millimeter-wave radar data of each lane traffic target at the near point and the far point, and the pixel coordinates of the center points of each lane traffic target at the near point and the far point;

[0064] A first calculation module, configured to calculate a projection transformation matrix according to the four corner points of the rectangular area of the section;

[0065] A second calculation module, configured to calculate a scaling matrix based on millimeter-wave radar data of each lane traffic target at the near point and the far point, and pixel coordinates of the center point of each lane traffic target at the near point and the far point;

[0066] A joint calibration module, configured to perform joint calibration of millimeter-wave radar data and video data based on a projection transformation matrix and a scaling matrix.

[0067] In this embodiment, by using the four corner points of the rectangular area of the video stream acquisition section to be jointly calibrated, millimeter-wave radar data of each lane traffic target at the near point and the far point, and pixel coordinates of the center point of each lane traffic target at the near point and the far point, and calculating the projection transformation matrix and the scaling matrix, the joint calibration of millimeter-wave radar data and video data can be effectively achieved. At the same time, the calculation steps are simple, which can greatly improve the processing efficiency.

[0068] It should be noted that the four corner points of the rectangular area of the section are respectively the upper left corner point, the upper right corner point, the lower left corner point, and the lower right corner point of the video stream to be jointly calibrated. In addition, the projection transformation matrix is used to enable mutual conversion between pixel coordinates and projection coordinates, and the scaling matrix is used to enable mutual conversion between millimeter-wave radar coordinates and projection coordinates, so as to indirectly realize the mutual conversion between pixel coordinates and millimeter-wave radar coordinates, and further realize the joint of millimeter-wave radar data and video data.

[0069] Preferably, the first calculation module includes:

[0070] A picture size determination module, configured to determine the picture size of the rectangular area of the section based on the four corner points of the rectangular area of the section;

[0071] A first comparison module, configured to compare the picture size of the video stream to be jointly calibrated with the picture size of the rectangular area of the section to obtain a projection transformation matrix.

[0072] In this embodiment, by first determining the picture size of the rectangular area of the section, and then performing matching projection on the picture size of the video stream to be jointly calibrated according to the picture size of the rectangular area of the section, the projection transformation matrix can be accurately obtained, and the accuracy of obtaining the projection transformation matrix can be improved.

[0073] Preferably, the second calculation module includes:

[0074] A transformation module, configured to perform projection transformation on the pixel coordinates of the center point of each lane traffic target at the near point and the far point to obtain the projection coordinates of the center point of each lane traffic target at the near point and the far point;

[0075] A second comparison module, configured to compare the projection coordinates of the center points of traffic targets in each lane at the near point and the far point with the millimeter-wave radar data of the traffic targets in each lane at the near point and the far point, so as to obtain a scaling matrix.

[0076] In this embodiment, the pixel coordinates of the center points of traffic targets in each lane at the near point and the far point are subjected to projection transformation through a projection transformation matrix to obtain corresponding projection coordinates, and then the projection coordinates are compared with the corresponding millimeter-wave radar data, so that the scaling matrix can be accurately obtained, and the accuracy of obtaining the scaling matrix is improved.

[0077] Preferably, the algorithm adopted by the second comparison module is the least squares method.

[0078] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any of the above millimeter-wave radar and video joint calibration methods are implemented.

[0079] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for joint calibration of millimeter-wave radar and video, characterized in that, Specifically, it includes: Obtain the video stream to be jointly calibrated; Collect the millimeter-wave radar data at the four corner points of the rectangular area of the video stream acquisition section to be jointly calibrated, the millimeter-wave radar data of each lane traffic target at the near point and the far point, and the pixel coordinates of the center point of each lane traffic target at the near point and the far point; Calculate the projection transformation matrix based on the four corner points of the rectangular area of the section; Calculate the scaling matrix based on the millimeter-wave radar data of each lane traffic target at the near point and the far point, and the pixel coordinates of the center point of each lane traffic target at the near point and the far point; Perform the joint of the millimeter-wave radar data and the video data based on the projection transformation matrix and the scaling matrix, so as to realize the joint calibration of the millimeter-wave radar and the video; Among them, the calculation of the scaling matrix based on the millimeter-wave radar data of each lane traffic target at the near point and the far point, and the pixel coordinates of the center point of each lane traffic target at the near point and the far point specifically includes: Perform projection transformation on the pixel coordinates of the center point of each lane traffic target at the near point and the far point to obtain the projection coordinates of the center point of each lane traffic target at the near point and the far point; Compare the projection coordinates of the center point of each lane traffic target at the near point and the far point with the millimeter-wave radar data of each lane traffic target at the near point and the far point to obtain the scaling matrix.

2. The millimeter-wave radar and video joint calibration method according to claim 1, wherein The calculation of the projection transformation matrix based on the four corner points of the rectangular area of the section specifically includes: Determine the picture size of the rectangular area of the section based on the four corner points of the rectangular area of the section; Compare the picture size of the video stream to be jointly calibrated with the picture size of the rectangular area of the section to obtain the projection transformation matrix.

3. A millimeter-wave radar and video joint calibration method according to claim 2, characterized in that, The algorithm used for comparing the projection coordinates of the center point of each lane traffic target at the near point and the far point with the millimeter-wave radar data of each lane traffic target at the near point and the far point is the least squares method.

4. A millimeter-wave radar and video joint calibration system, characterized in that, It includes: An acquisition module for obtaining the video stream to be jointly calibrated; A collection module for collecting the four corner points of the rectangular area of the video stream acquisition section to be jointly calibrated, the millimeter-wave radar data of each lane traffic target at the near point and the far point, and the pixel coordinates of the center point of each lane traffic target at the near point and the far point; A first calculation module for calculating the projection transformation matrix based on the four corner points of the rectangular area of the section; A second calculation module for calculating the scaling matrix based on the millimeter-wave radar data of each lane traffic target at the near point and the far point, and the pixel coordinates of the center point of each lane traffic target at the near point and the far point; A joint calibration module for jointly calibrating the millimeter-wave radar data and the video data based on the projection transformation matrix and the scaling matrix; Among them, the second calculation module includes: A transformation module for performing projection transformation on the pixel coordinates of the center point of each lane traffic target at the near point and the far point to obtain the projection coordinates of the center point of each lane traffic target at the near point and the far point; A second comparison module for comparing the projection coordinates of the center point of each lane traffic target at the near point and the far point with the millimeter-wave radar data of each lane traffic target at the near point and the far point to obtain the scaling matrix.

5. A millimeter-wave radar and video joint calibration system according to claim 4, characterized in that, The first calculation module includes: A screen size determination module, configured to determine the screen size of the road section rectangular area according to the four corner points of the road section rectangular area; A first comparison module, configured to compare the screen size of the video stream to be jointly calibrated with the screen size of the road section rectangular area to obtain a projection transformation matrix.

6. The millimeter-wave radar and video joint calibration system according to claim 5, characterized in that, The algorithm adopted by the second comparison module is the least squares method.

7. A terminal device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any one of the millimeter-wave radar and video joint calibration methods in claims 1-3 are implemented.

Images