Road-side multi-sensor joint calibration method based on high-precision map

Abstract

The present invention discloses a road-side multi-sensor joint calibration method based on high-precision maps, which relates to the field of sensor calibration technology. The present invention includes step S000: obtaining high-precision map information of the road within the sensor's field of view, and obtaining the radar corner reflector required for calibrating the millimeter-wave radar; step S001: finding the perspective transformation relationship between the original image feature points under the oblique field of view and the feature points in the bird's-eye view according to the principle of perspective transformation, and calibrating the camera sensor; step S002: finding the matrix transformation relationship between the three-dimensional coordinate points in the original coordinate system of the laser radar and the three-dimensional coordinate points in the bird's-eye view coordinate system according to the matching relationship of the feature points, and calibrating the laser radar sensor; step S003: setting up radar corner reflectors at the feature points, and calibrating the millimeter-wave radar sensor according to the coordinate transformation of the feature points. The present invention completes the joint calibration of the three sensors of laser radar, millimeter-wave radar, and camera through an extremely easy-to-operate method.

Description

Technical Field

[0001] The present invention belongs to the field of sensor calibration technology, and in particular relates to a road-side multi-sensor joint calibration method based on high-precision maps. Background Art

[0002] When using ranging sensors and image sensors for data fusion processing, a unified coordinate system is required. Therefore, the two sensors must be spatially self-calibrated before data fusion processing.

[0003] For example, Chinese patent CN113759364A, published on December 7, 2021, discloses a method and device for continuous positioning of a millimeter-wave radar based on a laser map. The main method is to obtain a priori two-dimensional laser map of the robot's working environment, obtain the current two-dimensional millimeter-wave radar measurement data of the mobile robot, extract the priori laser map and the current millimeter-wave radar features according to the neural network, and match them to complete the positioning of the robot. For example, Chinese patent CN107153186A, published on September 12, 2017, discloses a laser radar calibration method and laser radar. The main method is to move the laser radar and measure the relative displacement of the laser radar between the current sampling moment and the previous sampling moment; obtain the original point cloud data of the laser radar at the two moments, generate relative displacement data based on the difference in the original point cloud, compare the two parts of data, and adjust the parameter value of the laser radar. For example, Chinese patent CN105758426A, published on July 13, 2016, discloses a multi-sensor joint calibration method for mobile robots. The main idea is to rigidly connect the lidar and the camera. The feature points obtained by the lidar in the world coordinate system correspond one-to-one with the feature points obtained by the camera in the pixel coordinate system, and the rotation and translation matrices are calibrated.

[0004] However, when calibrating the position, laser / radar / sensor / camera and external measurement equipment are required to perform joint calibration of the position, and the calibration method is complicated. Summary of the Invention

[0005] The purpose of the present invention is to provide a road-side multi-sensor joint calibration method based on high-precision maps, which completes the joint calibration of three sensors: lidar, millimeter-wave radar, and camera through an extremely easy-to-operate method, solving the existing problems in the background technology.

[0006] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0007] The present invention provides a road-side multi-sensor joint calibration method based on high-precision maps, comprising the following steps:

[0008] Step S000: Obtain high-precision map information of the road within the sensor's field of view and obtain the radar corner reflector required for calibrating the millimeter-wave radar;

[0009] Step S001: Find the perspective transformation relationship between the feature points of the original image under the oblique field of view and the feature points in the bird's-eye view according to the principle of perspective transformation, and calibrate the camera sensor;

[0010] Step S002: Find the matrix transformation relationship between the three-dimensional coordinate points in the original coordinate system of the lidar and the three-dimensional coordinate points in the bird's-eye view coordinate system based on the matching relationship of the feature points, and calibrate the lidar sensor;

[0011] Step S003: Setting up a radar corner reflector at the feature point, and calibrating the millimeter-wave radar sensor according to the coordinate transformation of the feature point.

[0012] Furthermore, the perspective transformation matrix is obtained by matching multiple points, and then the matrix transformation relationship between the image pixel coordinate system and the bird's-eye view coordinate system established based on the high-precision map is established;

[0013] The perspective transformation matrix is:

[0014] The perspective transformation relationship is:

[0015]

[0016] The coordinates x and y after perspective transformation are: x = x′ / w′, y = y′ / w′;

[0017] Where: u and v are the coordinates of the feature points in the image pixel coordinate system, and x and y are the coordinates of the corresponding feature points in the bird's-eye view coordinate system established based on the high-precision map.

[0018] Furthermore, the coordinate system is established with the center point of the laser radar mirror as the origin. Let the three-dimensional coordinate point of the feature point in the original coordinate system be A in the three-dimensional space. S The three-dimensional coordinate point of the corresponding feature point in the bird's-eye view coordinate system established based on the high-precision map is A T ,but:

[0019] A T =R·A S +T;

[0020] Where R is the rotation matrix and T is the translation matrix. The angles of rotation around the three coordinate axes (x, y, z) required to complete the coordinate transformation are (α, β, θ), and the displacements generated along the three coordinate axes in space are (a, b, c). Then:

[0021]

[0022]

[0023] Furthermore, the selected feature points are the high-intensity reflection points of the millimeter-wave radar sensor. The high-intensity reflection points in the polar coordinate system are converted to the rectangular coordinate system, and the relationship is:

[0024] x=rcosθ,y=rsinθ;

[0025] After completing the coordinate conversion, a rectangular coordinate system is established with the center point of the millimeter-wave radar sensor surface as the origin. Then, the lidar coordinate point conversion calculation method is introduced to establish the matching relationship of multiple feature points. The matrix transformation relationship between the original coordinate system of the millimeter-wave radar sensor and the bird's-eye view coordinate system established based on the high-precision map can be obtained.

[0026] Furthermore, after the camera sensor, lidar sensor, and millimeter-wave radar sensor are installed, the original image data, original point cloud data, and high-intensity reflection point position data in the original polar coordinate system are collected respectively, and high-precision map information of the roads within the field of view of the camera sensor, lidar sensor, and millimeter-wave radar sensor is obtained before calibration.

[0027] Furthermore, when calibrating the camera sensor, the number of feature points selected is at least 8.

[0028] Furthermore, when calibrating the laser radar sensor, the number of feature points selected is greater than 6.

[0029] Furthermore, when calibrating the millimeter-wave radar sensor, the number of selected feature points is greater than 6.

[0030] Furthermore, the number of feature points selected is determined in the following manner:

[0031] Step 1: First, set a basic number of feature points, which is calibrated as the basic number; the basic number is preset by the management personnel.

[0032] Step 2: Then increase the base number by a fixed increment. After each increment, re-mark the sum as the incremented number. Continue increasing until you get fifteen sets of incremented numbers including the base number. The fixed increment is preset by the management staff.

[0033] Step 3: Get the calibration error. The calibration error is the absolute value of the difference between the map positioning of the corresponding device after calibration and the actual measured standard map.

[0034] Step 4: Obtain the calibration differences of thirty groups of incremented numbers, and mark the calibration differences as Bi, where i=1...15;

[0035] Step 5: Automatically obtain the cost of each device corresponding to the set increment value, which is the sum of the basic price and the monthly maintenance price; obtain 30 cost values Ci, i = 1...15; then calculate the policy value Pi according to the formula, the specific calculation formula is:

[0036] Pi=0.54*Bi+0.46 / Ci;

[0037] In the formula, 0.54 and 0.46 are preset weights used to highlight the different importance of different factors;

[0038] Step 6: Sort the increasing numbers in descending order of Pi value, select the top two increasing numbers, obtain all the values between them, and mark the obtained value interval as the target segment;

[0039] Step 7: According to the principles of steps 4 to 5, all corresponding strategy values in the target segment are obtained, and the corresponding increment number with the largest strategy value is marked as the number of feature points;

[0040] Step 8: Get the number of selected feature points.

[0041] The present invention has the following beneficial effects:

[0042] The present invention completes the joint calibration of three sensors: lidar, millimeter-wave radar, and camera through an extremely easy-to-operate method. After the three sensors are calibrated, they will be unified in the bird's-eye view world coordinate system established by the high-precision map. The calibration results can be qualitatively observed by the human eye or quantitatively calculated for errors, which is beneficial to improving the accuracy of the joint calibration results and is of great significance for the fusion of multi-sensor recognition results.

[0043] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 This is a multi-sensor calibration flow chart of the present invention;

[0046] Figure 2 Matching diagram between oblique perspective image and high-precision map;

[0047] Figure 3 This is the matching diagram between the oblique-view LiDAR point cloud image and the high-precision map;

[0048] Figure 4 It is the matching diagram of the top-view coordinates of the millimeter-wave radar strong reflection points and the high-precision map;

[0049] Figure 5 This is the radar corner reflector (triangular cone) diagram;

[0050] Figure 6 This is the principle diagram of perspective transformation;

[0051] Figure 7 This is the conversion diagram from polar coordinates to rectangular coordinates. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0053] In the description of the present invention, it should be understood that terms such as "bird's eye view", "up", "down", "tilt", "perspective transformation" and the like indicating orientation or positional relationship are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the components or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore should not be understood as limiting the present invention.

[0054] See also Figure 1-7 As shown, the present invention is a road-side multi-sensor joint calibration method based on high-precision maps, comprising the following steps:

[0055] Step S000: Obtain high-precision map information of the road within the sensor's field of view, and obtain the radar corner reflector (triangular cone) required for calibrating the millimeter-wave radar. Figure 5 As shown;

[0056] Step S001: Find the perspective transformation relationship between the feature points of the original image under the tilted field of view and the feature points in the bird's eye view according to the principle of perspective transformation, and calibrate the camera sensor. The perspective transformation diagram is as follows: Figure 6 As shown; further, the perspective transformation matrix is obtained by matching multiple points, and then the matrix transformation relationship between the image pixel coordinate system and the bird's-eye view coordinate system established based on the high-precision map is established;

[0057] The perspective transformation relationship is:

[0058]

[0059] After expansion, it becomes:

[0060]

[0061]

[0062] The coordinates x and y after perspective transformation are: x = x′ / w′, y = y′ / w′;

[0063] Where u and v are the coordinates of the feature points in the image pixel coordinate system, and x and y are the coordinates of the corresponding feature points in the bird's-eye view coordinate system based on the high-precision map. The perspective transformation matrix can be obtained by matching multiple points:

[0064] The perspective transformation matrix is: Then, a matrix transformation relationship is established between the image pixel coordinate system and the bird's-eye view world coordinate system based on the high-precision map;

[0065] Step S002: Find the matrix transformation relationship between the three-dimensional coordinate points in the original coordinate system of the laser radar and the three-dimensional coordinate points in the bird's-eye view coordinate system based on the matching relationship of the feature points, and calibrate the laser radar sensor, that is, the matrix transformation relationship between the coordinate system established with the center point of the laser radar mirror as the origin and the world coordinate system of the bird's-eye view established based on the high-precision map; the coordinate system established with the center point of the laser radar mirror as the origin, let the three-dimensional coordinate point of the feature point in the original coordinate system be A S The three-dimensional coordinate point of the corresponding feature point in the bird's-eye view coordinate system established based on the high-precision map is A T , the relationship between the two is as follows:

[0066] A T =R·A S +T;

[0067] Where R is the rotation matrix and T is the translation matrix. The angles of rotation around the three coordinate axes (x, y, z) required to complete the coordinate transformation are (α, β, θ), and the displacements generated along the three coordinate axes in space are (a, b, c). Then:

[0068]

[0069]

[0070] By matching multiple feature points, the translation and rotation matrices can be obtained to complete the labeling.

[0071] Step S003: Set up a radar corner reflector at the feature point, and calibrate the millimeter-wave radar sensor based on the coordinate transformation of the feature point. Specifically, set up a radar corner reflector (triangular pyramid) at the feature point. Due to the effect of the radar corner reflector, the selected feature point will become a high-intensity reflection point of the millimeter-wave radar.

[0072] Furthermore, the selected feature points are the high-intensity reflection points of the millimeter-wave radar sensor. The high-intensity reflection points in the polar coordinate system are converted to the rectangular coordinate system, and the relationship is:

[0073] x=rcosθ,y=rsinθ;

[0074] After completing the coordinate conversion, a rectangular coordinate system is established with the center point of the millimeter-wave radar sensor surface as the origin. Then, the lidar coordinate point conversion calculation method is introduced to establish the matching relationship of multiple feature points. The matrix transformation relationship between the original coordinate system of the millimeter-wave radar sensor and the bird's-eye view coordinate system established based on the high-precision map can be obtained.

[0075] Furthermore, after the camera sensor, lidar sensor, and millimeter-wave radar sensor are installed, they collect raw image data, raw point cloud data, and high-intensity reflection point position data in the original polar coordinate system, and obtain high-precision map information of the roads within the field of view of the camera sensor, lidar sensor, and millimeter-wave radar sensor before calibration. After the camera sensor, lidar sensor, and millimeter-wave radar sensor are installed, they collect raw data, the camera obtains raw image data, the lidar obtains raw point cloud data, and the millimeter-wave radar obtains high-intensity reflection point position data in the original polar coordinate system.

[0076] Furthermore, when calibrating the camera sensor, the number of feature points selected is at least 8.

[0077] Furthermore, when calibrating the laser radar sensor, the number of feature points selected is greater than 6.

[0078] Furthermore, when calibrating the millimeter-wave radar sensor, the number of selected feature points is greater than 6.

[0079] Furthermore, the number of feature points selected is determined in the following manner:

[0080] Step 1: First, set a basic number of feature points, which is calibrated as the basic number. The basic number is preset by the management personnel.

[0081] Step 2: Then increase the base number by a fixed increment. After each increment, re-mark the sum as the incremented number. Continue increasing until you get fifteen sets of incremented numbers including the base number. The fixed increment is preset by the management staff.

[0082] Step 3: Get the calibration error. The calibration error is the absolute value of the difference between the map positioning of the corresponding device after calibration and the actual measured standard map.

[0083] Step 4: Obtain the calibration differences of thirty groups of incremented numbers, and mark the calibration differences as Bi, where i=1...15;

[0084] Step 5: Automatically obtain the cost of each device corresponding to the set increment value, which is the sum of the basic price and the monthly maintenance price; obtain 30 cost values Ci, i = 1...15; then calculate the policy value Pi according to the formula, the specific calculation formula is:

[0085] Pi=0.54*Bi+0.46 / Ci;

[0086] In the formula, 0.54 and 0.46 are preset weights used to highlight the different importance of different factors;

[0087] Step 6: Sort the increasing numbers in descending order of Pi value, select the top two increasing numbers, obtain all the values between them, and mark the obtained value interval as the target segment;

[0088] Step 7: According to the principles of steps 4 to 5, all corresponding strategy values in the target segment are obtained, and the corresponding increment number with the largest strategy value is marked as the number of feature points;

[0089] Step 8: Get the number of selected feature points.

[0090] The high-precision map-based road-side multi-sensor joint calibration method completes the joint calibration of three sensors: lidar, millimeter-wave radar, and camera in an extremely easy-to-operate method. After the three sensors are calibrated, they will be unified in the top-down world coordinate system established by the high-precision map. The calibration results can be qualitatively observed by the human eye or quantitatively calculated for errors, which is conducive to improving the accuracy of the joint calibration results and is of great significance for the fusion of multi-sensor recognition results.

[0091] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0092] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A road-side multi-sensor joint calibration method based on high-precision maps, characterized by: The following steps are involved: Step S000: Obtain high-precision map information of the road within the sensor's field of view and obtain the radar corner reflector required for calibrating the millimeter-wave radar; Step S001: Find the perspective transformation relationship between the feature points of the original image under the oblique field of view and the feature points in the bird's-eye view according to the principle of perspective transformation, and calibrate the camera sensor; Step S002: Find the matrix transformation relationship between the three-dimensional coordinate points in the original coordinate system of the lidar and the three-dimensional coordinate points in the bird's-eye view coordinate system based on the matching relationship of the feature points, and calibrate the lidar sensor; Step S003: Setting up a radar corner reflector at the feature point, and calibrating the millimeter wave radar sensor according to the coordinate transformation of the feature point; The coordinate system is established with the center point of the laser radar mirror as the origin. Let the three-dimensional coordinate point of the feature point in the original coordinate system be A in the three-dimensional space. S The three-dimensional coordinate point of the corresponding feature point in the bird's-eye view coordinate system established based on the high-precision map is A T ,but: A T =R·A S +T; Where R is the rotation matrix and T is the translation matrix. The angles of rotation around the three coordinate axes (x, y, z) required to complete the coordinate transformation are (α, β, θ), and the displacements generated along the three coordinate axes in space are (a, b, c). Then: The selected feature points are the high-intensity reflection points of the millimeter-wave radar sensor. The high-intensity reflection points in the polar coordinate system are converted to the rectangular coordinate system. The relationship is: x=rcosθ,y=rsinθ; After completing the coordinate conversion, a rectangular coordinate system is established with the center point of the millimeter-wave radar sensor surface as the origin. Then, the lidar coordinate point conversion calculation method is introduced to establish the matching relationship of multiple feature points. The matrix transformation relationship between the original coordinate system of the millimeter-wave radar sensor and the bird's-eye view coordinate system established based on the high-precision map can be obtained.

2. The road-side multi-sensor joint calibration method based on high-precision maps according to claim 1 is characterized in that: The perspective transformation matrix is obtained by matching multiple points, and then the matrix transformation relationship between the image pixel coordinate system and the bird's-eye view coordinate system based on the high-precision map is established; The perspective transformation matrix is: The perspective transformation relationship is: The coordinates x and y after perspective transformation are: x = x′ / w′, y = y′ / w′; Where: u and v are the coordinates of the feature points in the image pixel coordinate system, and x and y are the coordinates of the corresponding feature points in the bird's-eye view coordinate system established based on the high-precision map.

3. The road-side multi-sensor joint calibration method based on high-precision maps according to claim 1 is characterized in that: After the camera sensor, lidar sensor, and millimeter-wave radar sensor are installed, original image data, original point cloud data, and high-intensity reflection point position data in the original polar coordinate system are collected respectively, and high-precision map information of the roads within the field of view of the camera sensor, lidar sensor, and millimeter-wave radar sensor is obtained before calibration.

4. The road-side multi-sensor joint calibration method based on high-precision maps according to claim 2 is characterized in that: When calibrating the camera sensor, the number of feature points selected is at least 8.

5. The road-side multi-sensor joint calibration method based on high-precision maps according to claim 1 is characterized in that: When calibrating the laser radar sensor, the number of feature points selected is greater than 6.

6. The road-side multi-sensor joint calibration method based on high-precision maps according to claim 1 is characterized in that: When calibrating the millimeter-wave radar sensor, the number of selected feature points is greater than 6.

7. The method for road-side multi-sensor joint calibration based on high-precision maps according to any one of claims 4 to 6, characterized in that: The number of feature points selected is determined by the following method: Step 1: First, set a basic number of feature points, which is calibrated as the basic number; the basic number is preset by the management personnel. Step 2: Then increase the base number by a fixed increment. After each increment, re-mark the sum as the incremented number. Continue increasing until you get fifteen sets of incremented numbers including the base number. The fixed increment is preset by the management staff. Step 3: Get the calibration error. The calibration error is the absolute value of the difference between the map positioning of the corresponding device after calibration and the actual measured standard map. Step 4: Obtain the calibration differences of thirty groups of incremented numbers, and mark the calibration differences as Bi, where i=1...15; Step 5: Automatically obtain the cost of each device corresponding to the set increment value, which is the sum of the basic price and the monthly maintenance price; obtain 30 cost values Ci, i = 1...15; then calculate the policy value Pi according to the formula, the specific calculation formula is: Pi=0.54*Bi+0.46 / Ci; In the formula, 0.54 and 0.46 are preset weights used to highlight the different importance of different factors; Step 6: Sort the increasing numbers in descending order of Pi value, select the top two increasing numbers, obtain all the values between them, and mark the obtained value interval as the target segment; Step 7: According to the principles of steps 4 to 5, all corresponding strategy values in the target segment are obtained, and the corresponding increment number with the largest strategy value is marked as the number of feature points; Step 8: Get the number of selected feature points.

Images