Multi-sensor fusion calibration method and device based on calibration board and storage medium

Abstract

The invention provides a multi-sensor fusion calibration method and device based on a calibration board and a storage medium. The method comprises the following steps: acquiring a calibration plate image and a radar point cloud data set; obtaining a first calibration plate frame according to the calibration plate image and the physical size of the calibration plate; extracting a target point cloud data set of the calibration plate from the radar point cloud data set, and performing plane fitting on the target point cloud data set to obtain a calibration plate plane; projecting the radar point cloud data set to a calibration plate plane, and determining an initial calibration plate frame of the calibration plate on the calibration plate plane according to the physical size of the calibration plate; obtaining a first candidate rotation angle, and determining a target rotation angle and a target displacement; obtaining a second calibration plate frame according to the target rotation angle, the target displacement and the initial calibration plate frame; and an external parameter matrix between the visual sensor and the radar is adjusted through an iterative optimization algorithm, so that the projection residual error between the second calibration plate frame and the first calibration plate frame is minimum. According to the invention, the accuracy of external parameter calibration is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-sensor joint calibration, and particularly relates to a multi-sensor fusion calibration method based on a calibration board, a computer device, and a storage medium. Background Art

[0002] In the positioning, mapping, and navigation tasks of autonomous mobile devices, multiple sensors such as cameras and radars are often used to make up for each other's deficiencies, improve the accuracy of positioning and mapping, and the accuracy of environmental perception. Usually, before using multiple sensors such as cameras and radars, the external parameters between the radar and the camera and other multiple sensors need to be calibrated offline. At present, the external parameters between the radar and the camera and other multiple sensors are mainly calibrated based on the external parameter calibration method of the calibration board. The external parameter calibration method based on the calibration board needs to extract the point, line, or plane features of the calibration board from the image data of the calibration board collected by the camera and the point cloud data of the calibration board obtained by the radar scanning the calibration board, and then use the non-linear optimization algorithm to calibrate the external parameters between the radar and the camera using the extracted point, line, or plane features of the calibration board.

[0003] However, when using the non-linear optimization algorithm, point-point, point-line, and line-line constraints are relatively easy to converge to the global optimum. However, the point cloud data obtained by the radar scanning the calibration board is sparse, and it is difficult to extract the point and line features of the calibration board from the sparse point cloud data, and the accuracy of the point and line features cannot be guaranteed, resulting in inaccurate external parameters between the calibrated radar and the camera. Therefore, how to improve the calibration accuracy of the external parameters between the radar and the camera is an urgent problem to be solved at present. Summary of the Invention

[0004] The embodiments of the present invention provide a multi-sensor fusion calibration method based on a calibration board, a computer device, and a storage medium, aiming to improve the calibration accuracy of the external parameters between the radar and the camera.

[0005] In a first aspect, the embodiments of the present invention provide a multi-sensor fusion calibration method from radar to camera, including:

[0006] Obtain a calibration board image and a radar point cloud data set;

[0007] According to the calibration board image and the physical size of the calibration board in the calibration board image, obtain the two-dimensional coordinates of multiple vertices of the calibration board in the image coordinate system to obtain a first calibration board frame;

[0008] Extract the target point cloud data set of the calibration board from the radar point cloud data set, and perform plane fitting on the target point cloud data set to obtain a calibration board plane;

[0009] Project the radar point cloud dataset onto the calibration board plane to obtain a projected point cloud dataset, and determine an initial calibration board frame of the calibration board on the calibration board plane according to the physical size of the calibration board;

[0010] Obtain a first candidate rotation angle, and iteratively adjust the rotation angle and displacement of the initial calibration board frame in a way of reducing step size to determine a target rotation angle and a target displacement, where the first candidate rotation angle is the rotation angle corresponding to the largest number of point clouds falling within the initial calibration board frame in the projected point cloud dataset;

[0011] According to the target rotation angle, the target displacement, and the initial calibration board frame, obtain the three-dimensional coordinates of multiple vertices of the calibration board in the radar coordinate system to obtain a second calibration board frame;

[0012] Adjust the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm to minimize the projection residual between the second calibration board frame and the first calibration board frame, and obtain a target external parameter matrix.

[0013] In a second aspect, an embodiment of the present invention further provides a computer device, which includes a processor, a memory, a computer program stored on the memory and executable by the processor, and a data bus for realizing the connection and communication between the processor and the memory. When the computer program is executed by the processor, it implements the calibration method for multi-sensor fusion based on a calibration board as described in the first aspect.

[0014] In a third aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the calibration method for multi-sensor fusion based on a calibration board as described in the first aspect.

[0015] An embodiment of the present invention provides a multi-sensor fusion calibration method, a computer device, and a storage medium based on a calibration board. Based on the calibration board image and the physical size of the calibration board, the two-dimensional coordinates of multiple vertices of the calibration board in the image coordinate system are obtained to obtain a first calibration board frame, which can ensure the accuracy of extracting the point-line features of the calibration board from the calibration board image. And based on the physical size of the calibration board, the initial calibration board frame is determined, and then the first candidate rotation angle is obtained so that the number of point clouds falling within the initial calibration board frame at the first candidate rotation angle in the projected point cloud dataset is the largest, thereby realizing the rough extraction of the point-line features of the calibration board. Then, the rotation angle and displacement of the initial calibration board frame are iteratively adjusted in a decreasing step manner to determine the target rotation angle and target displacement, and further realize the fine extraction of the point-line features of the calibration board, that is, the more accurate point-line features of the calibration board are further extracted with the point-line features of the roughly extracted calibration board as the screening range, reducing the error of extracting the point-line features of the calibration board from the radar point cloud data. In this way, based on the accurately extracted first calibration board frame and the accurately extracted second calibration board frame, the external parameter matrix between the vision sensor and the radar can be accurately determined, improving the calibration accuracy of the external parameters between the radar and the camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0017] Figure 1 is a schematic flowchart of a multi-sensor fusion calibration method based on a calibration board provided by an embodiment of the present invention;

[0018] Figure 2 is an exemplary diagram of a calibration board image in an embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of a point cloud map corresponding to a target point cloud dataset in an embodiment of the present invention;

[0020] Figure 4 is Figure 1 a schematic sub-step flowchart of the multi-sensor fusion calibration method based on a calibration board in

[0021] Figure 5 is an exemplary diagram of an initial calibration board frame in an embodiment of the present invention;

[0022] Figure 6 is Figure 1 a schematic sub-step flowchart of the multi-sensor fusion calibration method based on a calibration board in

[0023] Figure 7 It is a structural schematic block diagram of a computer device provided by an embodiment of the present invention. Detailed implementation manners

[0024] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0025] The flowchart shown in the accompanying drawings is only an example illustration, and does not necessarily include all contents and operations / steps, nor does it necessarily need to be executed in the described order. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual execution order may change according to the actual situation.

[0026] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in this specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0027] Currently, the external parameters between multiple sensors such as radar and camera are mainly calibrated based on the external parameter calibration method of the calibration board. The external parameter calibration method based on the calibration board needs to extract the point, line or surface features of the calibration board from the image data of the calibration board collected by the camera and the point, line or surface features of the calibration board from the point cloud data obtained by the radar scanning the calibration board, and then use the non-linear optimization algorithm to calibrate the external parameters between the radar and the camera using the extracted point, line or surface features of the calibration board. However, when using the non-linear optimization algorithm, the point-point, point-line and line-line constraints are relatively easy to converge to the global optimum, but the point cloud data obtained by the radar scanning the calibration board is sparse, and it is difficult to extract the point and line features of the calibration board from the sparse point cloud data, and the accuracy of the point and line features cannot be guaranteed, resulting in inaccurate external parameters between the calibrated radar and the camera.

[0028] To solve the above problems, an embodiment of the present invention provides a multi-sensor fusion calibration method, a computer device, and a storage medium based on a calibration board. Based on the calibration board image and the physical size of the calibration board, the two-dimensional coordinates of multiple vertices of the calibration board in the image coordinate system are obtained to obtain a first calibration board frame, which can ensure the accuracy of extracting the point-line features of the calibration board from the calibration board image. And based on the physical size of the calibration board, the initial calibration board frame is determined, and then the first candidate rotation angle is obtained so that the number of point clouds falling within the initial calibration board frame at the first candidate rotation angle in the projected point cloud dataset is the largest, thereby realizing the rough extraction of the point-line features of the calibration board. Then, the rotation angle and displacement of the initial calibration board frame are iteratively adjusted in a way of reducing the step size to determine the target rotation angle and the target displacement, and further realizing the fine extraction of the point-line features of the calibration board, that is, taking the point-line features of the roughly extracted calibration board as the screening range to further extract more accurate point-line features of the calibration board, reducing the error of extracting the point-line features of the calibration board from the radar point cloud data. In this way, based on the accurately extracted first calibration board frame and the accurately extracted second calibration board frame, the external parameter matrix between the vision sensor and the radar can be accurately determined, improving the calibration accuracy of the external parameters between the radar and the camera.

[0029] The multi-sensor fusion calibration method based on a calibration board provided by an embodiment of the present application can be applied to a computer device, which may include a terminal device, a self-mobile device, or a server. The terminal device may include a personal computer or a laptop computer, etc. The self-mobile device may include a sweeping robot or a lawn mowing robot. The server may be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0030] The following will, with reference to the accompanying drawings, elaborate on some embodiments of the present invention. Without conflict, the following embodiments and the features in the embodiments may be combined with each other.

[0031] The following will, in conjunction with Figure 1 the scenarios in Figure 1 detail the multi-sensor fusion calibration method provided by the embodiments of the present invention. It should be noted that

[0032] Please refer to Figure 1 Figure 1 which is a schematic flowchart of a multi-sensor fusion calibration method provided by an embodiment of the present invention.

[0033] As Figure 1 shown, the multi-sensor fusion calibration method includes steps S101 to S107.

[0034] Step S101, obtaining a calibration board image and a radar point cloud dataset.

[0035] In this embodiment, the calibration board image is obtained by a vision sensor collecting a target scene provided with a calibration board, and the radar point cloud dataset is obtained by a radar scanning the target scene. Among them, the vision sensor may include one or more, and the radar may include a lidar or a millimeter-wave radar.

[0036] In some embodiments, one or more calibration boards are provided in the target scene. The calibration board may be a checkerboard calibration board, a Charuco calibration board, an AprilTag calibration board, etc. For example, the vision sensor collecting a target scene provided with an AprilTag calibration board can obtain a calibration board image as Figure 2 shown.

[0037] In some embodiments, obtaining the calibration board image and the radar point cloud dataset may include: controlling the vision sensor to capture a target scene provided with a calibration board to obtain a calibration board image; controlling the radar to scan the target scene to obtain a radar point cloud dataset. This embodiment can automatically control the vision sensor and the radar to collect corresponding data, reduce manual intervention, and improve the efficiency of the external parameters of multi-sensor fusion calibration.

[0038] Step S102, according to the calibration board image and the physical size of the calibration board, obtaining the two-dimensional coordinates of multiple vertices of the calibration board in the image coordinate system to obtain a first calibration board frame.

[0039] In this embodiment, the physical size of the calibration board refers to the actual size of the calibration board in the real world, and the physical size of the calibration board may include the length, width, diagonal length, thickness, etc. of the calibration board. By introducing the physical size of the calibration board when obtaining the first calibration board frame in this embodiment, the influence of noise and common error sources (such as illumination changes, blurring, or non-linear distortion of the calibration board image) on the extraction of the first calibration board frame can be reduced, so as to improve the accuracy and robustness of extracting the first calibration board frame from the calibration board image.

[0040] In some embodiments, obtaining the two-dimensional coordinates of multiple vertices of the calibration board in the image coordinate system based on the calibration board image and the physical size of the calibration board may include: determining the two-dimensional coordinates of multiple corner points of the calibration board in the calibration board image in the image coordinate system, and determining the three-dimensional coordinates of the multiple corner points according to the physical size of the calibration board; determining the homography matrix according to the two-dimensional coordinates of the multiple corner points and the three-dimensional coordinates of the multiple corner points; determining the three-dimensional coordinates of multiple vertices of the calibration board according to the physical size of the calibration board; and determining the two-dimensional coordinates of multiple vertices of the calibration board in the image coordinate system according to the homography matrix and the three-dimensional coordinates of the multiple vertices of the calibration board. Among them, the corner point may be a feature point corresponding to the calibration board in the calibration board image, such as a corner point on the calibration board. When there are multiple calibration boards, its corner points correspond to the corner points of multiple calibration boards. In this embodiment, by introducing the physical size of the calibration board to obtain the two-dimensional coordinates of multiple vertices of the calibration board in the image coordinate system, the influence of noise and common error sources (such as illumination changes, blurring, or non-linear distortion of the calibration board image) on the vertex extraction of the calibration board can be reduced, so as to improve the accuracy and robustness of extracting the two-dimensional coordinates of multiple vertices of the calibration board from the calibration board image.

[0041] In some embodiments, determining the two-dimensional coordinates of multiple corner points of the calibration board in the calibration board image in the image coordinate system may include: performing a distortion removal process on the calibration board image; extracting multiple corner points from the calibration board image after the distortion removal process to obtain the two-dimensional coordinates of the multiple corner points in the image coordinate system. Among them, multiple corner points may be extracted from the calibration board image after the distortion removal process based on a preset corner point extraction algorithm. For example, the preset corner point extraction algorithm may include the Harris corner detection algorithm, the Shi-Tomasi corner detection algorithm, or the Apritag algorithm, etc. In this embodiment, by performing a distortion removal process on the calibration board image and then extracting corner points, the accuracy of corner point extraction can be ensured.

[0042] In some embodiments, determining the three-dimensional coordinates of multiple corner points according to the physical size of the calibration board may include: constructing a three-dimensional coordinate system according to the physical size of the calibration board, determining the positions of multiple corner points in the calibration board in the three-dimensional coordinate system, and obtaining the three-dimensional coordinates of multiple corner points in the calibration board. Among them, the origin of the three-dimensional coordinate system may be any corner point in the calibration board. For example, the origin of the three-dimensional coordinate system is the upper left vertex of the calibration board. The X-axis in the three-dimensional coordinate system is along the horizontal direction of the calibration board (such as from left to right), the Y-axis in the three-dimensional coordinate system is along the vertical direction of the calibration board (such as from top to bottom or from bottom to top), the Z-axis in the three-dimensional coordinate system is perpendicular to the plane where the calibration board is located, and the z-coordinates of all corner points are zero. For example, the length of the calibration board is L, the width is W, and the three-dimensional coordinates of the four corner points P1, P2, P3, and P4 of the calibration board are P1(0, 0, 0), P2(L, 0, 0), P3(L, W, 0), and P4(0, W, 0) respectively, and the three-dimensional coordinate of the center point of the calibration board is (L / 2, W / 2, 0).

[0043] In some embodiments, determining the homography matrix according to the two-dimensional coordinates of multiple corner points and the three-dimensional coordinates of multiple corner points may include: pairing the two-dimensional coordinates of multiple corner points with the three-dimensional coordinates of multiple corner points according to the actual physical positions of each corner point in the calibration board to obtain multiple coordinate pairs, where the coordinate pair includes a two-dimensional coordinate and a three-dimensional coordinate; determining the homography matrix according to the multiple coordinate pairs. In this embodiment, through the actual physical positions of each corner point in the calibration board, the two-dimensional coordinates of multiple corner points can be accurately paired with the three-dimensional coordinates of multiple corner points, and then based on the multiple coordinate pairs obtained by pairing, the homography matrix can be accurately determined.

[0044] Step S103, extract the target point cloud dataset of the calibration board from the radar point cloud dataset, and perform plane fitting on the target point cloud dataset to obtain the calibration board plane.

[0045] In this embodiment, since one or more calibration boards set in the target scene are arranged above the ground, therefore, the ground points can be first removed from the radar point cloud dataset, and then the target point cloud dataset of the calibration board can be extracted from the point cloud dataset after removing the ground points, so that the target point cloud dataset of the calibration board can be extracted from the radar point cloud dataset more quickly.

[0046] In some embodiments, extracting the target point cloud dataset of the calibration board from the radar point cloud dataset may include: extracting the non-ground point cloud dataset from the radar point cloud dataset; performing Euclidean distance clustering on the non-ground point cloud dataset to obtain multiple point cloud clusters, performing plane fitting on each point cloud cluster to obtain multiple fitted planes; removing the points in the non-ground point cloud dataset that are not located in each fitted plane to obtain the target point cloud dataset of the calibration board. For example, the target point cloud dataset of the calibration board under the target scene corresponds Figure 3The point cloud diagram shown. In this embodiment, the target point cloud dataset of the calibration board can be quickly and accurately extracted from the radar point cloud dataset.

[0047] It should be noted that the plane fitting algorithm in the embodiments of the present invention may include the least squares method or the random sample consensus (RANSAC) algorithm, etc. For example, using the random sample consensus algorithm, plane fitting is performed on the target point cloud dataset to obtain the calibration board plane. Or using the random sample consensus algorithm, plane fitting is performed on each point cloud cluster to obtain multiple fitting planes. Specifically, using the random sample consensus algorithm to perform plane fitting on the target point cloud dataset to obtain the calibration board plane may include: randomly selecting at least three points from the target point cloud dataset, and performing plane fitting on the at least three points to obtain a candidate plane; determining the distance from each point in the target point cloud dataset to the candidate plane, and determining the points with a distance less than a preset distance threshold as the inliers of the candidate plane; repeating the steps of randomly selecting at least three points from the target point cloud dataset, performing plane fitting on the at least three points to obtain a candidate plane; determining the distance from each point in the target point cloud dataset to the candidate plane, and determining the points with a distance less than a preset distance threshold as the inliers of the candidate plane n times, to obtain n + 1 candidate planes and the inliers of each candidate plane, where n is an integer greater than or equal to 2; determining the candidate plane with the largest number of inliers as the calibration board plane.

[0048] In some embodiments, extracting the non-ground point cloud dataset from the radar point cloud dataset may include: randomly selecting at least three candidate ground points from the radar point cloud dataset, and performing plane fitting on the at least three candidate ground points to obtain a candidate ground plane, where the height of the candidate ground points is less than or equal to a preset height threshold; determining the distance from each point in the radar point cloud dataset to the candidate ground plane, and determining the points with a distance less than a preset distance threshold as the inliers of the candidate ground plane; repeating the steps of randomly selecting at least three candidate ground points from the radar point cloud dataset, performing plane fitting on the at least three candidate ground points to obtain a candidate ground plane and determining the distance from each point in the radar point cloud dataset to the candidate ground plane, and determining the points with a distance less than a preset distance threshold as the inliers of the candidate ground plane n times, to obtain n + 1 candidate ground planes and the inliers of each candidate ground plane, where n is an integer greater than or equal to 2; determining the candidate ground plane with the largest number of inliers as the target ground plane, and removing the points located in the target ground plane from the radar point cloud dataset to obtain the non-ground point cloud dataset. This embodiment can accurately extract the non-ground point cloud dataset from the radar point cloud dataset.

[0049] In some embodiments, randomly selecting at least three candidate ground points from a radar point cloud dataset may include: screening out points with a height less than or equal to a preset height threshold from the radar point cloud dataset to obtain a candidate ground point set; determining the number of neighboring points of each candidate ground point in the candidate ground point set, and removing candidate ground points with a number of neighboring points less than or equal to a preset number threshold from the candidate ground point set to update the candidate ground point set; randomly selecting at least three candidate ground points from the updated candidate ground point set. Wherein, the number of neighboring points of a candidate ground point refers to the number of points with a distance less than a preset neighborhood distance from the candidate ground point. The preset height threshold can be determined according to the installation height of the radar, and the preset number threshold and preset distance threshold can be set based on actual situations. The embodiments of the present invention do not make specific limitations in this regard.

[0050] In some embodiments, determining the candidate ground plane with the largest number of inliers as the target ground plane includes: determining the candidate ground plane with the largest number of inliers as the plane to be grown, and using the inliers of the plane to be grown as seed points; searching for neighborhood points of the seed points in the radar point cloud dataset, and determining the vertical distance from the neighborhood points to the plane to be grown; when there is at least one neighborhood point with a vertical distance to the plane to be grown less than a preset distance threshold, using at least one neighborhood point as new inliers of the plane to be grown to update the plane to be grown; using the new inliers as new seed points, and returning to execute the step of searching for neighborhood points of the seed points in the radar point cloud dataset and determining the vertical distance from the neighborhood points to the plane to be grown; when the vertical distances of all neighborhood points to the plane to be grown are greater than or equal to the preset distance threshold, stop updating the plane to be grown, and determine the latest plane to be grown as the target ground plane. Wherein, the neighborhood points of the seed points are points with a distance less than a preset neighborhood distance from the seed points, and the preset distance threshold can be set based on actual situations. The embodiments of the present invention do not make specific limitations in this regard. This embodiment further expands the candidate ground plane with the largest number of inliers, so as to obtain a more accurate target ground plane.

[0051] Step S104, project the radar point cloud dataset onto the calibration board plane to obtain a projected point cloud dataset, and determine the initial calibration board frame of the calibration board on the calibration board plane according to the physical size of the calibration board.

[0052] For example, both the length and width of the calibration board are 1. Taking the geometric center of the calibration board as the origin, the positive X-axis direction as the direction from left to right of the calibration board, and the positive Y-axis direction as the direction from bottom to top of the calibration board to establish a plane coordinate system, then the plane coordinates of the lower left vertex, upper left vertex, upper right vertex, and lower right vertex of the initial calibration board frame of the calibration board on the calibration board plane can be expressed as p_pv1 = [-l / 2, -l / 2] T 、p_pv2 = [l / 2, -l / 2] T, p_pv3 = [l / 2, l / 2] T and p_pv4 = [-l / 2, l / 2] T .

[0053] In some embodiments, as Figure 4 shown, step S104 may include sub-steps S1041 to S1043.

[0054] Sub-step S1041, project the radar point cloud data set onto the calibration plate plane to obtain a set of plane points.

[0055] In this embodiment, the radar point cloud data set can be projected onto the calibration plate plane along the laser beam direction according to the normal vector and distance value of the calibration plate plane to obtain a set of plane points. For example, according to the formula P_p = P * (-n T P / d), the radar point cloud data set can be projected onto the calibration plate plane to obtain a set of plane points, where P = [x, y, z] T is a radar point in the radar point cloud data set, P_p is the plane point after the radar point is projected onto the calibration plate plane, n T is the transpose of the normal vector of the calibration plate plane, and d is the distance value of the calibration plate plane.

[0056] Sub-step S1042, construct a target plane coordinate system according to the normal vector of the calibration plate plane and the center point of the set of plane points.

[0057] In this embodiment, the average value of the abscissas can be calculated according to the abscissas of each plane point in the set of plane points, and the average value of the ordinates can be calculated according to the ordinates of each plane point in the set of plane points. The plane point corresponding to the average value of the abscissas and the average value of the ordinates is determined as the center point of the set of plane points.

[0058] In some embodiments, constructing a target plane coordinate system according to the normal vector of the calibration plate plane and the center point of the set of plane points may include: determining a first unit vector perpendicular to the normal vector of the calibration plate plane, and determining a second unit vector perpendicular to the normal vector of the calibration plate plane and the first unit vector; using the center point of the set of plane points as the origin, using the first unit vector as the x-axis, and using the second unit vector as the y-axis to construct a target plane coordinate system.

[0059] Sub-step S1043, determine the plane points of each plane point in the set of plane points in the target plane coordinate system to obtain a projected point cloud data set.

[0060] In this embodiment, the radar point cloud data set is converted into plane points belonging to the calibration plate plane, so as to subsequently extract the point-line features of the calibration plate first roughly and then finely, in order to reduce the error of extracting the point-line features of the calibration plate from the radar point cloud data.

[0061] In some embodiments, determining the planar points of each planar point in the planar point set in the target planar coordinate system to obtain a projected point cloud dataset may include: determining the planar points of each planar point in the planar point set in the target planar coordinate system according to the first unit vector, the second unit vector, and the center point of the planar point set. Specifically, determining the difference between the planar points in the planar point set and the center point of the planar point set, multiplying the transpose of the first unit vector by the difference and multiplying the transpose of the second unit vector by the difference to obtain the planar points (planar coordinates) of the planar points in the planar point set in the target planar coordinate system. For example, according to the formula p_p = [n1 T (P_p - P_o), n2 T (P_p - P_o)] T the planar points of each planar point in the planar point set in the target planar coordinate system can be determined. p_p is the planar point of the planar point in the planar point set in the target planar coordinate system, P_p is the planar point in the planar point set, n1 T is the transpose of the first unit vector, P_o is the center point of the planar point set, and n2 T is the transpose of the second unit vector.

[0062] Step S105, obtain a first candidate rotation angle, and iteratively adjust the rotation angle and displacement of the initial calibration plate frame in a way of reducing the step size to determine the target rotation angle and the target displacement. The first candidate rotation angle is the rotation angle corresponding to the largest number of point clouds falling within the initial calibration plate frame in the projected point cloud dataset.

[0063] In this embodiment, by obtaining the first candidate rotation angle, the number of point clouds falling within the initial calibration plate frame at the candidate rotation angle is maximized, thereby realizing the rough extraction of the point-line features of the calibration plate. Then, the rotation angle and displacement of the initial calibration plate frame are iteratively adjusted in a way of reducing the step size to determine the target rotation angle and the target displacement, thereby realizing the fine extraction of the point-line features of the calibration plate.

[0064] In some embodiments, the accuracy score corresponding to the target rotation angle and the target displacement is the highest. The accuracy score corresponding to the target rotation angle and the target displacement is related to the number of point clouds falling within the initial calibration plate frame at the target rotation angle and the target displacement and the number of target points. The target points are located outside the calibration area frame in the initial calibration plate frame at the target rotation angle and the target displacement, and the reflection intensity of the target points is greater than a preset threshold. For example, as Figure 5 shown, the initial calibration plate frame 10 includes 4 calibration area frames 11, Figure 4The white area 12 between the four calibration area frames 11 in [the object] is the area outside the calibration area frames 11 within the initial calibration plate frame 10. Among them, the preset threshold can be set based on the actual situation, and the embodiments of the present invention do not make specific limitations thereto. In this embodiment, by introducing the reflection intensity of the point cloud when extracting the point-line features of the calibration plate, the point-line features of the calibration plate can be extracted more accurately.

[0065] In some embodiments, the accuracy scores corresponding to the target rotation angle and the target displacement are positively correlated with the number of point clouds in the projection point cloud dataset that fall within the initial calibration plate frame at the target rotation angle and the target displacement, and the accuracy scores corresponding to the target rotation angle and the target displacement are positively correlated with the number of target points. That is, the more the number of point clouds in the projection point cloud dataset that fall within the initial calibration plate frame at the target rotation angle and the target displacement, the higher the accuracy scores corresponding to the target rotation angle and the target displacement; the fewer the number of point clouds in the projection point cloud dataset that fall within the initial calibration plate frame at the target rotation angle and the target displacement, the lower the accuracy scores corresponding to the target rotation angle and the target displacement; the more the number of target points, the higher the accuracy scores corresponding to the target rotation angle and the target displacement; the fewer the number of target points, the lower the accuracy scores corresponding to the target rotation angle and the target displacement.

[0066] It should be noted that the trigger conditions for stopping the iterative adjustment of the rotation angle and displacement of the initial calibration plate frame may include that the reduced step size is less than or equal to the preset step size, and the trigger conditions may also include that the number of iterations for adjusting the rotation angle and displacement of the initial calibration plate frame reaches the set number of iterations, etc. Among them, the number of iterations for adjusting the rotation angle and displacement of the initial calibration plate frame can be set based on the actual situation, and the embodiments of the present invention do not make specific limitations thereto. For example, the number of iterations for adjusting the rotation angle and displacement of the initial calibration plate frame includes 2 times or 3 times, etc., and the accuracy scores corresponding to the rotation angle and displacement determined in each iteration are the highest or the number of point clouds in the projection point cloud dataset that fall within the initial calibration plate frame at the rotation angle and displacement determined in each iteration is the largest. The following takes the number of iterations for adjusting the rotation angle and displacement of the initial calibration plate frame being 2 times as an example for explanation.

[0067] In some embodiments, the step size used to currently adjust the rotation angle and displacement of the initial calibration plate frame is smaller than the step size used to adjust the rotation angle and displacement of the initial calibration plate frame in the previous time. Wherein, each iterative adjustment of the rotation angle and displacement of the initial calibration plate frame starts from the rotation angle and displacement obtained in the previous iteration. Moreover, the accuracy score corresponding to the rotation angle and displacement determined in each iteration is the highest, or the number of point clouds in the projection point cloud dataset that fall within the initial calibration plate frame at the rotation angle and displacement determined in each iteration is the largest. The number of iterations for adjusting the rotation angle and displacement of the initial calibration plate frame can be set according to the actual situation, and the embodiments of the present invention do not make specific limitations thereon. For example, the number of iterations for adjusting the rotation angle and displacement of the initial calibration plate frame includes 2 times, 3 times, etc. Taking the number of iterations for adjusting the rotation angle and displacement of the initial calibration plate frame as 2 times as an example below, the iterative adjustment of the rotation angle and displacement of the initial calibration plate frame in a way of reducing the step size is explained to determine the target rotation angle and the target displacement.

[0068] In some embodiments, the iterative adjustment of the rotation angle and displacement of the initial calibration plate frame in a way of reducing the step size to determine the target rotation angle and the target displacement may include: starting from the first candidate rotation angle, adjusting the rotation angle of the initial calibration plate frame with the first preset rotation step size within the preset angle range and adjusting the displacement of the initial calibration plate frame with the first preset displacement step size within the preset displacement range to determine the second candidate rotation angle and the candidate displacement, wherein the number of point clouds in the projection point cloud dataset that fall within the initial calibration plate frame at the second candidate rotation angle and the candidate displacement is the largest; starting from the second candidate rotation angle and the candidate displacement, adjusting the rotation angle of the initial calibration plate frame with the second preset rotation step size within the preset angle range and adjusting the displacement of the initial calibration plate frame with the second preset displacement step size within the preset displacement range to determine the target rotation angle and the target displacement, wherein the accuracy score corresponding to the target rotation angle and the target displacement is the highest, the accuracy score corresponding to the target rotation angle and the target displacement is related to the number of point clouds in the projection point cloud dataset that fall within the initial calibration plate frame at the target rotation angle and the target displacement and the number of target points, the target points are located outside the calibration area frame in the initial calibration plate frame at the target rotation angle and the target displacement, and the reflection intensity of the target points is greater than the preset threshold, the second preset rotation step size is smaller than the first preset rotation step size, and the second preset displacement step size is smaller than the first preset displacement step size.

[0069] It should be noted that the preset angle range, preset displacement range, first preset rotation step, first preset displacement step, second preset rotation step, and second preset displacement step can be set based on actual situations, and the embodiments of the present invention do not make specific limitations thereto. For example, the preset angle range includes [-90°, 90°] or [-90°, 85°], the preset displacement range includes [-0.2m, 0.2m], the first preset rotation step includes 1°, the second preset rotation step includes 0.1°, the first preset displacement step includes 0.01m, and the second preset displacement step includes 0.005m.

[0070] In some embodiments, as Figure 6 shown, step S105 may include sub-steps S1051 to S1052.

[0071] Sub-step S1051, adjust the rotation angle of the initial calibration plate frame within the first angle range with the first rotation step to determine the first candidate rotation angle.

[0072] In this embodiment, the first angle range and the first rotation step can be set based on actual situations, and the embodiments of the present invention do not make specific limitations thereto. For example, the first angle range includes [-90°, 90°] or [-90°, 85°], etc., and the first rotation step includes 1° or 2°, etc.

[0073] In some embodiments, adjusting the rotation angle of the initial calibration plate frame within the first angle range with the first rotation step to determine the first candidate rotation angle may include: taking the center of the initial calibration plate frame as the center of the circle, traversing the rotation angle of the initial calibration plate frame within the first angle range with the first rotation step; determining the number of point clouds in the initial calibration plate frame that fall at each rotation angle obtained by traversing in the projection point cloud dataset, and determining the rotation angle corresponding to the largest number of point clouds as the first candidate rotation angle.

[0074] Sub-step S1052, starting from the first candidate rotation angle, iteratively adjust the rotation angle and displacement of the initial calibration plate frame in a way of narrowing the range and with a narrowing step to determine the target rotation angle and target displacement.

[0075] In this embodiment, iteratively adjusting the rotation angle and displacement of the initial calibration plate frame in a way of narrowing the range and with a narrowing step not only realizes the fine replacement of the point-line features of the calibration plate, but also can reduce the computational amount of iteration and improve the efficiency of extracting the point-line features of the calibration plate.

[0076] In some embodiments, the accuracy score corresponding to the target rotation angle and the target displacement is the highest. The accuracy score corresponding to the target rotation angle and the target displacement is related to the number of point clouds in the initial calibration plate frame that fall at the target rotation angle and the target displacement in the projected point cloud dataset and the number of target points. The target points are located outside the calibration area frame in the initial calibration plate frame at the target rotation angle and the target displacement, and the reflection intensity of the target points is greater than a preset threshold.

[0077] In some embodiments, the angle range and the displacement range used to currently adjust the rotation angle and the displacement of the initial calibration plate frame are smaller than the angle range and the displacement range used to adjust the rotation angle and the displacement of the initial calibration plate frame in the previous time, and the step size used to currently adjust the rotation angle and the displacement of the initial calibration plate frame is also smaller than the step size used to adjust the rotation angle and the displacement of the initial calibration plate frame in the previous time. Each iterative adjustment of the rotation angle and the displacement of the initial calibration plate frame starts from the rotation angle and the displacement obtained in the previous iteration.

[0078] It should be noted that the triggering conditions for stopping the iterative adjustment of the rotation angle and the displacement of the initial calibration plate frame may include that the reduced range is less than or equal to a preset range, and the reduced step size is less than or equal to a preset step size. The triggering conditions may also include that the number of iterations for adjusting the rotation angle and the displacement of the initial calibration plate frame reaches a set number of iterations, etc. Among them, the number of iterations for adjusting the rotation angle and the displacement of the initial calibration plate frame can be set based on actual situations, and the embodiments of the present invention do not make specific limitations in this regard. For example, the number of iterations for adjusting the rotation angle and the displacement of the initial calibration plate frame includes 2 times or 3 times, etc., and the accuracy score corresponding to the rotation angle and the displacement determined in each iteration is the highest or the number of point clouds in the initial calibration plate frame that fall at the rotation angle and the displacement determined in each iteration in the projected point cloud dataset is the largest. The following takes the number of iterations for adjusting the rotation angle and the displacement of the initial calibration plate frame being 2 times as an example for explanation.

[0079] In some embodiments, starting from the first candidate rotation angle, iteratively adjusting the rotation angle and displacement of the initial calibration plate frame in a way that narrows the range and with a decreasing step size to determine the target rotation angle and target displacement may include: starting from the first candidate rotation angle, adjusting the rotation angle of the initial calibration plate frame within a second angular range with a second rotation step size and adjusting the displacement of the initial calibration plate frame within a first displacement range with a first displacement step size to determine a second candidate rotation angle and a candidate displacement, where the number of point clouds within the initial calibration plate frame at the second candidate rotation angle and the candidate displacement in the projected point cloud dataset is the largest, the second angular range is smaller than the first angular range, and the second rotation step size is smaller than the first rotation step size; starting from the second candidate rotation angle and the candidate displacement, adjusting the rotation angle of the initial calibration plate frame within a third angular range with a third rotation step size and adjusting the displacement of the initial calibration plate frame within a second displacement range with a second displacement step size to determine the target rotation angle and the target displacement, where the accuracy score corresponding to the target rotation angle and the target displacement is the highest, the accuracy score corresponding to the target rotation angle and the target displacement is related to the number of point clouds within the initial calibration plate frame at the target rotation angle and the target displacement and the number of target points, the target points are located outside the calibration area frame in the initial calibration plate frame at the target rotation angle and the target displacement, and the reflection intensity of the target points is greater than a preset threshold, the third angular range is smaller than the second angular range, the third rotation step size is smaller than the second rotation step size, the second displacement range is smaller than the first displacement range, and the second displacement step size is smaller than the first displacement step size.

[0080] It should be noted that the second angular range, the first displacement range, the second rotation step size, the first displacement step size, the third angular range, the third rotation step size, the second displacement range, and the second displacement step size can be set based on actual situations, and the embodiments of the present invention do not make specific limitations in this regard. For example, the second angular range includes [-5°, 5°], the first displacement range includes [-0.2 m, 0.2 m], the second rotation step size includes 0.1°, the first displacement step size includes 0.01 m, the third angular range includes [-0.5°, 0.5°], the third rotation step size includes 0.02°, the second displacement range includes [-0.02 m, 0.02 m], and the second displacement step size includes 0.005 m.

[0081] In some embodiments, starting from the first candidate rotation angle, adjusting the rotation angle of the initial calibration plate frame in the second angular range with a second rotation step and adjusting the displacement of the initial calibration plate frame in the first displacement range with a first displacement step to determine the second candidate rotation angle and the candidate displacement may include: starting from the first candidate rotation angle, with the center of the initial calibration plate frame as the center of the circle, traversing the rotation angle of the initial calibration plate frame in the second angular range with the second rotation step, and traversing the displacement of the initial calibration plate frame in the first displacement range with the first displacement step, to obtain a plurality of first transformation matrices, where one first transformation matrix includes a rotation angle and a displacement; determining the number of point clouds in the initial calibration plate frame that fall at each of the traversed first transformation matrices in the projection point cloud dataset, and determining the rotation angle in the first transformation matrix corresponding to the largest such number of point clouds as the second candidate rotation angle, and determining the displacement in the first transformation matrix corresponding to the largest such number of point clouds as the candidate displacement.

[0082] In some embodiments, starting from the first candidate rotation angle, adjusting the rotation angle of the initial calibration plate frame in the second angular range with a second rotation step and adjusting the displacement of the initial calibration plate frame in the first displacement range with a first displacement step to determine the second candidate rotation angle and the candidate displacement may include: starting from the first candidate rotation angle, with the center of the initial calibration plate frame as the center of the circle, traversing the rotation angle of the initial calibration plate frame in the second angular range with the second rotation step, and traversing the displacement of the initial calibration plate frame in the first displacement range with the first displacement step, to obtain a plurality of first transformation matrices, where one first transformation matrix includes a rotation angle and a displacement; determining the accuracy score corresponding to each of the traversed first transformation matrices, the accuracy score corresponding to the first transformation matrix being related to the number of point clouds in the initial calibration plate frame that fall at the first transformation matrix and the number of target points in the projection point cloud dataset, the target points being outside the calibration region box in the initial calibration plate frame at the first transformation matrix and the reflection intensity of the target points being greater than a preset threshold; determining the rotation angle in the first transformation matrix corresponding to the highest such accuracy score as the second candidate rotation angle, and determining the displacement in the first transformation matrix corresponding to the highest such accuracy score as the candidate displacement.

[0083] In some embodiments, starting from the second candidate rotation angle and the candidate displacement, adjusting the rotation angle of the initial calibration plate frame by a third rotation step within a third angular range and adjusting the displacement of the initial calibration plate frame by a second displacement step within a second displacement range to determine the target rotation angle and the target displacement may include: starting from the second candidate rotation angle and the candidate displacement, taking the center of the initial calibration plate frame as the center of a circle, traversing the rotation angle of the initial calibration plate frame within the third angular range by the third rotation step, and traversing the displacement of the initial calibration plate frame within the second displacement range by the second displacement step, to obtain a plurality of second transformation matrices, where one second transformation matrix includes a rotation angle and a displacement; determining the accuracy score corresponding to each second transformation matrix obtained by traversal, the accuracy score corresponding to the second transformation matrix being related to the number of point clouds in the initial calibration plate frame that fall at the second transformation matrix in the projected point cloud dataset and the number of target points, the target points being outside the calibration region box in the initial calibration plate frame at the second transformation matrix and the reflection intensity of the target points being greater than a preset threshold; determining the rotation angle in the second transformation matrix corresponding to the highest accuracy score as the target rotation angle, and determining the displacement in the second transformation matrix corresponding to the highest accuracy score as the target displacement.

[0084] For example, taking the center of the initial calibration plate frame as the center of a circle, traversing the rotation angle of the initial calibration plate frame within [-90°, 90°] by a step of 1°, determining the number of point clouds in the initial calibration plate frame that fall at each rotation angle obtained by traversal in the projected point cloud dataset, and determining the rotation angle corresponding to the largest number of point clouds as the first candidate rotation angle θ1; starting from the first candidate rotation angle θ1, taking the center of the initial calibration plate frame as the center of a circle, traversing the rotation angle of the initial calibration plate frame within [-5°, 5°] by a step of 0.1°, and traversing the displacement of the initial calibration plate frame within [-0.2 m, 0.2 m] by a step of 0.01 m, to obtain a plurality of first transformation matrices, where one first transformation matrix includes a rotation angle and a displacement; determining the number of point clouds in the initial calibration plate frame that fall at each first transformation matrix obtained by traversal in the projected point cloud dataset, and selecting the first transformation matrix corresponding to the largest number of point clouds as the rough matching result T_coarse = [θ, x, y] T ; then with T_coarse = [θ, x, y] TStarting from [[ID=]], with the center of the initial calibration board frame as the center of the circle, traverse the rotation angle of the initial calibration board frame within [-0.5°, 0.5°] with a step size of 0.02°, and traverse the displacement of the initial calibration board frame within [-0.02m, 0.02m] with a step size of 0.005m to obtain multiple second transformation matrices. A second transformation matrix includes a rotation angle and a displacement; determine the accuracy score corresponding to each second transformation matrix obtained by traversing, and use the second transformation matrix corresponding to the highest accuracy score as the final matching result T_refine = [θ, x, y] T , T_refine = [θ, x, y] T where θ in [[ID=]] is the target rotation angle, and T_refine = [θ, x, y] T and [x, y] in [[ID=]] is the target displacement.

[0085] Step S106: According to the target rotation angle, target displacement, and initial calibration board frame, obtain the three-dimensional coordinates of multiple vertices of the calibration board in the radar coordinate system to obtain the second calibration board frame.

[0086] In this embodiment, the target rotation angle and target displacement are obtained through rough matching and fine matching, which can ensure the accuracy of the target rotation angle and target displacement. In this way, through the accurate target rotation angle and target displacement and the initial calibration board frame, the three-dimensional coordinates of multiple vertices of the calibration board in the radar coordinate system can be accurately obtained, thereby reducing the error of extracting the point and line features of the calibration board from the radar point cloud data.

[0087] In some embodiments, according to the target rotation angle, target displacement, and initial calibration board frame, obtaining the three-dimensional coordinates of multiple vertices of the calibration board in the radar coordinate system to obtain the second calibration board frame may include: transforming the plane coordinates of multiple vertices of the initial calibration board frame in the target plane coordinate system according to the target rotation angle and target displacement to obtain the target plane coordinates of multiple vertices; determining the homogeneous transformation matrix between the target plane coordinate system and the radar coordinate system; and determining the three-dimensional coordinates of multiple vertices of the calibration board in the radar coordinate system according to the homogeneous transformation matrix and the target plane coordinates of multiple vertices to obtain the second calibration board frame. Among them, the specific explanation of the target plane coordinate system can refer to the corresponding process in the foregoing embodiments and will not be elaborated here. This embodiment can accurately determine the three-dimensional coordinates of multiple vertices of the calibration board in the radar coordinate system, improving the accuracy of extracting the point and line features of the calibration board from the radar point cloud data.

[0088] In some embodiments, determining the homogeneous transformation matrix between the target plane coordinate system and the radar coordinate system may include: obtaining the normal vector of the calibration board plane, the first unit vector corresponding to the x-axis of the target plane coordinate system, and the second unit vector corresponding to the y-axis of the target plane coordinate system; determining the homogeneous transformation matrix between the target plane coordinate system and the radar coordinate system according to the normal vector of the calibration board plane, the first unit vector, the second unit vector, and the origin of the target plane coordinate system. This embodiment can accurately determine the homogeneous transformation matrix between the target plane coordinate system and the radar coordinate system.

[0089] For example, the formula P_v = [n1, n2, n, P_o][p_pv(x), p_pv(y), 0, 1] can be followed. T Determine the three-dimensional coordinates of multiple vertices of the calibration board in the radar coordinate system. P_v is the three-dimensional coordinate of the vertex of the calibration board in the radar coordinate system, [n1, n2, n, P_o] is the homogeneous transformation matrix between the target plane coordinate system and the radar coordinate system, n1 is the first unit vector, n2 is the second unit vector, n is the normal vector of the calibration board plane, P_o is the origin of the target plane coordinate system, [p_pv(x), p_pv(y), 0, 1] is the target plane coordinate of the vertex of the initial calibration board frame, its z-axis coordinate is 0, and 1 represents the identifier of the homogeneous coordinate, used to indicate that [p_pv(x), p_pv(y), 0, 1] is a point rather than a vector.

[0090] Step S107, adjust the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm to minimize the projection residual between the second calibration board frame and the first calibration board frame, and obtain the target external parameter matrix.

[0091] Based on the calibration board image and the physical size of the calibration board, this embodiment obtains the two-dimensional coordinates of multiple vertices of the calibration board in the image coordinate system to obtain the first calibration board frame, which can ensure the accuracy of extracting the point-line features of the calibration board from the calibration board image. And based on the physical size of the calibration board, determine the initial calibration board frame of the calibration board, and then obtain the first candidate rotation angle to maximize the number of point clouds in the projection point cloud dataset that fall within the initial calibration board frame at the first candidate rotation angle, so as to achieve the rough extraction of the point-line features of the calibration board. Then, iteratively adjust the rotation angle and displacement of the initial calibration board frame in a decreasing step size to determine the target rotation angle and target displacement, and further achieve the fine extraction of the point-line features of the calibration board, that is, further extract more accurate point-line features of the calibration board with the rough-extracted point-line features of the calibration board as the screening range, reducing the error of extracting the point-line features of the calibration board from the radar point cloud data. In this way, based on the accurately extracted first calibration board frame and the accurately extracted second calibration board frame, the external parameter matrix between the vision sensor and the radar can be accurately determined, improving the calibration accuracy of the external parameters between the radar and the camera.

[0092] In some embodiments, the extrinsic parameter matrix between the vision sensor and the radar is adjusted by an iterative optimization algorithm to minimize the projection residual between the second calibration board frame and the first calibration board frame. Obtaining the target extrinsic parameter matrix may include: determining the direction vector and midpoint of each first calibration board edge in the first calibration board frame, and extracting a plurality of calibration board points from each second calibration board edge in the second calibration board frame; determining the extrinsic parameter matrix between the vision sensor and the radar; for each calibration board point, determining the point-line residual between the calibration board point and the matched first calibration board edge according to the direction vector, midpoint, this extrinsic parameter matrix, and the intrinsic parameter matrix of the vision sensor; determining the projection residual between the second calibration board frame and the first calibration board frame according to the point-line residual between each calibration board point and the matched first calibration board edge; in the case where this projection residual is not the minimum, updating the extrinsic parameter matrix between the vision sensor and the radar, and returning to execute the step of determining, for each calibration board point, the point-line residual between the calibration board point and the matched first calibration board edge according to the direction vector, midpoint, this extrinsic parameter matrix, and the intrinsic parameter matrix of the vision sensor; in the case where this projection residual is the minimum, stopping the iteration, and determining the latest extrinsic parameter matrix as the target extrinsic parameter matrix between the vision sensor and the radar. In this embodiment, the iterative optimization algorithm for adjusting the extrinsic parameter matrix between the vision sensor and the radar is constrained by the point-line residual, which can reduce the dependence of the iterative optimization algorithm on the initial value, avoid falling into a local optimum, and make full use of the edge information of the calibration board extracted from the radar point cloud data (a plurality of calibration board points on each second calibration board edge in the second calibration board frame), improving the calibration accuracy of the extrinsic parameters between the radar and the camera.

[0093] In some embodiments, determining the point-line residual between the calibration board point and the matched first calibration board edge according to the direction vector, midpoint, this extrinsic parameter matrix, and the intrinsic parameter matrix of the vision sensor may include: projecting the calibration board point into the image coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix to obtain a projected point, normalizing the coordinates of the projected point to obtain a normalized projected point; determining the residual between the normalized projected point and the midpoint of the matched first calibration board edge, and multiplying the residual between the normalized projected point and the midpoint of the matched first calibration board edge by the direction vector of the matched first calibration board edge to obtain the point-line residual between the calibration board point and the matched first calibration board edge. For example, the process of determining the point-line residual described above can be described by the following formula:

[0094] r_l(x) = (p_l - p_lo)n_l

[0095] p_l = [P_lc(x) / P_lc(z), P_lc(y) / P_lc(z)] T ,

[0096] P_lc = K * T_cl * P_l

[0097] Where r_l(x) is the point-line residual, p_l is the normalized projection point, p_lo is the midpoint of the matched first calibration board edge, n_l is the direction vector of the first calibration board edge, P_lc is the projection point, K is the internal parameter matrix of the vision sensor, T_cl is the external parameter matrix between the vision sensor and the radar, and P_l is the calibration board point.

[0098] In some embodiments, determining the projection residual between the second calibration board frame and the first calibration board frame according to the point-line residual between each calibration board point and the matched first calibration board edge may include: accumulating the point-line residuals between each calibration board point and the matched first calibration board edge to obtain the total point-line residual, and dividing the total point-line residual by the total number of calibration board points to obtain the projection residual between the second calibration board frame and the first calibration board frame.

[0099] In some embodiments, adjusting the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm to minimize the projection residual between the second calibration board frame and the first calibration board frame to obtain the target external parameter matrix may include: adjusting the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm with a preset external parameter matrix between the vision sensor and the radar as the initial value to minimize the projection residual between the second calibration board frame and the first calibration board frame to obtain the target external parameter matrix, where the preset external parameter matrix is related to the positional relationship between the radar and the vision sensor. The preset external parameter matrix in this embodiment is related to the positional relationship between the radar and the vision sensor. Therefore, the preset external parameter matrix is closer to the external parameter matrix between the vision sensor and the radar to be calibrated. In this way, by using the iterative optimization algorithm with the preset external parameter matrix between the vision sensor and the radar as the initial value to optimize the external parameter matrix between the vision sensor and the radar, the number of optimizations can be reduced, the calibration efficiency can be improved, and getting stuck in local optima can be avoided, thereby improving the calibration accuracy.

[0100] In some embodiments, the positional relationship between the radar and the vision sensor includes the relative distance and relative azimuth between the radar and the vision sensor. For example, both the radar and the vision sensor are installed on a self - moving robot, and the positional relationship between the radar and the vision sensor includes the relative distance and relative azimuth between the installation position of the radar in the self - moving robot and the installation position of the vision sensor in the self - moving robot. Among them, in the case of determining the positional relationship between the radar and the vision sensor, the preset external parameter matrix between the radar and the vision sensor can be accurately determined.

[0101] In some embodiments, the external parameter matrix between the vision sensor and the radar is adjusted through an iterative optimization algorithm to minimize the projection residual between the second calibration board frame and the first calibration board frame, and obtaining the target external parameter matrix may include: adjusting the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm to minimize the target residual, and obtaining the target external parameter matrix, where the target residual is determined based on the projection residual between the second calibration board frame and the first calibration board frame and the point-plane residual between the target point cloud dataset and the image plane of the calibration board. In this embodiment, by combining the projection residual between the second calibration board frame and the first calibration board frame and the point-plane residual between the target point cloud dataset and the image plane of the calibration board to constrain the iterative optimization algorithm to adjust the external parameter matrix between the vision sensor and the radar, it is possible to reduce the dependence of the iterative optimization algorithm on the initial value, avoid falling into a local optimum, and make full use of the calibration board point cloud (target point cloud dataset) and its edge information (multiple calibration board points on each second calibration board edge in the second calibration board frame), thereby improving the calibration accuracy of the external parameters between the radar and the camera.

[0102] In some embodiments, adjusting the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm to minimize the target residual and obtaining the target external parameter matrix may include: adjusting the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm with the preset external parameter matrix between the vision sensor and the radar as the initial value to minimize the target residual, and obtaining the target external parameter matrix, where the target residual is determined based on the projection residual between the second calibration board frame and the first calibration board frame and the point-plane residual between the target point cloud dataset and the image plane of the calibration board. The preset external parameter matrix in this embodiment is related to the positional relationship between the radar and the vision sensor. Therefore, the preset external parameter matrix is closer to the external parameter matrix between the vision sensor and the radar to be calibrated. In this way, by using the iterative optimization algorithm with the preset external parameter matrix between the vision sensor and the radar as the initial value to optimize the external parameter matrix between the vision sensor and the radar, the number of optimizations can be reduced, the calibration efficiency can be improved, and it is possible to avoid falling into a local optimum. At the same time, by combining the projection residual between the second calibration board frame and the first calibration board frame and the point-plane residual between the target point cloud dataset and the image plane of the calibration board to constrain the iterative optimization algorithm to adjust the external parameter matrix between the vision sensor and the radar, it is possible to further avoid falling into a local optimum, thereby effectively improving the calibration accuracy.

[0103] In some embodiments, that the target residual is determined based on the projection residual between the second calibration board frame and the first calibration board frame and the point-plane residual between the target point cloud dataset and the image plane of the calibration board may include: the target residual is the sum of the projection residual between the second calibration board frame and the first calibration board frame and the point-plane residual between the target point cloud dataset and the image plane of the calibration board.

[0104] In some embodiments, the point-plane residual between the target point cloud dataset and the image plane of the calibration board can be determined in the following manner: Obtain the normal vector and distance value of the image plane of the calibration board; According to the external parameter matrix between the vision sensor and the radar, project each point in the target point cloud dataset onto the image coordinate system to obtain the projected position points of each point in the target point cloud dataset; According to the projected position points of each point in the target point cloud dataset and the normal vector and distance value of the image plane of the calibration board, determine the residual between each point in the target point cloud dataset and the image plane of the calibration board; According to the residual between each point in the target point cloud dataset and the image plane of the calibration board, determine the point-plane residual between the target point cloud dataset and the image plane of the calibration board. For example, the process of determining the residual between a point in the target point cloud dataset and the image plane of the calibration board can be described by the following formula: r_p(x) = n_c T (T_cl * P) + d_c, where r_p(x) is the point-plane residual, n_c T is the transpose of the normal vector of the image plane of the calibration board, T_cl is the external parameter matrix between the vision sensor and the radar, and d_c is the distance value of the image plane of the calibration board.

[0105] In some embodiments, determining the point-plane residual between the target point cloud dataset and the image plane of the calibration board according to the residual between each point in the target point cloud dataset and the image plane of the calibration board may include: Accumulating the residuals between each point in the target point cloud dataset and the image plane of the calibration board to obtain the total residual, and dividing the total residual by the total number of points included in the target point cloud dataset to obtain the point-plane residual between the target point cloud dataset and the image plane of the calibration board.

[0106] In some embodiments, obtaining the normal vector and distance value of the image plane of the calibration board may include: Determining the two-dimensional coordinates of multiple corner points of the calibration board in the calibration board image in the image coordinate system, and determining the three-dimensional coordinates of the multiple corner points according to the physical size of the calibration board; According to the two-dimensional coordinates of the multiple corner points and the three-dimensional coordinates of the multiple corner points, determining the homography matrix; According to the homography matrix, determining the normal vector and distance value of the image plane of the calibration board.

[0107] For example, the multi-sensor fusion calibration method provided by the present invention may include: acquiring a calibration board image and a radar point cloud dataset; obtaining the two-dimensional coordinates of multiple vertices of the calibration board in the image coordinate system according to the calibration board image and the physical size of the calibration board, so as to obtain a first calibration board frame; extracting a target point cloud dataset of the calibration board from the radar point cloud dataset, and performing plane fitting on the target point cloud dataset to obtain a calibration board plane; projecting the radar point cloud dataset onto the calibration board plane to obtain a projected point cloud dataset, and determining an initial calibration board frame of the calibration board on the calibration board plane according to the physical size of the calibration board; adjusting the rotation angle of the initial calibration board frame within a first angle range with a first rotation step to determine a first candidate rotation angle, wherein the number of point clouds in the projected point cloud dataset falling within the initial calibration board frame at the first candidate rotation angle is the largest; starting from the first candidate rotation angle, adjusting the rotation angle of the initial calibration board frame within a second angle range with a second rotation step and adjusting the displacement of the initial calibration board frame within a first displacement range with a first displacement step to determine a second candidate rotation angle and a candidate displacement, wherein the number of point clouds in the projected point cloud dataset falling within the initial calibration board frame at the second candidate rotation angle and the candidate displacement is the largest, the second angle range is smaller than the first angle range, and the second rotation step is smaller than the first rotation step; starting from the second candidate rotation angle and the candidate displacement, adjusting the rotation angle of the initial calibration board frame within a third angle range with a third rotation step and adjusting the displacement of the initial calibration board frame within a second displacement range with a second displacement step to determine a target rotation angle and a target displacement, wherein the accuracy score corresponding to the target rotation angle and the target displacement is the highest, the accuracy score corresponding to the target rotation angle and the target displacement is related to the number of point clouds in the projected point cloud dataset falling within the initial calibration board frame at the target rotation angle and the target displacement and the number of target points, the target points are located outside the calibration area frame in the initial calibration board frame at the target rotation angle and the target displacement, and the reflection intensity of the target points is greater than a preset threshold, the third angle range is smaller than the second angle range, the third rotation step is smaller than the second rotation step, the second displacement range is smaller than the first displacement range, and the second displacement step is smaller than the first displacement step; obtaining the three-dimensional coordinates of multiple vertices of the calibration board in the radar coordinate system according to the target rotation angle, the target displacement and the initial calibration board frame, so as to obtain a second calibration board frame; adjusting the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm with the preset external parameter matrix between the vision sensor and the radar as the initial value, so that the target residual is minimized to obtain a target external parameter matrix, wherein the target residual is determined according to the projection residual between the second calibration board frame and the first calibration board frame and the point-plane residual between the target point cloud dataset and the image plane of the calibration board.

[0108] In the multi-sensor fusion calibration method provided by way of example above, when extracting the point and line features of the calibration board from the radar point cloud data, the influence of noise in the laser beam direction is reduced by plane fitting, and based on the physical size of the calibration board and the reflection intensity of the points in the radar point cloud data, more accurate point and line features of the calibration board can be extracted from the radar point cloud data through a method of traversing from coarse to fine, reducing the error of extracting the point and line features of the calibration board from the sparse radar point cloud data. At the same time, by combining point-line and point-plane constraints, the external parameter matrix of the vision sensor and the radar is estimated through an iterative optimization method, reducing the dependence on the initial value, overcoming the defect that point-plane features are prone to falling into local optima, and making full use of all calibration board point clouds and their edge information. Therefore, the multi-sensor fusion calibration method provided by way of example above greatly improves the calibration accuracy of the external parameter matrix of the vision sensor and the radar.

[0109] Please refer to Figure 7 , Figure 7 which is a schematic block diagram of the structure of a computer device provided by an embodiment of the present invention.

[0110] As Figure 7 shown, the computer device 100 includes a processor 101 and a memory 102, and the processor 101 and the memory 102 are connected through a bus 103, and this bus is, for example, an I2C (Inter-integrated Circuit) bus.

[0111] Specifically, the processor 101 is used to provide computing and control capabilities to support the operation of the entire computer device. The processor 101 can be a central processing unit (CPU), and this processor 101 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or this processor can also be any conventional processor, etc.

[0112] Specifically, the memory 102 can be a Flash chip, a read-only memory (ROM), a magnetic disk, an optical disc, a USB flash drive, or a mobile hard disk, etc.

[0113] Those skilled in the art can understand that Figure 7The structure shown is only a block diagram of some structures related to the solution of the embodiment of the present invention, and does not constitute a limitation on the computer device to which the solution of the embodiment of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0114] Wherein, the processor is used to run a computer program stored in the memory, and when executing the computer program, implement any one of the multi-sensor fusion calibration methods provided by the embodiments of the present invention.

[0115] In one embodiment, the processor is used to run a computer program stored in the memory, and when executing the computer program, implement the following steps:

[0116] Obtain a calibration board image and a radar point cloud data set;

[0117] According to the calibration board image and the physical size of the calibration board, obtain the two-dimensional coordinates of multiple vertices of the calibration board in the image coordinate system to obtain a first calibration board frame;

[0118] Extract the target point cloud data set of the calibration board from the radar point cloud data set, and perform plane fitting on the target point cloud data set to obtain a calibration board plane;

[0119] Project the radar point cloud data set onto the calibration board plane to obtain a projected point cloud data set, and determine an initial calibration board frame of the calibration board on the calibration board plane according to the physical size of the calibration board;

[0120] Obtain a first candidate rotation angle, and iteratively adjust the rotation angle and displacement of the initial calibration board frame in a way of reducing the step size to determine a target rotation angle and a target displacement. The first candidate rotation angle is the rotation angle corresponding to the largest number of point clouds falling within the initial calibration board frame in the projected point cloud data set;

[0121] According to the target rotation angle, the target displacement and the initial calibration board frame, obtain the three-dimensional coordinates of multiple vertices of the calibration board in the radar coordinate system to obtain a second calibration board frame;

[0122] Adjust the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm to minimize the projection residual between the second calibration board frame and the first calibration board frame, and obtain a target external parameter matrix.

[0123] In some embodiments, the accuracy score corresponding to the target rotation angle and the target displacement is the highest. The accuracy score corresponding to the target rotation angle and the target displacement is related to the number of point clouds within the initial calibration plate frame at the target rotation angle and the target displacement in the projected point cloud dataset and the number of target points. The target points are located outside the calibration area frame in the initial calibration plate frame at the target rotation angle and the target displacement, and the reflection intensity of the target points is greater than a preset threshold.

[0124] In some embodiments, when the processor realizes obtaining a first candidate rotation angle and iteratively adjusting the rotation angle and displacement of the initial calibration plate frame in a way of reducing step sizes to determine the target rotation angle and the target displacement, it is used to realize:

[0125] Adjust the rotation angle of the initial calibration plate frame within a first angle range with a first rotation step size to determine the first candidate rotation angle;

[0126] Starting from the first candidate rotation angle, iteratively adjust the rotation angle and displacement of the initial calibration plate frame in a way of reducing the range and reducing step sizes to determine the target rotation angle and the target displacement.

[0127] In some embodiments, when the processor realizes starting from the first candidate rotation angle and iteratively adjusting the rotation angle and displacement of the initial calibration plate frame in a way of reducing the range and reducing step sizes to determine the target rotation angle and the target displacement, it is used to realize:

[0128] Starting from the first candidate rotation angle, adjust the rotation angle of the initial calibration plate frame within a second angle range with a second rotation step size and adjust the displacement of the initial calibration plate frame within a first displacement range with a first displacement step size to determine a second candidate rotation angle and a candidate displacement, where the number of point clouds within the initial calibration plate frame at the second candidate rotation angle and the candidate displacement in the projected point cloud dataset is the largest, the second angle range is smaller than the first angle range, and the second rotation step size is smaller than the first rotation step size;

[0129] Starting from the second candidate rotation angle and the candidate displacement, adjust the rotation angle of the initial calibration plate frame within a third angular range with a third rotation step and adjust the displacement of the initial calibration plate frame within a second displacement range with a second displacement step to determine a target rotation angle and a target displacement, where the accuracy score corresponding to the target rotation angle and the target displacement is the highest, and the accuracy score corresponding to the target rotation angle and the target displacement is related to the number of point clouds within the initial calibration plate frame at the target rotation angle and the target displacement in the projected point cloud dataset and the number of target points. The target points are located outside the calibration area frame in the initial calibration plate frame at the target rotation angle and the target displacement, and the reflection intensity of the target points is greater than a preset threshold. The third angular range is smaller than the second angular range, the third rotation step is smaller than the second rotation step, the second displacement range is smaller than the first displacement range, and the second displacement step is smaller than the first displacement step.

[0130] In some embodiments, when the processor realizes adjusting the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm to minimize the projection residual between the second calibration plate frame and the first calibration plate frame and obtain a target external parameter matrix, it is used to realize:

[0131] Adjust the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm with a preset external parameter matrix between the vision sensor and the radar as the initial value to minimize the projection residual between the second calibration plate frame and the first calibration plate frame and obtain a target external parameter matrix, where the preset external parameter matrix is related to the positional relationship between the radar and the vision sensor.

[0132] In some embodiments, when the processor realizes adjusting the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm to minimize the projection residual between the second calibration plate frame and the first calibration plate frame and obtain a target external parameter matrix, it is used to realize:

[0133] Adjust the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm to minimize the target residual and obtain a target external parameter matrix, where the target residual is determined based on the projection residual between the second calibration plate frame and the first calibration plate frame and the point-plane residual between the target point cloud dataset and the image plane of the calibration plate.

[0134] In some embodiments, when the processor realizes obtaining the two-dimensional coordinates of multiple vertices of the calibration plate in the image coordinate system according to the calibration plate image and the physical size of the calibration plate, it is used to realize:

[0135] Determine the two-dimensional coordinates of multiple corner points of the calibration board in the calibration board image in the image coordinate system, and determine the three-dimensional coordinates of the multiple corner points according to the physical size of the calibration board;

[0136] Determine the homography matrix according to the two-dimensional coordinates of the multiple corner points and the three-dimensional coordinates of the multiple corner points;

[0137] Determine the three-dimensional coordinates of multiple vertices of the calibration board according to the physical size of the calibration board;

[0138] Determine the two-dimensional coordinates of the multiple vertices of the calibration board in the image coordinate system according to the homography matrix and the three-dimensional coordinates of the multiple vertices of the calibration board.

[0139] In some embodiments, when the processor implements extracting the target point cloud dataset of the calibration board from the radar point cloud dataset, it is used to implement:

[0140] Extract the non-ground point cloud dataset from the radar point cloud dataset;

[0141] Perform Euclidean distance clustering on the non-ground point cloud dataset to obtain multiple point cloud clusters, and perform plane fitting on each point cloud cluster to obtain multiple fitted planes;

[0142] Remove the points in the non-ground point cloud dataset that are not located in each of the fitted planes to obtain the target point cloud dataset of the calibration board.

[0143] In some embodiments, when the processor implements projecting the radar point cloud dataset onto the calibration board plane to obtain the projected point cloud dataset, it is used to implement:

[0144] Project the radar point cloud dataset onto the calibration board plane to obtain a set of plane points;

[0145] Construct a target plane coordinate system according to the normal vector of the calibration board plane and the center point of the set of plane points;

[0146] Determine the plane points of each plane point in the set of plane points in the target plane coordinate system to obtain the projected point cloud dataset.

[0147] In some embodiments, when the processor implements constructing a target plane coordinate system according to the normal vector of the calibration board plane and the center point of the set of plane points, it is used to implement:

[0148] Determine a first unit vector perpendicular to the normal vector of the calibration board plane, and determine a second unit vector perpendicular to the normal vector of the calibration board plane and the first unit vector;

[0149] Taking the center point of the set of planar points as the origin, using the first unit vector as the x-axis, and using the second unit vector as the y-axis, construct the target plane coordinate system.

[0150] In some embodiments, when the processor implements determining the planar points of each planar point in the set of planar points in the target plane coordinate system, it is used to implement:

[0151] According to the first unit vector, the second unit vector, and the center point of the set of planar points, determine the planar points of each planar point in the set of planar points in the target plane coordinate system.

[0152] In some embodiments, when the processor implements obtaining the three-dimensional coordinates of multiple vertices of the calibration board in the radar coordinate system according to the target rotation angle, the target displacement, and the initial calibration board frame to obtain a second calibration board frame, it is used to implement:

[0153] According to the target rotation angle and the target displacement, transform the planar coordinates of multiple vertices of the initial calibration board frame in the target plane coordinate system to obtain the target planar coordinates of the multiple vertices;

[0154] Determine the homogeneous transformation matrix between the target plane coordinate system and the radar coordinate system;

[0155] According to the homogeneous transformation matrix and the target planar coordinates of the multiple vertices, determine the three-dimensional coordinates of the multiple vertices of the calibration board in the radar coordinate system to obtain a second calibration board frame.

[0156] In some embodiments, when the processor implements determining the homogeneous transformation matrix between the target plane coordinate system and the radar coordinate system, it is used to implement:

[0157] Obtain the normal vector of the calibration board plane, the first unit vector corresponding to the x-axis of the target plane coordinate system, and the second unit vector corresponding to the y-axis of the target plane coordinate system;

[0158] According to the normal vector of the calibration board plane, the first unit vector, the second unit vector, and the origin of the target plane coordinate system, determine the homogeneous transformation matrix between the target plane coordinate system and the radar coordinate system.

[0159] In some embodiments, the calibration board image is obtained by a vision sensor collecting a target scene provided with a calibration board, and the radar point cloud data set is obtained by a radar scanning the target scene.

[0160] It should be noted that those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working process of the computer device described above can refer to the corresponding process in the embodiment of the multi-sensor fusion calibration method described above, and will not be repeated here.

[0161] An embodiment of the present invention further provides a storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any multi-sensor fusion calibration method provided in the specification of the embodiment of the present invention.

[0162] Among them, the storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as the hard disk or memory of the computer device. The storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the computer device.

[0163] Those of ordinary skill in the art will understand that all or some of the steps in the methods disclosed above, and the functional modules / units in systems and devices, can be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division of functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be executed by several physical components in cooperation. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cassette, tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that a communication medium typically contains computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium.

[0164] It should be understood that the term "and / or" used in the specification and appended claims of the present invention refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. It should be noted that in this text, the term "comprises," "comprising," or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a series of elements includes not only those elements but also other elements not expressly listed, or elements that are inherent to such process, method, article, or system. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article, or system that comprises the element.

[0165] The serial numbers of the embodiments of the present invention above are only for description and do not represent the superiority or inferiority of the embodiments. As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A multi-sensor fusion calibration method based on a calibration board, characterized in that, Including: Obtaining a calibration board image and a radar point cloud data set; According to the calibration board image and the physical size of the calibration board, obtaining the two-dimensional coordinates of multiple vertices of the calibration board in the image coordinate system to obtain a first calibration board frame; Extracting a target point cloud data set of the calibration board from the radar point cloud data set and performing plane fitting on the target point cloud data set to obtain a calibration board plane; Projecting the radar point cloud data set onto the calibration board plane to obtain a projected point cloud data set, and determining an initial calibration board frame of the calibration board on the calibration board plane according to the physical size of the calibration board; Obtaining a first candidate rotation angle, and iteratively adjusting the rotation angle and displacement of the initial calibration board frame in a way of reducing step size to determine a target rotation angle and a target displacement, where the first candidate rotation angle is the rotation angle corresponding to the largest number of point clouds falling within the initial calibration board frame in the projected point cloud data set; According to the target rotation angle, the target displacement and the initial calibration board frame, obtaining the three-dimensional coordinates of multiple vertices of the calibration board in the radar coordinate system to obtain a second calibration board frame; Adjusting the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm to minimize the projection residual between the second calibration board frame and the first calibration board frame, and obtaining a target external parameter matrix.

2. The multi-sensor fusion calibration method according to claim 1, wherein The accuracy score corresponding to the target rotation angle and the target displacement is the highest. The accuracy score corresponding to the target rotation angle and the target displacement is related to the number of point clouds falling within the initial calibration board frame at the target rotation angle and the target displacement and the number of target points. The target points are located outside the calibration area frame in the initial calibration board frame at the target rotation angle and the target displacement, and the reflection intensity of the target points is greater than a preset threshold.

3. The multi-sensor fusion calibration method according to claim 1, wherein The obtaining a first candidate rotation angle, and iteratively adjusting the rotation angle and displacement of the initial calibration board frame in a way of reducing step size to determine a target rotation angle and a target displacement includes: Adjusting the rotation angle of the initial calibration board frame at a first rotation step size within a first angle range to determine the first candidate rotation angle; Starting from the first candidate rotation angle, iteratively adjusting the rotation angle and displacement of the initial calibration board frame in a way of reducing the range and reducing the step size to determine a target rotation angle and a target displacement.

4. The multi-sensor fusion calibration method according to claim 3, wherein The iteratively adjusting the rotation angle and displacement of the initial calibration board frame in a way of reducing the range and reducing the step size starting from the first candidate rotation angle to determine a target rotation angle and a target displacement includes: Starting from the first candidate rotation angle, adjust the rotation angle of the initial calibration plate frame with a second rotation step within a second angular range and adjust the displacement of the initial calibration plate frame with a first displacement step within a first displacement range to determine a second candidate rotation angle and a candidate displacement, where the number of point clouds within the initial calibration plate frame at the second candidate rotation angle and the candidate displacement in the projected point cloud dataset is the largest, the second angular range is smaller than the first angular range, and the second rotation step is smaller than the first rotation step; Starting from the second candidate rotation angle and the candidate displacement, adjust the rotation angle of the initial calibration plate frame with a third rotation step within a third angular range and adjust the displacement of the initial calibration plate frame with a second displacement step within a second displacement range to determine a target rotation angle and a target displacement, where the accuracy score corresponding to the target rotation angle and the target displacement is the highest, the accuracy score corresponding to the target rotation angle and the target displacement is related to the number of point clouds within the initial calibration plate frame at the target rotation angle and the target displacement and the number of target points, the target points are outside the calibration area frame in the initial calibration plate frame at the target rotation angle and the target displacement, and the reflection intensity of the target points is greater than a preset threshold, the third angular range is smaller than the second angular range, the third rotation step is smaller than the second rotation step, the second displacement range is smaller than the first displacement range, and the second displacement step is smaller than the first displacement step.

5. The multi-sensor fusion calibration method according to claim 1, wherein Adjusting the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm to minimize the projection residual between the second calibration plate frame and the first calibration plate frame to obtain the target external parameter matrix, includes: Adjusting the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm with the preset external parameter matrix between the vision sensor and the radar as the initial value to minimize the projection residual between the second calibration plate frame and the first calibration plate frame to obtain the target external parameter matrix, where the preset external parameter matrix is related to the positional relationship between the radar and the vision sensor.

6. The multi-sensor fusion calibration method according to claim 1, characterized in that, Adjusting the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm to minimize the projection residual between the second calibration plate frame and the first calibration plate frame to obtain the target external parameter matrix, includes: Adjusting the external parameter matrix between the vision sensor and the radar through an iterative optimization algorithm to minimize the target residual to obtain the target external parameter matrix, where the target residual is determined according to the projection residual between the second calibration plate frame and the first calibration plate frame and the point-plane residual between the target point cloud dataset and the image plane of the calibration plate.

7. The multi-sensor fusion calibration method according to claim 1, characterized in that According to the calibration plate image and the physical size of the calibration plate, obtaining the two-dimensional coordinates of multiple vertices of the calibration plate in the image coordinate system, includes: Determine the two-dimensional coordinates of multiple corner points of the calibration board in the calibration board image in the image coordinate system, and determine the three-dimensional coordinates of the multiple corner points according to the physical size of the calibration board; Determine the homography matrix according to the two-dimensional coordinates of the multiple corner points and the three-dimensional coordinates of the multiple corner points; Determine the three-dimensional coordinates of multiple vertices of the calibration board according to the physical size of the calibration board; Determine the two-dimensional coordinates of the multiple vertices of the calibration board in the image coordinate system according to the homography matrix and the three-dimensional coordinates of the multiple vertices of the calibration board.

8. The multi-sensor fusion calibration method according to claim 1, wherein, The extracting the target point cloud dataset of the calibration board from the radar point cloud dataset includes: Extract the non-ground point cloud dataset from the radar point cloud dataset; Perform Euclidean distance clustering on the non-ground point cloud dataset to obtain multiple point cloud clusters, and perform plane fitting on each point cloud cluster to obtain multiple fitted planes; Remove the points in the non-ground point cloud dataset that are not located in each of the fitted planes to obtain the target point cloud dataset of the calibration board.

9. The multi-sensor fusion calibration method according to any one of claims 1-8, characterized in that, The projecting the radar point cloud dataset onto the calibration board plane to obtain a projected point cloud dataset includes: Project the radar point cloud dataset onto the calibration board plane to obtain a set of plane points; Construct a target plane coordinate system according to the normal vector of the calibration board plane and the center point of the set of plane points; Determine the plane points of each plane point in the set of plane points in the target plane coordinate system to obtain the projected point cloud dataset.

10. The multi-sensor fusion calibration method according to claim 9, wherein The constructing a target plane coordinate system according to the normal vector of the calibration board plane and the center point of the set of plane points includes: Determine a first unit vector perpendicular to the normal vector of the calibration board plane, and determine a second unit vector perpendicular to the normal vector of the calibration board plane and the first unit vector; Construct the target plane coordinate system with the center point of the set of plane points as the origin, the first unit vector as the x-axis, and the second unit vector as the y-axis.

11. The multi-sensor fusion calibration method according to claim 10, wherein The determining the plane points of each plane point in the set of plane points in the target plane coordinate system includes: Determine the plane points of each plane point in the set of plane points in the target plane coordinate system according to the first unit vector, the second unit vector, and the center point of the set of plane points.

12. The multi-sensor fusion calibration method according to any one of claims 1-8, characterized in that The obtaining the three-dimensional coordinates of multiple vertices of the calibration board in the radar coordinate system according to the target rotation angle, the target displacement, and the initial calibration board frame to obtain a second calibration board frame includes: Transform the plane coordinates of the multiple vertices of the initial calibration board frame in the target plane coordinate system according to the target rotation angle and the target displacement to obtain the target plane coordinates of the multiple vertices; Determine the homogeneous transformation matrix between the target plane coordinate system and the radar coordinate system; Determine the three-dimensional coordinates of the multiple vertices of the calibration board in the radar coordinate system according to the homogeneous transformation matrix and the target plane coordinates of the multiple vertices to obtain a second calibration board frame.

13. The multi-sensor fusion calibration method according to claim 12, characterized in that, The determining the homogeneous transformation matrix between the target plane coordinate system and the radar coordinate system includes: Obtain the normal vector of the calibration board plane, the first unit vector corresponding to the x-axis of the target plane coordinate system, and the second unit vector corresponding to the y-axis of the target plane coordinate system; Determine the homogeneous transformation matrix between the target plane coordinate system and the radar coordinate system according to the normal vector of the calibration board plane, the first unit vector, the second unit vector, and the origin of the target plane coordinate system.

14. The multi-sensor fusion calibration method according to any one of claims 1-8, characterized in that, The calibration board image is obtained by a vision sensor collecting a target scene provided with a calibration board, and the radar point cloud data set is obtained by a radar scanning the target scene.

15. A computer device, characterized in that, The computer device includes a processor, a memory, a computer program stored on the memory and executable by the processor, and a data bus for realizing the connection and communication between the processor and the memory. When the computer program is executed by the processor, the multi-sensor fusion calibration method according to any one of claims 1 to 14 is realized.

16. A storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to realize the multi-sensor fusion calibration method according to any one of claims 1 to 14.

Images