Laser radar-camera target-object-free online calibration method based on point cloud azimuth diagram

Abstract

The invention discloses a laser radar-camera target-object-free online calibration method based on a point cloud azimuth diagram. The method comprises the following steps: based on a CT-ICP algorithm, obtaining a coarse estimation pose of a current frame according to each frame of interpolation laser radar attitude point cloud and laser radar target point cloud, and obtaining an optimal pose; all the optimal poses are converted into the laser radar point cloud of initial frame scanning to obtain global static point cloud; acquiring a local static point cloud based on initialization external parameters of a given laser radar and a camera, and performing projection transformation on the acquired local static point cloud to acquire a point cloud azimuth angle diagram; obtaining a feature point matching pair set according to the point cloud azimuth angle diagram and the camera image based on a pre-trained cross-modal deep learning network Super; performing wrong matching point elimination on the obtained feature point matching set pair through a rotating RANSAC algorithm to obtain an optimized feature point matching pair; and a Levenberg-Marquardt optimizer is adopted to minimize and optimize the re-projection error of the feature point matching pair, so that a laser radar-camera transformation matrix can be obtained, and the target object-free online calibration of the laser radar-camera can be realized. The problems that a traditional calibration method is a calibration method based on a target object, the calibration method based on the target object needs to depend on a specific object to conduct feature extraction calibration, most calibration methods are off-line calibration, and the calibration methods are tedious and cannot meet the requirements of some scenes are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of laser radar-camera calibration, and in particular to a laser radar-camera non-target online calibration method based on a point cloud azimuth diagram. Background Art

[0002] Multi-Sensor Fusion is the process of integrating data from multiple sensors to provide more accurate and reliable information or enhance the system's perception capabilities. In modern technologies and applications, especially in the fields of autonomous driving, robotics, drones, smart manufacturing, and the Internet of Things, multi-sensor fusion has important application value and a wide range of backgrounds. Compared with traditional fixed laser radars, external rotating laser radars have the advantages of a larger field of view and a denser point cloud, which can make the generated point cloud scene details richer and more delicate. Traditional cameras can provide rich texture and color information, and have high resolution, which can be used for subsequent visual tasks such as recognition. By integrating the data of the two sensors, more comprehensive, accurate, and reliable information can be provided.

[0003] The traditional calibration method is the target-based calibration method, which relies on specific objects for feature extraction and calibration, and most of them are offline calibrations, which are not only cumbersome but also cannot meet the needs of some scenarios. Summary of the invention

[0004] The present invention provides a target-free online calibration method for a laser radar-camera based on a point cloud azimuth diagram to overcome the above technical problems.

[0005] In order to achieve the above object, the technical solution of the present invention is:

[0006] A laser radar-camera online calibration method without a target based on a point cloud azimuth diagram comprises the following steps:

[0007] S1: Obtain several frames of laser radar point clouds and camera images scanned under the same preset scene; and interpolate the laser radar point clouds of each frame to obtain the interpolated laser radar attitude point cloud of the corresponding frame;

[0008] S2: Based on the CT-ICP algorithm, the rough estimated pose of the current frame is obtained according to the interpolated lidar pose point cloud of each frame and the preset lidar target point cloud, and the optimal pose is obtained according to the rough estimated pose of the current frame and the rough estimated pose of the previous frame; and each optimal pose is converted to the lidar point cloud scanned by the initial frame to obtain the global static point cloud of the preset scene;

[0009] S3: Based on the initialization external parameters of the given laser radar and camera, a local static point cloud is obtained according to the global static point cloud, and a projective transformation is performed on the obtained local static point cloud to obtain a point cloud azimuth angle map;

[0010] S4: Based on the pre-trained cross-modal deep learning network Superglue, feature point matching pairs are obtained according to the point cloud azimuth map and camera image;

[0011] S5: Using the rotational RANSAC algorithm, the wrong matching points are removed from the acquired feature point matching set pairs to obtain optimized feature point matching pairs;

[0012] S6: The Levenberg-Marquardt optimizer is used to minimize the reprojection error of the optimized feature point matching pairs to obtain the lidar-camera transformation matrix to achieve online calibration of the lidar-camera without targets.

[0013] Furthermore, the expression for obtaining the optimal posture described in S2 is:

[0014]

[0015] Where: Indicates the optimal pose of the lidar point cloud scanned in the previous frame; Indicates the optimal pose of the lidar point cloud scanned in the current frame; L b Indicates the rough estimated pose of the lidar point cloud scanned in the previous frame; L e Indicates the rough estimated pose of the lidar point cloud scanned in the current frame; F ICP represents the ICP iterative algorithm; C loc represents the position consistency constraint of the lidar sensor; C vel represents the uniform velocity constraint of the lidar sensor; t b Represents the translation matrix of the lidar point cloud scanned in the previous frame; t e Represents the translation matrix of the lidar point cloud scanned in the current frame; n represents the number of scans; q1 and q2 represent the coefficients of the position consistency constraint and the uniform speed constraint, respectively.

[0016] Furthermore, the projective transformation of the acquired local static point cloud in S3 to acquire the point cloud azimuth angle map specifically includes the following steps:

[0017] S31: Performing a plane projection on any three-dimensional cloud point in the acquired local static point cloud to obtain a plane projection point cloud, and the expression for performing the plane projection is:

[0018] P0=MP

[0019]

[0020] Where: P0 represents the projection point of any point P in the XYZ-O three-dimensional space in a plane Q parallel to the YOZ plane; N represents the distance between plane Q and the YOZ plane; M represents the plane projection matrix; x0, y0, z0 represent the position coordinates of the corresponding point P0; x, y, z represent the position coordinates of the corresponding point P;

[0021] S32: Gridding the plane projection point cloud to obtain the coordinate cloud point P of the projection point P0 in the set plane Q two-dimensional grid coordinate system Q (a0,b0), its expression is

[0022]

[0023] Where: a represents the number of columns into which the point cloud is divided horizontally in plane Q; b represents the number of rows into which the point cloud is divided vertically in plane Q; W represents the width of the point cloud projected on plane Q; H represents the height of the point cloud projected on plane Q; D represents the side length of the set square grid; y l Represents the Y-axis coordinate of the set plane Q two-dimensional grid coordinate system; z l Represents the Z-axis coordinate of the set plane Q two-dimensional grid coordinate system;

[0024] S33: coordinate cloud point P in the two-dimensional grid coordinate system of plane Q Q (a0, b0) is serialized to obtain optimized point cloud data; and the serialization method is to obtain the coordinate cloud points corresponding to each grid in the plane Q two-dimensional grid coordinate system, and obtain the coordinate cloud point closest to the center point of each grid, and use it as the optimized cloud point corresponding to the grid;

[0025] S34: Calculate the point cloud azimuth according to the optimized point cloud data, and the calculation formula of the point cloud azimuth is:

[0026]

[0027] Where: θ i,j Indicates the point cloud azimuth; P i,j represents the optimized cloud point in the i-th row and j-th column in the two-dimensional grid coordinate system of the corresponding plane Q; P i-1,j-1 represents the optimized cloud point at the i-1th row and j-1th column in the two-dimensional grid coordinate system of the corresponding plane Q; P view Represents the viewpoint of optimizing point cloud data;

[0028] S35: Map the point cloud azimuth angle into a two-dimensional image grayscale value to obtain a point cloud azimuth angle map; and the expression for mapping the point cloud azimuth angle into a two-dimensional image grayscale value is:

[0029]

[0030] Where: img i,j Represents the grayscale value of the image in the i-th row and j-th column in the two-dimensional grid coordinate system of the corresponding plane Q.

[0031] Furthermore, the S4 specifically includes the following steps:

[0032] S41: The pre-trained cross-modal deep learning network Superglue includes a SuperPoint deep neural network and a graph neural network, and the pre-trained SuperPoint deep neural network is used to extract the positions of feature points in the point cloud azimuth map and the camera image, respectively, to obtain descriptors of the corresponding feature points;

[0033] And the descriptor includes an image feature descriptor and a point cloud azimuth descriptor;

[0034] S42: Input the image feature descriptor and the point cloud azimuth descriptor into a pre-trained graph neural network, and obtain a feature point matching pair set of the image feature descriptor and the point cloud azimuth descriptor according to the pre-trained graph neural network.

[0035] Furthermore, the S5 specifically includes the following steps:

[0036] S51: Set the maximum number of iterations M and the internal point threshold N of the rotation RANSAC algorithm, and initialize the optimal rotation matrix R best and the maximum number of interior points N best ; Define the feature point position K of the feature point matching pair point cloud azimuth descriptor in the set as {x k1 ,x k2 ,...,x kn}, the feature point position D of the image feature descriptor in the feature point matching pair = {L pk1 ,L pk2 ,...,L pkn};

[0037] S52: Randomly select any two sets of feature points (x ki ,L pki ) and (x kj ,L pkj ), calculate and obtain the current rotation matrix R ij , whose expression is

[0038]

[0039] Where: I represents the unit matrix; K represents the antisymmetric matrix;

[0040] S53: Get the current rotation matrix R ijAs the optimal rotation matrix R best , through the optimal rotation matrix R best Rotate the remaining feature points in the feature point matching pair to obtain the current feature point;

[0041] Get the current feature distance between the current feature point and other corresponding feature points, and determine the size of the current feature distance and the inner point threshold N;

[0042] If the current feature distance is less than or equal to the inlier threshold N, the feature point corresponding to the current feature distance is stored in the current inlier set, and step S54 is executed; if the current feature distance is greater than the inlier threshold N, the feature point corresponding to the current feature distance is removed;

[0043] S54: Determine the number of interior points in the current interior point set and the maximum number of interior points N best size;

[0044] If the number of interior points in the current interior point set is less than or equal to the maximum number of interior points N best , then return to step S52;

[0045] Otherwise, the number of inliers in the current inlier set is updated to the maximum number of inliers N in the current iteration. best , and return to step S52 until the maximum number of iterations M is reached, and the final optimal rotation matrix R is obtained. best The obtained internal point set is used to obtain the optimized feature point matching pair.

[0046] Furthermore, the Levenberg-Marquardt optimizer is used in S6 to minimize the reprojection error of the optimized feature point matching pairs, and the expression of the lidar-camera transformation matrix is ​​obtained as follows:

[0047]

[0048] Where: represents the six-degree-of-freedom lidar-camera transformation matrix; n represents the number of feature point groups in the optimized feature point matching pair; ρ represents the abbreviated form of ρ(r), which is the Cauchy robust kernel function; r represents the residual; c represents an adjustment parameter.

[0049] Beneficial effect: The present invention provides a method for online calibration of a laser radar and a camera without a target based on a point cloud azimuth map. The method obtains the optimal pose based on the rough estimated pose of the current frame and the rough estimated pose of the previous frame based on the CT-ICP algorithm, and converts each optimal pose into the laser radar point cloud scanned by the initial frame to obtain a global static point cloud of a preset scene. The method obtains a local static point cloud by giving the initial external parameters of the laser radar and the camera, and performs a projection transformation on the obtained local static point cloud to obtain a point cloud azimuth map. The point cloud azimuth map has the advantages of strong ability to depict details, small distortion, and rich texture details, which is different from the traditional two-dimensional The images are closer, which can improve the accuracy and reliability of matching. At the same time, the local point cloud scene is used for feature extraction and matching. By determining the local scene corresponding to the image in the global static point cloud scene, calibration samples are provided for joint calibration to greatly improve the calibration efficiency. The RANSAC algorithm is used to remove the wrong matching points of the acquired feature point matching set pairs, and the Levenberg-Marquardt optimizer is used to minimize the reprojection error of the optimized feature point matching pairs to obtain the lidar-camera transformation matrix, which greatly improves the accuracy and efficiency of the online calibration of the lidar-camera without targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 It is a flow chart of the laser radar-camera online calibration method without target based on point cloud azimuth diagram of the present invention;

[0052] Figure 2 Schematic diagram of a global static point cloud in this embodiment;

[0053] Figure 3 Schematic diagram of the field of view angle of the XOY plane in the camera coordinate system in this embodiment;

[0054] Figure 4 Schematic diagram of the field of view angle of the YOZ plane in the camera coordinate system in this embodiment;

[0055] Figure 5 Schematic diagram of a local point cloud within the field of view of the corresponding camera in this embodiment;

[0056] Figure 6 This is a schematic diagram of rasterization of a local point cloud in this embodiment;

[0057] Figure 7 Schematic diagram of serialization of three-dimensional unordered point cloud in this embodiment;

[0058] Figure 8 Schematic diagram of calculation of point cloud azimuth in this embodiment;

[0059] Fig. 9 Schematic diagram of the point cloud azimuth diagram in this embodiment;

[0060] Fig.10 In this embodiment, according to the laser radar-camera transformation Schematic diagram of the result of projecting the external rotation 3D LiDAR point cloud into a 2D image at different times of a frame;

[0061] Fig.11 In this embodiment, according to the laser radar-camera transformation Schematic diagram of the result of projecting another frame of externally rotated 3D LiDAR point cloud at different times into a 2D image;

[0062] Fig.12 This is a core flow chart of the laser radar-camera online calibration method without target based on point cloud azimuth diagram in this embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] This embodiment provides a method for online calibration of a laser radar-camera without a target based on a point cloud azimuth diagram. Figure 1 and Fig.12 The following steps are shown:

[0065] S1: Obtain several frames of laser radar point clouds and camera images scanned under the same preset scene; and interpolate the laser radar point clouds of each frame to obtain the interpolated laser radar attitude point cloud of the corresponding frame; wherein the interpolation method for interpolating the laser radar point clouds of each frame includes linear interpolation or spline interpolation, etc., which can estimate the attitude of the laser radar at any time point, that is, the attitude of the corresponding point in the motion model at the same current scanning time point;

[0066] S2: Based on the CT-ICP algorithm, a rough estimated pose of the current frame is obtained according to the interpolated lidar pose point cloud of each frame and the preset lidar target point cloud, and the optimal pose is obtained according to the rough estimated pose of the current frame and the rough estimated pose of the previous frame; the pose is expressed as L = (R, t), R and t represent rotation and translation respectively; and each optimal pose is converted to the lidar point cloud scanned by the initial frame to obtain the global static point cloud of the preset scene;

[0067] Specifically, the expression for obtaining the optimal posture is:

[0068]

[0069] Where: Indicates the optimal pose of the lidar point cloud scanned in the previous frame; Indicates the optimal pose of the lidar point cloud scanned in the current frame; L b Indicates the rough estimated pose of the lidar point cloud scanned in the previous frame; L e Indicates the rough estimated pose of the lidar point cloud scanned in the current frame; F ICP represents the ICP iterative algorithm; C loc represents the position consistency constraint of the lidar sensor; C vel represents the uniform velocity constraint of the lidar sensor; t b Represents the translation matrix of the lidar point cloud scanned in the previous frame; t e Represents the translation matrix of the laser radar point cloud scanned in the current frame; n represents the number of scans; q1 and q2 represent the coefficients of the position consistency constraint and the uniform speed constraint respectively;

[0070] like Figure 2 As shown, in this embodiment, an external rotating laser radar and a fixed visible light camera are used. Compared with traditional laser radars, the external rotating laser radar has the advantages of a large field of view and dense point clouds. Therefore, it is necessary to align and accumulate the dynamic single-frame radar point cloud. First, in order to estimate the motion posture of the radar, the CT-ICP algorithm is used to minimize the distance between the scanning target point of the current radar and the interpolated radar posture point to jointly optimize the rough estimated posture at the beginning and end of the scan. For each new scan of the laser radar, this embodiment uses an iVox linear voxel container to save points, and based on the CT-ICP algorithm, the motion distortion of the input point cloud is corrected according to the two optimal postures estimated by the rough estimated posture of the current frame and the rough estimated posture of the previous frame. Then, the points are converted to the radar coordinate system at the time of the first scan and added to the global map to obtain the global static point cloud of the preset scene;

[0071] S3: Based on the initialization external parameters of the given laser radar and camera, a local static point cloud is obtained according to the global static point cloud, and a projective transformation is performed on the obtained local static point cloud to obtain a point cloud azimuth angle map;

[0072] Specifically, the field of view of the global point cloud map generated by an external rotating laser radar is often many times that of a traditional visible light camera. By giving an initial external parameter of a laser radar and a camera, the corresponding radar area with overlapping parts can be determined according to the size of the camera's field of view. Figure 3 is the camera's field of view, such as Figure 3 As shown, the angle α is the field of view angle of the XOY plane in the camera coordinate system, and the angle β is the field of view angle of the YOZ plane; and the angle values ​​of the angles α and β can be calculated by an algorithm based on the two-dimensional image using the camera's intrinsic parameters, camera model, and camera resolution, and the expression of the camera's intrinsic parameters is

[0073]

[0074] Among them, f x ,f y represents the focal length multiplied by the pixel density of the image sensor; c x With c y Represents the principal point of a two-dimensional image (usually the center of the image);

[0075] Horizontal field of view (HF):

[0076]

[0077] Vertical Field of View (VF):

[0078]

[0079] Where d represents the distance from the center of the camera's field of view to the edge corner of the matrix image; imgWidth represents the width of the two-dimensional image; imgHeight represents the height of the two-dimensional image;

[0080] Given an initial external parameter of a laser radar and a camera:

[0081] Constrain the field of view in the camera coordinate system to the radar coordinate system, such as Figure 4 The corresponding field of view in the radar coordinate system is shown, and then the local point cloud in the area is determined; Figure 5 Shown is the local point cloud (local static point cloud) within the field of view of the corresponding camera;

[0082] In a specific embodiment, performing a projection transformation on the acquired local static point cloud in S3 to acquire a point cloud azimuth angle map specifically includes the following steps:

[0083] S31: After filtering out the local point cloud, a projection transformation from the 3D model to the 2D model is required, such as Figure 6As shown, a plane projection is performed on any three-dimensional cloud point in the obtained local static point cloud to obtain a plane projection point cloud, and the expression for performing the plane projection is:

[0084] P0=MP

[0085]

[0086] Where: P0 represents the projection point of any point P in the XYZ-O three-dimensional space in a plane Q parallel to the YOZ plane; N represents the distance between plane Q and the YOZ plane; M represents the plane projection matrix; x0, y0, z0 represent the position coordinates of the corresponding point P0; x, y, z represent the position coordinates of the corresponding point P;

[0087] S32: Gridding the plane projection point cloud to obtain the coordinate cloud point P of the projection point P0 in the set plane Q two-dimensional grid coordinate system Q (a0,b0), its expression is

[0088]

[0089] Where: a represents the number of columns into which the point cloud is divided horizontally in plane Q; b represents the number of rows into which the point cloud is divided vertically in plane Q; W represents the width of the point cloud projected on plane Q; H represents the height of the point cloud projected on plane Q; D represents the side length of the set square grid; y l Represents the Y-axis coordinate of the set plane Q two-dimensional grid coordinate system; z l Represents the Z-axis coordinate of the set plane Q two-dimensional grid coordinate system;

[0090] S33: coordinate cloud point P in the two-dimensional grid coordinate system of plane Q Q (a0, b0) is serialized to obtain optimized point cloud data; Figure 7 As shown, the serialization processing method is to obtain the coordinate cloud points corresponding to each grid in the plane Q two-dimensional grid coordinate system, and obtain the coordinate cloud points closest to the center points of each grid, and use them as the optimized cloud points corresponding to the grid to complete the serialization of the three-dimensional disordered point cloud;

[0091] S34: Figure 8 As shown, the point cloud azimuth is calculated based on the optimized point cloud data, and the calculation formula of the point cloud azimuth is:

[0092]

[0093] Where: θ i,j Indicates the point cloud azimuth; P i,j represents the optimized cloud point in the i-th row and j-th column in the two-dimensional grid coordinate system of the corresponding plane Q; P i-1,j-1represents the optimized cloud point at the i-1th row and j-1th column in the two-dimensional grid coordinate system of the corresponding plane Q; P view Represents the viewpoint of optimizing point cloud data;

[0094] S35: Map the point cloud azimuth to a two-dimensional image grayscale value between [0, 255] to obtain a point cloud azimuth image (Bearing-Angle image, abbreviated as BA image); and the expression for mapping the point cloud azimuth to a two-dimensional image grayscale value is:

[0095]

[0096] Where: img i,j Represents the image grayscale value of the i-th row and j-th column in the two-dimensional grid coordinate system of the corresponding plane Q; the disordered point cloud in three-dimensional space is ordered and converted into a two-dimensional graph model point cloud azimuth graph, such as Fig. 9 Shown is the generated point cloud azimuth diagram;

[0097] S4: Based on the pre-trained cross-modal deep learning network Superglue, feature point matching pairs are obtained according to the point cloud azimuth map and the camera image, which specifically includes the following steps:

[0098] S41: The pre-trained cross-modal deep learning network Superglue includes a SuperPoint deep neural network and a graph neural network, and the pre-trained SuperPoint deep neural network is used to extract the positions of feature points in the point cloud azimuth map and the camera image, respectively, to obtain descriptors of the corresponding feature points, and the descriptors include image feature descriptors and point cloud azimuth descriptors;

[0099] Among them, the position of the feature point is the key point position. The key point refers to the point marked by the superpoint deep neural network, that is, some points in the two images to be matched by features; descriptor is a professional term, often used in the field of feature matching. The feature descriptor is a vector that describes the feature information of the local area around the key point. For example, the SIFT descriptor is a vector calculated based on the position and scale of the local extreme value in the scale space, and the gradient within its neighborhood;

[0100] S42: inputting the image feature descriptor and the point cloud azimuth descriptor into a pre-trained graph neural network, and obtaining a feature point matching pair set of the image feature descriptor and the point cloud azimuth descriptor according to the pre-trained graph neural network; wherein the correspondence relationship of the feature point matching pair refers to finding the same or similar positions of key points, i.e., feature points, in two images;

[0101] In order to find the feature matching relationship between the point cloud azimuth map and the camera image, this embodiment uses the cross-modal deep learning network Superglue, which is an image matching method based on graph neural network. It consists of two parts: Superpoint and graph neural network. The former performs key point detection and can detect key points from the image and extract their descriptors, while the latter can find the corresponding relationship between the key points of different images. In this training, the pre-trained weights of the deep image dataset MegaDepth were used and trained with a small amount of point cloud azimuth maps and traditional image pairing datasets. Although the inferred results can obtain feature matching pairs, there are a large number of false matches;

[0102] S5: By using the rotating RANSAC algorithm, the wrong matching points are eliminated from the acquired feature point matching set pairs to obtain optimized feature point matching pairs, which specifically includes the following steps:

[0103] S51: Set the maximum number of iterations M and the internal point threshold N of the rotation RANSAC algorithm, and initialize the optimal rotation matrix R best and the maximum number of interior points N best ; Define the feature point position K of the feature point matching pair point cloud azimuth descriptor in the set as {x k1 ,x k2 ,...,x kn}, the feature point position D of the image feature descriptor in the feature point matching pair = {L pk1 ,L pk2 ,...,L pkn};

[0104] S52: Randomly select any two sets of feature points (x ki ,L pki ) and (x kj ,L pkj ), calculate and obtain the current rotation matrix R ij , whose expression is

[0105]

[0106] Where: I represents the unit matrix; K represents the antisymmetric matrix;

[0107] S53: Get the current rotation matrix R ij As the optimal rotation matrix R best , through the optimal rotation matrix R best Rotate the remaining feature points in the feature point matching pair to obtain the current feature point;

[0108] Get the current feature distance between the current feature point and other corresponding feature points, and determine the size of the current feature distance and the inner point threshold N;

[0109] If the current feature distance is less than or equal to the inlier threshold N, the feature point corresponding to the current feature distance is stored in the current inlier set, and step S54 is executed; if the current feature distance is greater than the inlier threshold N, the feature point corresponding to the current feature distance is removed;

[0110] S54: Determine the number of interior points in the current interior point set and the maximum number of interior points N best size;

[0111] If the number of interior points in the current interior point set is less than or equal to the maximum number of interior points N best , then return to step S52;

[0112] Otherwise, the number of inliers in the current inlier set is updated to the maximum number of inliers N in the current iteration. best , and return to step S52 until the maximum number of iterations M is reached, and the final optimal rotation matrix R is obtained. best The obtained internal point set is used to obtain the optimized feature point matching pair.

[0113] S6: The Levenberg-Marquardt optimizer is used to minimize the reprojection error of the feature point matching pair to obtain the lidar-camera transformation matrix, which is the external parameter of the lidar-camera, including the rotation matrix R and the translation matrix t to realize the online calibration of the lidar-camera without the target object, and the expression of the lidar-camera transformation matrix is ​​obtained as follows:

[0114]

[0115] Where: represents the six-degree-of-freedom laser radar-camera transformation matrix; n represents the number of feature point groups in the optimized feature point matching pair; ρ represents the abbreviated form of ρ(r), which is the Cauchy robust kernel function; r represents the residual; c represents an adjustment parameter used to control the degree of suppression of outliers. By using this kernel function, the optimization process is no longer sensitive to large errors (usually caused by outliers, where outliers are relative to inliers, and refer to matching pairs that do not meet the model assumptions or whose errors exceed a predefined threshold), thereby improving the robustness of the registration. Figure 10 to Figure 11 is based on the final lidar-camera transformation The result of projecting two frames of external rotation 3D lidar point clouds at different times into a 2D image.

[0116] This embodiment obtains the optimal pose based on the rough estimated pose of the current frame and the rough estimated pose of the previous frame based on the CT-ICP algorithm, and converts each optimal pose into the lidar point cloud scanned by the initial frame to obtain the global static point cloud of the preset scene, and obtains the local static point cloud by giving the initialization external parameters of the lidar and the camera, and performs a projection transformation on the obtained local static point cloud to obtain the point cloud azimuth map; by using the point cloud azimuth map, it has the advantages of strong ability to depict details, small distortion, rich texture details, etc., which is closer to the traditional two-dimensional image, and can improve the matching accuracy and Reliability, and at the same time, the local point cloud scene is used for feature extraction and matching. By determining the local scene corresponding to the image in the global static point cloud scene, calibration samples are provided for joint calibration to greatly improve the calibration efficiency; the obtained feature point matching set pairs are picked out by the rotating RANSAC algorithm, and the Levenberg-Marquardt optimizer is used to minimize the reprojection error of the optimized feature point matching pairs to obtain the lidar-camera transformation matrix, which greatly improves the accuracy and efficiency of the lidar-camera online calibration without targets.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A laser radar-camera non-target online calibration method based on point cloud azimuth diagram, characterized in that: The following steps are involved: S1: Obtain several frames of laser radar point clouds and camera images scanned under the same preset scene; and interpolate the laser radar point clouds of each frame to obtain the interpolated laser radar attitude point cloud of the corresponding frame; S2: Based on the CT-ICP algorithm, the rough estimated pose of the current frame is obtained according to the interpolated lidar pose point cloud of each frame and the preset lidar target point cloud, and the optimal pose is obtained according to the rough estimated pose of the current frame and the rough estimated pose of the previous frame; and each optimal pose is converted to the lidar point cloud scanned by the initial frame to obtain the global static point cloud of the preset scene; S3: Based on the initialization external parameters of the given laser radar and camera, a local static point cloud is obtained according to the global static point cloud, and a projective transformation is performed on the obtained local static point cloud to obtain a point cloud azimuth angle map; S4: Based on the pre-trained cross-modal deep learning network Superglue, feature point matching pairs are obtained according to the point cloud azimuth map and camera image; S5: Using the rotational RANSAC algorithm, the wrong matching points are removed from the acquired feature point matching set pairs to obtain optimized feature point matching pairs; S6: The Levenberg-Marquardt optimizer is used to minimize the reprojection error of the optimized feature point matching pairs to obtain the lidar-camera transformation matrix to achieve online calibration of the lidar-camera without targets.

2. The method for online calibration of a laser radar-camera without a target based on a point cloud azimuth diagram according to claim 1, characterized in that: The expression for obtaining the optimal posture described in S2 is: Where: Indicates the optimal pose of the lidar point cloud scanned in the previous frame; Indicates the optimal pose of the lidar point cloud scanned in the current frame; L b Indicates the rough estimated pose of the lidar point cloud scanned in the previous frame; L e Indicates the rough estimated pose of the lidar point cloud scanned in the current frame; F ICP represents the ICP iterative algorithm; C loc represents the position consistency constraint of the lidar sensor; C vel represents the uniform velocity constraint of the lidar sensor; t b Represents the translation matrix of the lidar point cloud scanned in the previous frame; t e Represents the translation matrix of the lidar point cloud scanned in the current frame; n represents the number of scans; q1 and q2 represent the coefficients of the position consistency constraint and the uniform speed constraint, respectively.

3. The method for online calibration of a laser radar-camera without a target based on a point cloud azimuth diagram according to claim 2, characterized in that: The step of performing a projection transformation on the acquired local static point cloud to obtain a point cloud azimuth angle map described in S3 specifically includes the following steps: S31: Performing a plane projection on any three-dimensional cloud point in the acquired local static point cloud to obtain a plane projection point cloud, and the expression for performing the plane projection is P0=MP Where: P0 represents the projection point of any point P in the XYZ-O three-dimensional space in a plane Q parallel to the YOZ plane; N represents the distance between plane Q and the YOZ plane; M represents the plane projection matrix; x0, y0, z0 represent the position coordinates of the corresponding point P0; x, y, z represent the position coordinates of the corresponding point P; S32: Gridding the plane projection point cloud to obtain the coordinate cloud point P of the projection point P0 in the set plane Q two-dimensional grid coordinate system Q (a0,b0), its expression is Where: a represents the number of columns into which the point cloud is divided horizontally in plane Q; b represents the number of rows into which the point cloud is divided vertically in plane Q; W represents the width of the point cloud projected on plane Q; H represents the height of the point cloud projected on plane Q; D represents the side length of the set square grid; y l Represents the Y-axis coordinate of the set plane Q two-dimensional grid coordinate system; z l Represents the Z-axis coordinate of the set plane Q two-dimensional grid coordinate system; S33: coordinate cloud point P in the two-dimensional grid coordinate system of plane Q Q (a0, b0) is serialized to obtain optimized point cloud data; and the serialization method is to obtain the coordinate cloud points corresponding to each grid in the plane Q two-dimensional grid coordinate system, and obtain the coordinate cloud point closest to the center point of each grid, and use it as the optimized cloud point corresponding to the grid; S34: Calculate the point cloud azimuth according to the optimized point cloud data, and the calculation formula of the point cloud azimuth is: Where: θ i,j Indicates the point cloud azimuth; P i,j represents the optimized cloud point in the i-th row and j-th column in the two-dimensional grid coordinate system of the corresponding plane Q; P i-1,j-1 represents the optimized cloud point at the i-1th row and j-1th column in the two-dimensional grid coordinate system of the corresponding plane Q; P view Represents the viewpoint of optimizing point cloud data; S35: Map the point cloud azimuth angle into a two-dimensional image grayscale value to obtain a point cloud azimuth angle map; and the expression for mapping the point cloud azimuth angle into a two-dimensional image grayscale value is: Where: img i,j Represents the grayscale value of the image in the i-th row and j-th column in the two-dimensional grid coordinate system of the corresponding plane Q.

4. The method for online calibration of a laser radar-camera without a target based on a point cloud azimuth diagram according to claim 3, characterized in that: The S4 specifically comprises the following steps: S41: The pre-trained cross-modal deep learning network Superglue includes a SuperPoint deep neural network and a graph neural network, and the pre-trained SuperPoint deep neural network is used to extract the positions of feature points in the point cloud azimuth map and the camera image, respectively, to obtain descriptors of the corresponding feature points; And the descriptor includes an image feature descriptor and a point cloud azimuth descriptor; S42: Input the image feature descriptor and the point cloud azimuth descriptor into a pre-trained graph neural network, and obtain a feature point matching pair set of the image feature descriptor and the point cloud azimuth descriptor according to the pre-trained graph neural network.

5. The method for online calibration of a laser radar-camera without a target based on a point cloud azimuth diagram according to claim 3, characterized in that: The S5 specifically includes the following steps: S51: Set the maximum number of iterations M and the internal point threshold N of the rotation RANSAC algorithm, and initialize the optimal rotation matrix R best and the maximum number of interior points N best ; Define the feature point position K of the feature point matching pair point cloud azimuth descriptor in the set as {x k1 ,x k2 ,...,x kn }, the feature point position D of the image feature descriptor in the feature point matching pair = {L pk1 ,L pk2 ,...,L pkn }; S52: Randomly select any two sets of feature points (x ki ,L pki ) and (x kj ,L pkj ), calculate and obtain the current rotation matrix R ij , whose expression is Where: I represents the unit matrix; K represents the antisymmetric matrix; S53: Get the current rotation matrix R ij As the optimal rotation matrix R best , through the optimal rotation matrix R best Rotate the remaining feature points in the feature point matching pair to obtain the current feature point; Get the current feature distance between the current feature point and other corresponding feature points, and determine the size of the current feature distance and the inner point threshold N; If the current feature distance is less than or equal to the inlier threshold N, the feature point corresponding to the current feature distance is stored in the current inlier set, and step S54 is executed; if the current feature distance is greater than the inlier threshold N, the feature point corresponding to the current feature distance is removed; S54: Determine the number of interior points in the current interior point set and the maximum number of interior points N best size; If the number of interior points in the current interior point set is less than or equal to the maximum number of interior points N best , then return to step S52; Otherwise, the number of inliers in the current inlier set is updated to the maximum number of inliers N in the current iteration. best , and return to step S52 until the maximum number of iterations M is reached, and the final optimal rotation matrix R is obtained. best The obtained internal point set is used to obtain the optimized feature point matching pair.

6. The method for online calibration of a laser radar-camera without a target based on a point cloud azimuth diagram according to claim 3, characterized in that: In S6, the Levenberg-Marquardt optimizer is used to minimize the reprojection error of the optimized feature point matching pairs, and the expression of the lidar-camera transformation matrix is ​​obtained as follows: Where: represents the six-degree-of-freedom lidar-camera transformation matrix; n represents the number of feature point groups in the optimized feature point matching pair; ρ represents the abbreviated form of ρ(r), which is the Cauchy robust kernel function; r represents the residual; c represents an adjustment parameter.

Images