A method, apparatus, and intelligent terminal for lidar point cloud and image registration.

Abstract

This invention discloses a method for registering point clouds and images from a vehicle-mounted LiDAR. The method includes: acquiring point cloud data collected by a vehicle-mounted LiDAR and image data collected by a camera in the same time and space; obtaining three-dimensional point cloud features of a checkerboard boundary based on the point cloud data; obtaining two-dimensional features of the checkerboard boundary based on the image data; performing registration based on the three-dimensional point cloud features and the two-dimensional features of the checkerboard boundary to obtain an external calibration matrix; and performing point cloud registration based on the external calibration matrix. This invention can automatically process data from different dimensions measured by LiDAR and cameras using various algorithms to reduce errors and achieve high-precision registration using a checkerboard pattern.

Description

Technical Field

[0001] This invention relates to the field of visual measurement, and more specifically to a method, apparatus, and intelligent terminal for registering lidar point clouds with images. Background Technology

[0002] Cameras and LiDAR provide supplementary information about the environment. Cameras capture color, texture, and appearance information, while LiDAR provides 3D structural information about the environment. Therefore, they are often used together to perceive the environment. External parameters consisting of rotation and translation are prerequisites for fusing information from the two sensors. With the widespread use of unmanned mechanical devices, the number of vehicles equipped with LiDAR and cameras is rapidly increasing. The point cloud scenarios for autonomous driving perception tasks are diverse, and perception tasks based on autonomous driving scenarios are relatively complex. Moreover, the point cloud attribute information determines the accuracy of the perception task. Therefore, large-scale commercial applications require an effective and efficient external calibration algorithm.

[0003] Since laser point clouds are three-dimensional data while images are two-dimensional data, the two have different dimensions, necessitating registration between laser radar point cloud data and camera images. Registration between laser radar and camera can be achieved by converting the three-dimensional laser point cloud data into two-dimensional image data, and then projecting the three-dimensional point cloud onto a two-dimensional plane by solving a high-precision rigid body transformation matrix. However, existing methods require manual selection of registration points within a checkerboard pattern during joint calibration, which is cumbersome and lacks a system workflow that can be automated simply by inputting data. Furthermore, there is a lack of processing to address the impact of camera field of view and distortion on the results. Additionally, given the need to sample multiple sets of data to complete the registration process, there is a lack of error assessment and analysis for registration quality. Current registration methods also include converting two-dimensional image data into three-dimensional data. However, because images lack depth information, even images taken with depth cameras have low precision in depth information, making it difficult to accurately reconstruct three-dimensional data.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method for registering point clouds and images of vehicle-mounted LiDAR, which addresses the above-mentioned deficiencies of the prior art. The method aims to solve the problems of cumbersome manual registration, lack of error assessment and analysis of registration quality, and low accuracy of depth information in the prior art.

[0006] The technical solution adopted by this invention to solve the technical problem is as follows:

[0007] In a first aspect, the present invention provides a method for registering point clouds and images of a vehicle-mounted lidar, wherein the method includes:

[0008] Acquire point cloud data collected by vehicle-mounted LiDAR and image data collected by camera in the same time and space;

[0009] Based on the point cloud data, the three-dimensional point cloud features of the chessboard grid boundary are obtained;

[0010] Based on the image data, the two-dimensional features of the chessboard boundary are obtained;

[0011] The external calibration matrix is ​​obtained by registering the three-dimensional point cloud features of the chessboard boundary and the two-dimensional features of the chessboard boundary.

[0012] Point cloud registration is performed based on the external calibration matrix.

[0013] In one implementation, after acquiring the point cloud data collected by the vehicle-mounted lidar and the image data collected by the camera in the same time and space, the method further includes:

[0014] The point cloud data and image data are preprocessed to obtain processed point cloud data and image data; wherein, the data preprocessing includes timestamp synchronization and distortion correction.

[0015] In one implementation, obtaining the 3D point cloud features of the checkerboard boundary based on the point cloud data includes:

[0016] Noise in the point cloud data is removed using a method based on neighborhood density distribution to obtain a noise-free point cloud file frame.

[0017] Clustering methods are used to locate the noise-free point cloud file frames to obtain checkerboard point cloud clusters;

[0018] The optimal plane of the chessboard grid point cloud cluster is fitted using the RANSAC method to obtain the optimal cloud plane of the chessboard grid.

[0019] The boundary of the optimal point cloud plane of the chessboard is detected to obtain the three-dimensional point cloud features and three-dimensional corner points of the chessboard boundary.

[0020] In one implementation, the step of fitting the optimal plane of the chessboard grid point cloud cluster using the RANSAC method to obtain the optimal chessboard grid cloud plane includes:

[0021] The parameter constraint model is constructed as follows:

[0022]

[0023] Where A, B, C, and D are the estimated parameters of the fitted plane, and the point set P c It is a cluster of checkerboard-like dots, P' c Let ε be the first optimal plane, and let ε be the threshold for iteration.

[0024] The shrinkage threshold ε is used, and the model is repeatedly fitted and filtered according to the parameter constraints. When all point clouds are within the iterative threshold ε, the noisy point cloud is projected onto the ideal plane using the projection transformation model, resulting in the second optimal plane:

[0025] P'' c ={(x0t,y0t,z0t)|(x0,y0,z0)∈P' c}

[0026] Where t = -D / (Ax0 + By) 0+ Cz0);

[0027] The second optimal plane is divided into multiple rectangular grids, and the point cloud in each rectangular grid is randomly sampled to obtain the chessboard optimal point cloud plane.

[0028] In one implementation, obtaining the two-dimensional features of the checkerboard boundary based on the image data includes:

[0029] The internal corner points of the chessboard are detected by element feature points, and the chessboard is located by similarity, thus obtaining the two-dimensional features of the chessboard boundary.

[0030] The image data is detected using a chessboard corner detection algorithm to obtain two-dimensional corner points;

[0031] The two-dimensional corner points are sorted to obtain the index relationship.

[0032] In one implementation, the step of registering the three-dimensional point cloud features of the checkerboard boundary and the two-dimensional features of the checkerboard boundary to obtain the external calibration matrix includes:

[0033] The point cloud data of the three-dimensional point cloud features of the chessboard boundary are unfolded onto a two-dimensional plane to obtain two-dimensional point cloud features.

[0034] Based on the index relationship, the association relationship between the three-dimensional and two-dimensional corner point pairs of the chessboard boundary is obtained;

[0035] Based on the aforementioned correlation, the two-dimensional point cloud features and the two-dimensional features of the checkerboard boundary in each frame of data are registered to obtain an external calibration matrix.

[0036] In one implementation, the step of registering the two-dimensional point cloud features and the two-dimensional features of the checkerboard boundary in each frame of data according to the association relationship to obtain an external calibration matrix includes:

[0037] Based on the pose relationship between the lidar coordinate system and the camera coordinate system, the three-dimensional corner points and two-dimensional corner points in any frame of data are solved using the RANSCA-based PnP pose estimation algorithm to obtain the initial solution matrix.

[0038] Based on the initial solution matrix, a three-dimensional to two-dimensional projection is performed on each frame of data to obtain a two-dimensional chessboard estimate.

[0039] The two-dimensional features of the chessboard boundary are used as pseudo-true values. Based on the two-dimensional chessboard estimation and pseudo-true values, the reprojection error is calculated, and a target threshold for the error is set.

[0040] The reprojection error is compared with the target error threshold. If the reprojection error is greater than the target error threshold, the two-dimensional chessboard estimate corresponding to the reprojection error is deleted, and the initial solution matrix is ​​updated.

[0041] The steps of re-exercising the three-dimensional and two-dimensional corner points in any frame of data and using the RANSCA-based PnP pose estimation algorithm to obtain the initial solution matrix are repeated until the reprojection error in each frame of data is less than or equal to the error target threshold, thus obtaining the external calibration matrix.

[0042] Secondly, embodiments of the present invention also provide a vehicle-mounted lidar point cloud and image registration device, wherein the device includes:

[0043] The data and image acquisition module is used to acquire point cloud data collected by the vehicle-mounted LiDAR and image data collected by the camera in the same time and space.

[0044] The three-dimensional point cloud feature acquisition module is used to obtain the three-dimensional point cloud features of the chessboard boundary based on the point cloud data.

[0045] A two-dimensional feature acquisition module is used to obtain two-dimensional features of the checkerboard boundary based on the image data;

[0046] The external calibration matrix acquisition module is used to register the three-dimensional point cloud features of the chessboard boundary and the two-dimensional features of the chessboard boundary to obtain the external calibration matrix.

[0047] The registration module is used to perform point cloud registration based on the external calibration matrix.

[0048] Thirdly, embodiments of the present invention also provide a smart terminal, wherein the smart terminal includes a memory, a processor, and a vehicle-mounted LiDAR point cloud and image registration program stored in the memory and executable on the processor. When the processor executes the vehicle-mounted LiDAR point cloud and image registration program, it implements the steps of the vehicle-mounted LiDAR point cloud and image registration method as described in any of the above claims.

[0049] Fourthly, embodiments of the present invention also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a vehicle-mounted LiDAR point cloud and image registration program, and when the vehicle-mounted LiDAR point cloud and image registration program is executed by a processor, it implements the steps of the vehicle-mounted LiDAR point cloud and image registration method as described in any of the above claims.

[0050] Beneficial Effects: Compared with existing technologies, this invention provides a method for registering point clouds and images from vehicle-mounted LiDAR. First, multiple pairs of corresponding points between checkerboard boundary features in the LiDAR point cloud data and image data are selected and iteratively registered. By unifying the spatiotemporal context, the 3D point cloud data and 2D image data are synchronized. Then, based on the multiple pairs of corresponding point data, a RANSCA-based PnP algorithm is used to obtain an external calibration matrix. This external calibration matrix forms the basis for automated registration, and error data is adjusted by iteratively updating the external calibration matrix. Finally, point cloud registration is performed using the external calibration matrix, thus achieving automated, high-precision registration even when the position and attitude parameters of the panoramic image are unknown. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a schematic diagram of the process of vehicle-mounted lidar point cloud and image registration method provided in an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of the RANSAC method provided in the embodiments of the present invention.

[0054] Figure 3 This is a schematic diagram of the coordinate system correspondence provided in an embodiment of the present invention.

[0055] Figure 4 This is a schematic diagram of the principle of the vehicle-mounted lidar point cloud and image registration device provided in the embodiment of the present invention.

[0056] Figure 5 This is a block diagram illustrating the internal structure of a smart terminal provided in an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0058] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0059] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0060] Multi-sensor registration has always been essential for robotics and autonomous driving systems to accurately perceive the surrounding environment. Among the registration of different types of sensors, the most common is the combination of LiDAR and optical cameras, especially in vision-based ranging and mapping tasks, object detection, and tracking. Perception tasks utilize various sensors (LiDAR / cameras) installed on vehicles to sense the surrounding environment, collect data, and perform system calculations and analyses while the vehicle is in motion. Since LiDAR point clouds are three-dimensional data while images are two-dimensional data, their dimensions differ. This invention provides a method and system for registering vehicle-mounted LiDAR point clouds and images. It can automatically process data from LiDAR and cameras of different dimensions using various algorithms to reduce errors, and then perform high-precision registration using a checkerboard pattern.

[0061] Exemplary methods

[0062] This embodiment provides a method for registering point clouds and images using a vehicle-mounted LiDAR system. For example... Figure 1 As shown, the method includes the following steps:

[0063] Step S100: Acquire point cloud data collected by vehicle-mounted lidar and image data collected by camera in the same time and space;

[0064] Specifically, cameras and LiDAR provide supplementary information about the environment. Cameras capture color, texture, and appearance information, while LiDAR provides three-dimensional structural information about the environment. Therefore, they are often used together to perceive the environment. Extrinsic parameters consisting of rotation and translation are prerequisites for fusing information from both sensors. This embodiment reads point cloud and image files corresponding to the same time and space, along with the camera intrinsic parameter matrix.

[0065] Specifically, a certain type of LiDAR and camera are mounted on a vehicle to scan the real-world scene, capturing multiple sets of point cloud and image data. When acquiring point cloud data, the distance to the sensor is 5-10 meters, the checkerboard is tilted at 45 degrees, and the calibration board is placed diagonally, with its plane as perpendicular as possible to the sensor to expose more of its area to the LiDAR sensor's scan lines, increasing the amount of point cloud projected onto the calibration board. The front grid points must not be obstructed, and the calibration board must be unobstructed, free of reflections, and without shadows. Data between the LiDAR and camera sensors must be time-synchronized, meaning images and point clouds from the same frame are captured simultaneously from both sensors, capturing data without motion blur.

[0066] In one implementation, after step S200 of this embodiment, the method further includes:

[0067] The point cloud data and image data are preprocessed to obtain processed point cloud data and image data; wherein, the data preprocessing includes timestamp synchronization and distortion correction.

[0068] Specifically, in this embodiment, the multi-frame point cloud data and image data acquired have noise and outliers when the LiDAR acquires point cloud data with a checkerboard pattern to be calibrated. That is, there is overlap and accumulation of point clouds within a certain range. Therefore, the acquired file data to be registered needs to be preprocessed first. File preprocessing includes randomly extracting a preset number of random data frames. For each frame of random data, the point cloud image in the random data is adjusted to be consistent with the camera image. Consistency requires timestamp synchronization. Only point cloud and image data in the same time and space can be registered.

[0069] Specifically, distortion correction is performed on images with a preset number of frames:

[0070] The image plane coordinates formed by the aforementioned spatial points are deviated by lens distortion or similar effects; therefore, the influence of camera distortion can be expressed by the following formula:

[0071]

[0072] Where, r 2 =x 2 +y 2 x and y are the actual imaging point coordinates, x′ and y′ are the expected imaging point coordinates, and k1, k2, k3, p1, and p2 are the camera model distortion parameters.

[0073] Step S200: Based on the point cloud data, obtain the three-dimensional point cloud features of the chessboard boundary;

[0074] Specifically, this embodiment starts by converting the three-dimensional data of laser point cloud into two-dimensional data of image, and analyzes and obtains the current joint calibration of lidar and camera using a checkerboard pattern.

[0075] In one implementation, step S200 of this embodiment includes the following steps:

[0076] Step S201: Remove noise from the point cloud data using a method based on neighborhood density distribution to obtain a noise-free point cloud file frame;

[0077] Specifically, in this embodiment, random point cloud data of a preset number of frames are analyzed respectively to calculate a noise-free point cloud file that conforms to the density distribution of the point cloud domain;

[0078] The calculation of the density distribution in the point cloud domain includes:

[0079] m-frame point cloud: Q = {Q1, Q2, ..., Q} m}; The number of point clouds in a single frame file is n: (frame i) i∈m, Q i ={q1, q2, ..., q n}; Single point cloud q x (x∈n) Neighborhood threshold: k; Scaling factor: Sc;

[0080] Step 1: For any t∈n, q t ∈Q i , where in a single point cloud q t Calculate the distance between all point clouds within the Euclidean distance threshold k and q. t The average Euclidean distance, Ave;

[0081] Step 2: For a single-frame point cloud Q i ={q1, q2, ..., q n}, note AVE i ={Ave1, Ave2, ..., Ave n};

[0082] Step 3: For Q i, Te = Aave + Sc * STDave, where Aave is the average value of {Ave1, Ave2,..., Ave n}, and STDave is the standard deviation of {Ave1, Ave2,..., Ave n}.

[0083]

[0084]

[0085] Step Four: For any t ∈ n, q t ∈ Q i , if its Ave < Te, then remove q from Q i . t

[0086] Step Five: Go back to Step One until all m frame point clouds are processed.

[0087] Use the method based on domain density distribution to statistically remove the noisy point cloud data in each incoming frame, making the density distribution of points more uniform, so as to obtain a noise-free point frame in the point cloud frame group.

[0088] Step S202: Use the clustering method to locate the noise-free point cloud file frame to obtain a checkerboard point cloud cluster;

[0089] Step S203: Use the RANSAC method to fit the optimal plane of the checkerboard point cloud cluster to obtain the optimal point cloud plane of the checkerboard; <{}

[0090] Step S204: Detect the boundary of the optimal point cloud plane of the checkerboard to obtain the three-dimensional point cloud features and three-dimensional corner points of the boundary of the checkerboard.

[0091] Specifically, detect the candidate clusters containing the checkerboard, sort the clusters through the standard object similarity metric L to measure the difference between the clusters and the calibration target, and only keep the cluster with the smallest difference as the located checkerboard.

[0092]

[0093] ​Where n is the number of candidate checkerboard point cloud clusters, L1 is the similarity metric of the checkerboard point cloud clusters regarding the position of the points within them, and L2 is the similarity metric of the intensity of the points within the checkerboard point cloud clusters. Considering that after the checkerboard is segmented from the background, background points may be connected to the checkerboard, we first remove these points. It is necessary to determine the upper and lower boundaries of the checkerboard boundary points along the z-axis in the lidar coordinate system. For the upper boundary, the lower boundary is calculated based on the width distribution of the checkerboard points along the z-axis and the checkerboard size constraint: due to systematic measurement errors, the point cloud scatters along the axis, while ideally, all points should fall on the checkerboard plane, making it difficult to estimate accurate boundary coordinates. Therefore, we design an iterative refinement method to fit the point cloud to the ideal plane containing the checkerboard. In each iteration, such as... Figure 2 As shown, this embodiment first uses a RANSAC-based method to fit the optimal plane. Point cloud coordinates where the distance from the point cloud to the plane is less than a threshold are preserved.

[0094] In one implementation, step S203 of this embodiment includes the following steps:

[0095] Step S2031: Construct the parameter constraint model as follows:

[0096]

[0097] Where A, B, C, and D are the estimated parameters of the fitted plane. Point set P c It is a cluster of checkerboard-like dots, P' c Let ε be the first optimal plane, and let ε be the threshold for iteration.

[0098] Step S2032: Shrink the threshold ε and repeat the fitting and filtering process according to the parameter constraints of the model. When all point clouds are within the iterative threshold ε, use the projection transformation model to project the noisy point clouds onto the ideal plane to obtain the second optimal plane as follows:

[0099] P'' c ={(x0t,y0t,z0t)|(x0,y0,z0)∈P' c},

[0100] Where t = -D / (Ax0 + By) 0+ Cz0);

[0101] Step S2033: Divide the second optimal plane into multiple rectangular grids, and randomly sample the point cloud in each rectangular grid to obtain the chessboard optimal point cloud plane.

[0102] Specifically, such as Figure 2As shown, the RANSAC method uses four sets of point pairs for sampling, ensuring that the initial dataset contains the sampling points, and performs corresponding fitting of the model through parameter constraints. Within the limits of the mathematical model's critical threshold, the set of interior points meeting the threshold requirement is added to the largest set of interior points. The steps are as follows:

[0103]

[0104] Where A, B, C, and D are the estimated parameters of the fitted plane. Now, the point set P c It is a cluster of original chessboard grid points, P' c We get closer to the ideal checkerboard plane than Pc, then we shrink the threshold ε and repeat the fitting and filtering process.

[0105] The calculation stops when all point clouds are within the threshold ε of the current iteration. Based on the ranging principle of lidar, the ranging measurement of the laser beam may include errors in the radial direction (from the obstacle to the sensor center), while errors in the azimuth direction can be ignored. Based on this experience, a projection transformation model is then used to project the noisy point cloud onto an ideal plane.

[0106] P'' c ={(x0t,y0t,z0t)|(x0,y0,z0)∈P' c}

[0107] in:

[0108] t = -D / (Ax0 + By) 0+ Cz0)

[0109] To reduce the impact of point cloud non-uniformity, we will uniformly apply P'' c The data is divided into multiple grids, and then random sampling is performed on each grid to maintain the range of point cloud density.

[0110] The point cloud within the checkerboard pattern is divided into multiple rectangular grids. Random sampling is performed on the point cloud within each grid, and this process is repeated. By combining the sampled grids, this embodiment obtains a checkerboard point cloud with uniform density. This operation reduces the influence of high-density regions during subsequent corner fitting and yields better overall feature point detection results.

[0111] Step S300: Obtain the two-dimensional features of the chessboard boundary based on the image data;

[0112] In one implementation, step S300 of this embodiment includes the following steps:

[0113] Step S301: Detect the internal corner points of the chessboard grid by element feature points, and locate the chessboard grid by similarity to obtain the two-dimensional features of the chessboard grid boundary;

[0114] Step S302: Detect the image data using a chessboard corner detection algorithm to obtain two-dimensional corner points;

[0115] Step S303: Sort the two-dimensional corner points to obtain the index relationship.

[0116] Specifically, this embodiment performs two-dimensional corner detection from image data. A chessboard corner detection algorithm is used to detect two-dimensional corners in the image, while the image file corresponds one-to-one with a file processed from the point cloud's neighborhood density distribution data obtained from the LiDAR. Since the corners are symmetrically distributed along the diagonal of the chessboard grid, the order of the detected corners in the image and point cloud may be ambiguous. This embodiment reorders the detected corners, indexing them from the bottom left corner.

[0117] Step S400: Register the three-dimensional point cloud features of the chessboard boundary and the two-dimensional features of the chessboard boundary to obtain the external calibration matrix;

[0118] In one implementation, step S400 of this embodiment includes the following steps:

[0119] Step S401: Unfold the point cloud data of the three-dimensional point cloud features of the chessboard boundary onto a two-dimensional plane to obtain two-dimensional point cloud features;

[0120] Step S402: Based on the index relationship, obtain the association relationship between the three-dimensional and two-dimensional corner point pairs of the chessboard boundary;

[0121] Step S403: Register the two-dimensional point cloud features and the two-dimensional features of the checkerboard boundary in each frame of data according to the association relationship to obtain the external calibration matrix.

[0122] Specifically, this embodiment determines the association relationship between four sets of three-dimensional and two-dimensional corner point pairs of the chessboard boundary based on the index relationship, and registers the point cloud and image in each frame of data according to the association relationship. For example... Figure 2 As shown, the joint calibration principle of lidar and camera is to transform the checkerboard target in the same scene through rotation and translation matrices, and project it from the lidar coordinate system onto a two-dimensional image in the camera coordinate system. This projection process can be explained by imaging transformation.

[0123] Specifically, such as Figure 3 As shown, the world coordinate system consists of X w Y w Z w Axis configuration. Because the camera can be placed anywhere in the real scene, a reference coordinate system must be selected in the real scene to represent the camera's position, and then this reference system is used to characterize the target in the real scene.

[0124] Therefore, the world coordinate system can be directly regarded as the lidar coordinate system, that is, the origin of the world coordinate system coincides with the origin of the lidar coordinate system, i.e., (X w Y w Z w Let be the point cloud coordinates, and according to the relationship between the world coordinate system and the camera coordinate system, as shown in the following equation:

[0125]

[0126] Where P(x, y, z) is a point in the camera coordinate system in space, R is a 3×3 orthogonal identity matrix, t is a three-dimensional translation vector, and O = (0, 0, 0). T M2 is the 4×4 external calibration matrix.

[0127] In one implementation, step S403 of this embodiment includes the following steps:

[0128] Step S4031: Based on the pose relationship between the lidar coordinate system and the camera coordinate system, the three-dimensional corner points and two-dimensional corner points in any frame of data are solved using the RANSCA-based PnP pose estimation algorithm to obtain the initial solution matrix.

[0129] Step S4032: Based on the initial solution matrix, perform a three-dimensional-two-dimensional projection on each frame of data to obtain a two-dimensional chessboard estimate;

[0130] Step S4033: Use the two-dimensional features of the chessboard boundary as the pseudo-true value, and calculate the reprojection error based on the two-dimensional chessboard estimation and the pseudo-true value, and set the error target threshold.

[0131] Step S4034: Compare the reprojection error with the target error threshold. If the reprojection error is greater than the target error threshold, delete the two-dimensional chessboard estimate corresponding to the reprojection error and update the initial solution matrix.

[0132] Step S4035: Re-execute the step of using the RANSCA-based PnP pose estimation algorithm to obtain the initial solution matrix for the three-dimensional corner points and two-dimensional corner points in any frame of data, until the reprojection error in each frame of data is less than or equal to the error target threshold, and obtain the external calibration matrix.

[0133] Specifically, the pose relationship between the lidar coordinate system and the camera coordinate system is calculated and obtained through checkerboard calibration. Extrinsic parameters are then solved iteratively.

[0134]

[0135] Where u and v are pixel coordinates, X w Yw Z w The point cloud coordinates are in the lidar coordinate system. The scale factor is on the u-axis; R is the scale factor on the v-axis; M1 is the camera intrinsic parameter matrix; M2 is the rotation and translation matrix, where R is the rotation matrix from the lidar coordinate system to the camera coordinate system, and t is the translation matrix. M is a 3×3 projection matrix composed of the camera intrinsic parameters and the rotation and translation matrices.

[0136] Due to the perspective projection of the camera, the distance from which the chessboard is placed may affect the scale of the reprojected points (for example, if placed too far away, the reprojection error is less than the actual value). Therefore, this embodiment uses distance normalization to rescale the reprojection to reduce this deviation. The specific steps are as follows:

[0137] Step 1:

[0138] First, the point cloud data in the LiDAR coordinate system is mapped onto the pixel plane to obtain planar data. Simultaneously, the checkerboard targets identified in the camera coordinate system are matched with the checkerboard point cloud identified in the LiDAR coordinate system. Since the LiDAR has a larger FOV (field of view) than the camera, point clouds not falling within the image resolution are filtered out. Specifically, the 3D point cloud data in the LiDAR coordinate system is unfolded into a 2D plane, point clouds outside the image plane are filtered out, and the coordinates in the unconverted preset coordinate system are preserved, i.e., radial distance coordinates are ignored.

[0139] Step Two:

[0140] Considering the limited sample size of the checkerboard measurement, for the obtained 3D-2D corner index pairs, PnP based on RANSCA is used to obtain the initial solution matrix E0; the process is as follows:

[0141] (1) First, given the intrinsic parameters of the camera (M1), select three pairs of associated points and substitute them into the following formula to transform the solution into solving the E0 matrix:

[0142]

[0143] Where, X c Y c Z c These are the point cloud coordinates in the camera coordinate system.

[0144] Then, based on the calculated E0 matrix, three-dimensional-two-dimensional projection is performed on other sample frames;

[0145] Projection error calculation:

[0146]

[0147] in:

[0148]

[0149] (X ci ,Y ci Z ci (X) represents the actual coordinates. ci ',Y ci ',Z ci ') represents the coordinates of the corresponding projected imaging point, δx i δy i and δz i Let n be the error value, n be the number of projected point pairs, and ∝ be the target error value.

[0150] Based on the camera's intrinsic parameter matrix M1 and the estimated extrinsic matrix M2, the chessboard-estimated 3D point cloud is projected onto a 2D space. Then, feature points from the 2D image are treated as pseudo-true values ​​to calculate the reprojection error (i.e., the difference between the projected 3D point cloud values ​​and the 2D points in the image).

[0151] Step 3:

[0152] Delete samples with high reprojection errors, recalculate the E0 matrix, return to step two, and continue repeating the PnP solution and reprojection evaluation process until all errors are below the target value ∝. Represent the final external calibration matrix as M2.

[0153] Step S500: Perform point cloud registration based on the external calibration matrix.

[0154] In summary, this embodiment addresses outlier examples based on the density distribution of point cloud domains, improving the accuracy of sample data. Insufficient target placement sampling leads to a lack of constraints when solving the PnP problem. Factors affecting reprojection error are considered. Better performance is achieved by adding samples of target placements with different distances and poses, and the error remains stable with minimal samples across 5-6 poses. Optimizing variables, the minimum value of the similarity metric L, and the reprojection error corresponding to similar regions means that optimal extrinsic parameters can be solved accordingly, improving accuracy.

[0155] It should be noted that the method in this embodiment includes not only obtaining point cloud data from lidar, but also acquiring point cloud data from semi-solid-state lidar.

[0156] Exemplary device

[0157] like Figure 4 As shown in the illustration, this embodiment also provides a vehicle-mounted LiDAR point cloud and image registration device, the device comprising:

[0158] The data and image acquisition module 10 is used to acquire point cloud data collected by the vehicle-mounted lidar and image data collected by the camera in the same time and space.

[0159] The three-dimensional point cloud feature acquisition module 20 is used to obtain the three-dimensional point cloud features of the chessboard boundary based on the point cloud data.

[0160] The two-dimensional feature acquisition module 30 is used to obtain the two-dimensional features of the checkerboard boundary based on the image data;

[0161] The external calibration matrix acquisition module 40 is used to register the three-dimensional point cloud features of the chessboard boundary and the two-dimensional features of the chessboard boundary to obtain the external calibration matrix.

[0162] The registration module 50 is used to perform point cloud registration based on the external calibration matrix.

[0163] In one implementation, the vehicle-mounted lidar point cloud and image registration device further includes:

[0164] The preprocessing unit is used to perform data preprocessing on the point cloud data and image data to obtain processed point cloud data and image data; wherein, the data preprocessing includes timestamp synchronization and distortion correction.

[0165] In one implementation, the 3D point cloud feature acquisition module 20 includes:

[0166] A noise-free point cloud file frame acquisition unit is used to remove noise from the point cloud data using a method based on neighborhood density distribution to obtain a noise-free point cloud file frame.

[0167] The chessboard point cloud cluster acquisition unit is used to locate the noise-free point cloud file frame by using a clustering method to obtain the chessboard point cloud cluster.

[0168] The checkerboard optimal cloud plane acquisition unit is used to fit the optimal plane of the checkerboard point cloud cluster using the RANSAC method to obtain the checkerboard optimal cloud plane.

[0169] The three-dimensional point cloud feature acquisition unit is used to detect the boundary of the optimal point cloud plane of the chessboard grid, and obtain the boundary three-dimensional point cloud features and three-dimensional corner points of the chessboard grid.

[0170] In one implementation, the chessboard grid optimal point cloud plane acquisition unit includes:

[0171] The modeling sub-unit, used to construct the parametrically constrained model, is as follows:

[0172]

[0173] Where A, B, C, and D are the estimated parameters of the fitted plane, and the point set Pc It is a cluster of checkerboard-like dots, P' c Let ε be the first optimal plane, and let ε be the threshold for iteration.

[0174] The fitting subunit is used to shrink the threshold ε and repeat the fitting and filtering process according to the parameter constraints of the model. When all point clouds are within the iterative threshold ε, the projection transformation model is used to project the noisy point cloud onto the ideal plane, and the second optimal plane is obtained as follows:

[0175] P'' c ={(x0t,y0t,z0t)|(x0,y0,z0)∈P' c}

[0176] Where t = -D / (Ax0 + By) 0+ Cz0);

[0177] The optimal point cloud plane acquisition sub-unit of the chessboard pattern is used to divide the second optimal plane into multiple rectangular grids and randomly sample the point cloud in each rectangular grid to obtain the optimal point cloud plane of the chessboard pattern.

[0178] In one implementation, the two-dimensional feature acquisition module 30 of this embodiment includes:

[0179] The checkerboard boundary two-dimensional feature acquisition unit is used to detect the internal corner points of the checkerboard through element feature points and locate the checkerboard through similarity to obtain the two-dimensional features of the checkerboard boundary.

[0180] The two-dimensional corner point acquisition unit is used to detect the image data using a chessboard corner point detection algorithm to obtain two-dimensional corner points;

[0181] The index relationship acquisition unit is used to sort the two-dimensional corner points to obtain the index relationship.

[0182] In one implementation, the external calibration matrix acquisition module 40 of this embodiment includes:

[0183] The two-dimensional point cloud feature acquisition unit is used to unfold the point cloud data of the three-dimensional point cloud features of the chessboard boundary onto a two-dimensional plane to obtain two-dimensional point cloud features.

[0184] The association relationship acquisition unit is used to obtain the association relationship between the three-dimensional and two-dimensional corner point pairs of the chessboard boundary based on the index relationship;

[0185] The external calibration matrix acquisition unit is used to register the two-dimensional point cloud features and the two-dimensional features of the checkerboard boundary in each frame of data according to the correlation relationship, so as to obtain the external calibration matrix.

[0186] In one implementation, the external calibration matrix acquisition unit of this embodiment includes:

[0187] The initial solution matrix acquisition sub-unit is used to solve the three-dimensional corner points and two-dimensional corner points in any frame of data according to the pose relationship between the lidar coordinate system and the camera coordinate system, and obtain the initial solution matrix.

[0188] The two-dimensional chessboard estimation acquisition sub-unit is used to perform three-dimensional-two-dimensional projection on each frame of data according to the initial solution matrix to obtain the two-dimensional chessboard estimation.

[0189] The reprojection error calculation subunit is used to take the two-dimensional features of the chessboard boundary as pseudo-true values, and to calculate the reprojection error based on the two-dimensional chessboard estimation and pseudo-true values, and to set the error target threshold.

[0190] The initial solution matrix update sub-unit is used to compare the reprojection error with the error target threshold. If the reprojection error is greater than the error target threshold, the two-dimensional chessboard estimate corresponding to the reprojection error is deleted, and the estimated initial solution matrix is ​​updated.

[0191] The iterative subunit is used to re-execute the step of obtaining the initial solution matrix by using the RANSCA-based PnP pose estimation algorithm for the three-dimensional and two-dimensional corner points in any frame of data until the reprojection error in each frame of data is less than or equal to the error target threshold, and then obtains the external calibration matrix.

[0192] Based on the above embodiments, the present invention also provides a smart terminal, the principle block diagram of which can be as follows: Figure 5 As shown, the intelligent terminal includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for registering point clouds and images of an in-vehicle LiDAR system. The display screen can be an LCD screen or an e-ink screen. The temperature sensor is pre-installed within the intelligent terminal to detect the operating temperature of internal devices.

[0193] Those skilled in the art will understand that Figure 5The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the smart terminal to which the present invention is applied. A specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0194] In one embodiment, a smart terminal is provided, comprising a memory, a processor, and an onboard LiDAR point cloud and image registration program stored in the memory and executable on the processor. When the processor executes the onboard LiDAR point cloud and image registration program, it implements the following operation instructions:

[0195] Acquire point cloud data collected by vehicle-mounted LiDAR and image data collected by camera in the same time and space;

[0196] Based on the point cloud data, the three-dimensional point cloud features of the chessboard grid boundary are obtained;

[0197] Based on the image data, the two-dimensional features of the chessboard boundary are obtained;

[0198] The external calibration matrix is ​​obtained by registering the three-dimensional point cloud features of the chessboard boundary and the two-dimensional features of the chessboard boundary.

[0199] Point cloud registration is performed based on the external calibration matrix.

[0200] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, operational databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual operating data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0201] In summary, this invention discloses a method for registering point clouds and images from a vehicle-mounted LiDAR. The method includes: acquiring point cloud data collected by a vehicle-mounted LiDAR and image data collected by a camera in the same time and space; obtaining three-dimensional point cloud features of a checkerboard boundary based on the point cloud data; obtaining two-dimensional features of the checkerboard boundary based on the image data; performing registration based on the three-dimensional and two-dimensional checkerboard boundary features to obtain an external calibration matrix; and performing point cloud registration based on the external calibration matrix. This invention can automatically process data from different dimensions measured by LiDAR and cameras using various algorithms to reduce errors and achieve high-precision registration using a checkerboard pattern.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for registering point clouds and images of a vehicle-mounted lidar system, characterized in that, The method includes: Acquire point cloud data collected by vehicle-mounted LiDAR and image data collected by camera in the same time and space; Based on the point cloud data, the three-dimensional point cloud features of the chessboard grid boundary are obtained; Based on the image data, the two-dimensional features of the chessboard boundary are obtained; The external calibration matrix is ​​obtained by registering the three-dimensional point cloud features of the chessboard boundary and the two-dimensional features of the chessboard boundary. Point cloud registration is performed based on the external calibration matrix; The step of obtaining the two-dimensional features of the checkerboard boundary based on the image data includes: The internal corner points of the chessboard are detected by element feature points, and the chessboard is located by similarity, thus obtaining the two-dimensional features of the chessboard boundary. The image data is detected using a chessboard corner detection algorithm to obtain two-dimensional corner points; Sort the two-dimensional corner points to obtain the index relationship; The registration process based on the 3D point cloud features and 2D features of the checkerboard boundary to obtain the external calibration matrix includes: The point cloud data of the three-dimensional point cloud features of the chessboard boundary are unfolded onto a two-dimensional plane to obtain two-dimensional point cloud features. Based on the index relationship, the association relationship between the three-dimensional and two-dimensional corner point pairs of the chessboard boundary is obtained; Based on the aforementioned correlation, the two-dimensional point cloud features and the two-dimensional features of the checkerboard boundary in each frame of data are registered to obtain an external calibration matrix; The step of registering the two-dimensional point cloud features and the two-dimensional features of the checkerboard boundary in each frame of data according to the association relationship to obtain an external calibration matrix includes: Based on the pose relationship between the lidar coordinate system and the camera coordinate system, the PnP pose estimation algorithm based on RANSCA is used to solve for the three-dimensional corner points and two-dimensional corner points in any frame of data to obtain the initial solution matrix; Based on the initial solution matrix, a three-dimensional to two-dimensional projection is performed on each frame of data to obtain a two-dimensional chessboard estimate. The two-dimensional features of the chessboard boundary are used as pseudo-true values. Based on the two-dimensional chessboard estimation and pseudo-true values, the reprojection error is calculated, and a target threshold for the error is set. The reprojection error is compared with the target error threshold. If the reprojection error is greater than the target error threshold, the two-dimensional chessboard estimate corresponding to the reprojection error is deleted, and the initial solution matrix is ​​updated. The steps of re-exercising the three-dimensional and two-dimensional corner points in any frame of data and using the RANSCA-based PnP pose estimation algorithm to obtain the initial solution matrix are repeated until the reprojection error in each frame of data is less than or equal to the error target threshold, thus obtaining the external calibration matrix.

2. The method for registering point clouds and images of vehicle-mounted lidar according to claim 1, characterized in that, After acquiring the point cloud data collected by the vehicle-mounted LiDAR and the image data collected by the camera in the same time and space, the method further includes: The point cloud data and image data are preprocessed to obtain processed point cloud data and image data; wherein, the data preprocessing includes timestamp synchronization and distortion correction.

3. The method for registering point clouds and images of vehicle-mounted lidar according to claim 2, characterized in that, The step of obtaining the 3D point cloud features of the chessboard boundary based on the point cloud data includes: Noise in the point cloud data is removed using a method based on neighborhood density distribution to obtain a noise-free point cloud file frame. Clustering methods are used to locate the noise-free point cloud file frames to obtain checkerboard point cloud clusters; The optimal plane of the chessboard grid point cloud cluster is fitted using the RANSAC method to obtain the optimal cloud plane of the chessboard grid. The boundary of the optimal point cloud plane of the chessboard is detected to obtain the three-dimensional point cloud features and three-dimensional corner points of the chessboard boundary.

4. The method for registering point clouds and images of vehicle-mounted lidar according to claim 3, characterized in that, The process of fitting the optimal plane of the chessboard point cloud cluster using the RANSAC method to obtain the optimal chessboard point cloud plane includes: The parameter constraint model is constructed as follows: Where A, B, C, and D are the estimated parameters of the fitted plane, and the point set... It is a chessboard of dots and clusters of clouds. The first optimal plane, The threshold for iteration; Shrinkage threshold The model is then repeatedly fitted and filtered according to the parameters, until all point clouds are within the iterative threshold. When the noise point cloud is projected onto an ideal plane using a projection transformation model, the second optimal plane is obtained as follows: in ; The second optimal plane is divided into multiple rectangular grids, and the point cloud in each rectangular grid is randomly sampled to obtain the chessboard optimal point cloud plane.

5. A vehicle-mounted lidar point cloud and image registration device, characterized in that, The apparatus is used to implement the steps of the method for registering point clouds and images of a vehicle-mounted LiDAR as described in any one of claims 1-4, wherein the apparatus comprises: The data and image acquisition module is used to acquire point cloud data collected by the vehicle-mounted LiDAR and image data collected by the camera in the same time and space. The three-dimensional point cloud feature acquisition module is used to obtain the three-dimensional point cloud features of the chessboard boundary based on the point cloud data. A two-dimensional feature acquisition module is used to obtain two-dimensional features of the checkerboard boundary based on the image data; The external calibration matrix acquisition module is used to register the three-dimensional point cloud features of the chessboard boundary and the two-dimensional features of the chessboard boundary to obtain the external calibration matrix. The registration module is used to perform point cloud registration based on the external calibration matrix.

6. A smart terminal, characterized in that, The intelligent terminal includes a memory, a processor, and a vehicle-mounted LiDAR point cloud and image registration program stored in the memory and capable of running on the processor. When the processor executes the vehicle-mounted LiDAR point cloud and image registration program, it implements the steps of the vehicle-mounted LiDAR point cloud and image registration method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a vehicle-mounted LiDAR point cloud and image registration program. When the vehicle-mounted LiDAR point cloud and image registration program is executed by the processor, it implements the steps of the vehicle-mounted LiDAR point cloud and image registration method as described in any one of claims 1-4.

Images