An online calibration method for lidar and camera in ecological park scenes

Abstract

The present invention discloses an online calibration method for a laser radar and camera in an eco-park environment, comprising the following steps: installing a laser radar and a camera on an automated cleaning vehicle; utilizing the laser radar and camera to collect data, respectively, to obtain point cloud data and an RGB image; utilizing the imaging characteristics of solar panels in the RGB image to extract solar panel feature pixels from the RGB image; utilizing the varying reflection intensities of the laser radar on different materials to extract solar panel feature point clouds from the point cloud data; filtering and centering the feature pixels and feature point clouds using a dynamic programming algorithm to obtain image and point cloud feature point sets; and solving the image and point cloud feature point sets using an ICP algorithm based on kdtree nearest neighbor matching to obtain an extrinsic parameter matrix. This method can automatically, quickly, and accurately determine the extrinsic parameter matrices for the laser radar and camera, laying the foundation for efficient operation of unmanned vehicles in eco-parks.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving technology, and in particular to an online calibration method for a laser radar and a camera for an ecological park scene. Background Art

[0002] In recent years, autonomous driving technology has developed rapidly. Autonomous driving not only alleviates traffic pressure and improves people's travel experience, but also plays a crucial role in enclosed areas such as eco-parks. Eco-parks are remote, abandoned mines that have been transformed into enclosed grasslands capable of solar power generation and environmental protection. However, many mines are located in harsh, windy and sandy environments, requiring regular cleaning of solar panels. Autonomous eco-park vehicles are well-suited for this task. A wide range of sensors are the cornerstone for autonomous vehicles to perform perception and positioning in complex scenarios. Sensor calibration is fundamental to their efficient operation. Sensor calibration is divided into intrinsic and extrinsic calibration. Intrinsic calibration includes camera dedistortion and LiDAR internal bias calibration. Extrinsic calibration refers to the relative position and attitude between different sensors, commonly between the laser and inertial navigation units, and between the laser and camera. The accuracy of the extrinsic calibration between different sensors directly affects the perception accuracy of autonomous vehicles.

[0003] In the field of extrinsic calibration between LiDAR and cameras, accuracy and automation have always been two conflicting goals. Feature extraction-based calibration methods achieve very high calibration accuracy by extracting common features from laser point clouds and images to solve the PnP problem. However, this method requires the production of special calibration plates and has a low degree of automation. Motion-based methods treat extrinsic calibration as a hand-eye calibration problem, which improves automation to a certain extent, but their accuracy depends on the motion estimation of the camera and LiDAR. Methods based on maximizing mutual information assume that there is a certain correlation between image grayscale and reflection intensity. While they have relatively high accuracy and automation, they are limited to specific environments. Deep learning-based methods have a high degree of automation, but their training and processing are time-consuming and lack generalization capabilities.

[0004] Chinese patent publication number CN115100292A, the invention name is "A method for online calibration of external parameters between a lidar and a camera in a road environment", which mainly provides a method for calibrating external parameters between a lidar and a camera using multi-frame sliding window constraints in a road environment. By using the error construction method to construct a 2D-3D correlation relationship between camera and lidar data, a nonlinear optimization target is obtained. This method can well calculate the external parameter matrix of the lidar and camera. However, this method requires offline calibration of the external parameters between the lidar and the camera using checkerboard markers as the initial value of the external parameters for online calibration, which greatly reduces the degree of automation of the unmanned vehicle.

[0005] Chinese patent publication number CN114862968A, titled "Automatic Calibration Method and Apparatus for LiDAR and Camera Based on Attention Mechanism," encodes RGB images and LiDAR point clouds into a pre-trained cross-modal attention target association network. The resulting feature maps are then post-processed to generate extrinsic parameters. The device required for this invention is complex, requiring a large number of computer devices to form various modules, making it difficult to quickly deploy. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide an online calibration method for lidar and camera in an eco-park environment. The purpose is to automatically, quickly and accurately determine the external parameter matrix of the lidar and camera, laying the foundation for the efficient operation of unmanned vehicles in the eco-park.

[0007] The present invention adopts the following technical solutions:

[0008] A method for online calibration of lidar and camera in an ecological park environment includes the following steps:

[0009] Step 1: Install the LiDAR and camera on the autonomous cleaning vehicle;

[0010] Step 2: The autonomous cleaning vehicle cleans solar panels in the eco-park, using LiDAR and cameras to collect data, generating point cloud data and RGB images.

[0011] Step 3: Using the imaging characteristics of the solar panel in the RGB image, extract the solar panel feature pixels in the RGB image;

[0012] Step 4: Using the different reflection intensities of LiDAR on different materials, extract the solar panel feature point cloud from the point cloud data;

[0013] Step 5: Filter and center the feature pixels and feature point cloud using a dynamic programming algorithm to obtain a set of image and point cloud feature points.

[0014] Step 6: Solve the image and point cloud feature point sets based on the ICP algorithm of kdtree nearest neighbor matching to obtain the external parameter matrix.

[0015] Furthermore, in step 1, the laser radar and camera are installed above the front of the vehicle.

[0016] Furthermore, in step 3, pixels whose RGB channel values are both less than 5 in the image data are extracted and marked as feature pixels.

[0017] Furthermore, in step 4, points with reflection intensity less than a threshold value are extracted from the point cloud data and marked as feature point clouds.

[0018] Furthermore, the step 5, filtering and centering the feature pixels using a dynamic programming algorithm, includes the following sub-steps:

[0019] (1) The feature pixels obtained in step 3 constitute an image feature pixel set V. A point v0 is randomly selected from the image feature pixel set V, added to the cluster center point set K0, and v0 is deleted from V.

[0020] (2) Mark v0 as processed, and search for the number of feature points in its 4×4 area centered on v0;

[0021] (3) If the number of feature points in the v0 domain is greater than 3, all feature points in the v0 domain are deleted from V, added to K0, and marked as unprocessed;

[0022] (4) Search for unprocessed points in K0 and operate on them according to the method of processing v0 in steps (2) and (3);

[0023] (5) If all points in K0 have been processed, return to step (1) and randomly select v k Add v k To the cluster center point set K k At the same time, k Delete from V until the point set V becomes an empty set;

[0024] (6) Search set K 0～n Sets with fewer than 5 elements are marked as outliers and removed;

[0025] (7) K after removing outliers 0～n That is the set of image feature points.

[0026] Furthermore, the step 5, filtering and centering the feature point cloud using a dynamic programming algorithm, includes the following sub-steps:

[0027] S1, let the rotation matrix between the lidar and the camera be R, the translation matrix be T, the camera intrinsic parameter matrix be K, the coordinates of the point cloud to be projected be (x, y, z), and the coordinates after projection be (u, v). According to the projection rule:

[0028]

[0029] According to the initial calibration R, T matrix, the three-dimensional point cloud is projected onto the two-dimensional image and the corresponding relationship is recorded. The projection points constitute the projection point set of the point cloud data;

[0030] S2, using dynamic programming algorithm to filter and center the point cloud data projection point set:

[0031] (1) Randomly select a point v0′ from the point cloud data projection point set V′, add v0′ to the cluster center point set K0′, and delete v0′ from V′;

[0032] (2) Mark v0′ as processed, and search for the number of feature points in its 4×4 area with v0′ as the center;

[0033] (3) If the number of feature points in the domain v0′ is greater than 3, all feature points in the domain v0′ are deleted from V′, added to K0′, and marked as unprocessed;

[0034] (4) Search for unprocessed points in K0′ and operate them according to the method of processing v0′ in steps (2) and (3);

[0035] (5) If all points in K0′ have been processed, return to step (1) and randomly select v k ′Add v k ′ to the cluster center point set K k ′, and at the same time k ' is deleted from V' until the point set V' becomes an empty set;

[0036] (6) Search set K 0～n Sets with fewer than 5 elements in ′ are marked as outliers and removed;

[0037] (7) K after removing outliers 0～n ′ is the feature set of the central point cloud of the projected image;

[0038] S3, compare the number of point cloud features K in the projected image center 0～n ′ and the image feature point set K 0～nIs the number of sets in the same? If not, it is necessary to continuously change R and T to reproject, and repeat step S2 until the number of point cloud features in the projected image is consistent with the number of image feature point sets. If they are consistent, it means that there are no omissions after the three-dimensional feature projection. According to the corresponding relationship, the point cloud feature point set Q is obtained. 0～n .

[0039] Furthermore, step 6 includes the following sub-steps:

[0040] (1) Based on set K 0～n Construct kdtree;

[0041] (2) According to the initial calibration R, T matrix, Q 0～n Project onto the image and find Q through kdtree 0～n Every point in K 0～n The nearest neighbor in

[0042] (3) Judge Q 0～n and K 0～n Check whether the points in the equation correspond one to one; if not, change R and T and continue with step 2; if they do, proceed to the next step;

[0043] (4) Calculate the centroid positions p, p′ of a set of corresponding points, and then calculate the centroid coordinates of each point:

[0044] q i =p i -p,q i ′=p i ′-p′

[0045] Among them, p i , p i ′ is the coordinates of a set of corresponding points;

[0046] (5) Calculate the rotation matrix according to the following optimization problem:

[0047]

[0048] (6) Calculate T based on R in step 2 * :

[0049] T * =p-Rp′

[0050] (7) Get the external parameter matrix R * and T * .

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] (1) The present invention proposes a method for extracting image and point cloud features, which fully utilizes the solar panel features in the ecological park scene to quickly and efficiently extract the corresponding features of the image and point cloud.

[0053] (2) In the process of further processing the image and point cloud features, the present invention reduces noise and centers the image and point cloud features through a dynamic programming algorithm, which greatly improves the accuracy and speed of calibration.

[0054] (3) In the process of point cloud and image extrinsic parameter calibration, the present invention uses the ICP algorithm based on kdtree nearest neighbor search to greatly improve the calibration speed of lidar and camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flow chart of an example of the present invention.

[0056] Figure 2 It is a schematic diagram of the specific steps of filtering and centralization.

[0057] Figure 3 This is a schematic diagram of the specific steps of the ICP algorithm. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only for illustration and are not intended to limit the present invention.

[0059] This paper proposes an online calibration method for lidar and camera for ecological park scenes. The overall process is as follows: Figure 1 As shown, feature pixels and feature point clouds are extracted, feature filtering and centralization are performed to obtain the feature center point set, and then the external parameter matrix is solved by the ICP algorithm, as shown in Figure 1 The specific steps are as follows:

[0060] Step 1: Install the LiDAR and camera on the autonomous cleaning vehicle;

[0061] After obtaining permission, cameras and lidars are installed on the automatic cleaning cart, and are required to be placed as high as possible above the front of the vehicle to prevent them from being blocked by obstacles, while also maximizing the perception range.

[0062] Step 2: Collect lidar point cloud data and camera video data;

[0063] The camera and lidar installed in step 1 are used to collect data. The collected data are RGB images and point cloud data of the ecological park scene.

[0064] Step 3: Using the imaging characteristics of the solar panel in the RGB image, extract the solar panel feature pixels in the RGB image;

[0065] Pixels whose RGB channel values in the extracted image data are less than 5 are marked as feature pixels.

[0066] Step 4: Using the different reflection intensities of LiDAR on different materials, extract the solar panel feature point cloud from the point cloud data;

[0067] Points with reflection intensity less than a threshold in the point cloud data are extracted and marked as feature point clouds.

[0068] Step 5: Filter and center the feature pixels and feature point cloud to obtain the image and point cloud feature point set.

[0069] In an embodiment of the present application, after extracting feature pixels and feature point clouds, outliers need to be filtered and the extracted features need to be centralized.

[0070] See Figure 2 , Figure 2 A schematic diagram of the specific steps of filtering and centralization processing is shown.

[0071] Clustering is performed on feature pixels based on the dynamic programming algorithm, and outliers are filtered out through the clustering results. The specific steps are as follows:

[0072] 1. Randomly select a point v0 from the image feature pixel set V, add v0 to the cluster center point set K0, and delete v0 from V;

[0073] 2. Mark v0 as processed and search for the number of feature points in its 4×4 area centered on v0;

[0074] 3. If the number of feature points in the v0 domain is greater than 3, all feature points in the v0 domain are deleted from V, added to K0, and marked as unprocessed;

[0075] 4. Search for unprocessed points in K0 and operate on them according to the method of processing v0 in steps 2 and 3;

[0076] 5. If all points in K0 have been processed, return to step 1 and randomly select v k Add v k To the cluster center point set K k At the same time, k Delete from V until the point set V becomes an empty set;

[0077] 6. Search set K 0～n Sets with fewer than 5 elements are marked as outliers and removed;

[0078] 7. K after removing outliers 0～n This is the clustering result.

[0079] Cluster the feature point cloud based on projection and filter out outliers through the clustering results. The specific steps are as follows:

[0080] 1. Assume that the rotation matrix between the lidar and the camera is R, the translation matrix is T, the camera intrinsic parameter matrix is K, the coordinates of the point cloud to be projected are (x, y, z), and the coordinates after projection are (u, v). According to the projection rules:

[0081]

[0082] According to the initial calibration R, T matrix, the three-dimensional point cloud is projected onto the two-dimensional image and the corresponding relationship is recorded. The projection points constitute the point cloud data projection point set.

[0083] 2. Use the dynamic programming algorithm for clustering image features to cluster the point cloud data projection point set to obtain the centralized point cloud features. The specific steps are as follows:

[0084] (1) Randomly select a point v0′ from the point cloud data projection point set V′, add v0′ to the cluster center point set K0′, and delete v0′ from V′;

[0085] (2) Mark v0′ as processed, and search for the number of feature points in its 4×4 area with v0′ as the center;

[0086] (3) If the number of feature points in the domain v0′ is greater than 3, all feature points in the domain v0′ are deleted from V′, added to K0′, and marked as unprocessed;

[0087] (4) Search for unprocessed points in K0′ and operate them according to the method of processing v0′ in steps (2) and (3);

[0088] (5) If all points in K0′ have been processed, return to step (1) and randomly select v k ′Add v k ′ to the cluster center point set K k ′, and at the same time k ' is deleted from V' until the point set V' becomes an empty set;

[0089] (6) Search set K 0～n Sets with fewer than 5 elements in ′ are marked as outliers and removed;

[0090] (7) K after removing outliers 0～n ′ is the feature set of the central point cloud of the projected image.

[0091] 3. Compare the number of central point cloud features K in the projected image 0～n ′ and the image feature point set K 0～n Is the number of sets in the same? If not, it is necessary to continuously change R and T to reproject, and repeat step 2 until the number of features in the projected image center point cloud is consistent with the number of feature point sets in the image. If they are consistent, it means that there are no omissions after the three-dimensional feature projection. According to the corresponding relationship, the three-dimensional feature point set Q of the centralized point cloud is obtained. 0～n , which is the clustering result.

[0092] Afterwards, based on the clustering results, the feature pixels and feature point clouds are centralized and the set K is recorded. 0～n and set Q 0～n The center of density of the midpoint set.

[0093] Step 6: After obtaining the image and point cloud feature point sets, the image and point cloud feature point sets are solved using the ICP algorithm based on kdtree nearest neighbor matching to obtain the extrinsic parameter matrix.

[0094] See Figure 3 , Figure 3 The following is a schematic diagram showing the specific steps of using the ICP algorithm to solve the external parameter matrix. The following content specifically describes the specific process of using the ICP algorithm to solve the external parameter matrix:

[0095] 1. Based on set K 0～n Construct kdtree;

[0096] 2. According to the initial calibration R, T matrix, Q 0～n Project onto the image and find Q through kdtree 0～n Every point in K 0～n The nearest neighbor in

[0097] 3. Judge Q 0～n and K 0～n Check whether the points in the image are in one-to-one correspondence. If not, change R and T and continue with step 2. If they are, proceed to the next step.

[0098] 4. Calculate the centroid positions p,p′ of a set of corresponding points, and then calculate the centroid coordinates of each point:

[0099] q i =p i -p,q i ′=p i ′-p′

[0100] Among them, p i , p i ′ is the coordinates of a set of corresponding points; q i ,q i′ is the centroid coordinate of the corresponding points of this group;

[0101] 5. Calculate the rotation matrix according to the following optimization problem:

[0102]

[0103] 6. Calculate T based on R in step 2 * :

[0104] T * =p-Rp′

[0105] 7. Get the external parameter matrix R * and T * .

[0106] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any form. Any person skilled in the art who, without departing from the scope of the technical solution of the present invention, utilizes the technical content disclosed above to make more possible changes and modifications to the technical solution of the present invention, or modifications are all equivalent embodiments of the present invention. Therefore, any equivalent and equivalent changes made in accordance with the ideas of the present invention without departing from the content of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for online calibration of lidar and camera for ecological park environment, characterized by: The following steps are involved: Step 1: Install the LiDAR and camera on the autonomous cleaning vehicle; Step 2: The autonomous cleaning vehicle cleans solar panels in the eco-park, using LiDAR and cameras to collect data, generating point cloud data and RGB images. Step 3: Using the imaging characteristics of the solar panel in the RGB image, extract the solar panel feature pixels in the RGB image; Step 4: Using the different reflection intensities of LiDAR on different materials, extract the solar panel feature point cloud from the point cloud data; Step 5: Filter and center the feature pixels and feature point cloud using a dynamic programming algorithm to obtain a set of image and point cloud feature points. Step 6: Solve the image and point cloud feature point sets based on the ICP algorithm of kdtree nearest neighbor matching to obtain the external parameter matrix; The step 5, filtering and centering the feature point cloud using a dynamic programming algorithm, includes the following sub-steps: S1, let the rotation matrix between the lidar and the camera be R, the translation matrix be T, the camera intrinsic parameter matrix be K, the coordinates of the point cloud to be projected be (x, y, z), and the coordinates after projection be (u, v). According to the projection rule: According to the initial calibration R, T matrix, the three-dimensional point cloud is projected onto the two-dimensional image and the corresponding relationship is recorded. The projection points constitute the projection point set of the point cloud data; S2, using dynamic programming algorithm to filter and center the point cloud data projection point set: (1) Randomly select a point v0′ from the point cloud data projection point set V′, add v0′ to the cluster center point set K0′, and delete v0′ from V′; (2) Mark v0′ as processed, and search for the number of feature points in its 4×4 area with v0′ as the center; (3) If the number of feature points in the domain v0′ is greater than 3, all feature points in the domain v0′ are deleted from V′, added to K0′, and marked as unprocessed; (4) Search for unprocessed points in K0′ and operate them according to the method of processing v0′ in steps (2) and (3); (5) If all points in K0′ have been processed, return to step (1) and randomly select v k ′Add v k ′ to the cluster center point set K k ′, and at the same time k ' is deleted from V' until the point set V' becomes an empty set; (6) Search set K 0～n Sets with fewer than 5 elements in ′ are marked as outliers and removed; (7) K after removing outliers 0～n ′ is the feature set of the central point cloud of the projected image; S3, compare the number of point cloud features K in the projected image center 0～n ′ and the image feature point set K 0～n Is the number of sets in the same? If not, it is necessary to continuously change R and T to reproject, and repeat step S2 until the number of point cloud features in the projected image is consistent with the number of image feature point sets. If they are consistent, it means that there are no omissions after the three-dimensional feature projection. According to the corresponding relationship, the point cloud feature point set Q is obtained. 0～n .

2. The online calibration method for lidar and camera in an ecological park environment according to claim 1, characterized in that: In step 1, the laser radar and camera are installed above the front of the vehicle.

3. The online calibration method of a lidar and camera for an ecological park environment according to claim 1, characterized in that: In step 3, pixels whose RGB channel values are both less than 5 in the image data are extracted and marked as feature pixels.

4. The online calibration method for lidar and camera in an ecological park environment according to claim 1, characterized in that: In step 4, points with reflection intensity less than a threshold value are extracted from the point cloud data and marked as feature point clouds.

5. The online calibration method of a lidar and camera for an ecological park environment according to claim 1, characterized in that: Step 6 includes the following sub-steps: (1) Based on set K 0～n Construct kdtree; (2) According to the initial calibration R, T matrix, Q 0～n Project onto the image and find Q through kdtree 0～n Every point in K 0～n The nearest neighbor in (3) Judge Q 0～n and K 0～n Check whether the points in the equation correspond one to one; if not, change R and T and continue with step 2; if they do, proceed to the next step; (4) Calculate the centroid positions p, p′ of a set of corresponding points, and then calculate the centroid coordinates of each point: q i =p i -p,q i ′=p i ′-p′ Among them, p i , p i ′ is the coordinates of a set of corresponding points; (5) Calculate the rotation matrix according to the following optimization problem: (6) Calculate T based on R in step 2 * : T * =p-Rp′ (7) Get the external parameter matrix R * and T * .

Images