Automatic registration method and system of multi-lens combined images and lidar point clouds

Abstract

The application discloses a kind of multi-lens combination image and laser radar point cloud automatic registration method and system thereof.The automatic registration method includes: constructing the calibration scene of multi-view combined camera and laser radar sensor;Calibrate the external parameter of multi-view combined camera and laser radar sensor;Solve the initial pose of multi-lens combination image;According to sequence image generation and laser radar point cloud scale consistent sparse image feature point cloud under world coordinate system;By the preset point cloud matching algorithm, the sparse image feature point cloud is registered with the laser radar point cloud, and the accurate registration result of the sparse image feature point cloud to the laser radar point cloud is obtained.The application effectively solves the scale uniformity and initial value calculation problem in 3D-3D point cloud alignment, has higher registration accuracy, and can realize automatic registration.

Description

TECHNICAL FIELD

[0001] The present application relates to a cross-modal data fusion between a three-dimensional laser radar and a vision system, in particular to an automatic registration method and system between mobile laser radar point cloud and multi-lens combined image, belonging to the technical field of space environment perception. BACKGROUND

[0002] Mobile mapping system (MMS) based on multi-sensor data fusion provides a new technical means for BIM modeling, urban planning, AR / VR. The typical MMS usually carries environmental perception sensors such as laser radar and combined panoramic camera. The visible light image and the laser radar point cloud data collected by the system have good complementarity. The visible light image provides rich texture information in the scene, while the laser radar provides depth information that the camera lacks. Effective fusion of the two data can quickly reconstruct the scene in the physical world in three dimensions.

[0003] In the registration and fusion of mobile laser radar point cloud and visible light image, there are mainly two challenge problems: (1) the calibration error of the sensor. Although the calibration method between multi-line sparse laser radar and visible light camera has been studied a lot, the calibration accuracy of the sensor level is still difficult to meet the task requirements of higher level data fusion, and it is expected to achieve sub-pixel accuracy. At the same time, the mechanical vibration that is difficult to avoid in the data collection process will also affect the external parameters of the sensors calibrated in advance. (2) The clock of the sensors is not synchronized. During the data collection process, it is often difficult to achieve strict clock synchronization between the laser radar and the camera, and between the cameras, which makes the camera pose calculated based on the laser radar odometry inaccurate. The above unavoidable factors require the use of data-driven registration method in the subsequent data processing process.

[0004] Scholars at home and abroad have carried out a lot of research on the registration method of mobile mapping system (MMS) laser scanning data and image. The existing registration methods are mainly divided into three categories: ① 2D-2D registration algorithm; ② 2D-3D registration algorithm; ③ 3D-3D registration algorithm of image feature point cloud and laser point cloud registration. The 2D-2D based method converts the image to point cloud registration problem into image to image registration process. The 2D-3D based method completely retains the information from different modes, which is theoretically more conducive to improving the registration accuracy. The 3D-3D based method needs a series of images for three-dimensional reconstruction, and then performs three-dimensional registration. The registration methods known in the literature are mostly based on the registration of panoramic images spliced and mobile laser radar point cloud, which are not suitable for high-precision automatic registration between multi-lens combined camera images and mobile laser radar point cloud that have not been spliced, and are difficult to effectively deal with the two challenge problems encountered in the registration and fusion of mobile laser radar point cloud and visible light image. SUMMARY

[0005] To address the challenges of clock asynchrony between sensors during data acquisition in Mobile Measurement Systems (MMS), which makes high-precision registration difficult even with initial extrinsic parameters for LiDAR and camera, and the fact that most feature-based registration methods often require manual intervention, are scene-dependent, and lack robust automated registration capabilities, this invention provides an automatic registration method and system based on multi-lens combined images and LiDAR point clouds with initial image pose, thereby overcoming the aforementioned problems in the prior art.

[0006] To achieve the aforementioned objectives, the present invention employs the following solution:

[0007] One aspect of the present invention provides an automatic registration method for multi-lens composite images and mobile lidar point clouds based on initial image pose, comprising the following steps:

[0008] Step 1: Construct a calibration scene for a multi-view combined camera and LiDAR sensor;

[0009] Step 2: Calibrate the extrinsic parameters of the multi-view combined camera and the LiDAR sensor to obtain their extrinsic parameters.

[0010] Step 3: Calculate the initial pose of the multi-camera composite image, specifically including: acquiring a point cloud frame M of the scene using sparse LiDAR. l Determine the point cloud frame data M l Initial pose to world coordinate system Based on the external parameters of the multi-view combined camera and lidar sensor and the initial pose Obtain the initial pose at the moment of image acquisition.

[0011] Step 4: Generate a sparse image feature point cloud M in a world coordinate system consistent with the scale of the lidar point cloud based on the image sequence. C The sequence of images was acquired by a visible light camera mounted on the mobile measurement system;

[0012] Step 5: Register the sparse image feature point cloud M using a preset point cloud matching algorithm. C With the lidar point cloud M L The sparse image feature point cloud M is obtained. C To the lidar point cloud M L The accurate registration results.

[0013] Understandably, the aforementioned lidar point cloud M L This can be understood as multi-frame point cloud frame data Ml a set of images.

[0014] In one embodiment, the constructing the calibration scene of the multi-view combined camera and the lidar sensor specifically comprises: selecting a room, and pasting a checkerboard pattern in each wall of the room, the number of the checkerboard pattern being consistent with the number of the multi-view combined camera.

[0015] In one embodiment, the calibrating the extrinsic parameters of the multi-view combined camera and the lidar sensor specifically comprises:

[0016] Step 2.1: acquiring panoramic images I of the calibration scene based on the rack-mounted lidar p and a scene point cloud M s , and acquiring images of the calibration scene by using the combined lens wherein N is the number of lenses of the combined panoramic camera, and each image contains a complete checkerboard pattern, and a point cloud frame data M of the scene is acquired by using the sparse lidar l ;

[0017] Step 2.2: calculating 2D coordinates of the corner points of the checkerboard in the panoramic images I p and the images of the calibration scene , obtaining 3D coordinates of the corner points of the checkerboard according to the correspondence between the panoramic images I p and the scene point cloud M S , and calculating the pose of the multi-view combined camera by using a preset pose estimation algorithm

[0018] Step 2.3: selecting three linearly independent planes from the scene point cloud M S and the point cloud frame data M l respectively, obtaining plane equations corresponding to the point clouds by using a plane fitting algorithm, and then calculating the pose of the sparse lidar with the initial value, and obtaining an accurate transformation from the point cloud frame data M l to the scene point cloud M S by using a preset point cloud matching algorithm, to obtain an optimized pose of the sparse lidar The extrinsic parameters of the multi-view combined camera and the lidar sensor are represented as:

[0019]

[0020] The inv() is an inverse matrix.

[0021] In one embodiment, the calculating the initial pose of the combined image of the multiple lenses specifically comprises: obtaining an initial pose of the point cloud frame data M l to the world coordinate system by using a lidar SLAM algorithm, and denoted as According to the extrinsic parameters of the multi-view combined camera and the laser radar sensor and the initial pose to obtain the initial pose at the image acquisition moment

[0022]

[0023] In one embodiment, the step 5 specifically comprises: taking the laser radar point cloud M L as a reference benchmark, realizing configuration parameter fine-tuning by minimizing the distance between the sparse feature point cloud and the laser radar point cloud; and obtaining the accurate registration result of the sparse image feature point cloud M C to the laser radar point cloud M L by ICP method iterative optimization.

[0024] Another aspect of the present application provides an automatic registration system of multi-lens combined images and mobile laser radar point clouds based on image initial pose, the registration system comprising:

[0025] a scene construction module, the scene construction module being used for constructing a calibration scene of a multi-view combined camera and a laser radar sensor;

[0026] an extrinsic parameter calibration module, the extrinsic parameter calibration module being used for calibrating the extrinsic parameters of the multi-view combined camera and the laser radar sensor to obtain the extrinsic parameters of the multi-view combined camera and the laser radar sensor

[0027] an initial pose solving module, the initial pose solving module being used for solving the initial pose of the multi-lens combined images, specifically comprising: acquiring one frame of point cloud frame data M l of a sparse laser radar acquisition scene; and determining the initial pose of the point cloud frame data M l to a world coordinate system According to the extrinsic parameters of the multi-view combined camera and the laser radar sensor and the initial pose to obtain the initial pose at the image acquisition moment

[0028] a sparse image feature point cloud generation module, the sparse image feature point cloud generation module being used for generating a sparse image feature point cloud M C in a world coordinate system according to a sequence image, the sequence image being acquired by a visible light camera carried in a mobile measurement system;

[0029] a registration module, the registration module being used for registering the sparse image feature point cloud M C and the laser radar point cloud M LThe sparse image feature point cloud M is obtained. C To the lidar point cloud M L The accurate registration results.

[0030] In one embodiment, the calibration scenario for constructing a multi-camera and lidar sensor combination specifically includes: selecting a room and pasting a checkerboard pattern on each wall of the room, matching the number of multi-camera combinations.

[0031] In one embodiment, calibrating the extrinsic parameters of the multi-view combined camera and lidar sensor specifically includes:

[0032] Panoramic images of the scene acquired and calibrated using a stationary LiDAR system (I) p With scene point cloud M S Using a combination lens to capture images of the calibration scene Where N is the number of lenses in the combined panoramic camera, ensuring that each image contains a complete checkerboard pattern, and point cloud frame data M of the scene is collected using sparse LiDAR. l ;

[0033] Calculate the panoramic image I p and the image of the calibrated scene The 2D coordinates of the corner points of the chessboard grid are based on the panoramic image I. p With scene point cloud M S The 3D coordinates of the chessboard corner points are obtained by establishing the correspondence, and then the pose of the multi-camera system is calculated using a preset pose estimation algorithm.

[0034] The point cloud M in the scene S and the point cloud frame data M l Three linearly independent planes are selected, and the plane equations corresponding to the point cloud are obtained through a plane fitting algorithm, which is then used to calculate the pose of the sparse lidar. Initial values ​​are used, and then a preset point cloud matching algorithm is used to obtain the point cloud frame data M. l To the scene point cloud M S The precise transformation yields the optimized pose of the sparse lidar. The extrinsic parameters of the multi-view combined camera and lidar sensor are expressed as follows:

[0035]

[0036] The inv() function is the inverse matrix.

[0037] In one embodiment, the initial pose calculation of the multi-lens composite image specifically includes: obtaining point cloud frame M through a lidar SLAM algorithm. l The initial pose to the world coordinate system is denoted as According to the extrinsic parameters of the multi-camera combined with the LiDAR sensor And the initial pose Get the initial pose at the image acquisition moment

[0038]

[0039] In one embodiment, the registration module is specifically configured to: take the LiDAR point cloud M L As a reference, the configuration parameter refinement is realized by minimizing the distance between the sparse feature point cloud and the LiDAR point cloud; and the accurate registration result of the sparse image feature point cloud M C To the LiDAR point cloud M L Is obtained by iterative optimization of the ICP method.

[0040] Compared with the prior art, the automatic registration method and system of multi-lens combined image and LiDAR point cloud provided by the application effectively solve the problems of scale unification and initial value calculation in 3D-3D point cloud alignment, have higher registration accuracy, and can realize automatic registration. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments described in the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0042] Figure 1a 、 Figure 1b is an example diagram of a combined image and LiDAR point cloud data acquisition device in an embodiment of the application;

[0043] Figure 2 is a calibration scene schematic diagram in an embodiment of the application;

[0044] Figure 3 is an indoor and outdoor four-camera combined image and LiDAR point cloud in an embodiment of the application;

[0045] Figure 4a 、 Figure 4b is an indoor and outdoor scene registration result in an embodiment of the application;

[0046] Figure 5 is an error quantification evaluation result in an embodiment of the application;

[0047] Figure 6is a schematic diagram of automatic registration method deployment in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In view of the deficiencies of the prior art, the present application provides a technical solution, which effectively solves the problems of scale unification and initial value calculation in 3D-3D point cloud alignment, has higher registration accuracy, and can realize automatic registration.

[0049] In order to make the objectives, technical solutions and advantages of the present application clearer, the specific embodiments of the present application will be described in detail below with reference to the drawings. Examples of these preferred embodiments are illustrated in the drawings. The embodiments of the present application shown in the drawings and described according to the drawings are merely exemplary, and the present application is not limited to these embodiments.

[0050] Here, it should also be noted that, in order to avoid obscuring the present application due to unnecessary details, only structures and / or processing steps closely related to the solution according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0051] In a more typical embodiment of the present application, first, a high-precision rack-mounted laser radar is used as an intermediate sensor to calibrate the external parameters of the multi-view combined camera and the sparse laser radar sensor. Then, the laser radar SLAM algorithm is used to obtain the odometer, and then the initial pose of each image is obtained. Then, the SfM algorithm with image pose is used to obtain the sparse feature point cloud consistent in scale with the sparse laser radar scene point cloud. Finally, the iterative closest point (ICP) algorithm is used to align the laser radar scene point cloud and the image feature point cloud.

[0052] Specifically, the process of constructing the calibration scene of the combined camera and the laser radar sensor can be: selecting an indoor room, and sticking a chessboard pattern consistent with the number of combined cameras in the middle of the four surrounding walls, wherein the chessboard pattern is printed on A4 paper, and the pattern array is 4x5.

[0053] The external parameter calibration process of the multi-view combined camera and the laser radar sensor can be: based on the millimeter-level precision of the rack-mounted laser radar, the panoramic image I p and the scene point cloud M S , the combined lens is used to collect the image of the calibration scene where N is the number of lenses of the combined panoramic camera, ensuring that each image contains a complete chessboard pattern, and the point cloud frame data Ml of the scene is collected by the sparse laser radar, which refers to a frame of point cloud data collected by the LiDAR, and is used to calculate the external parameters between the LiDAR and the visible light camera. The findChessboardCorners algorithm of OpenCV is used to calculate the image I p and 2D coordinates of the checkerboard corner points according to the panoramic image I p and the correspondence relationship of the scene point cloud M S , and then the pose of the multi-view combined camera is calculated by the PnP algorithm The input data of the PnP algorithm includes the 2D coordinates of the detected corner points in the image and the 3D coordinates of the corresponding corner points in the point cloud. Three linearly independent planes are selected in the point clouds M s and M l , and the plane equation corresponding to the point cloud is obtained by the plane fitting algorithm, and then the pose of the sparse laser radar is calculated The initial value is obtained, and the accurate transformation of the point cloud M l to M s is obtained by the ICP algorithm, and the optimized pose of the sparse laser radar is obtained The inv() is the inverse matrix, that is, the inverse operation of the matrix.

[0054] The initial pose of the multi-lens combined image is calculated. The initial pose of the laser radar point cloud frame to the world coordinate system is obtained by the laser radar SLAM algorithm, denoted as The initial pose of the image acquisition moment is obtained by the extrinsic parameters of the combined camera and the laser radar sensor obtained in step 2

[0055] At the same time, the SfM method with image pose is used to obtain the sparse image feature point cloud in the world coordinate system generated by the sequence image and the LiDAR point cloud.

[0056] For the sparse feature point cloud and the laser radar point cloud unified in the world coordinate system, taking the laser radar point cloud as the reference benchmark, the registration parameter refinement is realized by minimizing the distance between the two (i.e. between the sparse feature point cloud and the laser radar point cloud), and the laser radar point cloud is denoted as M L , where the point cloud is the dense scene point cloud registered by the SLAM algorithm in a state, and the sparse feature point cloud reconstructed by the SfM of the image sequence is denoted as M C . After step 4, the sparse reconstructed image feature point cloud and the laser radar point cloud are approximately registered in space, and then the ICP method is used for iterative optimization to obtain the accurate registration result of M C to M L .

[0057] Specifically, as shown in Figure 1a , Figure 1b , it is installed on the data acquisition system. The specific implementation steps are as follows:

[0058] 1) as shown in Figure 2 ​As shown, the indoor calibration scene is selected, and chessboard is pasted on the four walls to ensure that each camera image contains a chessboard. Then, a three-dimensional scanner is used to obtain the dense point cloud and panoramic image of the scene.

[0059] 2) For the point cloud obtained by the laser radar and the three-dimensional scanner, three linear independent planes are matched simultaneously to obtain the initial value of the extrinsic parameter between the laser radar and the scanner.

[0060] 3) The point cloud of the entire scene is used for ICP algorithm to obtain the accurate extrinsic parameter through iterative optimization.

[0061] 4) For the combined image, the three-dimensional coordinates of the chessboard corner points are obtained by using the three-dimensional scanner, and the PnP algorithm is used to optimize the accurate extrinsic parameter of the camera and the scanner.

[0062] 5) Based on the extrinsic parameter between the camera and the laser radar obtained by calibration, the initial pose of each image is obtained through the SLAM mileage calculation method, and the sparse feature point cloud of the image is obtained through the SfM algorithm with pose.

[0063] 6) The sparse feature point cloud of the image is iteratively registered with the laser radar scene point cloud. As shown, Figure 3 The data acquisition system obtains the point cloud data and the combined image data. As shown, Figure 4a , Figure 4b The registered point cloud is colored, and the qualitative observation of the texture-rich area has a good coloring effect. As shown, Figure 5 The registration error is within one pixel, Figure 5 The horizontal coordinate represents the number of accuracy evaluation images, and the vertical coordinate is the re-projection error, which is in pixels.

[0064] It should be understood that although the present specification is described in terms of embodiments, each embodiment does not contain only one independent technical solution, and the description manner of the specification is only for the sake of clarity. The skilled person should consider the specification as a whole, and the technical solutions of each embodiment can also be appropriately combined to form other embodiments that can be understood by the skilled person.

Claims

1. An automatic registration method of multi-camera combined images and mobile laser radar point clouds based on image initial pose, characterized in that, The registration method comprises the following steps: Step 1: constructing a calibration scene of the multi-view combined camera and the laser radar sensor, specifically comprising: selecting a room, and pasting a checkerboard consistent with the number of the multi-view combined camera in each wall of the room; Step 2: calibrate the extrinsic parameters of the multi-view combined camera and the lidar sensor to obtain the extrinsic parameters of the multi-view combined camera and the lidar sensor The external parameter calibration of the multi-view combined camera and the laser radar sensor specifically comprises: Step 2.1: Collecting panoramic images I of the calibration scene based on the rack-mounted laser radar p and the scene point cloud M S , collecting images of the calibration scene using a combined lens where N is the number of lenses of the combined panoramic camera, ensuring that each image contains a complete checkerboard pattern, and collecting point cloud frame data M l of the scene using a sparse laser radar Step 2.2: calculating the panoramic image I p and the image of the calibration scene 2D coordinates of the chessboard corner points in the panoramic image I p and the correspondence between the scene point cloud M S and the 3D coordinates of the chessboard corner points, and then the pose of the multi-view camera is calculated by a preset pose estimation algorithm Step 2.3: three linearly independent planes are selected from the scene point cloud M S and the point cloud frame data M l respectively, the plane equation corresponding to the point cloud is obtained through the plane fitting algorithm, and then the pose of the sparse laser radar is solved , and the initial value is obtained, and the preset point cloud matching algorithm is used to obtain the accurate transformation of the point cloud frame data M l to the scene point cloud M S , and the optimized pose of the sparse laser radar is obtained The external parameter of the multi-view combined camera and the laser radar sensor is represented as: The inv() is an inverse matrix; Step 3: solving the initial pose of the multi-lens combined image, specifically comprising: collecting a frame of point cloud frame data M of a scene by a sparse laser radar l ; determining the initial pose of the point cloud frame data M l to a world coordinate system According to the extrinsic parameters of the multi-lens combined camera and the laser radar sensor and the initial pose , the initial pose at the image collection time is obtained Step 4: generating a sparse image feature point cloud M in a world coordinate system consistent with the scale of the lidar point cloud according to sequence images, the sequence images being acquired by a visible light camera carried in a mobile measurement system C , the sequence images being acquired by a visible light camera carried in a mobile measurement system Step 5: registering the sparse image feature point cloud M by a preset point cloud matching algorithm C with the lidar point cloud M L , obtaining an accurate registration result of the sparse image feature point cloud M C to the lidar point cloud M L .

2. The automatic registration method of claim 1, wherein, The initial pose of the combined image of the multiple lenses is obtained by a laser radar SLAM algorithm to obtain point cloud frame data M l The initial pose to the world coordinate system is recorded as According to the external parameters of the multi-view combined camera and the laser radar sensor And the initial pose An initial pose at the image acquisition time is obtained 3. The registration method of claim 1, wherein, The step 5 specifically comprises: taking the laser radar point cloud M L As a reference, the configuration parameter refinement is realized by minimizing the distance between the sparse feature point cloud and the laser radar point cloud; and the accurate registration result of the sparse image feature point cloud M C to the laser radar point cloud M L is obtained through ICP method iterative optimization.

4. An automatic registration system of multi-camera combined imagery and mobile laser radar point cloud based on image initial pose, characterized in that, The registration system comprises: A scene construction module, which is configured to construct a calibration scene of the multi-view combined camera and the laser radar sensor, specifically comprising: selecting a room, and pasting a checkerboard consistent with the number of the multi-view combined camera in each wall of the room; An external parameter calibration module is configured to calibrate external parameters of the multi-view combined camera and the LiDAR sensor, and obtain the external parameters of the multi-view combined camera and the LiDAR sensor The calibration of the external parameters of the multi-view combined camera and the LiDAR sensor specifically comprises: Based on the frame station type laser radar collects the panoramic image I of the calibration scene p With the scene point cloud M S , using the combined lens to collect the image of the calibration scene Where N is the number of lenses of the combined panoramic camera, ensure that each image contains a complete checkerboard pattern, using sparse laser radar to collect scene point cloud frame data M l ; calculating the panoramic image I p and the image of the calibration scene 2D coordinates of the chessboard corner points in the panoramic image I p and the scene point cloud M S 3D coordinates of the chessboard corner points according to the correspondence between the panoramic image I and the scene point cloud M S and the point cloud frame data M l Three linearly independent planes are selected respectively in the scene point cloud M l and the point cloud frame data M S The plane equation corresponding to the point cloud is obtained by the plane fitting algorithm, and then the pose of the sparse laser radar is solved initial value, and then the accurate transformation of the point cloud frame data M l to the scene point cloud M S is obtained by using the preset point cloud matching algorithm, and the optimized pose of the sparse laser radar is obtained The external parameter of the multi-view combined camera and the laser radar sensor is represented as: The inv() is an inverse matrix; An initial pose solving module is configured to solve an initial pose of a multi-lens combined image, specifically comprising: collecting a point cloud frame M of a scene by using a sparse laser radar l ; determining an initial pose of the point cloud frame M l to a world coordinate system According to the external parameters of the multi-lens combined camera and the laser radar sensor and the initial pose to obtain an initial pose at an image collection time The sparse image feature point cloud generation module is used to generate a sparse image feature point cloud M in a world coordinate system with the same scale as the lidar point cloud based on the sequence of images. C The sequence of images was acquired by a visible light camera mounted on the mobile measurement system; a registration module for registering the sparse image feature point cloud M by a preset point cloud matching algorithm C to the lidar point cloud M L , obtaining an accurate registration result of the sparse image feature point cloud M C to the lidar point cloud M L .

5. The automatic registration system of claim 4, wherein, The initial pose of the combined image of the multiple lenses is obtained by a laser radar SLAM algorithm to obtain a point cloud frame M l The initial pose to the world coordinate system is recorded as According to the external parameters of the multi-view combined camera and the laser radar sensor And the initial pose An initial pose at an image acquisition time is obtained 6. The automatic registration system of claim 4, wherein, The registration module is specifically used for: registering the sparse image feature point cloud M L with the laser radar point cloud M C by taking the sparse image feature point cloud M L as a reference, refining the configuration parameters by minimizing the distance between the sparse feature point cloud and the laser radar point cloud, and obtaining the accurate registration result of the sparse image feature point cloud M C and the laser radar point cloud M L by iterative optimization through the ICP method.

Images