A point cloud and panoramic image fine registration method based on conjugate rod base element search

Abstract

The application discloses a kind of point cloud and panorama image fine registration method based on conjugate rod base element search, comprising: 1) extracting rod base element from vehicle-mounted laser point cloud and panorama image;2) construct the corresponding relationship of panoramic image and frame format image transformation, convert rod base element into frame format image;3) using template matching in frame format image, obtain rod base element matching result;4) according to the matching result, the corresponding 2D-3D corresponding point is obtained using conversion formula;5) by 2D-3D corresponding point, the accurate transformation relationship between vehicle-mounted laser point cloud and panorama image is solved by EPnP method, to complete registration.The method of the application converts 2D and 3D rod base element into frame format image, focuses on the translation transformation relationship between panoramic plane and frame format plane, converts the required 6 parameters into 2 parameters in X and Y directions in plane, reduces the dimension of solving space, realizes the high-precision registration of vehicle-mounted laser point cloud and panorama image.

Description

Technical Field

[0001] This invention belongs to the field of surveying and mapping technology, specifically relating to a method for precise registration of vehicle-mounted laser point clouds and panoramic images. Background Technology

[0002] The vehicle-mounted mobile mapping system (MMS) achieves coverage of ground-based side-view laser point cloud and image data, efficiently acquiring high-resolution vertical LiDAR (light detection and ranging) point clouds and panoramic images from the measured environment. It represents a novel photogrammetry and remote sensing method. The high-resolution LiDAR point clouds and panoramic images provided by the MMS can be applied to strip topographic mapping, highway asset management, and image-based city construction, among other fields.

[0003] In the process of registering vehicle-mounted laser point clouds with panoramic images, in order to quickly obtain accurate 2D-3D correspondence points, it is necessary to match 2D rod primitives and 3D rod primitives. However, the relationship between 3D rod primitives and 2D rod primitives is one of rotation and translation, which requires solving 6 parameters. Existing methods are difficult to solve this problem.

[0004] To address this, the present invention proposes a method for precise registration of point clouds and panoramic images based on conjugate rod primitive lookup. This method transfers 2D and 3D rod primitives into the frame-type image and constructs translation transformation relationships in different planes. This transforms the six parameters required for the solution into two parameters in the X and Y directions of the plane, reducing the dimensionality of the solution space and making it possible. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for precise registration of point clouds and panoramic images based on conjugate rod primitive lookup. In the registration process between vehicle-mounted laser point clouds and panoramic images, 2D and 3D rod primitives are transferred to the frame image, and a translation transformation relationship between the panoramic plane and the frame plane is constructed. This transforms the six parameters required for the solution into two parameters in the X and Y directions of the plane, reducing the dimensionality of the solution space and achieving precise registration between vehicle-mounted laser point clouds and panoramic images.

[0006] To achieve the above objectives, the present invention adopts the following technical solution.

[0007] A method for precise registration of vehicle-mounted laser point clouds and panoramic images based on conjugate rod primitive lookup includes the following steps:

[0008] Step 1: Extract pole primitives from vehicle-mounted laser point clouds and panoramic images;

[0009] 2D rod primitives in panoramic images are extracted using a multi-level convolutional neural network. Then, rod-shaped objects are extracted from the vehicle-mounted laser point cloud by slicing, point clustering, and cylinder fitting. An improved connected component analysis method is used to obtain rod primitives. The point cloud rod primitives are then transformed into the panoramic plane according to the initial pose obtained by the vehicle-mounted laser scanning system to obtain planar point cloud rod primitives, i.e. 3D rod primitives.

[0010] Step 2: Construct the correspondence between the panoramic image and the frame image in the X and Y directions of translation transformation, transform the 2D rod primitives and 3D rod primitives in the panoramic image to the frame image, realize the same transformation of rod primitives in the panoramic plane and the frame plane, and provide a theoretical basis for subsequent template matching.

[0011] The conversion of 2D rod primitives to frame images is achieved by a reverse solution method. The coordinates of each pixel in the frame image are reversed to the panoramic image using the panoramic image-frame image conversion formula. The nearest neighbor pixel value of the panoramic image is selected as the pixel value of the frame image, thus obtaining the frame image after the 2D rod primitive conversion.

[0012] The conversion of 3D rod primitives into frame images is achieved by using the initial transformation parameters provided by the vehicle-mounted laser scanning system, transforming the point cloud into the panoramic coordinate system using the point cloud-panoramic coordinate transformation formula, and then converting the 3D rod primitives into the frame image according to the panoramic image-frame image transformation formula.

[0013] Step 3: Use template matching in the frame image to obtain the optimal matching results for 2D and 3D rod primitives;

[0014] Step 4: Based on the best matching results of 2D and 3D rod primitives in the frame image, obtain a sufficient number of corresponding points. Then, convert the 2D corresponding points into the panoramic image using the panoramic image-frame image conversion formula, and convert the 3D corresponding points into the panoramic image using the panoramic image-frame image conversion formula. Finally, convert the point cloud into point cloud coordinates using the point cloud-panoramic coordinate conversion formula to obtain the corresponding 2D-3D corresponding points.

[0015] Step 5: Based on the corresponding 2D-3D points, the transformation relationship between the vehicle-mounted laser point cloud and the panoramic image is directly calculated using the EPnP method, thereby achieving high-precision registration between the vehicle-mounted laser point cloud and the panoramic image.

[0016] Specifically, the translational transformations in the X direction between the panoramic image and the frame image constructed in step 2 are as follows:

[0017] Left half of the panoramic image:

[0018]

[0019]

[0020] Right half of the panoramic image:

[0021]

[0022] In the above formula, b is the distance of translation in the X direction on the panoramic image, x is the distance of translation in the X direction on the corresponding frame image, W represents the width of the panoramic image, h is the focal length set when converting the panoramic image to a frame image, θ is the angle of rotation of the panoramic sphere corresponding to the translation of the panoramic image, α is the angle between the line connecting the translated pixel and the exposure point and the image plane in the X direction, and β is the angle between the line connecting the translated pixel and the exposure point and the image plane in the X direction.

[0023] Specifically, the translational transformations in the Y direction between the panoramic image and the frame image constructed in step 2 are as follows:

[0024] upper half of the panoramic image

[0025]

[0026] lower half of the panoramic image

[0027]

[0028] In the above formula, b is the distance translated in the Y direction on the panoramic image, y is the distance translated in the Y direction on the corresponding frame image, H represents the width of the panoramic image, h is the focal length set when converting the panoramic image to a frame image, θ is the angle of rotation of the panoramic sphere corresponding to the translation of the panoramic image, α is the angle between the line connecting the translated pixel and the exposure point and the image plane in the X direction, and β is the angle between the line connecting the original pixel and the exposure point and the image plane in the X direction.

[0029] Specifically, the formula for converting panoramic images to frame-type images in step 2 is as follows:

[0030]

[0031] In the above formula, (x p ,y p ,z p (x, y) is a point in the panoramic sphere coordinate system, (x, y) are the coordinates of a point in the image plane, f is the set camera focal length, (u, v) are the pixel coordinates in the frame image, and R... c The camera rotation matrix is ​​set, (u0, v0) is the frame size of the image, and R and K are the transformation matrix from point cloud to panoramic sphere coordinate system and the camera intrinsic parameter matrix, respectively.

[0032] The specific point cloud-panoramic coordinate transformation formula mentioned in step 2 is as follows:

[0033]

[0034] In the above formula, (x w ,y w ,z w ) represents the point cloud coordinates in the world coordinate system, and R is the transformation matrix from the point cloud to the panoramic sphere coordinate system.

[0035] The beneficial effects of this invention are:

[0036] This invention proposes a method for precise registration of point clouds and panoramic images based on conjugate rod primitive lookup. By transferring 2D and 3D rod primitives into the frame image during the registration process of vehicle-mounted laser point clouds and panoramic images, and for the first time constructing the translation transformation relationship between the panoramic plane and the frame plane, the six parameters required for the solution are converted into two parameters in the X and Y directions of the plane, reducing the dimensionality of the solution space and achieving precise registration of vehicle-mounted laser point clouds and panoramic images. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the extraction results of 2D rod primitives in the embodiment;

[0038] Figure 2 This is a schematic diagram of the extraction results of the point cloud rod element in the embodiment;

[0039] Figure 3 This is a schematic diagram of the extraction results of 3D rod primitives in the embodiment;

[0040] Figure 4 This is a schematic diagram showing the results of projecting 2D rod elements and 3D rod elements into a frame-type image in the embodiment (red represents 3D rod elements, and blue represents 2D rod elements);

[0041] Figure 5 This is a diagram illustrating the conversion relationship between panoramic images and frame-type images in the embodiment.

[0042] Figure 6 This is a diagram showing the relationship between the panoramic image and the frame-type image in the X-direction coordinate translation in the embodiment (left half of the image);

[0043] Figure 7 This is a diagram showing the relationship between the panoramic image and the frame-type image in the X-direction coordinate translation in the embodiment (right half of the image);

[0044] Figure 8 This is a diagram showing the relationship between the panoramic image and the frame-type image in the Y-direction coordinate translation in the embodiment (upper half of the image);

[0045] Figure 9 This is a diagram showing the relationship between the panoramic image and the frame-type image in the Y-direction coordinate translation in the embodiment (lower half of the image);

[0046] Figure 10 This is a diagram showing the initial positional relationship between the 2D rod elements and the 3D rod elements in the embodiment;

[0047] Figure 11 This is a schematic diagram of the matching results of 2D and 3D rod elements in the embodiment;

[0048] Figure 12 This is a high-precision registration result of the vehicle-mounted laser point cloud and panoramic image in the embodiment. Detailed Implementation

[0049] The present invention will be further described in detail below with reference to specific embodiments, but the scope of protection of the present invention is not limited to the content described.

[0050] Example

[0051] A method for fine registration of point clouds and panoramic images based on conjugate rod primitive lookup includes the following steps:

[0052] Step 1: Extract pole primitives from vehicle-mounted laser point clouds and panoramic images;

[0053] The 2D pole primitives in the panoramic image were extracted using a neural network. The extraction result is as follows: Figure 1 As shown, rod-shaped elements were then extracted from the vehicle-mounted laser point cloud using slicing, point clustering, and cylindrical fitting. An improved connected component analysis method was then used to obtain the point cloud rod primitives. The extraction results are shown below. Figure 2 As shown, the point cloud rod primitives are then transformed into the image plane based on the initial pose obtained by the vehicle-mounted laser scanning system, resulting in planar point cloud rod primitives (i.e., 3D rod primitives). Figure 3 As shown;

[0054] Step 2, as follows Figure 4 As shown, 2D rod primitives and 3D rod primitives are converted into frame-format images:

[0055] The conversion of 2D rod primitives to frame-type images is achieved using a reverse solution method, such as... Figure 5 As shown, the coordinates of each pixel in the frame image are reversed and solved into the panoramic image using the panoramic image-frame image conversion formula. The nearest neighbor pixel value summarized in the panoramic image is selected as the pixel value of the frame image, thus obtaining the frame image after 2D rod primitive conversion.

[0056] Specifically, the formula for converting panoramic images to frame-type images is as follows:

[0057]

[0058] In the above formula, (x p ,y p ,z p(x, y) is a point in the panoramic sphere coordinate system, (x, y) are the coordinates of a point in the image plane, f is the set camera focal length, (u, v) are the pixel coordinates in the frame image, and R... c The camera rotation matrix is ​​set, (u0, v0) is the frame size of the image, and R and K are the transformation matrix from point cloud to panoramic sphere coordinate system and the camera intrinsic parameter matrix, respectively.

[0059] The conversion of 3D rod primitives into frame images is achieved by using the initial transformation parameters provided by the vehicle-mounted laser scanning system, transforming the point cloud into the panoramic coordinate system using the point cloud-panoramic coordinate transformation formula, and then converting the 3D rod primitives into the frame image according to the panoramic image-frame image transformation formula.

[0060] Specifically, the point cloud-panoramic coordinate transformation formula is as follows:

[0061]

[0062] In the above formula, (x w ,y w ,z w ) represents the point cloud coordinates in the world coordinate system, and R is the transformation matrix from the point cloud to the panoramic sphere coordinate system.

[0063] Since panoramic images are obtained by unfolding a spherical model, they are deformed. As a result, the translation in the panoramic image is not in a 1:1 relationship with the translation in the frame image. Therefore, it is necessary to construct the correspondence between the translation transformation of the panoramic image and the frame image in the X and Y directions.

[0064] Specifically, the translation transformation relationship between panoramic images and frame-type images in the X direction is as follows:

[0065] like Figure 6 The diagram shows the relationship between the panoramic image and the frame image in the left half of the image in the X direction.

[0066] For the left half of the panoramic image, we have:

[0067]

[0068] like Figure 7 The diagram shows the relationship between the panoramic image and the frame image in the right half of the image in the X direction.

[0069] For the right half of the panoramic image, we have:

[0070]

[0071]

[0072] In the above formula, b is the distance of translation in the X direction on the panoramic image, x is the distance of translation in the X direction on the corresponding frame image, W represents the width of the panoramic image, h is the focal length set when converting the panoramic image to a frame image, θ is the angle of rotation of the panoramic sphere corresponding to the translation of the panoramic image, α is the angle between the line connecting the translated pixel and the exposure point and the image plane in the X direction, and β is the angle between the line connecting the translated pixel and the exposure point and the image plane in the X direction.

[0073] like Figure 8 The diagram shows the relationship between the panoramic image and the frame image in the upper half of the Y direction.

[0074] For the upper half of the panoramic image, we have:

[0075] upper half of the panoramic image

[0076]

[0077] like Figure 9 The diagram shows the relationship between the upper and lower halves of the panoramic and frame-type images in the Y direction.

[0078] For the lower half of the panoramic image, we have:

[0079]

[0080]

[0081] In the above formula, b is the distance translated in the Y direction on the panoramic image, y is the distance translated in the Y direction on the corresponding frame image, H represents the width of the panoramic image, h is the focal length set when converting the panoramic image to a frame image, θ is the angle of rotation of the panoramic sphere corresponding to the translation of the panoramic image, α is the angle between the line connecting the translated pixel and the exposure point and the image plane in the X direction, and β is the angle between the line connecting the original pixel and the exposure point and the image plane in the X direction.

[0082] Based on the aforementioned correspondence between panoramic and frame-type images in the X and Y directions of translation transformation, the rod element transformation in the panoramic image is converted to the transformation in the frame-type image. Therefore, all 2D and 3D rod elements are converted to the frame-type image, and the conversion result is as follows. Figure 10 As shown.

[0083] Step 4: Use template matching methods in frame-type imagery, such as... Figure 11 As shown, the optimal matching results for 2D rod primitives and 3D rod primitives are obtained;

[0084] Step 5: Based on the best matching results of 2D and 3D rod primitives in the frame image, obtain a sufficient number of corresponding points. Then, convert the 2D corresponding points into the panoramic image using the panoramic image-frame image conversion formula, and convert the 3D corresponding points into the panoramic image using the panoramic image-frame image conversion formula. Finally, convert them into point cloud coordinates using the point cloud-panoramic coordinate conversion formula to obtain the corresponding 2D-3D correspondence table, as shown in Table 1 below.

[0085] Table 1. 2D-3D Correspondence Points Table

[0086]

[0087] Step 6: Based on the 2D-3D correspondence point table, the transformation relationship between the vehicle-mounted laser point cloud and the panoramic image is directly calculated using the EPnP method, achieving high-precision registration between the vehicle-mounted laser point cloud and the panoramic image. The registration result is as follows: Figure 12 As shown.

[0088] In summary, this invention provides a method for precise registration of point clouds and panoramic images based on conjugate rod primitive lookup. During the registration process of vehicle-mounted laser point clouds and panoramic images, 2D and 3D rod primitives are transferred to the frame-type image, and for the first time, a transformation relationship between the panoramic plane and the frame-type plane is constructed. This transforms the six parameters required for the solution into two parameters in the X and Y directions of the plane, reducing the dimensionality of the solution space and achieving high-precision registration of vehicle-mounted laser point clouds and panoramic images.

[0089] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Any simple modifications, alterations, or imitations made to the above embodiments based on the technical content of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for precise registration of point clouds and panoramic images based on conjugate rod primitive lookup, characterized in that, Includes the following steps: Step 1: Extract pole primitives from vehicle-mounted laser point clouds and panoramic images; 2D rod primitives in panoramic images are extracted using a multi-level convolutional neural network. Then, rod-shaped objects are extracted from the vehicle-mounted laser point cloud by slicing, point clustering, and cylinder fitting. An improved connected component analysis method is used to obtain rod primitives. The point cloud rod primitives are then transformed into the panoramic plane according to the initial pose obtained by the vehicle-mounted laser scanning system to obtain planar point cloud rod primitives, i.e. 3D rod primitives. Step 2: Construct the correspondence between the panoramic image and the frame image in the X and Y directions of translation transformation, transform the 2D rod primitives and 3D rod primitives in the panoramic image to the frame image, realize the same transformation of rod primitives in the panoramic plane and the frame plane, and provide a theoretical basis for subsequent template matching. The conversion of 2D rod primitives to frame images is achieved by a reverse solution method. The coordinates of each pixel in the frame image are reversed to the panoramic image using the panoramic image-frame image conversion formula. The nearest neighbor pixel value of the panoramic image is selected as the pixel value of the frame image, thus obtaining the frame image after the 2D rod primitive conversion. The conversion of 3D rod primitives into frame images is achieved by using the initial transformation parameters provided by the vehicle-mounted laser scanning system, transforming the point cloud into the panoramic coordinate system using the point cloud-panoramic coordinate transformation formula, and then converting the 3D rod primitives into the frame image according to the panoramic image-frame image transformation formula. Step 3: Use template matching in the frame image to obtain the optimal matching results for 2D and 3D rod primitives; Step 4: Based on the best matching results of 2D and 3D rod primitives in the frame image, obtain a sufficient number of corresponding points. Then, convert the 2D corresponding points into the panoramic image using the panoramic image-frame image conversion formula, and convert the 3D corresponding points into the panoramic image using the panoramic image-frame image conversion formula. Finally, convert the point cloud into point cloud coordinates using the point cloud-panoramic coordinate conversion formula to obtain the corresponding 2D-3D corresponding points. Step 5: Based on the corresponding 2D-3D points, the transformation relationship between the vehicle-mounted laser point cloud and the panoramic image is directly calculated using the EPnP method, thereby achieving high-precision registration between the vehicle-mounted laser point cloud and the panoramic image.

2. The method for fine registration of point clouds and panoramic images based on conjugate rod primitive lookup according to claim 1, characterized in that, The translational transformations in the X direction between the panoramic image and the frame image constructed in step 2 are as follows: Left half of the panoramic image: Right half of the panoramic image: In the above formula, b is the distance of translation in the X direction on the panoramic image, x is the distance of translation in the X direction on the corresponding frame image, W represents the width of the panoramic image, h is the focal length set when converting the panoramic image to a frame image, θ is the angle of rotation of the panoramic sphere corresponding to the translation of the panoramic image, α is the angle between the line connecting the translated pixel and the exposure point and the image plane in the X direction, and β is the angle between the line connecting the translated pixel and the exposure point and the image plane in the X direction.

3. The method for fine registration of point clouds and panoramic images based on conjugate rod primitive search according to claim 1, characterized in that, The translational transformations in the Y direction between the panoramic image and the frame image constructed in step 2 are as follows: upper half of the panoramic image lower half of the panoramic image In the above formula, b is the distance translated in the Y direction on the panoramic image, y is the distance translated in the Y direction on the corresponding frame image, H represents the width of the panoramic image, h is the focal length set when converting the panoramic image to a frame image, θ is the angle of rotation of the panoramic sphere corresponding to the translation of the panoramic image, α is the angle between the line connecting the translated pixel and the exposure point and the image plane in the X direction, and β is the angle between the line connecting the original pixel and the exposure point and the image plane in the X direction.

4. The method for fine registration of point clouds and panoramic images based on conjugate rod primitive search according to claim 1, characterized in that, The formula for converting panoramic images to frame-type images in step 2 is as follows: In the above formula, (x p ,y p ,z p (x, y) is a point in the panoramic sphere coordinate system, (x, y) are the coordinates of a point in the image plane, f is the set camera focal length, (u, v) are the pixel coordinates in the frame image, and R... c The camera rotation matrix is ​​set, (u0, v0) is the frame size of the image, and R and K are the transformation matrix from point cloud to panoramic sphere coordinate system and the camera intrinsic parameter matrix, respectively.

5. The method for fine registration of point clouds and panoramic images based on conjugate rod primitive search according to claim 1, characterized in that, The point cloud to panoramic coordinate transformation formula mentioned in step 2 is: In the above formula, (x w ,y w ,z w ) represents the point cloud coordinates in the world coordinate system, and R is the transformation matrix from the point cloud to the panoramic sphere coordinate system.

Images