A method and system for fusing laser point cloud and visible light image

Abstract

The first aspect of this application proposes a method for fusing laser point cloud and visible light image, including: S1, performing spatial dimensionality reduction on the point cloud data to obtain two-dimensional point cloud data; S2, extracting feature points of the two-dimensional point cloud data and feature points of the visible light image data through the Scale-Invariant Feature Transform (SIFT) algorithm; S3, determining the projection plane, projecting the point cloud data through the projection model to generate a depth-echo intensity image, and constructing a 3D-2D coordinate mapping relationship between the point cloud data and the depth-echo intensity image; S4, mapping the feature points of the visible light image data to the point cloud data according to the 3D-2D coordinate mapping relationship. This solution can achieve a good image registration and fusion effect, can be applied to various platforms, and has a wide range of applicable scenarios. The second aspect of this application proposes a system for fusing laser point cloud and visible light image.

Description

Technical Field

[0001] This application belongs to the field of industrial vision, and specifically relates to a method and system for fusing laser point cloud and visible light image. Background Art

[0002] In the field of industrial vision technology, in some application scenarios, such as the autonomous driving of vehicles, etc., the application environment is complex and the real-time requirement for digital imaging is extremely high. Therefore, a single sensor can no longer meet the needs of actual applications. In digital imaging systems such as autonomous driving systems, a platform including different sensors is usually set up to work together, and multi-sensor data is fused to ensure better perception and understanding of the environment. At the same time, the fusion of heterogeneous data from multiple devices has also become a research hotspot in the field of image perception.

[0003] The fusion of laser point cloud and visible light image has become an important technical support for modules such as multi-object tracking and obstacle recognition in vehicle assistance systems. Compared with traditional image registration problems, laser point cloud and optical image are two types of cross-modal heterogeneous data, and the huge differences between them bring great difficulties to the registration problem. Existing technologies still have deficiencies in aspects such as registration accuracy, robustness, and automation level. Summary of the Invention

[0004] In view of the above problems, the first aspect of this application proposes a method for fusing laser point cloud and visible light image, including the following steps:

[0005] S1, perform spatial dimensionality reduction on the point cloud data to obtain two-dimensional point cloud data;

[0006] S2, extract the feature points of the two-dimensional point cloud data and the feature points of the visible light image data through the Scale-Invariant Feature Transform (SIFT) algorithm;

[0007] S3, determine the projection plane, project the point cloud data through the projection model to generate a depth-echo intensity image, and construct a 3D-2D coordinate mapping relationship between the point cloud data and the depth-echo intensity image;

[0008] S4, map the feature points of the visible light image data to the point cloud data according to the 3D-2D coordinate mapping relationship.

[0009] The solution proposed in this application first uses SIFT to extract the feature points of the point cloud data and the visible light image data, uses the feature points of the point cloud image data as reference points to perform projection processing on the point cloud, reduces the dimensionality and visualizes the point cloud data, obtains the 3D-2D coordinate mapping relationship between the point cloud data and the depth-echo intensity image, and performs registration and fusion of the point cloud data and the visible light image based on this mapping relationship. This solution can achieve a better image registration and fusion effect.

[0010] Preferably, S1 further includes first performing voxelization processing on the point cloud data. The formula for calculating the centroid of the point cloud voxel grid is as follows: where x c , y c , z c are the x, y, and z coordinates of the point cloud data in the voxel grid, p is the total number of points in the voxel grid, and i is the i-th point. By performing voxelization processing on the point cloud data, spatial dimensionality reduction is achieved, making it consistent with the visible light dimension, and thus enabling feature extraction and coordinate matching between the two; at the same time, the voxelization process can achieve ordered data storage and downsampling, improving the operation efficiency.

[0011] Preferably, S2 uses KFDA-SIFT (Kernel Fisher Discriminant Analysis-SIFT) to extract the feature points of the dimension-reduced point cloud data and the feature points of the visible light image data, specifically including using Kernel Principal Component Analysis (KPCA) for spatial dimensionality reduction and using Linear Discriminant Analysis (LDA) for feature extraction of the dimension-reduced data. Further, the kernel function of KPCA uses a polynomial kernel.

[0012] The KFDA algorithm projects the original sample data into a high-dimensional feature space through a non-linear mapping, and then performs discriminant analysis in this feature space. Therefore, using KFDA-SIFT can obtain sufficient feature information and scope from more dimensions, increasing the information depth and width of feature extraction.

[0013] Preferably, S3 includes using the Direct Linear Transformation (DLT) method to construct the 3D-2D coordinate mapping relationship with the feature points of the point cloud data as reference points. Further, the 3D-2D coordinate mapping relationship is constructed in combination with the least squares method.

[0014] Preferably, the calculation formula adopted by the projection model in S3 includes:

[0015]

[0016]

[0017] Z = d

[0018] where x and y are the x and y coordinates in the projection coordinate system, X and Y are the coordinates of X and Y in the world coordinate system, X res , Yres is the resolution of the depth map, FOV represents the projection vector, and FOV h is the scalar decomposed in the horizontal direction of FOV, and FOV v is the scalar decomposed in the vertical direction of FOV, and d is the depth value.

[0019] Preferably, the visible light image data is the RGB data of the visible light image.

[0020] A second aspect of the present application proposes a laser point cloud and visible light image fusion system, including:

[0021] A data dimensionality reduction module configured to perform spatial dimensionality reduction on the point cloud data and map it to the same dimension as the visible light image data;

[0022] A feature extraction module configured to extract the feature points of the dimensionality-reduced point cloud data and the feature points of the visible light image data through the SIFT algorithm;

[0023] A coordinate mapping module configured to determine the projection plane, project the point cloud data through the projection model to generate a depth-echo intensity image, and construct a 3D-2D coordinate mapping relationship between the point cloud data and the depth-echo intensity image;

[0024] A registration and fusion module that maps the feature points of the visible light image data to the point cloud data according to the 3D-2D coordinate mapping relationship.

[0025] The laser point cloud and visible light image fusion method and system proposed in this paper have good robustness and high registration accuracy, and can be applied to platforms such as airborne, vehicle-mounted, and ground, with a wide range of applicable scenarios. Description of the Drawings

[0026] The drawings help to further understand the present application. The elements of the drawings are not necessarily in proportion to each other. For the convenience of description, only the parts related to the relevant invention are shown in the drawings.

[0027] Figure 1 is a schematic flowchart of the laser point cloud and visible light image fusion method in an embodiment of the present application;

[0028] Figure 2 is the original visible light image in an embodiment of the present application;

[0029] Figure 3 is the point cloud visualization image in an embodiment of the present application;

[0030] Figure 4 is the effect diagram after the fusion of the point cloud and visible light in an embodiment of the present application;

[0031] Figure 5Schematic diagram of the structure of the laser point cloud and visible light image fusion system in an embodiment of the present application. Detailed implementation manners

[0032] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related invention and are not intended to limit the invention.

[0033] Figure 1 It is a schematic flowchart of the laser point cloud and visible light image fusion method in an embodiment of the present application. The process includes:

[0034] S1, perform spatial dimensionality reduction on the point cloud data to obtain two-dimensional point cloud data.

[0035] The data of the point cloud includes information such as the longitude and latitude coordinates, intensity, multiple echoes, and color of each point. The dimension and amount of information of the information are much higher than those of the visible light image data. In order to facilitate subsequent feature matching, the solution proposed in the present application first reduces the dimension of the point cloud data to the same dimension as the visible light image.

[0036] In a preferred embodiment, S1 includes first performing voxelization processing on the point cloud data. The formula for calculating the centroid of the point cloud voxel grid is: where x c , y c , z c are the x, y, and z coordinates of the point cloud data in the voxel grid, p is the total number of points in the voxel grid, and i is the i-th point.

[0037] S2, extract the feature points of the two-dimensional point cloud data and the feature points of the visible light image data through the Scale-Invariant Feature Transform (SIFT) algorithm.

[0038] As a stable feature matching algorithm, SIFT has been widely used in many fields. After reducing the dimension of the point cloud data, the same SIFT algorithm structure can be used to describe the features of the reduced-dimensional point cloud data and the visible light image data.

[0039] In a preferred embodiment, in S2, KFDA-SIFT (Kernel Fisher Discriminant Analysis-SIFT) is used to extract the feature points of the dimension-reduced point cloud data and the feature points of the visible light image data. In this embodiment, an improved SIFT algorithm with kernel discriminant analysis (KFDA-SIFT) is adopted, which can further improve the robustness of the SIFT algorithm. Specifically, it includes using Kernel Principal Component Analysis (KPCA) for spatial dimension reduction to obtain low-dimensional data in a non-linear space, and then using Linear Discriminant Analysis (LDA) to extract features from the dimension-reduced data. KPCA is an improved PCA (Principal Component Analysis) algorithm that can extract a feature set more suitable for classification than traditional PCA.

[0040] Furthermore, the kernel function of KPCA adopts a polynomial kernel.

[0041] S3, determine the projection plane, project the point cloud data through the projection model to generate a depth-echo intensity image, and construct a 3D-2D coordinate mapping relationship between the point cloud data and the depth-echo intensity image. Based on the feature points extracted in S2, the point cloud data can be converted into two-dimensional data through the projection model to achieve the visualization of the point cloud. The projection model is actually a parametric model, that is, a function model is set, and the input point cloud is projected onto the function model, so as to achieve the form of point cloud filtering.

[0042] In a preferred embodiment, taking the feature points of the point cloud data as reference points, the 3D-2D coordinate mapping relationship is constructed by using the Direct Linear Transformation (DLT) method combined with the least squares method.

[0043] The calculation formula adopted by the projection model includes:

[0044]

[0045]

[0046] Z = d

[0047] where x and y are the x and y coordinates in the projection coordinate system, X and Y are the coordinates of X and Y in the world coordinate system, X res 、Y res are the resolutions of the depth map, FOV represents the projection vector, FOV h is the scalar decomposed in the horizontal direction of FOV, FOV vis a scalar decomposed in the vertical direction of the FOV, and d is the depth value.

[0048] S4. According to the 3D-2D coordinate mapping relationship, map the feature points of the visible light image data to the point cloud data. After obtaining the 3D-2D coordinate mapping relationship between the point cloud data and the depth-echo intensity image, calculate the mapping matrix between the point cloud and the visible light according to this mapping relationship, and the feature points of the visible light image data can be correspondingly mapped into the point cloud data. Specifically, the visible light image data is the RGB data of the visible light image, and map the RGB values of the corresponding pixels of the visible light image into the point cloud to realize the fusion of the laser point cloud and the visible light image.

[0049] In a specific embodiment, Figure 2 is the original visible light image, Figure 3 is the point cloud visualization image obtained in S3, Figure 4 is the effect diagram after the fusion of the point cloud and the visible light. It can be seen that in the urban road environment, the method proposed in this application can achieve good fusion of the laser point cloud and the visible light data and is well applicable to the vehicle-mounted system.

[0050] Figure 5 is a schematic structural diagram of a laser point cloud and visible light image fusion system 500 in an embodiment of this application, including:

[0051] A data dimensionality reduction module 501, configured to perform spatial dimensionality reduction on the point cloud data and map it to the same dimension as the visible light image data;

[0052] A feature extraction module 502, configured to extract the feature points of the dimensionality-reduced point cloud data and the feature points of the visible light image data through the SIFT algorithm;

[0053] A coordinate mapping module 503, configured to determine the projection plane, project the point cloud data through the projection model to generate a depth-echo intensity image, and construct a 3D-2D coordinate mapping relationship between the point cloud data and the depth-echo intensity image;

[0054] A registration and fusion module 504, which maps the feature points of the visible light image data to the point cloud data according to the 3D-2D coordinate mapping relationship.

[0055] This application proposes a laser point cloud and visible light fusion registration scheme and system, which uses the SIFT algorithm to extract the feature points of the point cloud data and the visible light image data, and performs registration and fusion of the point cloud data and the visible light image based on the 3D-2D coordinate mapping relationship between the point cloud data and the depth-echo intensity image. It is applicable to multiple scenarios and platforms, especially applicable to assisting autonomous driving in vehicle-mounted digital imaging systems.

[0056] Although the content of the present application is specifically shown and described in connection with the preferred embodiments, those skilled in the art should understand that without departing from the spirit and scope of the present application defined by the appended claims, and without making any creative efforts, various changes made to the present application in form and details fall within the protection scope of the present application.

Claims

1. A method for fusing laser point cloud and visible light image, characterized in that, It includes the following steps: S1. Perform spatial dimensionality reduction on the point cloud data to obtain two-dimensional point cloud data; S2. Extract the feature points of the two-dimensional point cloud data and the feature points of the visible light image data through the SIFT algorithm; Use KFDA-SIFT to extract the feature points of the dimensionality-reduced point cloud data and the feature points of the visible light image data, specifically including using KPCA for spatial dimensionality reduction and using LDA for feature extraction of the dimensionality-reduced data; S3. Determine the projection plane, project the point cloud data through the projection model to generate a depth-echo intensity image, and construct a 3D-2D coordinate mapping relationship between the point cloud data and the depth-echo intensity image; S4. According to the 3D-2D coordinate mapping relationship, map the feature points of the visible light image data to the point cloud data.

2. The method for fusing laser point cloud and visible light image according to claim 1, characterized in that, S1 includes first performing voxelization processing on the point cloud data. The formula for calculating the centroid of the point cloud voxel grid is: Among them, x c , y c , z c are the x, y, and z coordinates of the point cloud data in the voxel grid, p is the total number of points in the voxel grid, and i is the i-th point.

3. The method for fusing laser point cloud and visible light image according to claim 2, characterized in that, Among them, the kernel function of KPCA uses a polynomial kernel.

4. The method for fusing laser point cloud and visible light image according to claim 1, characterized in that, S3 includes using the feature points of the point cloud data as reference points and constructing the 3D-2D coordinate mapping relationship by using the DLT method.

5. The method for fusing laser point cloud and visible light image according to claim 4, characterized in that, Further combine the least squares method to construct the 3D-2D coordinate mapping relationship.

6. The method for fusing laser point cloud and visible light image according to claim 1, characterized in that, The calculation formula adopted by the projection model in S3 includes: Z = d Among them, x and y are the x and y coordinates in the projection coordinate system, X and Y are the coordinates of X and Y in the world coordinate system, X res , Y res are the resolutions of the depth map, FOV represents the projection vector, FOV h is the scalar decomposed in the horizontal direction of FOV, FOV v is the scalar decomposed in the vertical direction of FOV, and d is the depth value.

7. The method for fusing laser point cloud and visible light image according to claim 1, characterized in that, The visible light image data is the RGB data of the visible light image.

8. A system for fusing laser point cloud and visible light image, characterized in that, It includes: A data dimensionality reduction module configured to perform spatial dimensionality reduction on the point cloud data and map it to the same dimension as the visible light image data; A feature extraction module configured to extract the feature points of the dimensionality-reduced point cloud data and the feature points of the visible light image data through the SIFT algorithm; The feature extraction module uses KFDA-SIFT to extract the feature points of the dimensionality-reduced point cloud data and the feature points of the visible light image data, specifically including using KPCA for spatial dimensionality reduction and using LDA for feature extraction of the dimensionality-reduced data; A coordinate mapping module configured to determine the projection plane, project the point cloud data through the projection model to generate a depth-echo intensity image, and construct a 3D-2D coordinate mapping relationship between the point cloud data and the depth-echo intensity image; A registration and fusion module that maps the feature points of the visible light image data to the point cloud data according to the 3D-2D coordinate mapping relationship.

Images