Three-dimensional lidar ranging and intensity information fusion and image representation method

Abstract

This invention belongs to the field of image fusion technology and provides a method for fusing and visually representing 3D LiDAR ranging and intensity information. An azimuth map is generated based on the spatial coordinates of the LiDAR point cloud, and an intensity map is generated based on the intensity information. An encoder-decoder network-based image fusion method is employed, constructing the encoder-decoder network. The encoder is responsible for extracting image features, and the decoder is responsible for reconstructing the image. Image fusion of the azimuth and intensity maps is performed at the feature layer to obtain a fused image with clear edge textures and good discriminability. This invention can provide effective support for tasks such as scene understanding and segmentation of LiDAR point clouds.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image fusion, and relates to a three-dimensional laser radar ranging and intensity information fusion and image expression method, in particular to a three-dimensional laser point cloud azimuth angle graph and intensity graph fusion method. BACKGROUND

[0002] Vision is an important way for robots to perceive the environment and is an important part of a robot system. In general, the imaging process of a vision sensor is limited by the lighting conditions in the environment and cannot normally image in a dark environment. In addition, a monocular vision sensor cannot obtain spatial depth information and cannot provide the distance of objects in the scene.

[0003] Laser radar collects environmental point cloud information and is not affected by lighting conditions, and can provide accurate three-dimensional spatial position information. Therefore, most current perception tasks use a multi-sensor system composed of laser radars and cameras, etc., to take advantage of different sensors to obtain multiple feature information of the same scene, improve the scene perception dimension and decision reliability, and make up for the limitations of a single sensor, but also cause problems such as complex system setup and relatively high cost. Since the information obtained by laser radar is richer than that of ordinary vision sensors, a single laser radar imaging technology is considered. On the one hand, it reduces the cost, and on the other hand, it can use mature image processing technology. Currently, the main method for generating a two-dimensional image model from laser point cloud data is a depth map.

[0004] A depth map is a two-dimensional image obtained by mapping the distance information of point cloud data to the pixel value of a grayscale image. The advantage of a depth map is that the image model is simple to calculate and has less computation time, but the pixel mapping process is not sensitive to distance value changes, the pixel representation error is large, resulting in poor texture performance of the image edge, and the scene details are weakened. Currently, some work has optimized the depth map, and patent CN105574905A, a two-dimensional image expression method for three-dimensional laser point cloud data, proposes a new method for calculating a depth image. The three-dimensional point cloud is projected onto a two-dimensional horizontal plane, the main direction of the point cloud distribution is determined using statistical principles, and finally the distance value of each point on the horizontal plane to the main direction is calculated as the pixel value of the grayscale image. Compared with the traditional depth map, the image generated by this method has clearer texture and better boundary discrimination, but the image effect generated by this method depends on a large amount of point cloud data, and an external device needs to be added to the laser radar to obtain dense point cloud data.

[0005] For standalone multi-line lidar, the azimuth map is also a two-dimensional image model of the laser point cloud. It maps azimuth angle values ​​to pixel values ​​to obtain a grayscale image (see reference: Scaramuzza D, Harati A, Siegwart R. Extrinsic self-calibration of a camera and a 3D laser range finder from natural scenes[C]. 2007 IEEE / RSJ International Conference on Intelligent Robots and Systems, 2007: 4164-4169.). When calculating the azimuth angle, for each laser ranging point, first determine the vector from the lidar viewpoint to the selected ranging point, then determine the vector from the selected ranging point to its adjacent ranging points in a specific direction, and calculate the angle between these two vectors as the azimuth angle. Currently, azimuth maps are mainly used in lidar and camera calibration tasks due to their strong edge description capabilities and rich texture details.

[0006] There is an urgent need for a two-dimensional image imaging method that is only applicable to three-dimensional point cloud data from multi-line lidar, and that meets the requirements of convenient operation, low computational load, and relatively good scene segmentation effect. Summary of the Invention

[0007] This invention addresses the problems existing in the current image imaging process by proposing a method for fusing three-dimensional lidar ranging and intensity information and representing it in an image. Specifically, it utilizes a deep learning-based image fusion method to fuse the azimuth and intensity maps of three-dimensional lidar point cloud data, generating a two-dimensional grayscale image with rich texture and preserving the object's intensity information.

[0008] An azimuth map maps azimuth angle values ​​to pixel values, resulting in a grayscale image. When calculating the azimuth, for each laser ranging point, first, the vector from the lidar viewpoint to the selected ranging point is determined. Then, the vectors from the selected ranging point to adjacent ranging points in a specific direction (including horizontal, vertical, main diagonal, and oblique diagonal directions) are determined. The angle between these two vectors is calculated as the azimuth. Azimuth maps have strong edge description capabilities and rich image texture information. However, because pixel values ​​are only related to the vector angle, the grayscale pixel representations of different objects can be similar. Furthermore, image quality is significantly affected by the viewpoint selection.

[0009] An intensity map maps the pulse echo intensity value of a laser ranging point to grayscale pixel values, generating an intensity image. For the emitted light from a LiDAR, the intensity of reflected light varies depending on the material of the object. Generally, objects with high reflectivity have strong reflective ability, resulting in a high intensity value in the point cloud and a corresponding large pixel value in the grayscale image; conversely, objects with low reflectivity have weak reflective ability, resulting in a low intensity value in the point cloud and a corresponding small pixel value in the grayscale image. This characteristic of reflectivity can be used as a basis for determining the category of objects in an image. The advantage of intensity maps is their excellent ability to distinguish objects with different reflectivities in a scene. However, they are affected by factors such as the incident angle of the emitted light and whether the object is moving, which can lead to adjacent objects having similar intensity values, resulting in indistinct object edges.

[0010] The specific technical solution of the present invention is as follows:

[0011] A method for fusing and visually representing three-dimensional lidar ranging and intensity information, characterized by the following steps:

[0012] Step 1: Construct an image fusion dataset, including the azimuth map and intensity map of the laser point cloud data;

[0013] Step 2, construct a deep learning network; the deep learning network includes an encoder network and a decoder network; the specific network structure diagram is as follows. Figure 1 As shown. The encoder network contains three or more convolutional layers for extracting features from the input image and outputting a feature layer; the input image is an azimuth map or intensity map of laser point cloud data; the decoder network contains one or more convolutional layers for re-decoding the feature layer extracted by the encoder network and outputting a single-channel fused image;

[0014] Step 3: Train the deep learning network described in Step 2. The specific formula for calculating the required loss function is as follows: Loss = MSE + λ(1-SSIM)

[0015] Where Loss represents the joint loss function, MSE represents the mean squared error loss function, SSIM represents the structural similarity loss function, and λ is the coefficient;

[0016] Step 4: Using the deep learning network trained in Step 3, fuse the azimuth map and intensity map of the laser point cloud data: The fusion process is as follows... Figure 2 As shown.

[0017] Step 4.1: Use the trained encoder network to extract features from the azimuth map and intensity map respectively to obtain feature layers for the two images;

[0018] Step 4.2: Using a preset fusion strategy, fuse the feature layers corresponding to the two images obtained in Step 4.1 by channel.

[0019] Step 4.3: Use the trained decoder network to decode the feature layers fused in Step 4.2 to obtain the final fused image.

[0020] The image fusion dataset is primarily constructed by generating azimuth and intensity maps of the laser point cloud data. Each frame of point cloud data acquired by the lidar contains information such as three-dimensional spatial coordinates and reflection intensity. The three-dimensional spatial coordinates are used to calculate the azimuth and generate the azimuth map. The reflection intensity is used to generate the intensity map. Since the point cloud data is represented in three-dimensional space, it needs to be mapped to a two-dimensional plane to represent it as a two-dimensional image model. The number of rows in the two-dimensional plane is determined based on the number of lines in the point cloud, and the number of columns in the two-dimensional plane is determined based on the number of points in each line. Each point in the two-dimensional plane corresponds to an actual three-dimensional point cloud coordinate. Further, in step 1, the method for obtaining the azimuth map of the laser point cloud data is as follows: calculate the azimuth values ​​of all selected points in the laser point cloud data, and use the azimuth values ​​as pixel values ​​of the grayscale image to obtain the azimuth map of each frame of point cloud data. Figure 5 Provide an example of an azimuth diagram for a given scene.

[0021] The formula for calculating the azimuth angle of the selected point is as follows:

[0022]

[0023] Among them, BA m Indicates that the selected point P m The calculated azimuth angle is used to select point P. m (x m ,y m ,z m (x) m ,y m ,z m ) represents point P m Three-dimensional spatial coordinates; This represents the three-dimensional spatial vector from the lidar viewpoint to the selected point, where the spatial coordinates of the lidar viewpoint are O(x). o ,y o ,z o ), where (x o ,y o ,z o () represents the three-dimensional spatial coordinates of viewpoint O; This represents the three-dimensional spatial vector from the selected point to its horizontal right-side adjacent point. The selected point P... m The adjacent point to the right horizontally is P. n (x n ,y n ,z n ), where (x n ,y n ,zn ) represents point P n The formula uses three-dimensional spatial coordinates; the numerator calculates the dot product of two vectors, and the denominator is the product of the magnitudes of the two vectors. A schematic diagram of the calculation is shown below. Figure 3 As shown.

[0024] Furthermore, the method for obtaining the intensity map of the laser point cloud data is as follows: the reflection intensity value is mapped to the range of 0 to 255 as the grayscale pixel value to obtain the intensity map; the specific mapping formula is as follows:

[0025]

[0026] Where Pixel represents the pixel value of the intensity map, and In represents the intensity value. max In min These represent the maximum and minimum intensity values ​​in the current frame's point cloud data, respectively. Figure 6 Provide an example of an intensity map for a given scene.

[0027] Furthermore, in the encoder network described in step 2, the first convolutional layer preprocesses the input image, extracting features while simultaneously increasing the number of image channels from a single channel to an A-dimensional output. The remaining convolutional layers are connected in series using a channel concatenation mechanism to form a deep feature extraction module. Due to the introduction of the channel concatenation mechanism, the channel depth needs to be expanded before the last three convolutional layers in this module extract features. Specifically, the input channel of the second convolutional layer is A-dimensional, and the output channel is B-dimensional; the input channel of the third convolutional layer is A+B-dimensional, and the output channel is B-dimensional; when the number of convolutional layers is greater than 3, the input channels of all convolutional layers after the third convolutional layer are 2B-dimensional, and the output channels are all B-dimensional.

[0028] Furthermore, in the decoder network described in step 2, when there is more than one convolutional layer, each convolutional layer progressively reduces the feature layer dimension until the final output is a single-channel image. This achieves the effect of making full use of the feature layer information as much as possible.

[0029] During the network training phase, the input and ground truth images are the same, with the aim of training the encoder and decoder's feature extraction and image reconstruction capabilities. The Adam optimizer is used during training, which, in calculating the update step size, considers both the mean and squared gradients, rather than being directly determined by the current gradient, thus achieving adaptive adjustment. The loss function consists of two parts: MSE (Mean Squared Error) and SSIM (Structure Similarity Index Measure), used to evaluate the similarity between two images.

[0030] Furthermore, in step 3, the mean squared error loss function (MSE) is used to calculate the error between the network's predicted output image and the ground truth image pixel by pixel. The specific calculation formula is as follows:

[0031]

[0032] Where x and y represent the network output image and ground truth image, respectively, MSE is the calculated error, i represents the pixel index, and x i It is the value of the i-th pixel in the network output image, y i is the value of the i-th pixel in the true image, and m is the total number of pixels.

[0033] Furthermore, in step 3, the structural similarity loss function SSIM is designed based on the perceptual characteristics of human eyes regarding images. It considers the brightness, contrast, and structural information of the image, and can better reflect the human eye's perception of image quality. The specific calculation formula for SSIM is as follows:

[0034]

[0035] Where x and y represent the network's predicted output image and ground truth image, respectively, u x and u y Let σ represent the pixel mean values ​​of x and y, respectively. x 2 and σ y 2 Let σ represent the pixel variances of x and y, respectively. xy This represents the pixel covariance of x and y, where c1 and c2 are constants, typically set to (0.01L). 2 and (0.03L) 2 , where L is the dynamic range of the image, and its value is 255.

[0036] Furthermore, based on the difference in magnitude between MSE and SSIM, λ can be dynamically set to 1, 10, 100, or 1000 to make them of the same magnitude, thereby increasing the impact of SSIM on the training process.

[0037] The network is trained using all azimuth and intensity maps from the dataset. A fixed number of azimuth and intensity maps are randomly selected for each batch and fed into the network for reconstruction training. Training with the complete dataset improves the network's generalization and robustness.

[0038] After network training is completed, the weight parameters of the encoder-decoder network are obtained, which can be used to fuse features from the azimuth map and intensity map.

[0039] Furthermore, in step 4.1, the azimuth map and the intensity map have the same channel dimension in their feature layers. Since the pixels of the azimuth map and the intensity map are in one-to-one correspondence, there is no need to register them again, and image fusion can be performed directly.

[0040] Furthermore, in step 4.2, the preset fusion strategy is a method of calculating the mean value pixel by pixel.

[0041] The beneficial effects of this invention are that it provides a visual representation method specifically for multi-line lidar that effectively combines azimuth and intensity information. The fused image generated by this method possesses clear edge textures and other features, and can clearly distinguish different objects. This invention can effectively assist in tasks such as scene understanding and segmentation of lidar point cloud data. Attached Figure Description

[0042] Figure 1 The encoder-decoder network is used for the training process.

[0043] Figure 2 For integration process structure diagram.

[0044] Figure 3 This is a schematic diagram for azimuth calculation.

[0045] Figure 4 This is a frame of 3D laser point cloud data from the dataset.

[0046] Figure 5 for Figure 4 The corresponding azimuth diagram.

[0047] Figure 6 for Figure 4 The corresponding intensity map.

[0048] Figure 7 for Figure 4 The corresponding fusion diagram of the present invention.

[0049] Figure 8 for Figure 5 The superpixel segmentation results.

[0050] Figure 9 for Figure 6 The superpixel segmentation results.

[0051] Figure 10 for Figure 7 The superpixel segmentation results. Detailed Implementation

[0052] The specific embodiments of the present invention are described in detail below with reference to the technical solutions and accompanying drawings.

[0053] A method for fusing and visualizing 3D lidar ranging and intensity information includes the following steps:

[0054] Step 1: Construct the training dataset

[0055] The dataset used is the SemanticKITTI dataset provided by the research team at the University of Bonn, Germany. Azimuth and intensity maps were generated using the point cloud data from the `data_odometry_velodyne` set within this dataset. The LiDAR used in this dataset is the Velodyne HDL-64E. The 00 sequence from the dataset was selected for creating the azimuth and intensity maps, resulting in a total of 4541 point cloud frames, which generated 4541 azimuth and intensity maps. These 9082 images were placed in the same folder as the training dataset.

[0056] Step 2: Constructing a deep learning network

[0057] This embodiment utilizes an encoder-decoder network to fuse the azimuth and intensity maps of 3D point cloud data. The network was built, trained, and tested using the PyTorch framework, and its specific structure is shown in Table 1. The convolutional layers use ordinary convolutions with a size of 3x3 and a stride of 1. During convolution operations, to maintain the consistency of edge pixels and prevent them from being stretched or compressed, reflection padding is applied around the edges of the image. A mirror-symmetric padding is applied around the outermost pixel of the input image. No activation function is applied after the last convolutional layer of the decoder network; all other convolutional layers use the ReLU activation function. In the experiment, both the input and output images were 64*1024 pixels with 1 channel.

[0058] Table 1. Network Structure Table

[0059]

[0060] Step 3: Training the Neural Network

[0061] This invention uses an Nvidia 1080Ti graphics card as the training platform. A neural network is built using the PyTorch framework, and the training and testing of the network are implemented. During network training, the batch size is set to 8, the total number of epochs is set to 20, and the weight file is saved after each training epoch. The learning rate of the Adam optimizer is set to a fixed value of 0.0001.

[0062] Step 4: Test the network

[0063] After training, the weight file with the smallest training loss function is selected as the weights of the encoder-decoder network. The encoder extracts features from the azimuth map and intensity map respectively, and fuses the feature layers using a selected fusion strategy. The decoder reconstructs the fused feature layers into a fused image, completing the fusion of the azimuth map and intensity map.

[0064] To intuitively evaluate the advantages of this invention, a specific frame of laser point cloud data from the dataset (e.g., ...) was selected. Figure 3 The diagram is displayed as shown, and the corresponding azimuth diagram is generated (e.g., ...). Figure 4 As shown), intensity map (as shown) Figure 5 (as shown) and the fusion diagram of the present invention (as shown) Figure 6 As shown in the image, the azimuth map is not good at distinguishing objects in the scene, while the edge texture of the intensity map is not clear enough. Superpixel segmentation is performed on the azimuth map, intensity map, and blended map using the SLIC superpixel segmentation algorithm, with the superpixel block size dynamically adjustable. Setting the average superpixel size to 20, the segmentation results are as follows: Figure 7 , Figure 8 , Figure 9 As shown. Precision is used as a metric to evaluate the results of superpixel segmentation. The formula for calculating precision is as follows:

[0065]

[0066] Where Precision represents the accuracy, TP indicates that the model predicted a positive example and it actually was, and FP indicates that the model predicted a positive example but it actually was a negative example. The ground truth is constructed from the semantic labels provided in the dataset. During testing, 100 images each of the azimuth map, intensity map, and fused image were selected, superpixel segmentation was performed, and the accuracy was calculated. The accuracy results calculated based on different superpixel block sizes are shown in the table below:

[0067] Average superpixel size Azimuth map Presion Intensity map Presion Fusion map Presion 20 70.21％ 71.96％ 73.87％ 30 71.24％ 73.18％ 74.85％ 40 70.72％ 73.00％ 73.80％ 50 66.54％ 66.03％ 69.64％

[0068] Compared to azimuth and intensity maps, the fused image of this invention achieves better segmentation results, preserving both scene boundary texture information and intensity information of different objects, thus improving segmentation performance. Therefore, this invention can effectively assist in tasks such as object recognition, image segmentation, and scene understanding.

Claims

1. A method for fusing and image representation of range and intensity information of a three-dimensional laser radar, characterized in that, The method comprises the following steps: Step 1, constructing a data set for image fusion, including an azimuth angle graph and an intensity graph of laser point cloud data; Step 2, constructing a deep learning network; the deep learning network comprises an encoder network and a decoder network; the encoder network comprises more than three convolutional layers, and is used for extracting features of an input image and outputting a feature layer; the input image is an azimuth angle graph or an intensity graph of laser point cloud data; the decoder network comprises more than one convolutional layer, and is used for decoding the feature layer extracted by the encoder network and outputting a single-channel fusion image; Step 3, training the deep learning network in step 2, and a specific calculation formula of a required loss function is as follows: Loss = MSE + λ (1-SSIM) Wherein, Loss represents a joint loss function, MSE represents a mean square error loss function, SSIM represents a structural similarity loss function, and λ is a coefficient; Step 4, using the deep learning network trained in step 3 to fuse the azimuth angle graph and the intensity graph of laser point cloud data: Step 4.1, using the trained encoder network to extract features of the azimuth angle graph and the intensity graph respectively, to obtain feature layers of the two kinds of images; Step 4.2, using a preset fusion strategy to fuse the feature layers of the two kinds of images obtained in step 4.1 according to channels; Step 4.3, using the trained decoder network to decode the feature layer fused in step 4.2, to obtain a final fusion image.

2. The method according to claim 1, wherein, In step 1, the method for obtaining the azimuth angle graph of the laser point cloud data is as follows: the azimuth angle values of selected points of all laser point cloud data are calculated, the azimuth angle values are taken as pixel values of a gray image, and an azimuth angle graph of each frame of point cloud data is obtained; and a calculation formula of the azimuth angle of the selected point is as follows: where BA m represents the azimuth angle calculated by the selected point P m ; m (x m ,y m ,z m ) where (x m ,y m ,z m ) represents the three-dimensional space coordinates of the point P m ; represents the three-dimensional space vector from the laser radar viewpoint to the selected point, the space coordinates of the laser radar viewpoint being O(x o ,y o ,z o ), where (x o ,y o ,z o ) represents the three-dimensional space coordinates of the viewpoint O; represents the three-dimensional space vector from the selected point to the horizontally right adjacent point, the selected point P m ; the horizontally right adjacent point of P n (x n ,y n ,z n ), where (x n ,y n ,z n ) represents the three-dimensional space coordinates of the point P n ; the numerator part of the formula calculates the dot product between the two vectors, and the denominator part is the product of the norms of the two vectors.

3. The three-dimensional laser radar ranging and intensity information fusion and imaging representation method according to claim 1, characterized in that, The method for obtaining the intensity graph of the laser point cloud data is as follows: a reflection intensity value is mapped to a range of 0-255 as a gray pixel value, and an intensity graph is obtained; and a specific mapping formula is as follows: wherein, Pixel represents the pixel value of the intensity map, In represents the intensity value, In max , In min respectively represent the maximum value and the minimum value of the intensity in the current frame point cloud data.

4. The three-dimensional laser radar ranging and intensity information fusion and imaging representation method according to claim 1, characterized in that, In the encoder network in step 2, a first convolutional layer pre-processes an input image, extracts features, and at the same time, upgrades the number of image channels from a single channel to A-dimensional channel output; the remaining convolutional layers are connected in series by using a channel splicing mechanism to form a deep feature extraction module, wherein the input channel of a second convolutional layer is A-dimensional, and the output channel is B-dimensional; the input channel of a third convolutional layer is A+B-dimensional, and the output channel is B-dimensional; when the number of convolutional layers is greater than 3, the input channel of the convolutional layer after the third convolutional layer is 2B-dimensional, and the output channel is B-dimensional.

5. The three-dimensional laser radar ranging and intensity information fusion and imaging representation method according to claim 1, characterized in that, In the decoder network in step 2, when the number of convolutional layers is greater than 1, each convolutional layer gradually reduces the dimension of the feature layer, until the last output is a single-channel output image.

6. The method of claim 1, wherein, In step 3, the mean square error loss function MSE is used to calculate the error between a network prediction output image and a true value image pixel by pixel, and a specific calculation formula is as follows: where x and y represent the network output image and the ground truth image, respectively, MSE is the calculated error, i represents the pixel index, x i is the i-th pixel value of the network output image, y i is the i-th pixel value of the ground truth image, and m is the total number of pixels.

7. The method of claim 1, wherein, In step 3, a specific calculation formula of the structural similarity loss function SSIM is as follows: where x and y represent the network predicted output image and ground truth image, respectively, u x and u y represent the pixel mean of x and y, respectively, σ x 2 and σ y 2 represent the pixel variance of x and y, respectively, σ xy represents the pixel covariance of x and y, and c1 and c2 are constants.

8. The method of claim 1, wherein, In step 4.1, the channel dimension of the feature layer of the azimuth angle graph and the intensity graph is the same.

9. The three-dimensional laser radar ranging and intensity information fusion and imaging expression method according to claim 1, characterized in that In step 4.2, the preset fusion strategy is a method of calculating the mean value pixel by pixel.

Images