An end-to-end unsupervised laser odometry combined with semantic segmentation method
By combining laser odometry and semantic segmentation in a deep learning network, and utilizing Siamese networks and self-attention mechanisms, the problems of insufficient pose estimation accuracy and resource waste in laser odometry in dynamic environments are solved, achieving an efficient combination of pose and semantic segmentation.
Patent Information
- Application Number
- CN202310325830.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-30
- Publication Date
- 2026-02-24
- Estimated Expiration
- 2043-03-30
AI Technical Summary
Existing laser odometry methods struggle to effectively remove the influence of dynamic objects on pose estimation in dynamic environments and lack direct integration with semantic segmentation, resulting in wasted computational resources and insufficient accuracy.
Laser odometry and semantic segmentation are combined into the same deep learning network. Motion and semantic features are extracted through Siamese networks and self-attention mechanisms. The network is trained using a unified loss function and outputs inter-frame pose and semantic segmentation results.
This improves the pose estimation and semantic segmentation accuracy of laser odometry in dynamic environments, saves computing resources, and achieves efficient end-to-end pose and semantic information extraction.
Smart Images
Figure CN116342883B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a laser odometry method in the field of deep learning, and more particularly to a method combining end-to-end unsupervised laser odometry with semantic segmentation. Background Technology
[0002] Laser odometry refers to a method that uses LiDAR (Light Detection and Ranging) for recursive position and pose estimation. The most well-known of these methods is LOAM (LiDAR odometry and mapping in real-time). Most current leading laser odometry solutions utilize the overall architecture and approach of the LOAM algorithm, with improvements and refinements in certain aspects.
[0003] The goal of semantic segmentation is to divide a given point cloud into subsets based on the semantic meaning of the points, and assign a class label to each data point in the input modality. Classic point cloud semantic segmentation tools include PointNet, RangeNet++, and others.
[0004] In recent years, deep learning has surpassed traditional manual methods in pattern recognition. It can learn effective features from massive amounts of training data, greatly advancing tasks such as detection and classification. Given the rapid development of deep learning, laser odometry and semantic segmentation have also begun to incorporate deep learning methods.
[0005] Due to the success of deep learning in the image domain, the mainstream method for laser semantic segmentation currently uses deep learning methods. However, the laser odometry field uses more traditional geometric methods. These methods require first extracting point and surface features from the point cloud, then matching feature points in two frames of point clouds using some designed rules, and finally using optimization methods such as Newton's method to optimize the feature point matching error between frames based on an initial pose, thereby obtaining a better pose estimation result.
[0006] The theory of laser odometry is relatively mature, and existing methods can provide very accurate results in ideal static environments. However, many dynamic objects exist in real-world environments, which significantly impact pose estimation. To mitigate this impact, semantic segmentation methods are incorporated into the laser odometry framework. By identifying the semantic information in point clouds, geometric laser odometry infers which points might belong to dynamic objects, and then uses rules to remove the influence of these points on pose estimation. This is a relatively direct approach.
[0007] Deep learning-based laser odometry methods have been increasingly studied in recent years. A common approach combining deep learning is to use deep learning networks to replace certain modules of geometric laser odometry, such as feature extraction and feature matching. Another more direct method is to use a single network to perform all the work of laser odometry, taking two frames of point clouds as input and directly outputting the pose transformation between the point clouds end-to-end.
[0008] Unlike geometric methods, there are currently few end-to-end laser odometry methods that directly combine it with semantic segmentation. However, in fields such as autonomous driving, both laser odometry and semantic segmentation are necessary tasks. Both end-to-end laser odometry and semantic segmentation utilize deep learning networks, and these two tasks are deeply interconnected, mutually reinforcing each other. Combining them would not only allow the use of a single network, saving computational resources, but also enable the network to learn both geometric motion information and semantic information simultaneously, improving the accuracy of pose prediction and semantic segmentation. Summary of the Invention
[0009] To address the problems and needs in the background technology, the present invention aims to provide a method combining end-to-end unsupervised laser odometry and semantic segmentation. This method integrates self-supervised end-to-end laser odometry and semantic segmentation into the same deep learning network, tightly coupling the two tasks, removing the influence of dynamic objects on laser odometry, and simultaneously estimating the inter-frame pose transformation of point clouds and predicting the semantic segmentation results of point clouds. This provides a deep learning-based laser odometry and semantic segmentation method for autonomous driving.
[0010] The objective of this invention is achieved through the following technical solution:
[0011] Step 1: After preprocessing the collected point cloud data, a training set is obtained;
[0012] Step 2: Build a laser odometry and semantic segmentation network, construct a network loss function, and train the laser odometry and semantic segmentation network using the training set based on the network loss function to obtain the trained laser odometry and semantic segmentation network.
[0013] Step 3: Input the vertex graphs of the two adjacent frames to be predicted into the trained laser odometry and semantic segmentation network, and predict and output the semantic segmentation results of the current two frames and the corresponding inter-frame poses.
[0014] Step 1 specifically involves:
[0015] Project two adjacent point clouds onto a spherical coordinate system to obtain the corresponding two-frame vertex maps; traverse the remaining two adjacent point clouds and perform spherical projection on each, finally obtaining all groups of network inputs and forming a training set.
[0016] In step 2, the laser odometry and semantic segmentation network includes two backbone networks, two semantic segmentation networks, a feature aggregation module, an uncertainty prediction module, and a pose prediction network.
[0017] The first backbone network is connected to the first semantic segmentation network, and the second backbone network is connected to the second semantic segmentation network. Both the first and second backbone networks are connected to the feature aggregation module. Both the first and second semantic segmentation networks are connected to the feature aggregation module. The feature aggregation module is connected to the uncertainty prediction module and the pose prediction network. The vertex graph of the previous frame is input into the first semantic segmentation network, which outputs the semantic segmentation result of the previous frame. The vertex graph of the next frame is input into the second semantic segmentation network, which outputs the semantic segmentation result of the next frame. The uncertainty prediction module outputs the uncertainty graph, and the pose prediction network outputs the predicted inter-frame pose.
[0018] The first backbone network and the second backbone network are twin networks, and the first semantic segmentation network and the second semantic segmentation network are twin networks. Specifically, the first backbone network includes five downsampling modules, and the first semantic segmentation network includes five upsampling modules. The first to fifth downsampling modules are connected in sequence, and the first to fifth upsampling modules are connected in sequence. The fifth upsampling module is connected to the feature aggregation module. The first downsampling module is connected to the fifth upsampling module, and the first downsampling module is connected to the feature aggregation module. The second downsampling module is connected to the fourth upsampling module, and the second downsampling module is connected to the feature aggregation module. The third downsampling module is connected to the third upsampling module, and the third downsampling module is connected to the feature aggregation module. The fourth downsampling module is connected to the second upsampling module, and the fourth downsampling module is connected to the feature aggregation module. The fifth downsampling module is connected to the first upsampling module, and the fifth downsampling module is connected to the feature aggregation module. The fifth upsampling module outputs the semantic segmentation result.
[0019] The feature aggregation module includes at least one self-attention combination module. Each self-attention combination module includes two self-attention mechanisms and one cross-self-attention mechanism. The outputs of the first backbone network and the first semantic segmentation network are both input into the first self-attention mechanism of each self-attention combination module. The outputs of the second backbone network and the second semantic segmentation network are both input into the second self-attention mechanism of each self-attention combination module. The first and second self-attention mechanisms are both connected to the cross-self-attention mechanism. If there are multiple self-attention combination modules, the output of the previous self-attention combination module is input into the first and second self-attention mechanisms of the next self-attention combination module, respectively. The output of the last self-attention combination module is used as the output of the feature aggregation module.
[0020] In step 2, the network loss function is obtained by adding the loss function of laser odometry and the loss function of semantic segmentation.
[0021] The loss function of the laser odometry is obtained by adding the projection loss and the field of view loss, as shown in the following formula:
[0022]
[0023]
[0024] Among them, L icp L represents the projection loss function value. fov V represents the field of view loss function value. t+1 This represents the vertex graph of the next frame, v t+1 σ represents a point at a pixel position in the vertex graph of the next frame. t+1 T represents the uncertainty of the uncertainty map corresponding to the pixel position. t,t+1 Represents the pose predicted by the network between two frames; |d(v t+1 |1 represents the distance of the pixel in the next frame from the normal vector of the pixel in the previous frame when the pixel in the vertex graph of the next frame is projected onto the coordinate system of the point cloud of the previous frame using the predicted pose; II() is the unit step function, w and h are the width and height of the vertex graph, respectively, and π() represents the projection function from the camera coordinate system to the pixel coordinate system.
[0025] The initial weight coefficient for the vision loss is 1 in the first cycle of the second training phase, and is adjusted to 0 after one training cycle.
[0026] The training process for the laser odometry and semantic segmentation network is as follows:
[0027] In the first training phase, the first and second backbone networks, as well as the first and second semantic segmentation networks, are trained, along with the fixed feature aggregation module, uncertainty prediction module, and pose prediction network, with a training period of N1. In the second training phase, the learning rates of the first and second backbone networks and the first and second semantic segmentation networks are reduced, and the entire laser odometry and semantic segmentation network is trained again, with a training period of N2.
[0028] The normal vector direction of the point in the previous frame is calculated using the following method:
[0029] For each pixel in the vertex graph of the previous frame, first subtract the 3D coordinates of the current pixel from the 3D coordinates of its four adjacent pixels to obtain the up, down, left, and right direction vectors respectively. Then, perform a cross product between the up and left direction vectors to obtain the first normal vector, and perform a cross product between the down and right direction vectors to obtain the second normal vector. Next, normalize the first and second normal vectors respectively and then add them together to obtain the initial normal vector. Finally, normalize the initial normal vector to obtain the normal vector of the current pixel.
[0030] Due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0031] This invention solves the problem of lack of semantic information in end-to-end laser odometry by allowing semantic segmentation and laser odometry to share the same backbone network in terms of network structure. Therefore, the extracted features have both motion geometry information and semantic information.
[0032] This invention combines laser odometry and semantic segmentation into the same neural network framework, and incorporates both pose and semantic errors into the loss function, enabling the use of the same network for pose and semantic prediction.
[0033] This invention uses self-attention and cross-attention mechanisms alternately in the feature aggregation module, which gives the features between frames a larger receptive field when they are combined, thus improving the effect of feature fusion. Attached Figure Description
[0034] Figure 1 This is a schematic diagram of the basic process of the present invention;
[0035] Figure 2 This is a schematic diagram of the feature fusion module of the present invention. Detailed Implementation
[0036] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the objectives and effects of the present invention will become more apparent.
[0037] Taking the publicly available autonomous driving dataset KITTI as an example, the 00-08 sequence was selected as the training set, and the 09-10 sequence as the validation set. The LiDAR device used was a 64-line Velodyne 3D LiDAR with a frequency of 10Hz. Two NVIDIA GTX 1080Ti graphics cards were used in the experiment, and the batch size was set to 16.
[0038] The specific implementation steps are as follows:
[0039] Step 1: After preprocessing the collected point cloud data, a training set is obtained;
[0040] 1.1) Project two adjacent point cloud frames onto a spherical coordinate system to obtain corresponding vertex maps. The three channels of each vertex map represent the corresponding 3D coordinates in the point cloud. Specifically, the point clouds at two time points after motion distortion removal are projected onto a spherical coordinate system, with the two angles used as the x-axis coordinates. After truncating the coordinate values, the point clouds are rasterized into image-like data. The x, y, and z coordinates of the camera coordinate system are used as the projected 3-channel data. The projected data is called a vertex map. If multiple points are projected onto the same pixel coordinate system, only the closest point is selected. The projection formula is as follows:
[0041]
[0042] Where u and v are two pixel coordinates, and d is the horizontal distance from the 3D coordinates to the LiDAR, d = (p x2 +p y2 ) 1 / 2 p x p y p z Let p represent the three coordinates of the point. x2 p x The square of p y2 p y The square of f h and f v The horizontal and vertical viewpoints are respectively, f. v =f vu +f vl f vu f represents the upper part of the vertical viewpoint. vl δ represents the lower part of the vertical viewing angle. h and δ v It refers to the pixel resolution in both the horizontal and vertical directions;
[0043] Traverse the remaining two adjacent point clouds, perform spherical projection on each, and finally obtain all the network inputs and form a training set.
[0044] Step 2: Build a laser odometry and semantic segmentation network, construct a network loss function, and train the laser odometry and semantic segmentation network using the training set based on the network loss function to obtain the trained laser odometry and semantic segmentation network.
[0045] like Figure 1 As shown, in step 2, the laser odometry and semantic segmentation network includes two backbone networks, two semantic segmentation networks, a feature aggregation module, an uncertainty prediction module, and a pose prediction network.
[0046] The first backbone network is connected to the first semantic segmentation network, and the second backbone network is connected to the second semantic segmentation network. Both the first and second backbone networks are connected to the feature aggregation module. The feature aggregation module is connected to the uncertainty prediction module and the pose prediction network. The vertex graph of the previous frame is input into the first semantic segmentation network, which outputs the semantic segmentation result of the previous frame. The vertex graph of the next frame is input into the second semantic segmentation network, which outputs the semantic segmentation result of the next frame. The uncertainty prediction module outputs an uncertainty map, which is a graph of size equal to the input vertex graph. Figure 1 The diagram shows the pose prediction network outputting the predicted inter-frame pose, which has 6 degrees of freedom: 3 degrees of translation and 3 degrees of rotation represented by Euler angles.
[0047] The first and second backbone networks are twin networks, as are the first and second semantic segmentation networks. Specifically, the first backbone network includes five downsampling modules, and the first semantic segmentation network includes five upsampling modules. The first to fifth downsampling modules are sequentially connected, as are the first to fifth upsampling modules. The fifth upsampling module is connected to the feature aggregation module. The first downsampling module is connected to the fifth upsampling module, and the first downsampling module is connected to the feature aggregation module. The second downsampling module is connected to the fourth upsampling module, and the second downsampling module is connected to the feature aggregation module. The third downsampling module is connected to the third upsampling module, and the third downsampling module is connected to the feature aggregation module. The fourth downsampling module is connected to the second upsampling module, and the fourth downsampling module is connected to the feature aggregation module. The fifth downsampling module is connected to the first upsampling module, and the fifth downsampling module is connected to the feature aggregation module. The fifth upsampling module outputs the semantic segmentation result. The backbone network is used to extract common features of laser odometry and semantic segmentation in each frame. The backbone network adopts a modified residual network Darknet structure, utilizing basic modules with residual structures. Throughout the process, five downsampling steps of 2 were performed along the horizontal axis. After each downsampling, the features were skipped to the subsequent semantic segmentation network and pose prediction network. The semantic segmentation network is a decoding structure opposite to the backbone network. It uses basic modules with residual structures to perform five upsampling steps of 2, and skipped to the features of the backbone network before each upsampling.
[0048] like Figure 2As shown, the feature aggregation module includes at least one self-attention combination module. Each self-attention combination module includes two self-attention mechanisms and one cross-self-attention mechanism to integrate the motion and semantic features of two frames, resulting in fused aggregated features with inter-frame motion information. The outputs of the first backbone network and the first semantic segmentation network are input into the first self-attention mechanism of each self-attention combination module, and the outputs of the second backbone network and the second semantic segmentation network are input into the second self-attention mechanism of each self-attention combination module. Both the first and second self-attention mechanisms are connected to the cross-self-attention mechanism. If multiple self-attention combination modules exist, the output of the previous self-attention combination module is input into the first and second self-attention mechanisms of the next self-attention combination module, respectively. The output of the last self-attention combination module is used as the output of the feature aggregation module, i.e., the aggregated features are output.
[0049] In step 2, the network loss function is obtained by weighted summation of the loss function of laser odometry and the loss function of semantic segmentation, with the weights obtained by parameter tuning during the experiment.
[0050] The loss function of the laser odometry is an unsupervised loss function, obtained by weighted summation of projection loss and field-of-view loss. The weights are obtained by parameter tuning during the experiment, and the formula is as follows:
[0051]
[0052]
[0053] Among them, L icp L represents the projection loss function value. fov V represents the field of view loss function value. t+1 This represents the vertex graph of the next frame, v t+1 σ represents a point at a pixel position in the vertex graph of the next frame. t+1 T represents the uncertainty of the uncertainty map corresponding to the pixel position. t,t+1 Represents the pose predicted by the network between two frames; |d(v t+1 |1 represents the distance of the pixel in the next frame from the normal vector of the pixel in the previous frame when the pixel in the vertex graph of the next frame is projected onto the coordinate system of the point cloud of the previous frame using the predicted pose; II() is the unit step function, w and h are the width and height of the vertex graph, respectively, and π() represents the projection function from the camera coordinate system to the pixel coordinate system.
[0054] The normal vector direction of the point in the previous frame is calculated using the following method:
[0055] For each pixel in the vertex graph of the previous frame, first subtract the 3D coordinates of the current pixel from the 3D coordinates of its four adjacent pixels to obtain the up, down, left, and right direction vectors respectively. Then, perform a cross product between the up and left direction vectors to obtain the first normal vector, and perform a cross product between the down and right direction vectors to obtain the second normal vector. Next, normalize the first and second normal vectors respectively and then add them together to obtain the initial normal vector. Finally, normalize the initial normal vector to obtain the normal vector of the current pixel.
[0056] The initial weight coefficient for the vision loss is 1 in the first cycle of the second training phase, and is adjusted to 0 after one training cycle.
[0057] The loss function used for semantic segmentation is the cross-entropy loss function.
[0058] The training process for the laser odometry and semantic segmentation network is as follows:
[0059] In the first training phase, the first and second backbone networks, as well as the first and second semantic segmentation networks, are trained, while the feature aggregation module, uncertainty prediction module, and pose prediction network are fixed. The training period is N1. In the second training phase, the learning rates of the first and second backbone networks and the first and second semantic segmentation networks are reduced, and the entire laser odometry and semantic segmentation networks are trained again. The training period is N2, and training ends. In the specific implementation, N1=100, N2=300, and the learning rate of the first and second backbone networks and the first and second semantic segmentation networks in the second training phase is 0.1 times that in the first phase. This method of training the semantic segmentation network first and then the laser odometry network results in a poorer final semantic segmentation result, but it ensures the accuracy of the pose result.
[0060] Step 3: Input the vertex graphs of the two adjacent frames to be predicted into the trained laser odometry and semantic segmentation network, and predict and output the semantic segmentation results of the current two frames and the corresponding inter-frame poses.
[0061] The above are merely specific embodiments of the present invention and should not be construed as limiting the scope of the present invention. Equivalent variations made by those skilled in the art based on this invention, as well as changes well-known to those skilled in the art, should still fall within the scope of the present invention.
Claims
1. A method combining end-to-end unsupervised laser odometry with semantic segmentation, characterized in that, It includes the following steps: Step 1: After preprocessing the collected point cloud data, a training set is obtained; Step 1 specifically involves: Project two adjacent point clouds onto a spherical coordinate system to obtain the corresponding two-frame vertex maps; traverse the remaining two adjacent point clouds and perform spherical projection on each, finally obtaining all groups of network inputs and forming a training set; Step 2: Build a laser odometry and semantic segmentation network, construct a network loss function, and train the laser odometry and semantic segmentation network using the training set based on the network loss function to obtain the trained laser odometry and semantic segmentation network. The laser odometry and semantic segmentation network includes two backbone networks, two semantic segmentation networks, a feature aggregation module, an uncertainty prediction module, and a pose prediction network. The first backbone network is connected to the first semantic segmentation network, and the second backbone network is connected to the second semantic segmentation network. Both the first and second backbone networks are connected to the feature aggregation module. Both the first and second semantic segmentation networks are connected to the feature aggregation module. The feature aggregation module is connected to the uncertainty prediction module and the pose prediction network. The vertex graph of the previous frame is input into the first semantic segmentation network, and the first semantic segmentation network outputs the semantic segmentation result of the previous frame. The vertex graph of the next frame is input into the second semantic segmentation network, and the second semantic segmentation network outputs the semantic segmentation result of the next frame. The uncertainty prediction module outputs the uncertainty graph, and the pose prediction network outputs the predicted inter-frame pose. The network loss function is obtained by adding the loss function of laser odometry and the loss function of semantic segmentation; Step 3: Input the vertex graphs of the two adjacent frames to be predicted into the trained laser odometry and semantic segmentation network, and predict and output the semantic segmentation results of the current two frames and the corresponding inter-frame poses.
2. The method for combining end-to-end unsupervised laser odometry with semantic segmentation according to claim 1, characterized in that, The first backbone network and the second backbone network are twin networks, and the first semantic segmentation network and the second semantic segmentation network are twin networks. Specifically, the first backbone network includes five downsampling modules, and the first semantic segmentation network includes five upsampling modules. The first to fifth downsampling modules are connected in sequence, and the first to fifth upsampling modules are connected in sequence. The fifth upsampling module is connected to the feature aggregation module. The first downsampling module is connected to the fifth upsampling module, and the first downsampling module is connected to the feature aggregation module. The second downsampling module is connected to the fourth upsampling module, and the second downsampling module is connected to the feature aggregation module. The third downsampling module is connected to the third upsampling module, and the third downsampling module is connected to the feature aggregation module. The fourth downsampling module is connected to the second upsampling module, and the fourth downsampling module is connected to the feature aggregation module. The fifth downsampling module is connected to the first upsampling module, and the fifth downsampling module is connected to the feature aggregation module. The fifth upsampling module outputs the semantic segmentation result.
3. The method for combining end-to-end unsupervised laser odometry with semantic segmentation according to claim 1, characterized in that, The feature aggregation module includes at least one self-attention combination module. Each self-attention combination module includes two self-attention mechanisms and one cross-self-attention mechanism. The outputs of the first backbone network and the first semantic segmentation network are both input into the first self-attention mechanism of each self-attention combination module. The outputs of the second backbone network and the second semantic segmentation network are both input into the second self-attention mechanism of each self-attention combination module. The first and second self-attention mechanisms are both connected to the cross-self-attention mechanism. If there are multiple self-attention combination modules, the output of the previous self-attention combination module is input into the first and second self-attention mechanisms of the next self-attention combination module, respectively. The output of the last self-attention combination module is used as the output of the feature aggregation module.
4. The method for combining end-to-end unsupervised laser odometry with semantic segmentation according to claim 1, characterized in that, The loss function of the laser odometry is obtained by adding the projection loss and the field of view loss, as shown in the following formula: in, This represents the projection loss function value. This represents the value of the field of view loss function. This represents the vertex graph of the next frame. This represents a point at a pixel location in the vertex graph of the next frame. This represents the uncertainty of the uncertainty map corresponding to the pixel location. This represents the pose predicted by the network between two frames. This represents the distance of a pixel in the next frame from the normal vector of a pixel in the previous frame when the pixel in the vertex map of the next frame is projected onto the coordinate system of the point cloud of the previous frame using the predicted pose. It is a unit step function. These are the width and height of the vertex graph, respectively. This represents the projection function from the camera coordinate system to the pixel coordinate system.
5. The method for combining end-to-end unsupervised laser odometry with semantic segmentation according to claim 4, characterized in that, The initial weight coefficient for the vision loss is 1 in the first cycle of the second training phase, and is adjusted to 0 after one training cycle.
6. The method for combining end-to-end unsupervised laser odometry with semantic segmentation according to claim 1, characterized in that, The training process for the laser odometry and semantic segmentation network is as follows: In the first training phase, the first and second backbone networks, as well as the first and second semantic segmentation networks, are trained, along with the fixed feature aggregation module, uncertainty prediction module, and pose prediction network. The training period is N1. In the second training phase, the learning rates of the first and second backbone networks and the first and second semantic segmentation networks are reduced, and the entire laser odometry and semantic segmentation network is trained again, with a training cycle of N2.
7. The method for combining end-to-end unsupervised laser odometry with semantic segmentation according to claim 4, characterized in that, The normal vector direction of the point in the previous frame is calculated using the following method: For each pixel in the vertex graph of the previous frame, first subtract the 3D coordinates of the current pixel from the 3D coordinates of its four adjacent pixels to obtain the up, down, left, and right direction vectors respectively. Then, perform a cross product between the up and left direction vectors to obtain the first normal vector, and perform a cross product between the down and right direction vectors to obtain the second normal vector. Next, normalize the first and second normal vectors respectively and then add them together to obtain the initial normal vector. Finally, normalize the initial normal vector to obtain the normal vector of the current pixel.