A point cloud based multi-view semantic embedding scene recognition method
By using a multi-view semantic embedding method, point cloud data is projected into multi-view images and feature fusion is performed, which solves the shortcomings of point cloud scene recognition algorithms in terms of computational efficiency and recognition accuracy, and achieves efficient and highly accurate scene recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHEASTERN UNIV CHINA
- Filing Date
- 2024-07-04
- Publication Date
- 2026-05-12
AI Technical Summary
Existing point cloud scene recognition algorithms are insufficient in terms of computational efficiency and recognition accuracy, making it difficult to maintain high efficiency and high accuracy in complex environments, and the multi-view projection feature relationship is not effectively utilized.
A multi-view semantic embedding method is adopted to project point cloud data into forward-looking and bird's-eye view images. Combined with a semantic segmentation network and a self-attention module, descriptors are generated through feature adaptive fusion, which enhances the recognition capability by utilizing different perspectives and semantic information.
It achieves efficient computation and high accuracy in point cloud scene recognition, improves recognition precision and robustness, and maintains good recognition performance in complex environments.
Smart Images
Figure CN118747809B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of scene recognition technology, and in particular to a multi-view semantic embedding scene recognition method based on point clouds. Background Technology
[0002] The ability to identify and match scenes is crucial for the autonomous navigation of robots and self-driving cars. It is also the foundation for loop closure detection and relocalization in Simultaneous Localization and Mapping (SLAM) technology. In recent years, scene recognition technology has made significant progress, resulting in various vision-based methods. However, the performance of visual scene recognition remains severely constrained by factors such as seasonality, viewing angle, lighting, and weather conditions. In contrast, LiDAR sensors largely avoid the limitations of vision. Therefore, point cloud scene recognition methods have received widespread attention. Current point cloud scene recognition algorithms still suffer from low computational efficiency in point feature extraction and matching errors in similar scenes. On the one hand, projecting 3D point clouds onto a 2D plane before extracting features is more efficient and easier to implement. Scene recognition tasks focus more on the diversity of scenes; therefore, fusing the front view projection and bird's-eye view projection of 3D point clouds can clearly express the spatial specificity of the scene. On the other hand, semantic information plays a crucial role. It can help identify object categories in the scene and understand the overall structure and layout of the scene, which is beneficial for improving recognition accuracy and enhancing system robustness. Through semantic information, objects such as roads, vehicles, and pedestrians can be identified, and more robust features ensure that the system can cope with more complex scenes. Although a large number of point cloud-based scene recognition algorithms have emerged, few methods have achieved a balance between computational performance and accuracy, and there is still room for improvement in large-scale outdoor scene recognition.
[0003] The paper "Luo L, Cao SY, Han B, et al. BVMatch: Lidar-Based Place Recognition Using Bird's-Eye View Images[J]. IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 6076-6083. 2021." projects 3D point clouds onto bird's-eye view images and then uses a set of Log-Gabor filters to construct a maximum index map to encode the orientation information of structures in the image. Although this improves the computational efficiency of descriptor generation, it cannot avoid the loss of vertical orientation information of the scene. The paper "Komorowski J. MinkLoc3D: Point CloudBased Large-Scale Place Recognition[C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision, pp. 1790-1799. 2021." calculates discriminative 3D point cloud descriptors based on sparse voxelized point cloud representation and sparse 3D convolution. While sparse 3D convolutions are used to extract global features from point clouds, ensuring real-time computation, they cannot effectively handle scene rotation changes. The paper "Li L, Kong X, Zhao X, et al. RINet: Efficient 3D lidar-based place recognition using rotation invariant neural network[J]. IEEE Robotics and Automation Letters, vol.7, no.2, pp.4321-4328.2022." combines semantic and geometric features to improve descriptive capabilities, and then uses rotation-invariant Siamese neural networks to predict the similarity between descriptors. This combination of semantic and geometric information ensures information integrity, but its computational efficiency is lower than other methods. The paper "Zhao S, Yin P, Yi G, et al. Spherevlad++: Attention-based and signal-enhanced viewpoint invariant descriptor[J]. IEEE Robotics and Automation Letters, vol.8, no.1, pp.256-263.2023." projects point clouds onto a spherical viewpoint in multiple unique regions and captures the contextual connections between local features and their dependencies on the global 3D geometric distribution, exhibiting rotation invariance.Spherical convolution encodes equivalent features in local directions, but does not utilize the correlation between features.
[0004] In summary, current research cannot simultaneously guarantee recognition accuracy, real-time performance, and complete scene representation, and has not yet effectively addressed the relationship between multi-view projection features. Summary of the Invention
[0005] This invention proposes a multi-view semantic embedding scene recognition method based on point clouds. The method uses multi-view feature fusion and semantic embedding for scene recognition, effectively utilizing the projection features of point clouds from different perspectives and enhancing the recognition ability of similar structural locations with semantic information, achieving a balance between descriptor generation computational efficiency and robustness in complex environments.
[0006] The technical solution of the present invention is as follows: a multi-view semantic embedding scene recognition method based on point cloud, wherein point cloud data in two trajectories are processed into unique descriptors through a descriptor encoding network, and the descriptors of the two trajectories are used as a database and a query set, respectively; during the query process, the nearest neighbor algorithm is used to find the most similar point cloud in the database to complete the global scene recognition based on point cloud.
[0007] The two trajectories are data collected at different times along the same route; the descriptor of the trajectory collected earlier is the database; the descriptor of the trajectory collected later is the query set.
[0008] The descriptor coding network is divided into multi-view projection, semantic embedding feature learning, and feature adaptive fusion.
[0009] Point cloud data is projected onto the forward view and the bird's-eye view through multi-view projection to obtain distance view image and bird's-eye view image. The distance view image and the bird's-eye view image are fused through semantic embedding feature learning, and then fused through feature adaptive fusion to obtain the final point cloud descriptor.
[0010] The multi-view projection is specifically as follows:
[0011] Distance-view image generation: For any point p = {x, y, z} in point cloud data, project it onto the range view through the following transformation;
[0012] θ = atan2(y, x)
[0013]
[0014] The atan2 function calculates the azimuth angle θ from the origin to the point (x,y), which is the angle between the point and the X-axis; φ is the angle between the point cloud and the XY plane; r and c represent the horizontal and vertical coordinates of the point cloud projected onto the 2D plane, respectively; where Δθ and Δφ represent the horizontal and vertical resolutions of the radar; the projected image values are filled with a 5-tuple (range,x,y,z,intensity), where range is the distance from the point cloud point to the z-axis, and if there is no point at a specific location, the value of each channel is set to 0; Intensity refers to the reflection intensity value of the point cloud point;
[0015] Bird's-eye view image generation is achieved by projecting a point cloud onto the XY plane; creating a filter to retain only points within a specific interval of the point cloud; determining the planar coordinates of each point and specifying the fill value for the corresponding location; treating the height and reflection intensity of the original point cloud as physical quantities for differentiation and geographic representation; and obtaining a bird's-eye view image with two channels for z-coordinate and reflection intensity through the above operations.
[0016] The point cloud is set to a specific interval with a filtering range of 20 meters, a resolution of 5 centimeters, and the origin is shifted to minimize the data at position (0,0).
[0017] The semantic embedding feature learning includes a semantic segmentation network, a self-attention module, and a combined convolution module; the semantic segmentation network processes the distance view image to obtain a point cloud containing the semantic segmentation results;
[0018] The feature extraction part of the semantic segmentation network is connected to the self-attention module to further refine the features for scene recognition tasks; the point cloud containing the semantic segmentation results is converted into a semantic bird's-eye view image, which forms a three-channel bird's-eye view image; the three-channel bird's-eye view image is processed using a combined convolution module; the output features of the combined convolution module and the output features of the self-attention module are fused into a global feature representation through post-fusion, and the final point cloud descriptor is obtained by using GeM pooling.
[0019] The semantic segmentation network consists of two stages: a downsampling encoding stage and an upsampling decoding stage. The model parameters of the semantic segmentation network are pre-trained. The point cloud containing the semantic segmentation results is projected onto a bird's-eye view to obtain a semantic bird's-eye view image. In the distance view image branch, the features of the distance view image after the downsampling encoding stage are further extracted into global features for scene recognition tasks using a self-attention module. The bird's-eye view branch is constructed as a new three-channel bird's-eye view image and input into a combined convolution module for global feature extraction. The two branches are processed into their respective descriptors through GeM pooling. The descriptors of the two branches are fused from the descriptors obtained from the two heterogeneous networks through a feature adaptive fusion module to obtain the final feature representation.
[0020] The self-attention module first uses three convolutional layers to map the feature maps from the downsampling encoding stage; then, it further refines the features through a self-attention mechanism. The combined convolutional module, consisting of four convolutional layers and pooling layers, extracts the basic features of the bird's-eye view image. The combined convolutional module uses two sets of CNNs with different parameters to process the basic features separately, performing feature integration in the form of cross-layer summation of features at different levels. Finally, the dot product operation merges the feature integration results into a unified representation.
[0021] The adaptive feature fusion specifically involves: fusing and encoding the global features of the two branches into a unique point cloud descriptor; the fusion process specifically involves sequentially passing through the Transformer layer module and GeM pooling.
[0022] The beneficial effects of this invention are as follows: The descriptor coding network proposed in this invention is not only computationally efficient and capable of utilizing point cloud features from different perspectives, but also effectively leverages semantic information to enhance scene recognition. The proposed descriptor coding network can effectively describe point cloud scenes and is applied to scene recognition tasks. It achieves a significant improvement in recognition accuracy and outperforms point feature-based methods in generalization performance and real-time performance. Due to the superior overall network architecture design, this invention achieves superior performance by using only a triplet loss function to optimize the network. Attached Figure Description
[0023] Figure 1 Flowchart for point cloud scene recognition;
[0024] Figure 2 Generate a flowchart for the descriptor. Detailed Implementation
[0025] Figure 1 This is a flowchart of the overall technical solution of the present invention. The present invention proposes a multi-view semantic embedding scene recognition method based on point clouds. First, the point cloud data in the trajectory is processed into unique descriptors through a proposed descriptor encoding network. The descriptors of the two trajectories serve as the database and query set, respectively. Then, during the query process, the nearest neighbor algorithm is used to find the most similar point cloud in the database, thus completing the global scene recognition based on point clouds.
[0026] Figure 2This invention utilizes a descriptor encoding network that projects point clouds as 2D images for processing, significantly improving computational efficiency. We project the point cloud onto forward and bird's-eye view perspectives to obtain distance view (RV) and bird's-eye view (BEV) images, respectively. Then, a semantic segmentation network is used to process the RV images, and the feature extraction part of the semantic segmentation network is connected to a self-attention module to further refine the features for scene recognition tasks. Next, the point cloud containing the semantic segmentation results is converted into a semantic BEV image, and a combined convolution module is used to process the multi-channel BEV image. Finally, the features from the two branches are fused into a global feature representation using a post-fusion approach, and GeM pooling is used to obtain the final point cloud descriptor.
[0027] The RV image generation process is as follows: for any point p = {x, y, z} in the point cloud, it can be projected onto the range view through the following transformations:
[0028] @@=atl@n2@@y·x@
[0029]
[0030] Where θ is the angle between the point cloud and the X-axis, φ is the angle between the point cloud and the XY plane, and r and c represent the horizontal and vertical coordinates of the point cloud projected onto the 2D plane, respectively. Δθ and Δφ represent the radar's horizontal and vertical resolutions, respectively. The projected image values are filled with a 5-tuple of (range, x, y, z, intensity), where range is the distance from the point to the z-axis; if there is no point at a specific location, the value for each channel is set to 0. Intensity refers to the reflection intensity value of the point.
[0031] BEV image generation is achieved by projecting a point cloud onto the XY plane. First, a filter is created to retain only points within a specific interval of the point cloud. Points that are too far apart are too sparse and their impact on scene representation is negligible. We set the filter range to 20 meters, meaning we only retain points with x and y coordinates within the interval [-20, 20]. Then, we set the resolution to 5 centimeters and shift the origin to minimize the data at location (0, 0). After determining the planar coordinates for each point, we need to specify the padding values for that location. We consider the height and reflectance of the original point cloud as physical quantities important for distinguishing geographic representations. Through these operations, a two-channel (z-coordinate, reflectance) BEV image can be obtained.
[0032] Semantic embedding feature learning has two branches for feature representation learning. First, we use the RangeNet++ model to perform semantic segmentation on the RV image. The semantic segmentation network consists of two stages: a downsampling encoding stage and an upsampling decoding stage. The parameters of the semantic segmentation network model are pre-trained, so they do not need to be trained in this session. The point cloud is then projected onto the BEV viewpoint to obtain the semantic BEV image. In the RV branch, a self-attention module is used to further extract the features from the semantic downsampling stage into global features for the scene recognition task. The BEV branch has been constructed as a new three-channel image and input into a combined convolution module for global feature extraction. The two branches are processed into their respective descriptors using GeM pooling. The descriptors of the two branches can be used for retrieval independently, but we further use a feature adaptive fusion module to fuse the descriptors obtained from the two heterogeneous networks to obtain the final feature representation.
[0033] The self-attention module first maps the feature maps using three convolutional layers. Then, it further refines the features through a self-attention mechanism. The ensemble convolution module extracts the basic features of the BEV image using a basic CNN consisting of four convolutional layers and pooling layers. Then, it processes the features separately using two sets of CNNs with different parameters, performing feature ensemble in a cross-layer summation form of features at different levels. Finally, dot product operations fuse the results into a unified representation. Ensemble convolution can generate a large number of feature maps with a small number of parameters and operations, each representing a class of features. Compared to traditional convolution, it can encode more information under the same conditions.
[0034] Feature Adaptive Fusion: After processing the two branches and obtaining their respective global point cloud feature descriptions, the feature adaptive fusion module fuses and encodes the global features of the two branches into a unique point cloud descriptor. The fusion utilizes a simple Transformer layer module, and then the fused features are processed through GeM pooling. The reason for using post-fusion on two different viewpoint branches is that it allows each branch to exist and be used independently, and can be easily replaced and upgraded without adjusting and retraining the remaining branches.
[0035] To address the problems of low computational efficiency of point cloud features, inadequate representation of scene structure by single-view projection, and underutilization of semantic information in current point cloud scene recognition tasks, this invention proposes a multi-view semantic embedding scene recognition method based on point clouds. This method offers solutions to the aforementioned problems, achieving a balance between computational efficiency and accuracy in point cloud scene recognition. With its computational efficiency, high accuracy, and robustness, this invention can be effectively used for real-time global point cloud scene recognition tasks.
[0036] This invention was tested on publicly available KITTI and NCLT datasets, as well as in real-world scenarios. In the publicly available datasets, a designated data sequence was used as the database, while additional data sequences simulated the real-time operation of an autonomous vehicle. The point cloud data acquired in real-time was then queried from the database. In the real-world scenario, the invention used a vehicle equipped with a LiDAR sensor to first record point cloud sequences from the test site to build a database. Then, the vehicle was driven in real-time, and the acquired point cloud data was queried from the database. In both scenarios, scene recognition was achieved at a minimum frequency of 10Hz, and the recognition accuracy was superior to current methods of the same type.
[0037] To verify that the algorithm achieves a balance between real-time performance and accuracy, this invention was tested on the KITTI and NCLT datasets. The experimental platform consisted of a Core i7-10700 CPU@2.9GHz and an Nvidia RTX3090 GPU. The generation time of the single-frame point cloud descriptor in this invention is 28.95ms, ensuring real-time point cloud processing on a 10Hz LiDAR device. The AR@1% (average recall rate) on the two datasets were 97.8% and 89.5%, respectively. This demonstrates that our algorithm maintains good performance on datasets with different types of LiDAR sensors, ensuring both scene recognition accuracy and good real-time performance. It can be effectively used for tasks such as large-scale scene global localization for mobile robots.
Claims
1. A multi-view semantic embedding scene recognition method based on point clouds, characterized in that, The point cloud data in the two trajectories are processed into unique descriptors through a descriptor encoding network. The descriptors of the two trajectories are used as the database and query set, respectively. During the query process, the nearest neighbor algorithm is used to find the most similar point cloud in the database to complete the global scene recognition based on the point cloud. The descriptor encoding network is characterized by being divided into multi-view projection, semantic embedding feature learning, and feature adaptive fusion. Point cloud data is projected onto the forward view and the bird's-eye view through multi-view projection to obtain distance view image and bird's-eye view image; the distance view image and the bird's-eye view image are fused through semantic embedding feature learning, and then fused through feature adaptive fusion to obtain the final point cloud descriptor; The multi-view projection is specifically as follows: Distance-view image generation: For any point p = {x, y, z} in point cloud data, project it onto the range view through the following transformation; The atan2 function calculates the azimuth angle θ between the origin and the point (x, y), which is the angle between the point and the X-axis. φ is the angle between the point cloud and the XY plane, where r and c represent the horizontal and vertical coordinates of the point cloud projected onto the 2D plane, respectively; where θ and φ represents the horizontal and vertical resolution of the radar; the projected image values are filled with a 5-tuple (range, x, y, z, intensity), where range is the distance from the point cloud point to the z-axis, and if there is no point at a certain location, the value of each channel is set to 0; Intensity refers to the reflection intensity value of the point cloud point; Bird's-eye view image generation is achieved by projecting a point cloud onto the XY plane; a filter is created to retain only points within a specific interval of the point cloud; after determining the planar coordinates of each point, the fill value for the corresponding position is specified; the height and reflection intensity of the original point cloud are regarded as physical quantities in terms of differentiation and geographic representation; through the above operations, a bird's-eye view image with two channels of z-coordinate and reflection intensity is obtained. The semantic embedding feature learning includes a semantic segmentation network, a self-attention module, and a combined convolutional module; Semantic segmentation networks process distance-view images to obtain point clouds containing semantic segmentation results; The feature extraction part of the semantic segmentation network is connected to the self-attention module to further refine the features for scene recognition tasks; the point cloud containing the semantic segmentation results is converted into a semantic bird's-eye view image, which forms a three-channel bird's-eye view image; the three-channel bird's-eye view image is processed using a combined convolution module; the output features of the combined convolution module and the output features of the self-attention module are fused into a global feature representation through post-fusion, and the final point cloud descriptor is obtained by using GeM pooling. The adaptive feature fusion specifically involves: fusing and encoding the global features of the two branches into a unique point cloud descriptor; the fusion process specifically involves sequentially passing through the Transformer layer module and GeM pooling.
2. The multi-view semantic embedding scene recognition method based on point clouds according to claim 1, characterized in that, The two trajectories are data collected at different times along the same route; the descriptor of the trajectory collected earlier is the database; the descriptor of the trajectory collected later is the query set.
3. The multi-view semantic embedding scene recognition method based on point clouds according to claim 1, characterized in that, The point cloud is set to a specific interval with a filtering range of 20 meters, a resolution of 5 centimeters, and the origin is shifted to minimize the data at position (0,0).
4. The multi-view semantic embedding scene recognition method based on point cloud according to claim 1, characterized in that, The semantic segmentation network consists of two stages: a downsampling encoding stage and an upsampling decoding stage. The model parameters of the semantic segmentation network are pre-trained. The point cloud containing the semantic segmentation results is projected onto a bird's-eye view to obtain a semantic bird's-eye view image. In the distance view image branch, the features of the distance view image after the downsampling encoding stage are further extracted into global features for the scene recognition task using a self-attention module. The bird's-eye view branch is constructed as a new three-channel bird's-eye view image and input into a combined convolution module for global feature extraction. GeM pooling is used to process the two branches into their own descriptors; the descriptors of the two branches are then fused from the descriptors obtained from the two heterogeneous networks by the feature adaptive fusion module to obtain the final feature representation.
5. The multi-view semantic embedding scene recognition method based on point cloud according to claim 4, characterized in that, The self-attention module first uses three convolutional layers to map the feature maps from the downsampling encoding stage; then, it further refines the features through a self-attention mechanism. The combined convolutional module, consisting of four convolutional layers and pooling layers, extracts the basic features of the bird's-eye view image. The combined convolutional module uses two sets of CNNs with different parameters to process the basic features separately, performing feature integration in the form of cross-layer summation of features at different levels. Finally, the dot product operation merges the feature integration results into a unified representation.