A multi-scale feature fusion large-scale point cloud scene re-identification method

Abstract

The application belongs to the field of scene recognition, and discloses a large-scale point cloud scene recognition method based on multi-scale feature fusion. The point cloud sequence is subjected to static point cloud generation processing, near-range dynamic objects in the scene are filtered out, and pure static point cloud is obtained; the pure static point cloud is input into a descriptor coding network, global point cloud descriptors are obtained, and a database is constructed; in the query process, the nearest neighbor algorithm is used to find the most similar point cloud in the database. The application realizes the balance of point cloud scene recognition in the aspects of calculation efficiency and accuracy. The static point cloud generation method removes the shielding effect of dynamic objects and obtains spliced static point cloud of continuous frames. The point cloud descriptor coding network comprises a space transformation module, a multi-scale feature fusion module, a CapsNet module and a GeM pooling layer. The application can be effectively used for real-time point cloud global positioning tasks due to its high efficiency, high accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the field of scene re-identification technology, and in particular to a large-scale point cloud scene re-identification method using multi-scale feature fusion. Background Technology

[0002] Scene re-identification is a crucial task in mobile robotics and autonomous driving, and it forms the technological foundation for loop closure detection and relocalization in Simultaneous Localization and Mapping (SLAM) technology. Over the past two decades, scene re-identification has undergone significant development, giving rise to a series of vision-based methods. However, factors such as season, viewing angle, lighting, and weather continue to limit the performance of visual scene re-identification. Compared to vision, LiDAR is largely unaffected by changes in lighting. With the rapid development of LiDAR sensors, point cloud scene recognition methods have been extensively studied. LiDAR point clouds overcome some of the shortcomings of visual images, but still face many challenges. On the one hand, outdoor scenes inevitably contain many dynamic objects. Nearby dynamic objects can obstruct the LiDAR sensor's field of view, leading to large blind spots and affecting the integrity of the point cloud scene. Furthermore, the appearance of dynamic objects is random; if dynamic objects exist in the point cloud scene when the query database is built, but not during retrieval, the point cloud retrieval will fail, and vice versa. On the other hand, outdoor scenes typically contain buildings, trees, vehicles, pedestrians, and other complex elements with different sizes, which manifest as differences in the number of object points in the point cloud data. Single-frame point clouds also exhibit a pattern of denser features at close range and sparser features at distant locations. These factors significantly impact the performance of scene re-identification. Furthermore, the features extracted from point clouds are not isolated, and few studies have focused on the correlations between these features. Although numerous point cloud-based scene re-identification algorithms have emerged, large-scale outdoor scene re-identification remains an unsolved problem.

[0003] The paper "Luo L, Cao SY, Han B, et al. BVMatch: Lidar-Based Place Recognition Using Bird's-Eye View Images[J]. IEEE Robotics and Automation Letters, vol.6, no.3, pp.6076-6083.2021." projects 3D point clouds onto bird's-eye view images and then uses a set of Log-Gabor filters to construct a maximum index map to encode the orientation information of structures in the image. Although this improves the computational efficiency of descriptor generation, it cannot avoid the loss of vertical orientation information of the scene. The paper "Komorowski J. MinkLoc3D: Point CloudBased Large-Scale Place Recognition [C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision, pp. 1790-1799. 2021." calculates discriminative 3D point cloud descriptors based on sparse voxelized point cloud representation and sparse 3D convolution. Although it ensures real-time computation, it cannot effectively handle scene rotation changes. The paper "Li L, Kong X, Zhao X, et al. RINet: Efficient 3Dlidar-based place recognition using rotation invariant neural network [J]. IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 4321-4328. 2022." combines semantic and geometric features to improve descriptive ability, and then uses rotation-invariant Siamese neural networks to predict the similarity between descriptors. It uses the combination of semantic and geometric information to ensure the integrity of information, but its computational efficiency is lower than other methods. The paper "Zhao S, Yin P, Yi G, et al. Spherevlad++: Attention-based and signal-enhanced viewpoint invariant descriptor[J]. IEEE Robotics and Automation Letters, vol.8, no.1, pp.256-263.2023." projects point clouds on a spherical viewpoint in multiple unique regions and captures contextual connections between local features and their dependencies on the global 3D geometric distribution, while also exhibiting rotation invariance.While spherical convolution encodes equivalent features in local orientations, it fails to leverage the correlations between features. In summary, current research cannot simultaneously guarantee rotation invariance, real-time performance, and complete scene representation; it also lacks effective handling of dynamic object interference and utilization of feature correlations. Summary of the Invention

[0004] To address the problems in current point cloud scene re-identification tasks, such as the impact of dynamic objects occluding point clouds on point cloud structure, the inability of a single receptive field of the network to fully extract point cloud features, and the underutilization of correlations between features, this invention proposes a large-scale point cloud scene re-identification method with multi-scale feature fusion. This method effectively utilizes the correlations between point cloud features for encoding, can resist the influence of various dynamic objects in the environment, and achieves a balance between descriptor generation computation efficiency and point cloud retrieval accuracy.

[0005] The technical solution of this invention is a large-scale point cloud scene re-identification method based on multi-scale feature fusion. The method involves performing static point cloud generation processing on the point cloud sequence to filter out near-field dynamic objects in the scene and obtain a pure static point cloud; inputting the pure static point cloud into a descriptor encoding network to obtain a global point cloud descriptor and constructing a database; and using the nearest neighbor algorithm to find the most similar point cloud in the database during the query process.

[0006] The static point cloud generation is described in detail below:

[0007] Extracting three consecutive point cloud frames P i-1 P i and P i+1 P is obtained through IMU sensor parameters i Convert the point cloud of the previous frame and the point cloud of the next frame to P i The position is used to obtain the transformed point cloud P. T i-1 and P T i+1 The transformed point cloud's 3D coordinates are projected onto a 2D image coordinate system. The image pixels are then filled with the distance values ​​from each point in the point cloud to the center of the lidar sensor, resulting in the point cloud range image M. i Set a distance threshold τ. Objects within the range whose distance is less than the distance threshold τ are considered to be in the near range. Objects within the near range are retained to obtain M′. i Pixel values ​​exceeding the distance threshold τ are set to 0;

[0008] M i (u,v)=0,stM i (u,v)>τ.

[0009] By comparing pixel differences, the distance images of two adjacent frames are subtracted, and values ​​with a difference value less than the difference threshold γ are retained as the difference image G. i ;

[0010]

[0011] Detection regions are set for non-zero elements in the difference image, and pixel coordinates satisfying the following formula are marked. This refers to the detection region matrix, where L0 refers to the 0 norm; (α, β) are the width and height of the detection region, and δ is the threshold parameter.

[0012]

[0013] The detected image pixels are traced back to the point cloud coordinates of each frame. Region growing and clustering are performed on the point cloud, and dynamic objects are filtered out to obtain a static environment point cloud. The three consecutive frames of static point clouds are fused according to pose transformation to obtain the final clean static point cloud P'. i .

[0014] The descriptor encoding network includes a spatial transformation module, a multi-scale feature fusion module MSFF, a CapsNet module, and a GeM pooling layer, which are connected in sequence. The spatial transformation module is composed of T-net. The multi-scale feature fusion module is composed of EdgeConv and ECA attention modules with different numbers of layers. The CapsNet module includes a feature construction layer, a primary capsule layer, and a latent capsule layer. The GeM pooling layer is a single-layer structure.

[0015] The spatial transformation module is a T-net network; a clean static point cloud is input into the T-net for spatial transformation; the T-net generates an affine transformation matrix by learning the rotation of the existing point cloud, and performs normalization processing on the changes of the clean static point cloud.

[0016] The multi-scale feature fusion module MSFF consists of EdgeConv and ECA attention fusion at different levels, and includes three branches: the first branch connects the first-level EdgeConv to the first ECA attention layer; the second branch connects the second-level EdgeConv to the second ECA attention layer, with the input of the second-level EdgeConv connected to the output of the first-level EdgeConv; and the third branch connects the third-level EdgeConv to the third ECA attention layer, with the input of the third-level EdgeConv connected to the output of the second-level EdgeConv.

[0017] EdgeConv establishes the topological relationships between point clouds after spatial transformation. Stacks of EdgeConv at different levels are used to represent information of different ranges. ECA attention obtains the weights corresponding to each branch.

[0018] The i-th branch of the MSFF module, x iThis represents the features after ECA attention processing for each branch. The features x after ECA attention processing for each branch are... i ∈R N With weighted vector k∈R N Element-wise multiplication is performed to obtain the fused features, which are then input into the CapsNet module. It represents a linear mapping from the global descriptor to the shared weight vector, and GAP(.) represents global average pooling;

[0019]

[0020]

[0021] The CapsNet module includes a capsule building layer, a primary capsule layer, and a latent capsule layer. Each network layer in the CapsNet module stores features in vector form. A dynamic routing mechanism is used to map the vectors stored in the primary capsule layer to the latent capsule layer for implicit learning of the association between features. Finally, the global features of the point cloud are output. The global features of the point cloud are aggregated by the GeM pooling layer to obtain a 256-dimensional global descriptor of the point cloud.

[0022] The loss function of the descriptor encoding network is the triplet loss function;

[0023]

[0024] Where δa, δp, and δn represent the current query descriptor, positive sample, and negative sample, respectively; α represents the minimum difference between the distance from the query descriptor to the positive sample and the distance from the query descriptor to the negative sample.

[0025] The beneficial effects of this invention are as follows: This invention achieves a balance between computational efficiency and accuracy in point cloud scene re-identification. First, a novel static point cloud generation method is proposed to remove the occlusion effects of dynamic objects, resulting in a stitched static point cloud of consecutive frames. Then, a novel point cloud descriptor encoding network is proposed, which includes a point space transformation module, a multi-scale feature fusion module, a CapsNet module, and a GeM pooling layer. Finally, efficient model training can be achieved using only a triplet loss function. This invention, with its computational efficiency, high accuracy, and robustness, can be effectively used for real-time point cloud global localization tasks. Attached Figure Description

[0026] Figure 1 Flowchart of a large-scale point cloud scene re-identification method based on multi-scale feature fusion;

[0027] Figure 2 Static point cloud generation flowchart;

[0028] Figure 3Point cloud descriptor encoding network framework diagram. Detailed Implementation

[0029] Figure 1 This is a flowchart of the overall technical solution of the present invention. The present invention proposes a large-scale point cloud scene re-identification algorithm based on multi-scale feature fusion. First, the point cloud sequence undergoes static point cloud generation processing to filter out near-field dynamic objects in the scene and obtain a clean static point cloud. Then, the point cloud is input into a descriptor encoding network to obtain point cloud descriptors and construct a database. During the query process, the nearest neighbor algorithm is used to find the most similar point cloud in the database.

[0030] like Figure 2 The following is a flowchart of the static point cloud generation process:

[0031] Occlusion by dynamic objects can significantly impact the structure of point cloud scenes, blocking the view behind them and changing the direction of occlusion as their position changes. For example... Figure 2 As shown, we obtain data from three consecutive point clouds P. i-1 P i and P i+1 To begin, to more easily compare the differences in point clouds, we first convert the before and after point clouds to P using IMU sensor parameters. i The position is used to obtain the transformed point cloud P. T i-1 and P T i+1 Then, the point cloud is projected onto the image coordinate system, and pixels are filled with the distance values ​​from the points to the center of the sensor to obtain the point cloud range image M. i Distance images allow for a visual comparison of scene changes, especially of nearby dynamic objects, while differences in static environments are not readily apparent over short intervals. Reducing the spatial representation to retain only objects within the nearest range yields M′. i This is achieved by setting the pixel values ​​that exceed the distance threshold τ to 0.

[0032] M i (u,v)=0,stM i (u,v)>τ.

[0033] Then, the two distance images in this interval are subtracted by comparing pixel differences, and the values ​​that meet the difference threshold γ are retained as the difference image G. i .

[0034]

[0035] Detection regions are set for non-zero elements in the difference image, and pixel coordinates satisfying the following formula are marked, where (α, β) are the width and height of the detection region, and δ is the threshold parameter. Except for pixels of moving objects, all other regions in the difference image are 0. This means detecting whether non-zero pixels meet our set conditions; if they do, the region is considered a pixel of the moving object.

[0036]

[0037] The recorded image pixels are traced back to point cloud coordinates. Then, region growing and clustering are performed on the point cloud, and dynamic objects are filtered out to obtain a static environment point cloud. Three consecutive frames of static point clouds are fused to obtain the final point cloud P'. i This greatly reduces the impact of dynamic object occlusion.

[0038] like Figure 2 As shown, the proposed dynamic object filtering method for continuous point clouds can obtain static point cloud data acquired in real time by LiDAR. The output of this module is a stitched result of three consecutive static point cloud frames, minimizing the gaps left after dynamic object filtering. This module does not require model pre-training, features lightweight computation, and can effectively generate static point clouds in real time for other tasks.

[0039] like Figure 3 The descriptor encoding network shown mainly consists of four parts: spatial transformation module, multi-scale feature fusion (MSFF) module, CapsNet module, and GeM pooling layer.

[0040] 1. To address potential rotational changes in the input point cloud, T-Net is first used to learn the rotation of the point cloud and perform spatial transformations on it. T-Net is a miniature network used to generate an affine transformation matrix to normalize the rotation, translation, and other transformations of the point cloud.

[0041] 2. Then, EdgeConv is used to establish the topological relationships between points. Stacking EdgeConv with different numbers of layers can represent information of different ranges. The MSFF module adaptively fuses features extracted from receptive fields of different scales. For the i-th branch, the features xi∈R of the point are... N With weighted vector k∈R N Perform element-wise multiplication

[0042]

[0043]

[0044] 3. Next, the fused features are input into the CapsNet module, which mainly consists of three parts: the capsule construction layer, the primary capsule layer, and the latent capsule layer. Each capsule stores features in vector form, and a dynamic routing mechanism is used to map primary capsules to latent capsules to implicitly learn the associations between features.

[0045] 4. Finally, the features are aggregated using the GeM pooling layer to obtain a 256-dimensional global descriptor.

[0046] like Figure 3 As shown, the proposed global descriptor encoding network for point clouds not only resists scene rotation changes and utilizes point cloud features at different scales, but also effectively leverages the correlations between features. The proposed descriptor encoding network can effectively describe point cloud scenes and can be applied to scene re-identification tasks.

[0047] Due to the superior overall network architecture design, this invention achieves excellent performance by using only the triplet loss function to optimize the network. Here, δa, δp, and δn represent the current query descriptor, positive sample, and negative sample, respectively. α represents the minimum difference between the distance from the query descriptor to the positive sample and the distance from the query descriptor to the negative sample.

[0048]

[0049] To verify that the algorithm achieves a balance between real-time performance and accuracy, this invention was tested on three datasets: Oxford RobotCar, KITTI, and a self-collected dataset. The experimental platform consisted of a Core i7-10700 CPU@2.9GHz and an Nvidia RTX 3090 GPU. The generation time of the single-frame point cloud descriptor in this invention is 41.26ms, ensuring real-time point cloud processing on a 10Hz LiDAR device. The AR@1% (average recall rate) on the three datasets were 98.3%, 87.3%, and 93.1%, respectively. This demonstrates that our algorithm maintains good performance on datasets with different types of LiDAR sensors, ensuring both scene re-identification accuracy and good real-time performance, making it effective for tasks such as large-scale scene global localization for mobile robots.

Claims

1. A method for re-identifying large-scale point cloud scenes using multi-scale feature fusion, characterized in that, The point cloud sequence is processed into a static point cloud to filter out near-field dynamic objects in the scene and obtain a clean static point cloud. The clean static point cloud is then input into a descriptor encoding network to obtain a global point cloud descriptor and build a database. During the query process, the nearest neighbor algorithm is used to find the most similar point cloud in the database; The descriptor encoding network comprises a spatial transformation module, a multi-scale feature fusion module MSFF, a CapsNet module, and a GeM pooling layer, which are connected sequentially. The spatial transformation module is composed of T-net. The multi-scale feature fusion module is composed of EdgeConv and ECA attention modules with different numbers of layers. The CapsNet module includes a feature construction layer, a primary capsule layer, and a latent capsule layer. The GeM pooling layer is a single-layer structure. The multi-scale feature fusion module MSFF consists of EdgeConv and ECA attention fusion at different levels, and includes three branches: the first branch connects the first-level EdgeConv to the first ECA attention layer; the second branch connects the second-level EdgeConv to the second ECA attention layer, with the input of the second-level EdgeConv connected to the output of the first-level EdgeConv; and the third branch connects the third-level EdgeConv to the third ECA attention layer, with the input of the third-level EdgeConv connected to the output of the second-level EdgeConv. EdgeConv establishes the topological relationships between point clouds after spatial transformation, and stacks of EdgeConv at different levels are used to represent information of different ranges; ECA attention obtains the weights corresponding to each branch; The i-th branch of the MSFF module, x i This represents the features after ECA attention processing for each branch. The features x after ECA attention processing for each branch are... i ∈ R N With weighted vector k ∈ R N Element-wise multiplication is performed to obtain the fused features, which are then input into the CapsNet module. φ(.) represents a linear mapping from the global descriptor to the shared weight vector, and GAP(.) represents global average pooling; 。 2. The large-scale point cloud scene re-identification method based on multi-scale feature fusion according to claim 1, characterized in that, The static point cloud generation is described in detail below: Extracting three consecutive point cloud frames P i−1 P i and P i+1 P is obtained through IMU sensor parameters i Convert the point cloud of the previous frame and the point cloud of the next frame to P i The position is used to obtain the transformed point cloud P. T i−1 and P T i+1 The transformed point cloud's 3D coordinates are projected onto a 2D image coordinate system. The image pixels are then filled with the distance values ​​from each point in the point cloud to the center of the lidar sensor, resulting in the point cloud range image M. i Set a distance threshold τ. Objects within the range whose distance is less than the distance threshold τ are considered to be in the near range. Objects within the near range are retained to obtain M. ′ i Pixel values ​​exceeding the distance threshold τ are set to 0; By comparing pixel differences, the distance images of two adjacent frames are subtracted, and values ​​with a difference value less than the difference threshold γ are retained as the difference image G. i ; Detection regions are set for elements in the difference image that are not zero, and pixel coordinates that satisfy the following formula are marked. This refers to the detection region matrix, where L0 refers to the 0 norm; (α, β) are the width and height of the detection region, and δ is the threshold parameter. The detected image pixels are traced back to the point cloud coordinates of each frame. Region growing and clustering are performed on the point cloud, and dynamic objects are filtered out to obtain a static environment point cloud. The three consecutive frames of static point clouds are fused according to pose transformation to obtain the final clean static point cloud P. ’ i .

3. The large-scale point cloud scene re-identification method based on multi-scale feature fusion according to claim 1, characterized in that, The spatial transformation module is a T-net network; a clean static point cloud is input into the T-net for spatial transformation; the T-Net generates an affine transformation matrix by learning the rotation of the existing point cloud, and performs normalization processing on the changes of the clean static point cloud.

4. The large-scale point cloud scene re-identification method based on multi-scale feature fusion according to claim 1, characterized in that, The CapsNet module includes a capsule building layer, a primary capsule layer, and a latent capsule layer. Each network layer in the CapsNet module stores features in vector form. A dynamic routing mechanism is used to map the vectors stored in the primary capsule layer to the latent capsule layer for implicit learning of the association between features. Finally, the global features of the point cloud are output. The global features of the point cloud are aggregated by the GeM pooling layer to obtain a 256-dimensional global descriptor for the point cloud.

5. The large-scale point cloud scene re-identification method based on multi-scale feature fusion according to claim 4, characterized in that, The loss function of the descriptor encoding network is the triplet loss function; Where δa, δp, and δn represent the current query descriptor, positive sample, and negative sample, respectively; α represents the minimum difference between the distance from the query descriptor to the positive sample and the distance from the query descriptor to the negative sample.

Images