A two-stage point cloud registration method and system fusing depth map features

Abstract

The application discloses a kind of two-stage point cloud registration methods and systems of fusion depth map features, first, source point cloud and target point cloud are preprocessed and initial feature coding, extract multi-level geometric features using rotation invariant attention network and carry out global context aggregation and coarse matching, obtain initial transformation matrix;Then based on the matrix, target point cloud is transformed into source point cloud coordinate system and projected to generate depth map, and the multiscale semantic features of the depth map are extracted by the pre-trained diffusion model;Again, the extracted depth features and the underlying geometric features of point cloud are cross-modal mapping and fusion, to obtain joint features;Finally, based on joint features, mutual nearest neighbor and geometric consistency constraint are used for fine matching, and the final transformation matrix is solved by weighted singular value decomposition.The application enhances the distinguishing ability of point cloud features by introducing depth map semantic information, significantly improves the registration robustness and precision in low overlap, large rotation and symmetric structure scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D point cloud registration technology, and in particular relates to a two-stage point cloud registration method and system that integrates depth map features. Background Technology

[0002] With the widespread adoption of 3D sensors, point clouds, as the most direct 3D digital representation of the physical world, have become the core data foundation for cutting-edge fields such as autonomous driving, robot navigation, digital twins, and cultural relic digitization. Point cloud registration aims to solve the spatial transformation relationship between point cloud data acquired from different perspectives and at different times. It is a key foundational technology for achieving 3D scene reconstruction and high-precision positioning. Among these technologies, pairwise view point cloud registration aims to register two sets of point clouds with a certain overlap area to the same coordinate system. Since the classic Iterative Closest Point (ICP) algorithm was proposed in 1992, point cloud registration technology has evolved from traditional geometric optimization methods to feature descriptor-based matching methods, and is currently undergoing a transformation driven by data-driven methods represented by deep learning.

[0003] In recent years, deep learning-based point cloud matching methods have made significant progress, but they still face many challenges, such as difficulty in matching low-overlapping scenes and ambiguity in matching similar structures. In partially overlapping point cloud pairs, especially when the overlap rate is less than 30%, existing methods often struggle to accurately extract sufficiently discriminative local features from numerous interfering information, leading to an increase in mismatches. Rotation-invariant descriptors, while pursuing pose independence, often lose the ability to distinguish symmetrical structures, such as the left and right halves of a human body or the symmetrical facades of a building, resulting in matching ambiguities. In addition, existing image-point cloud cross-modal registration methods usually rely on metric learning to align features of different modalities. However, due to the fundamental differences in data representation between images and point clouds, these methods often have difficulty converging, limited generalization ability, and require a large amount of labeled data.

[0004] In summary, to address the issues of insufficient accuracy and poor robustness of existing deep learning-based point cloud registration methods in scenarios with low overlap rates, large rotational changes, and symmetrical or ambiguous structures, this invention proposes a two-stage point cloud registration scheme that integrates depth map features. Summary of the Invention

[0005] To address the above technical problems, this invention provides a two-stage point cloud registration method and system that integrates depth map features.

[0006] The technical solution adopted by this invention to solve its technical problem is: A two-stage point cloud registration method that integrates depth map features includes the following steps: S100: Obtain the source point cloud and the target point cloud, perform preprocessing and initial feature encoding on both respectively, and obtain the triplet representations of the source point cloud and the target point cloud; S200: A rotation-invariant attention encoding network that uses triplet representations to input shared weights to extract multi-level geometric features containing top-level and bottom-level features; S300: Perform global context aggregation and coarse matching based on top-level features to obtain the initial transformation matrix; S400: Align the source point cloud and the target point cloud using the initial transformation matrix, and project the initially aligned point cloud to generate a depth map; S500: Hole filling and repair of the depth map to obtain a dense depth map, extract multi-scale depth features from the dense depth map and perform dimensionality reduction to obtain depth map features; S600: Performs cross-modal fusion of low-level features and depth map features to obtain joint features; S700: Based on joint features, a refined matching of the source point cloud and the target point cloud is performed to obtain the final transformation matrix.

[0007] Preferably, S100 includes: S110: Input two frames of point cloud data to be registered, denoted as the source point cloud. and target point cloud ,in Include A three-dimensional point, Include A three-dimensional point, source point cloud The coordinates of each point in the array are represented as follows: Target point cloud The coordinates of each point in the array are represented as follows: ; S120: Yes and Sampling is performed at the farthest point to obtain a representative subset with uniform distribution. At the same time, principal component analysis or neighborhood-based covariance matrix decomposition is used to estimate the normal vector of each point. S130: Combine the original 3D coordinates with the normal vector to construct the initial input representation of each point, forming a triplet representation of the source point cloud. Triple representation of the target point cloud ,in, and Let represent the sets of normal vectors for each point in the source point cloud and the target point cloud, respectively. and These represent the point-by-point feature vectors of each point in the source point cloud and the target point cloud, respectively, and are both initialized as vectors of all 1s.

[0008] Preferably, S200 includes: S210: Input the triplet representations of the two point clouds obtained in S100 into a rotation-invariant feature encoding network with shared weights. The encoding network adopts a hierarchical encoder-decoder architecture, which contains multiple encoder blocks and corresponding decoder blocks. Each encoder block contains a downsampling layer and multiple attention layers, and each decoder block contains a transition upsampling layer and multiple attention layers. The specific calculation process for the attention layer is as follows: For each anchor point in the set of anchor points and its corresponding normal vector The set of anchor points is the set of points whose k nearest neighbors are retrieved from the set of support points to form a neighborhood point set; where the set of anchor points is the set of points in the current attention layer to be calculated for the output features, and the set of support points is the set of reference points used to provide neighborhood information. For each neighboring point and its normal vector Construct a local coordinate representation based on point-pair features, and calculate it using the following formula: in , indicating from the anchor point To neighboring points The translation vector, where ∠ represents the angle between the vectors, is completely invariant to global rotation; The aforementioned local coordinate representation is projected onto a geometric code using a linear layer, while the features of neighboring points are projected onto a context code using the same linear layer. During attention calculation, the query vector is dot-producted with both the positional encoding matrix of the geometric code and the key matrix of the context code. The sums are then scaled and normalized using Softmax to obtain attention weights. These attention weights are then used to perform a weighted summation on the geometric information matrix of the geometric code and the value matrix of the context code, calculated using the following formula: in For query vector, The key matrix, For value matrices, For position encoding, For geometric information matrix, For feature dimension, For the updated features, Attention weights; S220: By stacking multiple encoder blocks, the network progressively downsamples the point cloud, outputting multi-layered point cloud features, including bottom-layer dense point features, middle-layer superpoint features, and top-layer sparse superpoint features; where the top-layer sparse superpoint features are denoted as... and The dimensions are respectively and , and These represent the number of superpoints after downsampling the source point cloud and the target point cloud, respectively. The dimension of the superpoint feature; the pointwise geometric features output by the decoding network. As a feature of dense points at the bottom layer.

[0009] Preferably, S300 includes: S310: Top-level sparse superpoint features and The input is into a global transformer module, which contains multiple stacked global transformer blocks. Each global transformer block contains a geometry-aware self-attention module and a position-aware cross-attention module. S320: The geometry-aware self-attention module includes a geometry branch and a context branch. The geometry branch first calculates the pairwise geometric embeddings between superpoints. The geometric embeddings contain Euclidean distance and angle information and are encoded into a high-dimensional representation using a sine function. Using this geometric embedding, the geometry code and position code are linearly projected through two learnable matrices. The context branch linearly maps the superpoint features output by the encoder to query, key, and value through three learnable matrices. The query matrix, key matrix, and position code are injected into the attention mechanism to obtain a hybrid attention matrix. The hybrid attention matrix is ​​then weighted and summed with the geometry code and value matrix to obtain geometric information and context information. In the position-aware cross-attention module, the features of the source point cloud and the target point cloud enhanced by self-attention interact. The position-aware cross-attention module first adds the geometric information and context information calculated in the geometry-aware self-attention module to obtain position-aware features. Cross-attention calculation is performed using the source point cloud features as the query and the target point cloud features as the key and value to achieve information flow across point clouds and output superpoint features enhanced by cross-attention. S330: By stacking multiple global transformer blocks, the superpoint features are updated once after each global transformer block, following the process in S320; after stacking multiple blocks, the final enhanced superpoint features are output. and ; S340: Perform L2 normalization on the enhanced superpoint features and calculate the cosine similarity matrix. : in, This is the index of the superpoint in the source point cloud, with a value ranging from 1 to... , This represents the number of superpoints after downsampling from the source point cloud. The index of the superpoint in the target point cloud, with a value ranging from 1 to... , The number of superpoints after downsampling the target point cloud; S350: The cosine similarity matrix is ​​double-normalized to enhance the uniqueness of the matching, and the K pairs of superpoints with the highest similarity are selected as the initial coarse matching set. The initial transformation matrix is ​​estimated using these coarse matching point pairs via an adaptive weighting method. .

[0010] Preferably, S400 includes: S410: Using the initial transformation matrix , respectively target point cloud Heyuan Point Cloud Transform to the other's coordinate system to obtain a preliminarily aligned point cloud. , ; S420: Define a projection matrix K based on the acquisition viewpoints of all point clouds. The projection matrix contains the camera's intrinsic parameters, i.e., focal length. , and principal point coordinates , ; Align the point cloud Projected onto a two-dimensional imaging plane using the projection matrix K, for each three-dimensional point Calculate its corresponding pixel coordinates and the depth value Assign to this pixel to generate a depth map. .

[0011] Preferably, S500 includes: S510: Employing a multi-scale void filling algorithm for... Repair methods include: first, expanding the known depth region using dilation; then, filling holes using interpolation based on neighborhood mean; and finally, smoothing the depth map using median filtering to obtain a dense depth map. ; S520: For the generated dense depth map A pre-trained depth map conditional diffusion model is used for depth feature extraction. This process utilizes a joint architecture of ControlNet and Stable Diffusion to extract multi-scale features with strong semantic consistency and geometric structure preservation capabilities from the depth map. The specific implementation is as follows: Dense depth map The input is the deep encoding branch of ControlNet, which contains a lightweight feature extraction network that outputs multi-scale deep conditional feature maps. These feature maps are injected as residuals into the corresponding layers of the Stable Diffusion U-Net decoder, thereby guiding the generation process to conform to the structural constraints of the depth maps. At the same time, text prompts are constructed for the classifier to freely guide sampling. Positive prompts describe the scene content, while negative prompts contain unwanted content or are set to blank. Positive and negative prompts are encoded into text embeddings for conditional generation and unconditional generation, respectively. S530: Performs the inverse sampling process of the diffusion model: generates a pure Gaussian noise image with the same size as the depth map. Using this as the starting point, a DDIM sampling strategy is employed, with a preset number of sampling steps. Back-diffusion begins from this preset starting step, and noise is estimated at each step. Calculated using the classifier-guided formula: in To guide the scale, Conditions for receiving depth maps simultaneously and text conditions Noise in conditional model predictions To receive noise from the unconditional model predictions based solely on depth map conditions, For the first The noisy image of the step; S540: During the sampling process, feature maps of the intermediate layers of the U-Net decoder are extracted as depth features. The feature maps of each layer are upsampled to the resolution of the original depth map, i.e., H×W, through bilinear interpolation. Then, these feature maps are concatenated in the channel dimension to obtain a multi-scale feature tensor. S550: Principal component analysis is used to reduce the dimensionality of the stitched features. A PCA transformation matrix pre-trained on a large amount of image data is used to compress the stitched high-dimensional features to a preset low-dimensional feature dimension, thus obtaining the final depth map features.

[0012] Preferably, S600 includes: S610: Establish the spatial correspondence between point cloud and depth map, and use the depth map features obtained from S550. The projection mapping relationship is used to backfeed the original point cloud. For points that are outside the image range after projection, their corresponding depth features are set to zero vectors. S620: Combines the returned depth features with its own... Features are concatenated to obtain joint features. .

[0013] Preferably, S700 includes: S710: Based on the obtained joint features In the source cloud and target point cloud Fine-grained matching is performed between corresponding points: the nearest neighbor criterion is used to search in the feature space. For each point in the source point cloud, the nearest neighbor feature is searched in the feature set of the target point cloud to obtain candidate matching pairs. Similarly, for each point in the target point cloud, the nearest neighbor feature is searched in the feature set of the source point cloud. Only when the nearest neighbor results in the two directions are consistent is the candidate matching pair retained as the initial matching pair. S720: To filter out outlier matches, apply a geometric consistency constraint: for each initial matching pair Check whether other matches within its local neighborhood are consistent, i.e., in the point cloud. China and Israel With center radius Select several neighboring points within the sphere, and for each neighboring point... Find its matching point in the target point cloud. ,if lie in The radius is If a point is found within a sphere, its local neighborhood is considered to have a consistent match. The number of consistent matches is counted; if the proportion is lower than a preset threshold, the initial match is discarded. After this filtering process, a high-precision set of point correspondences is obtained. ; S730: Finally, using the point-to-point set, the final transformation matrix is ​​solved by the weighted singular value decomposition method: different weights are assigned to each matching pair, and the weighted centroids of the two point clouds are first calculated: in, This represents the weight assigned to each matching pair; Calculate the centroid-free covariance matrix: The rotation matrix is ​​obtained by performing singular value decomposition on H. With translation vector This constitutes the final transformation matrix. .

[0014] Preferably, it further includes an iterative optimization step: transforming the transformation matrix output by S700... As the new initial transformation matrix, it replaces the one in S300. The process repeats steps S400 to S700 to form an iterative optimization loop. When the change in the transformation matrix is ​​less than a preset threshold in two consecutive iterations, or when the maximum number of iterations is reached, the iteration is terminated, and the transformation matrix calculated in the last iteration is output as the final result.

[0015] A two-stage point cloud registration system that integrates depth map features includes: The point cloud acquisition and initial encoding module is used to acquire source point cloud and target point cloud, and to preprocess and initially encode the two respectively to obtain triplet representations of source point cloud and target point cloud; The geometric feature extraction module is used to extract multi-level geometric features, including top-level and bottom-level features, from a rotation-invariant attention encoding network with triplet representations as input and shared weights. The coarse matching module is used to perform global context aggregation and coarse matching based on the top-level features to obtain the initial transformation matrix; The depth map generation module is used to align the source point cloud and the target point cloud using an initial transformation matrix, and then project the initially aligned point cloud to generate a depth map. The depth feature extraction module is used to fill holes in the depth map to obtain a dense depth map, extract multi-scale depth features from the dense depth map and perform dimensionality reduction to obtain depth map features; The cross-modal fusion module is used to fuse low-level features and depth map features across modalities to obtain joint features; The refined matching and transformation solution module is used to perform refined matching of the source point cloud and the target point cloud based on joint features, and solve for the final transformation matrix.

[0016] This invention proposes a two-stage point cloud registration method and system that integrates depth map features. First, a rotation-invariant attention network is used to extract multi-level geometric features and perform coarse matching to obtain an initial transformation matrix. Based on this, the aligned point cloud is projected to generate a depth map, and a pre-trained diffusion model is innovatively used to extract multi-scale semantic features from the depth map. Then, through cross-modal feature mapping and fusion, geometric features and depth semantic features are organically combined, significantly enhancing the distinguishability of points in low-overlap regions and symmetrical structures. Finally, fine-grained matching and transformation matrix solving are performed based on the fused features, and an iterative optimization loop can be optionally introduced to further improve accuracy. This invention can effectively suppress mismatches and improve the robustness and accuracy of registration, and is particularly suitable for complex scenarios with low overlap rates and repetitive, symmetrical geometric structures, providing a reliable 3D registration solution for applications such as autonomous driving, robot navigation, and digital twins. Attached Figure Description

[0017] Figure 1 This is a flowchart of a point cloud registration method incorporating deep semantic enhancement in one embodiment of the present invention; Figure 2 This is a schematic diagram of a point cloud registration method that incorporates deep semantic enhancement in one embodiment of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0019] This invention relates to a point cloud registration method and system incorporating deep semantic enhancement, aiming to address the insufficient registration accuracy of existing technologies in scenarios with low overlap, large rotation, and symmetrical structures. This invention significantly improves the robustness and accuracy of the final registration by introducing a cross-modal feature fusion mechanism and embedding a depth map-based semantic feature enhancement step into the traditional point cloud geometric matching process. A detailed flowchart is shown below. Figure 1 As shown.

[0020] (I) Point cloud preprocessing and initial feature encoding S100: Obtain the source point cloud and the target point cloud, perform preprocessing and initial feature encoding on both respectively, and obtain the triplet representations of the source point cloud and the target point cloud.

[0021] Furthermore, S100 includes: S110: Input two frames of point cloud data to be registered, denoted as the source point cloud. and target point cloud ,in Include A three-dimensional point, Include A three-dimensional point, source point cloud The coordinates of each point in the array are represented as follows: Target point cloud The coordinates of each point in the array are represented as follows: ; S120: Yes and Sampling is performed at the farthest point to obtain a representative subset with uniform distribution. At the same time, principal component analysis or neighborhood-based covariance matrix decomposition is used to estimate the normal vector of each point. S130: Combine the original 3D coordinates with the normal vector to construct the initial input representation of each point, forming a triplet representation of the source point cloud. Triple representation of the target point cloud ,in, and Let these represent the sets of normal vectors for each point in the source point cloud and the target point cloud, respectively. and These represent the point-by-point feature vectors of each point in the source point cloud and the target point cloud, respectively, and are both initialized as vectors of all 1s.

[0022] This preprocessing step ensures that the input data for subsequent feature extraction contains basic geometric information.

[0023] (II) Multi-level geometric feature extraction based on rotation-invariant attention S200: A rotation-invariant attention encoding network that uses triples to represent inputs with shared weights, extracting multi-level geometric features that include top-level and bottom-level features.

[0024] Furthermore, S200 includes: S210: Input the triplet representations of the two point clouds obtained in S100 into a rotation-invariant feature encoding network with shared weights. The encoding network adopts a hierarchical encoder-decoder architecture, which contains multiple encoder blocks and corresponding decoder blocks. Each encoder block contains a downsampling layer and multiple attention layers, and each decoder block contains a transition upsampling layer and multiple attention layers. The specific calculation process for the attention layer is as follows: For each anchor point in the set of anchor points and its corresponding normal vector The set of anchor points is the set of points whose k nearest neighbors are retrieved from the set of support points to form a neighborhood point set; where the set of anchor points is the set of points in the current attention layer to be calculated for the output features, and the set of support points is the set of reference points used to provide neighborhood information. For each neighboring point and its normal vector Construct a local coordinate representation based on point-pair features, and calculate it using the following formula: in , indicating from the anchor point To neighboring points The translation vector, where ∠ represents the angle between the vectors, is completely invariant to global rotation; The aforementioned local coordinate representation is projected onto a geometric code using a linear layer, while the features of neighboring points are projected onto a context code using the same linear layer. During attention calculation, the query vector is dot-producted with both the positional encoding matrix of the geometric code and the key matrix of the context code. The sums are then scaled and normalized using Softmax to obtain attention weights. These attention weights are then used to perform a weighted summation on the geometric information matrix of the geometric code and the value matrix of the context code, calculated using the following formula: in For query vector, The key matrix, For value matrices, For position encoding, For geometric information matrix, For feature dimension, For the updated features, Attention weights; S220: By stacking multiple encoder blocks, the network progressively downsamples the point cloud, outputting multi-layered point cloud features, including bottom-layer dense point features, middle-layer superpoint features, and top-layer sparse superpoint features; where the top-layer sparse superpoint features are denoted as... and The dimensions are respectively and , and These represent the number of superpoints after downsampling the source point cloud and the target point cloud, respectively. The dimension of the superpoint feature; the pointwise geometric features output by the decoding network. As a feature of dense points at the bottom layer.

[0025] (III) Global Context Aggregation and Coarse Matching S300: Perform global context aggregation and coarse matching based on top-level features to obtain the initial transformation matrix.

[0026] Furthermore, the S300 includes: S310: Top-level sparse superpoint features and The input is into a global transformer module, which contains multiple stacked global transformer blocks. Each global transformer block contains a geometry-aware self-attention module and a position-aware cross-attention module. S320: The geometry-aware self-attention module includes a geometry branch and a context branch. The geometry branch first calculates the pairwise geometric embeddings between superpoints. The geometric embeddings contain Euclidean distance and angle information and are encoded into a high-dimensional representation using a sine function. Using this geometric embedding, the geometry code and position code are linearly projected through two learnable matrices. The context branch linearly maps the superpoint features output by the encoder to query, key, and value through three learnable matrices. The query matrix, key matrix, and position code are injected into the attention mechanism to obtain a hybrid attention matrix. The hybrid attention matrix is ​​then weighted and summed with the geometry code and value matrix to obtain geometric information and context information. In the position-aware cross-attention module, the features of the source point cloud and the target point cloud enhanced by self-attention interact. The position-aware cross-attention module first adds the geometric information and context information calculated in the geometry-aware self-attention module to obtain position-aware features. Cross-attention calculation is performed using the source point cloud features as the query and the target point cloud features as the key and value to achieve information flow across point clouds and output superpoint features enhanced by cross-attention. S330: By stacking multiple global transformer blocks, the superpoint features are updated once after each global transformer block, following the process in S320; after stacking multiple blocks, the final enhanced superpoint features are output. and ; S340: Perform L2 normalization on the enhanced superpoint features and calculate the cosine similarity matrix. : in, This is the index of the superpoint in the source point cloud, with a value ranging from 1 to... , This represents the number of superpoints after downsampling from the source point cloud. The index of the superpoint in the target point cloud, with a value ranging from 1 to... , The number of superpoints after downsampling the target point cloud; S350: The cosine similarity matrix is ​​double-normalized to enhance the uniqueness of the matching, and the K pairs of superpoints with the highest similarity are selected as the initial coarse matching set. The initial transformation matrix is ​​estimated using these coarse matching point pairs via an adaptive weighting method. .

[0027] (iv) Alignment of point cloud projection and generation of depth map S400: Align the source point cloud and the target point cloud using the initial transformation matrix, and project the initially aligned point cloud to generate a depth map.

[0028] Furthermore, the S400 includes: S410: Using the initial transformation matrix , respectively target point cloud Heyuan Point Cloud Transform to the other's coordinate system to obtain a preliminarily aligned point cloud. , ; S420: Define a projection matrix K based on the acquisition viewpoints of all point clouds. The projection matrix contains the camera's intrinsic parameters, i.e., focal length. , and principal point coordinates , ; Align the point cloud Projected onto a two-dimensional imaging plane using the projection matrix K, for each three-dimensional point Calculate its corresponding pixel coordinates and the depth value Assign to this pixel to generate a depth map. .

[0029] (v) Multi-scale feature extraction of depth maps S500: Hole filling and repair of the depth map to obtain a dense depth map, multi-scale depth features are extracted from the dense depth map and dimensionality reduction is performed to obtain depth map features.

[0030] Furthermore, the S500 includes: S510: Employs a multi-scale void filling algorithm for... Repair methods include: first, expanding the known depth region using dilation; then, filling holes using interpolation based on neighborhood mean; and finally, smoothing the depth map using median filtering to obtain a dense depth map. ; S520: For the generated dense depth map A pre-trained depth map conditional diffusion model is used for depth feature extraction. This process utilizes a joint architecture of ControlNet and Stable Diffusion to extract multi-scale features with strong semantic consistency and geometric structure preservation capabilities from the depth map. The specific implementation is as follows: Dense depth map The input is the deep encoding branch of ControlNet, which contains a lightweight feature extraction network that outputs multi-scale deep conditional feature maps. These feature maps are injected as residuals into the corresponding layers of the Stable Diffusion U-Net decoder, thereby guiding the generation process to conform to the structural constraints of the depth maps. At the same time, text prompts are constructed for the classifier to freely guide sampling. Positive prompts describe the scene content, while negative prompts contain unwanted content or are set to blank. Positive and negative prompts are encoded into text embeddings for conditional generation and unconditional generation, respectively. S530: Performs the inverse sampling process of the diffusion model: generates a pure Gaussian noise image with the same size as the depth map. As the starting point; the DDIM sampling strategy is adopted, and a preset number of sampling steps is set (in this embodiment, the sampling steps are set to 20 steps), starting from the preset starting step. Initiate reverse diffusion (i.e., progressively denoising the noisy image to the intermediate state corresponding to step 150). At each step, noise estimation... Calculated using the classifier-guided formula: in To guide the scale, Conditions for receiving depth maps simultaneously and text conditions Noise in conditional model predictions To receive noise from the unconditional model predictions based solely on depth map conditions, For the first The noisy image of the step; S540: During the sampling process, feature maps of the intermediate layers of the U-Net decoder are extracted as depth features (different intermediate layers have different resolutions, which can capture multi-scale information from global scene layout to local geometric details, and the number of channels of each feature map is 1280). For each layer, the feature maps of each layer are upsampled to the resolution of the original depth map, i.e., H×W, through bilinear interpolation. Then, these feature maps are concatenated in the channel dimension to obtain a multi-scale feature tensor with a dimension of H×W×(3×1280). S550: Principal component analysis is used to reduce the dimensionality of the stitched features. A PCA transformation matrix pre-trained on a large amount of image data is used to compress the stitched high-dimensional features (3840 dimensions) to a preset low-dimensional feature dimension (128 dimensions) to obtain the final depth map features with a size of H×W×128.

[0031] This feature map contains both semantic consistency information from the diffusion model and rich geometric structure information, and is robust to noise and missing data in the depth map.

[0032] (vi) Cross-modal feature fusion S600: Cross-modal fusion of low-level features and depth map features to obtain joint features.

[0033] Furthermore, the S600 includes: S610: Establish the spatial correspondence between point cloud and depth map, and use the depth map features obtained from S550. The projection mapping relationship is used to backfeed the original point cloud. For points that are outside the image range after projection, their corresponding depth features are set to zero vectors. S620: Combines the returned depth features with its own... Features are concatenated to obtain joint features. This feature possesses both geometric invariance and semantic distinctiveness.

[0034] (vii) Refined matching and transformation matrix solution S700: Based on joint features, a refined matching of the source point cloud and the target point cloud is performed to obtain the final transformation matrix.

[0035] Furthermore, the S700 includes: S710: Based on the obtained joint features In the source cloud and target point cloud Fine-grained matching is performed between corresponding points: the nearest neighbor criterion is used to search in the feature space. For each point in the source point cloud, the nearest neighbor feature is searched in the feature set of the target point cloud to obtain candidate matching pairs. Similarly, for each point in the target point cloud, the nearest neighbor feature is searched in the feature set of the source point cloud. Only when the nearest neighbor results in the two directions are consistent is the candidate matching pair retained as the initial matching pair. S720: To filter out outlier matches, apply a geometric consistency constraint: for each initial matching pair Check whether other matches within its local neighborhood are consistent, i.e., in the point cloud. China and Israel With center radius Select several neighboring points within the sphere, and for each neighboring point... Find its matching point in the target point cloud. ,if lie in The radius is If a point is found within a sphere, its local neighborhood is considered to have a consistent match. The number of consistent matches is counted; if the proportion is lower than a preset threshold, the initial match is discarded. After this filtering process, a high-precision set of point correspondences is obtained. ; S730: Finally, using the point-to-point set, the final transformation matrix is ​​solved by the weighted singular value decomposition method: different weights are assigned to each matching pair, and the weighted centroids of the two point clouds are first calculated: in, This represents the weight assigned to each matching pair; Calculate the centroid-free covariance matrix: The rotation matrix is ​​obtained by performing singular value decomposition on H. Translation vector This constitutes the final transformation matrix. .

[0036] Furthermore, to further improve registration accuracy, an iterative optimization step is also included: the transformation matrix output by S700... As the new initial transformation matrix, it replaces the one in S300. The process repeats steps S400 to S700 to form an iterative optimization loop. When the change in the transformation matrix is ​​less than a preset threshold in two consecutive iterations, or when the maximum number of iterations is reached, the iteration is terminated, and the transformation matrix calculated in the last iteration is output as the final result.

[0037] Compared with the prior art, the present invention has the following advantages: 1. Significantly Improved Registration Accuracy in Low-Overlap and Extreme Rotation Scenes: This invention employs a two-stage registration strategy: the first stage extracts geometric features and performs coarse matching using a rotation-invariant attention network to obtain an initial transformation matrix; the second stage projects the aligned point cloud into a depth map and utilizes a pre-trained diffusion model to extract rich semantic information implicit in the depth map, fusing it with the original point cloud geometric features across modalities. Since depth map features have a stronger ability to perceive the structural layout and object categories of a scene, even in extreme cases where the point cloud overlap rate is less than 30%, or when there is a large angle of rotation, the fused features can still provide sufficient discriminative power, effectively avoiding mismatches and significantly improving the accuracy and success rate of registration.

[0038] 2. Significantly Enhanced Robustness to Noise, Occlusion, and Symmetrical Structures: The depth map itself exhibits good tolerance to local noise, outliers, and occlusion during point cloud acquisition. This characteristic is further enhanced by the multi-scale depth features extracted using the diffusion model. When the depth features are fused with the rotation-invariant geometric features of the point cloud, the joint features retain both pose-invariant geometric descriptions and globally consistent semantic information. This enables the invention to accurately distinguish symmetrical structures that are inherently prone to confusion, while effectively suppressing erroneous matching caused by sensor noise or local missing data, thus maintaining stable registration performance in complex real-world environments.

[0039] In one embodiment, a two-stage point cloud registration system that fuses depth map features is also provided, comprising: The point cloud acquisition and initial encoding module is used to acquire source point cloud and target point cloud, and to preprocess and initially encode the two respectively to obtain triplet representations of source point cloud and target point cloud; The geometric feature extraction module is used to extract multi-level geometric features, including top-level and bottom-level features, from a rotation-invariant attention encoding network with triplet representations as input and shared weights. The coarse matching module is used to perform global context aggregation and coarse matching based on the top-level features to obtain the initial transformation matrix; The depth map generation module is used to align the source point cloud and the target point cloud using an initial transformation matrix, and then project the initially aligned point cloud to generate a depth map. The depth feature extraction module is used to fill holes in the depth map to obtain a dense depth map, extract multi-scale depth features from the dense depth map and perform dimensionality reduction to obtain depth map features; The cross-modal fusion module is used to fuse low-level features and depth map features across modalities to obtain joint features; The refined matching and transformation solution module is used to perform refined matching of the source point cloud and the target point cloud based on joint features, and solve for the final transformation matrix.

[0040] Specific limitations regarding a two-stage point cloud registration system that integrates depth map features can be found in the limitations of a two-stage point cloud registration method that integrates depth map features described above, and will not be repeated here. Each module in the aforementioned two-stage point cloud registration system that integrates depth map features can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0041] In one embodiment, a computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a two-stage point cloud registration method that fuses depth map features.

[0042] In one embodiment, a computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a two-stage point cloud registration method that fuses depth map features.

[0043] The foregoing has provided a detailed description of a two-stage point cloud registration method and system for fusing depth map features, as provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A two-stage point cloud registration method that integrates depth map features, characterized in that, Includes the following steps: S100: Obtain the source point cloud and the target point cloud, perform preprocessing and initial feature encoding on both respectively, and obtain triplet representations of the source point cloud and the target point cloud; S100 includes: S110: Input two frames of point cloud data to be registered, denoted as the source point cloud. and target point cloud ,in Include A three-dimensional point, Include A three-dimensional point, source point cloud The coordinates of each point in the array are represented as follows: Target point cloud The coordinates of each point in the array are represented as follows: ; S120: Yes and Sampling is performed at the farthest point to obtain a representative subset with uniform distribution. At the same time, principal component analysis or neighborhood-based covariance matrix decomposition is used to estimate the normal vector of each point. S130: Combine the original 3D coordinates with the normal vector to construct the initial input representation of each point, forming a triplet representation of the source point cloud. Triple representation of the target point cloud ,in, and Let these represent the sets of normal vectors for each point in the source point cloud and the target point cloud, respectively. and These represent the point-by-point feature vectors of each point in the source point cloud and the target point cloud, respectively, and are both initialized to vectors of all 1s. S200: A rotation-invariant attention encoding network that uses triples to represent input shared weights to extract multi-level geometric features containing top-level and bottom-level features; S200 includes: S210: Input the triplet representations of the two point clouds obtained in S100 into a rotation-invariant feature encoding network with shared weights. The encoding network adopts a hierarchical encoder-decoder architecture, which contains multiple encoder blocks and corresponding decoder blocks. Each encoder block contains a downsampling layer and multiple attention layers, and each decoder block contains a transition upsampling layer and multiple attention layers. The specific calculation process for the attention layer is as follows: For each anchor point in the set of anchor points and its corresponding normal vector The set of anchor points is the set of points whose k nearest neighbors are retrieved from the set of support points to form a neighborhood point set; where the set of anchor points is the set of points in the current attention layer to be calculated for the output features, and the set of support points is the set of reference points used to provide neighborhood information. For each neighboring point and its normal vector Construct a local coordinate representation based on point-pair features, and calculate it using the following formula: in , indicating from the anchor point To neighboring points The translation vector, where ∠ represents the angle between the vectors, is completely invariant to global rotation; The aforementioned local coordinate representation is projected onto a geometric code using a linear layer, while the features of neighboring points are projected onto a context code using the same linear layer. During attention calculation, the query vector is dot-producted with both the positional encoding matrix of the geometric code and the key matrix of the context code. The sums are then scaled and normalized using Softmax to obtain attention weights. These attention weights are then used to perform a weighted summation on the geometric information matrix of the geometric code and the value matrix of the context code, calculated using the following formula: ， in For query vector, The key matrix, For value matrices, For position encoding, For geometric information matrix, For feature dimension, For the updated features, Attention weights; S220: By stacking multiple encoder blocks, the network progressively downsamples the point cloud, outputting multi-layered point cloud features, including bottom-layer dense point features, middle-layer superpoint features, and top-layer sparse superpoint features; where the top-layer sparse superpoint features are denoted as... and The dimensions are respectively and , and These represent the number of superpoints after downsampling the source point cloud and the target point cloud, respectively. The dimension of the superpoint feature; the pointwise geometric features output by the decoding network. As a feature of dense points at the bottom layer; S300: Perform global context aggregation and coarse matching based on top-level features to obtain the initial transformation matrix; S400: Align the source point cloud and the target point cloud using the initial transformation matrix, and project the initially aligned point cloud to generate a depth map; S500: Hole filling and repair of the depth map to obtain a dense depth map, extract multi-scale depth features from the dense depth map and perform dimensionality reduction to obtain depth map features; S600: Performs cross-modal fusion of low-level features and depth map features to obtain joint features; S700: Based on joint features, a refined matching of the source point cloud and the target point cloud is performed to obtain the final transformation matrix.

2. The method according to claim 1, characterized in that, The S300 includes: S310: Top-level sparse superpoint features and The input is into a global transformer module, which contains multiple stacked global transformer blocks. Each global transformer block contains a geometry-aware self-attention module and a position-aware cross-attention module. S320: The geometry-aware self-attention module includes a geometry branch and a context branch. The geometry branch first calculates the pairwise geometric embeddings between superpoints. The geometric embeddings contain Euclidean distance and angle information and are encoded into a high-dimensional representation using a sine function. Using this geometric embedding, the geometry code and position code are linearly projected through two learnable matrices. The context branch linearly maps the superpoint features output by the encoder to query, key, and value through three learnable matrices. The query matrix, key matrix, and position code are injected into the attention mechanism to obtain a hybrid attention matrix. The hybrid attention matrix is ​​then weighted and summed with the geometry code and value matrix to obtain geometric information and context information. In the position-aware cross-attention module, the features of the source point cloud and the target point cloud enhanced by self-attention interact. The position-aware cross-attention module first adds the geometric information and context information calculated in the geometry-aware self-attention module to obtain position-aware features. Cross-attention calculation is performed using the source point cloud features as the query and the target point cloud features as the key and value to achieve information flow across point clouds and output superpoint features enhanced by cross-attention. S330: By stacking multiple global transformer blocks, the superpoint features are updated once after each global transformer block, following the process in S320; after stacking multiple blocks, the final enhanced superpoint features are output. and ; S340: Perform L2 normalization on the enhanced superpoint features and calculate the cosine similarity matrix. : in, This is the index of the superpoint in the source point cloud, with a value ranging from 1 to... , This represents the number of superpoints after downsampling from the source point cloud. The index of the superpoint in the target point cloud, with a value ranging from 1 to... , The number of superpoints after downsampling the target point cloud; S350: The cosine similarity matrix is ​​double-normalized to enhance the uniqueness of the matching, and the K pairs of superpoints with the highest similarity are selected as the initial coarse matching set. The initial transformation matrix is ​​estimated using these coarse matching point pairs via an adaptive weighting method. .

3. The method according to claim 2, characterized in that, The S400 includes: S410: Using the initial transformation matrix , respectively target point cloud Heyuan Point Cloud Transform to the other's coordinate system to obtain a preliminarily aligned point cloud. , ; S420: Define a projection matrix K based on the acquisition viewpoints of all point clouds. The projection matrix contains the camera's intrinsic parameters, i.e., focal length. , and principal point coordinates , ; Align the point cloud Projected onto a two-dimensional imaging plane using the projection matrix K, for each three-dimensional point Calculate its corresponding pixel coordinates and the depth value Assign to this pixel to generate a depth map. .

4. The method according to claim 3, characterized in that, The S500 includes: S510: Employing a multi-scale void filling algorithm for... Repair methods include: first, expanding the known depth region using dilation; then, filling holes using interpolation based on neighborhood mean; and finally, smoothing the depth map using median filtering to obtain a dense depth map. ; S520: For the generated dense depth map A pre-trained depth map conditional diffusion model is used for depth feature extraction. This process utilizes a joint architecture of ControlNet and Stable Diffusion to extract multi-scale features with strong semantic consistency and geometric structure preservation capabilities from the depth map. The specific implementation is as follows: Dense depth map The input is the deep encoding branch of ControlNet, which contains a lightweight feature extraction network that outputs multi-scale deep conditional feature maps. These feature maps are injected as residuals into the corresponding layers of the Stable Diffusion U-Net decoder, thereby guiding the generation process to conform to the structural constraints of the depth maps. At the same time, text prompts are constructed for the classifier to freely guide sampling. Positive prompts describe the scene content, while negative prompts contain unwanted content or are set to blank. Positive and negative prompts are encoded into text embeddings for conditional generation and unconditional generation, respectively. S530: Performs the inverse sampling process of the diffusion model: generates a pure Gaussian noise image with the same size as the depth map. Using this as the starting point, a DDIM sampling strategy is employed, with a preset number of sampling steps. Back-diffusion begins from this preset starting step, and noise is estimated at each step. Calculated using the classifier-guided formula: in To guide the scale, Conditions for receiving depth maps simultaneously and text conditions Noise in conditional model predictions To receive noise from the unconditional model predictions based solely on depth map conditions, For the first The noisy image of the step; S540: During the sampling process, feature maps of the intermediate layers of the U-Net decoder are extracted as depth features. The feature maps of each layer are upsampled to the resolution of the original depth map, i.e., H×W, through bilinear interpolation. Then, these feature maps are concatenated in the channel dimension to obtain a multi-scale feature tensor. S550: Principal component analysis is used to reduce the dimensionality of the stitched features. A PCA transformation matrix pre-trained on a large amount of image data is used to compress the stitched high-dimensional features to a preset low-dimensional feature dimension, thus obtaining the final depth map features.

5. The method according to claim 4, characterized in that, The S600 includes: S610: Establish the spatial correspondence between point cloud and depth map, and convert the depth features obtained from S550. The projection mapping relationship is used to backfeed the original point cloud. For points that are outside the image range after projection, their corresponding depth features are set to zero vectors. S620: Combines the returned depth features with its own... Features are concatenated to obtain joint features. .

6. The method according to claim 5, characterized in that, The S700 includes: S710: Based on the obtained joint features In the source cloud and target point cloud Fine-grained matching is performed between corresponding points: the nearest neighbor criterion is used to search in the feature space. For each point in the source point cloud, the nearest neighbor feature is searched in the feature set of the target point cloud to obtain candidate matching pairs. Similarly, for each point in the target point cloud, the nearest neighbor feature is searched in the feature set of the source point cloud. Only when the nearest neighbor results in the two directions are consistent is the candidate matching pair retained as the initial matching pair. S720: To filter out outlier matches, apply a geometric consistency constraint: for each initial matching pair Check whether other matches within its local neighborhood are consistent, i.e., in the point cloud. China and Israel With center radius Select several neighboring points within the sphere, and for each neighboring point... Find its matching point in the target point cloud. ,if lie in The radius is If a point is found within a sphere, its local neighborhood is considered to have a consistent match. The number of consistent matches is counted; if the proportion is lower than a preset threshold, the initial match is discarded. After this filtering process, a high-precision set of point correspondences is obtained. ; S730: Finally, using the point-to-point set, the final transformation matrix is ​​solved by the weighted singular value decomposition method: different weights are assigned to each matching pair, and the weighted centroids of the two point clouds are first calculated: in, This represents the weight assigned to each matching pair; Calculate the centroid-free covariance matrix: The rotation matrix is ​​obtained by performing singular value decomposition on H. With translation vector This constitutes the final transformation matrix. .

7. The method according to claim 6, characterized in that, It also includes an iterative optimization step: transforming the S700 output transformation matrix. As the new initial transformation matrix, it replaces the one in S300. And repeat S400 to S700 to form an iterative optimization loop; When the change in the transformation matrix is ​​less than a preset threshold in two consecutive iterations, or when the maximum number of iterations is reached, the iteration is terminated, and the transformation matrix calculated in the last iteration is output as the final result.

8. A two-stage point cloud registration system that fuses depth map features, for performing the steps of the method as described in any one of claims 1 to 7, characterized in that, include: The point cloud acquisition and initial encoding module is used to acquire source point cloud and target point cloud, and preprocess and initially encode the two respectively to obtain triplet representations of source point cloud and target point cloud; The geometric feature extraction module is used to extract multi-level geometric features, including top-level and bottom-level features, from a rotation-invariant attention encoding network with triplet representations as input and shared weights. The coarse matching module is used to perform global context aggregation and coarse matching based on the top-level features to obtain the initial transformation matrix; The depth map generation module is used to align the source point cloud and the target point cloud using an initial transformation matrix, and then project the initially aligned point cloud to generate a depth map. The depth feature extraction module is used to fill holes in the depth map to obtain a dense depth map, extract multi-scale depth features from the dense depth map and perform dimensionality reduction to obtain depth map features; The cross-modal fusion module is used to fuse low-level features and depth map features across modalities to obtain joint features; The refined matching and transformation solution module is used to perform refined matching of the source point cloud and the target point cloud based on joint features, and solve for the final transformation matrix.

Images