Point cloud completion method based on hierarchical multi-scale feature fusion

Abstract

This invention provides a point cloud completion method based on hierarchical multi-scale feature fusion, belonging to the field of point cloud completion technology. The method includes: acquiring an incomplete point cloud and performing multi-scale sampling to obtain a multi-scale point cloud set; inputting the multi-scale point cloud set into a multi-scale feature fusion encoder to extract hierarchical and global features and perform layer-by-layer fusion to obtain the final hierarchical feature vectors of point clouds at different scales; then inputting these vectors into a hierarchical refinement decoder to perform coarse point cloud feature transformation, hierarchical feature guidance, and cross-attention mechanisms to obtain fine missing point clouds at different scales; and processing the generated fine missing point clouds and the actual missing point clouds using a Transformer-enhanced discriminator to improve the point cloud completion effect. This invention solves the problems of insufficient multi-scale feature fusion, weak correlation between point clouds at different scales, and coarse decoding and generation processes in existing technologies, resulting in missing details, poor structural continuity, and low completion accuracy in the completed point cloud.

Description

Technical Field

[0001] This invention belongs to the field of point cloud completion technology, specifically relating to a point cloud completion method based on hierarchical multi-scale feature fusion. Background Technology

[0002] Point clouds are a commonly used form of 3D data representation, expressing richer 3D information while using less memory. They are widely used in various research fields, such as autonomous driving, 3D reconstruction, robotics, and medical imaging. However, due to limitations in sensor technology, the properties of the object itself, and complex environmental factors, the raw point clouds collected often suffer from incompleteness and sparsity, failing to accurately describe the complete information of the target.

[0003] Traditional point cloud completion methods are mostly based on geometric rules or template alignment, which have limited completion accuracy and are sensitive to noise and have weak generalization ability. With the development of deep learning, neural network-based point cloud completion methods have shown significant advantages, exhibiting higher computational efficiency and better robustness, effectively handling noise and outliers. However, these methods still have many limitations: some methods, such as the Point Completion Network (PCN), rely too heavily on global features and ignore local features, resulting in a lack of fine geometric details in the generated completed point cloud; some methods, such as PF-Net and PoinTr networks, directly concatenate features from multiple resolutions into one feature, ignoring the relationships between point clouds of different resolutions, and using max pooling to aggregate global information of points without considering the connectivity between points and the interrelationships between adjacent points, and also failing to handle the characteristics and correlations between local point cloud structures, leading to irreversible information loss; in addition, the coarse-to-fine refinement strategy requires building multiple decoders to enhance the coarse point cloud, which leads to complex model structure, reduced inference efficiency, and significantly increased inference time, and is prone to producing unnatural seams or holes in the transition regions of point clouds of different resolutions. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a point cloud completion method based on hierarchical multi-scale feature fusion, which solves the problems of insufficient multi-scale feature fusion, weak correlation between point clouds at different scales, and coarse decoding and generation processes, resulting in missing details, poor structural continuity, and low completion accuracy in the completed point cloud.

[0005] To achieve the above objectives, the technical solution adopted in this invention is as follows: a point cloud completion method based on hierarchical multi-scale feature fusion, implemented through a point cloud completion neural network, comprising the following steps:

[0006] Obtain an incomplete point cloud, perform multi-scale sampling on the incomplete point cloud, and obtain a multi-scale point cloud set with different resolutions;

[0007] The multi-scale point cloud sets with different resolutions are input into the multi-scale feature fusion encoder to extract the hierarchical and global features of the point cloud at each scale, and then fused layer by layer to obtain the final hierarchical feature vectors of the point cloud at different scales.

[0008] The final hierarchical feature vectors of point clouds at different scales are input into the hierarchical refinement decoder to perform coarse point cloud feature transformation, obtain coarse point cloud global features, and hierarchical feature guidance is applied to the coarse point cloud global features. Combined with the cross attention mechanism, fine missing point clouds at different scales are obtained.

[0009] Based on the detailed missing point cloud and the true missing point cloud, the Transformer enhancement discriminator is used to enhance the detailed missing point cloud to obtain a high-quality missing point cloud.

[0010] To address the problems of insufficient multi-scale feature fusion, weak correlation between point clouds at different scales, and coarse decoding and generation processes in existing technologies, resulting in missing details, poor structural continuity, and low accuracy in point cloud completion, this invention extracts hierarchical and global features at different scales through a multi-scale feature fusion encoder, enhances features using a self-attention mechanism, and then captures the connections between point clouds at different scales through layer-by-layer feature fusion. A hierarchical refinement decoder generates a coarse point cloud, extracting the correlation between incomplete and complete point clouds. To fully utilize the advantages of multi-scale processing, the hierarchical refinement decoder is combined with the U-Net architecture for point cloud completion tasks. A cross-attention mechanism combined with hierarchical features completes the point cloud from coarse to fine, achieving refined point cloud reconstruction. A Transformer-enhanced discriminator, based on adversarial network principles, enhances the quality of point cloud completion. This invention can preserve the feature details of the point cloud, enhance the structural coherence of the completion result, and generate higher-quality completion results that are closer to the real point cloud in different object categories and real-world scenes, with higher accuracy.

[0011] Furthermore: the multi-scale feature fusion encoder includes a Transformer enhanced hierarchical feature encoder, a Transformer enhanced global feature encoder, and a feature fusion module;

[0012] The Transformer-enhanced hierarchical feature encoder is used to process point clouds of different scales through convolutional layers, mixing pooling, and hierarchical feature enhancement Transformer to obtain hierarchical features.

[0013] The Transformer-enhanced global feature encoder is used to process point clouds of different scales through edge convolution and global feature enhancement Transformer to obtain global features;

[0014] The feature fusion module is used to fuse two adjacent features from hierarchical features and global features in a layer-by-layer fusion manner from low scale to high scale, so as to obtain the final hierarchical feature vector of point clouds at different scales.

[0015] In traditional point cloud completion techniques, the feature extraction module maps point cloud data layer by layer to different dimensional spaces, and then uses max pooling to extract features in the final feature dimension. However, this only obtains global features, and the features extracted using max pooling not only ignore low-dimensional feature information but are also easily affected by dimensional depth, resulting in the loss of a large amount of detailed information and a lack of detail in the point cloud completion result. The multi-scale feature fusion encoder of this invention enhances the hierarchical feature encoder through Transformer, using convolutional layers, hybrid pooling, and hierarchical feature enhancement Transformer to extract and fuse contextual information from different dimensions, reducing the loss of detail information in point cloud data and forming rich hierarchical features. It enhances the global feature encoder through Transformer, using edge convolution to capture local structure and complete the geometric details of the point cloud. The global feature is then aggregated through the global feature enhancement Transformer to obtain global features, representing the macroscopic structure of the point cloud data and providing overall shape information for point cloud completion. Through the feature fusion module, global features and hierarchical features are fused layer by layer from low scale to high scale to generate a final hierarchical feature vector with rich detail information and structure.

[0016] Furthermore, the expression for the Transformer-enhanced hierarchical feature encoder is as follows:

[0017]

[0018]

[0019]

[0020] in, For hierarchical features, Enhance the Transformer for hierarchical features. This refers to the fusion features resulting from splicing and hybrid pooling. To create a mixed pool that combines the maximum and average values, For splicing operations, and These are all point cloud features of different dimensions. Indexes for point cloud features in different dimensions. To map features to a fixed-dimensional multilayer perceptron;

[0021] The expression for the Transformer-enhanced global feature encoder is as follows:

[0022]

[0023]

[0024]

[0025] in, As a global feature, Enhance the Transformer for global features. For multi-scale point clouds with different resolutions. All are the first The features obtained by performing edge convolution between the hierarchical point cloud data and the features themselves. The intermediate features are generated by performing edge convolution on the point cloud data and features of its own layer and the point cloud data and features of the next layer. These represent the original features of incomplete point clouds at different resolutions. For different levels of indexing, For edge convolution;

[0026] The expression for the feature fusion module is as follows:

[0027]

[0028] in, These represent the final hierarchical feature vectors of point clouds at different scales. To enhance the hierarchical feature vectors of point clouds at different scales obtained by the Transformer hierarchical feature encoder. For splicing operations, To map the spliced ​​features to a fixed-dimensional multilayer perceptron.

[0029] To address the problems of incomplete feature information, insufficient fusion of local feature extraction and global semantics, and information fragmentation caused by simple splicing of multi-scale features in existing point cloud completion technologies, this invention enhances the hierarchical feature encoder using a Transformer. This utilizes a multilayer perceptron to map features to different dimensions, followed by splicing and hybrid pooling fusion. The hierarchical feature enhancement Transformer then further enhances the features, achieving effective integration of multi-level features. Furthermore, this invention enhances the global feature encoder using a Transformer. It extracts local geometric structures layer by layer through edge convolution, and then aggregates these into strong semantic global features using a global feature enhancement Transformer. Finally, this invention employs a feature fusion module, achieving deep multi-scale feature fusion of hierarchical and global features through a layer-by-layer fusion approach from low to high scales. This preserves more complete feature information and richer geometric details, improving the accuracy and detail reproduction of point cloud completion.

[0030] Furthermore, the expression for the hierarchical feature enhancement Transformer is as follows:

[0031]

[0032]

[0033]

[0034]

[0035] in, For hierarchical features, For layer normalization, It is a fully connected layer. These are intermediate features that have undergone attention enhancement and normalization. For self-attention mechanism, For activation function, To Mapped queries To Mapped keys, To The mapped value, It is the transpose symbol. for Dimensions To incorporate new features of positional encoding, For position encoding, The fused features are obtained after passing through the convolutional layer;

[0036] The expression for the global feature enhancement Transformer is as follows:

[0037]

[0038]

[0039]

[0040] in, As a global feature, This is an incomplete point cloud obtained by downsampling from the farthest point twice. This is a fused feature obtained after passing through multiple edge convolutional layers.

[0041] To address the shortcomings of traditional feature extraction methods in processing point cloud data, such as insufficient feature representation and incomplete preservation of geometric structure information, leading to missing details and structural distortion in subsequent point cloud completion, this invention designs a hierarchical feature enhancement Transformer and a global feature enhancement Transformer. The hierarchical feature enhancement Transformer fuses spatial location information through positional encoding and enhances hierarchical features using a self-attention mechanism, layer normalization, and fully connected layers. The global feature enhancement Transformer combines the downsampled incomplete point cloud with the fused features obtained through multiple edge convolutional layers, and enhances global features through a self-attention mechanism, layer normalization, and fully connected layers, achieving global feature fusion enhancement. This invention provides complete geometric detail information and can also characterize the macroscopic structure of point cloud data, providing rich features for subsequent coarse point cloud generation.

[0042] Furthermore: the hierarchical refinement decoder includes a coarse point cloud feature converter and a missing point cloud feature generator;

[0043] The coarse point cloud feature converter is used to process the final hierarchical feature vectors at different scales through edge convolution and global feature enhancement Transformer to obtain coarse point cloud global features.

[0044] The missing point cloud feature generation Transformer is used to process the global features of the coarse point cloud through a self-attention layer, a cross-attention layer, and a feedforward layer to obtain fine missing point clouds at different scales.

[0045] To address the issues of uneven transition and weak correlation between incomplete and missing point clouds in existing point cloud completion techniques, leading to distorted completion results and blurred details, this invention designs a hierarchical refinement decoder comprising a coarse point cloud feature converter and a missing point cloud feature generation Transformer. This decoder, while refining the point cloud structure layer by layer, introduces multiple feature layers of final hierarchical feature vectors at different scales based on the generated coarse point cloud. Guided by these hierarchical features, a cross-attention mechanism is used to generate refined point clouds at different scales. Through this cross-scale feature transfer mechanism, low-scale features retain global features while effectively transferring local geometric details to higher scales, forming a complete information flow from macroscopic structure to microscopic details. This allows the final generated point cloud to accurately recover the geometric details of the original point cloud while maintaining overall shape consistency.

[0046] Furthermore, the expression for the coarse point cloud feature converter is as follows:

[0047]

[0048] in, For coarse point cloud global features, This represents the final layer feature vector of the point cloud at the last scale. To map features to a fixed-dimensional multilayer perceptron, For edge convolution, Enhance the Transformer for global features;

[0049] The expression for generating the Transformer from the missing point cloud features is as follows:

[0050]

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[0057] in, For the generated missing point cloud, For a linear layer of linear transformation, For missing point cloud features, and To map different features to a fixed-dimensional multilayer perceptron, The hierarchical features obtained by the multi-scale feature fusion encoder Enhance the Transformer to include missing point cloud features. for Features obtained after dimensional transformation for Features obtained after dimensional transformation For cross-attention layer and input query The sum of, For the self-attention layer, for Query with its own input The sum of.

[0058] To address the issues of uneven transitions and weak correlations between incomplete and missing point clouds in existing point cloud completion techniques, leading to distorted completion results and blurred details, this invention's coarse point cloud feature converter uses edge convolution to capture the local geometric structure of the point cloud and enhances features through a global feature enhancement Transformer. Finally, a multilayer perceptron smoothly and accurately maps the final features of the incomplete point cloud to coarse point cloud global features representing the missing parts, establishing a correlation from known incomplete point clouds to unknown incomplete point clouds. The missing point cloud feature generation Transformer captures the geometric relationship between hierarchical features and the missing point cloud through MGT, generating missing point cloud features with more details. These features are then combined with the transformed final features. Finally, linear and dimensional transformations are performed to obtain a high-precision point cloud with low resolution and high precision for the missing point cloud. Layer by layer, medium-resolution and high-precision missing point clouds are obtained. This invention reduces the discontinuity of the generated point cloud, ensures the integrity of the macroscopic structure of the point cloud, and, guided by cross-attention, generates point clouds with local geometric details, improving the accuracy and quality of point cloud completion.

[0059] Furthermore, the expression for the Transformer enhanced discriminator is as follows:

[0060]

[0061] in, To enhance the binary classification results output by the discriminator of the Transformer, For fine-grained missing point clouds and true missing point clouds, Enhance the hierarchical feature encoder for Transformer. To map the output dimension 1920 layer by layer to a multilayer perceptron with a dimension of 1.

[0062] To address the problem of traditional discriminators failing to effectively capture the complex 3D structural features of point clouds, resulting in low-quality generated point clouds, this invention draws upon the idea of ​​adversarial training between the generator and discriminator in Generative Adversarial Networks (GANs). The generated high-precision missing point cloud and the real point cloud are input into the discriminator separately. After passing through a hierarchical feature enhancement Transformer, a feature vector with rich dimensional information is obtained. This vector is then transformed into 1s through multiple linear transformations, and the output is a binary classification value of true or false. Through adversarial training with the generator, this invention guides the Transformer to enhance the discriminator's output, resulting in a point cloud completion result that is geometrically more reasonable, more realistic in detail, and closer to the distribution of the real point cloud.

[0063] Furthermore, the expression for the loss function of the point cloud completion neural network is as follows:

[0064]

[0065]

[0066]

[0067]

[0068]

[0069] in, The loss function for a point cloud completion system based on hierarchical multi-scale feature fusion is: To complete the loss function The weight, To counteract loss function The weight, To complete the loss function, To counteract the loss function, The chamfer distance between two point cloud datasets. , , To complete point clouds at different scales, For coarse point clouds, For realistic point clouds, and The number of samples that need to be downsampled. For weight parameters, and For two different point cloud data, for The number of points in point cloud data, for The number of points in point cloud data, for Points in point cloud data for Points in point cloud data This represents the total number of samples in the batch. For the sample index of the batch samples, For norm, For the farthest point sampling operation, For multi-scale feature fusion encoder, To refine the decoder in layers, For generator, For discriminator, For fine-grained missing point clouds and true missing point clouds, This is for inputting incomplete point clouds.

[0070] The loss function of this invention includes a completion loss function and an adversarial loss function. The completion loss function supervises the multi-level output from coarse to fine layers to ensure the accuracy of macroscopic structure and microscopic geometry. The adversarial loss function promotes the generation distribution to approximate real data to improve the overall realism. Through the synergistic optimization of the two functions, this invention enhances the point cloud completion capability of complex local features while ensuring shape accuracy, improves the completion quality, and solves the problem of missing completion details caused by the difficulty in simultaneously taking into account global structure and local details in point cloud completion.

[0071] The beneficial effects of this invention are as follows: This invention proposes a point cloud completion method based on hierarchical multi-scale feature fusion, which is used to generate missing point clouds from incomplete point clouds in a hierarchical manner from coarse to fine. The multi-scale feature fusion encoder extracts highly abstract hierarchical and global features of the incomplete point cloud based on an attention mechanism, resulting in more distinct hierarchical features and improving the ability to preserve point cloud details. The feature fusion module effectively integrates secondary hierarchical relationships, achieving a smooth transition of hierarchical features and improving the accuracy of the completion result. The hierarchical refinement decoder first generates a coarse missing point cloud, then extracts the features of the missing point cloud, achieving a smooth transition from incomplete point cloud to missing point cloud. High-quality point cloud generation is then achieved using a layer-by-layer refinement structure, and the hierarchical refinement decoder combined with hierarchical features retains more hierarchical details, guiding the generation of missing point clouds from coarse to fine, and achieving refined point cloud reconstruction. Finally, the Transformer enhancement discriminator is used to further enhance the point cloud completion effect and improve the point cloud completion instructions. Attached Figure Description

[0072] Figure 1 The flowchart shows the point cloud completion method based on hierarchical multi-scale feature fusion.

[0073] Figure 2 Complete the overall structure of the neural network for point clouds;

[0074] Figure 3 A diagram illustrating the structure of an enhanced hierarchical feature encoder for the Transformer;

[0075] Figure 4 A diagram illustrating the structure of the global feature encoder for enhancing the Transformer;

[0076] Figure 5 Here is a structural diagram of the feature fusion module;

[0077] Figure 6 To refine the decoder structure diagram in a layered manner;

[0078] Figure 7 Generate a Transformer structure diagram for the missing point cloud features. Detailed Implementation

[0079] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0080] Example 1

[0081] like Figure 1 As shown, this invention provides a point cloud completion method based on hierarchical multi-scale feature fusion, implemented through a point cloud completion neural network, including the following steps:

[0082] Obtain an incomplete point cloud, perform multi-scale sampling on the incomplete point cloud, and obtain a multi-scale point cloud set with different resolutions;

[0083] The multi-scale point cloud sets with different resolutions are input into the multi-scale feature fusion encoder to extract the hierarchical and global features of the point cloud at each scale, and then fused layer by layer to obtain the final hierarchical feature vectors of the point cloud at different scales.

[0084] The final hierarchical feature vectors of point clouds at different scales are input into the hierarchical refinement decoder to perform coarse point cloud feature transformation, obtain coarse point cloud global features, and hierarchical feature guidance is applied to the coarse point cloud global features. Combined with the cross attention mechanism, fine missing point clouds at different scales are obtained.

[0085] Based on the detailed missing point cloud and the true missing point cloud, the Transformer enhancement discriminator is used to enhance the detailed missing point cloud to obtain a high-quality missing point cloud.

[0086] This invention provides a point cloud completion method based on hierarchical multi-scale feature fusion, outlining the specific operation process of the point cloud completion neural network. The overall structure of the proposed point cloud completion neural network is as follows: Figure 2As shown, the point cloud completion neural network includes a multi-scale feature fusion encoder, a hierarchical refinement decoder, and a Transformer-enhanced discriminator. In machine learning, Transformer represents a transformer or deformer; to avoid repetition and confusion, this paper directly uses "Transformer." First, an incomplete point cloud is input, and multi-scale sampling, such as sampling the furthest point, is used to obtain input point clouds at different resolutions. Second, the point cloud data at the three different resolutions are fed into the Transformer-enhanced hierarchical feature encoder and the Transformer-enhanced global feature encoder, respectively, to obtain hierarchical features and global features at each scale. These features are then fused... The fusion module performs feature fusion layer by layer from low scale to high scale to obtain the final hierarchical features and final features of the incomplete point cloud. Then, the final features are used to generate a coarse missing point cloud, which is then input into a global feature enhancement Transformer to transform it into missing point cloud features. After that, the missing point cloud features are used with the guidance of hierarchical features and a cross-attention mechanism, combined with the final features of the incomplete point cloud, to generate missing point clouds of different scales from coarse to fine. Finally, in order to improve the point cloud completion effect by co-training with the hierarchical refinement decoder, the highest level missing point cloud needs to be input into the Transformer enhanced discriminator, which outputs a binary classification result of true or false. The discriminator enhances the generation capability of the decoder.

[0087] In one embodiment of the present invention, an incomplete point cloud is acquired, and multi-scale sampling is performed on the incomplete point cloud to obtain a multi-scale point cloud set with different resolutions; in a specific embodiment of the present invention, the multi-scale point cloud set with different resolutions can be obtained as follows: , and ;

[0088] In one embodiment of the present invention, in view of the problem that traditional point cloud completion methods can only obtain global features, ignore low-dimensional feature information and are easily affected by the dimensional depth, resulting in the loss of a large amount of point cloud completion detail information, the present invention inputs multi-scale point cloud sets of different resolutions into a multi-scale feature fusion encoder, extracts the hierarchical features and global features of point clouds at each scale, and performs layer-by-layer fusion to obtain the final hierarchical feature vectors of point clouds at different scales.

[0089] The multi-scale feature fusion encoder includes a Transformer enhanced hierarchical feature encoder, a Transformer enhanced global feature encoder, and a feature fusion module.

[0090] The Transformer-enhanced hierarchical feature encoder is used to process point clouds of different scales through convolutional layers, mixing pooling, and hierarchical feature enhancement Transformer to obtain hierarchical features. The hierarchical features fuse contextual information from different dimensions, which can reduce the loss of detailed information in point cloud data.

[0091] The Transformer-enhanced global feature encoder is used to process point clouds at different scales through edge convolution and global feature enhancement Transformer to obtain global features. Global features can characterize the macroscopic structure of point cloud data and provide overall shape information for point cloud completion.

[0092] The feature fusion module is used to fuse two adjacent features from hierarchical features and global features in a layer-by-layer fusion manner from low scale to high scale, so as to obtain the final hierarchical feature vector of point cloud at different scales. The final hierarchical feature vector has rich detail information and a structured representation.

[0093] In specific embodiments of the present invention, such as Figure 3 The diagram shows the structure of the Transformer enhanced hierarchical feature encoder, which uses five convolutional layers to encode multi-scale point clouds at different resolutions. Mapped to different dimensions, the five convolutional layers sequentially output features with dimensions (N, 64), (N, 128), (N, 256), (N, 512), and (N, 1024). These features are then fused to obtain a latent feature vector of dimension (N, 1920). This latent feature vector is then processed by a hybrid pooling process consisting of max pooling and average pooling to obtain a composite latent feature vector. This composite latent feature vector is further enhanced by a hierarchical feature enhancement Transformer to obtain the final hierarchical features. The Transformer-enhanced hierarchical feature encoder fuses multi-dimensional point cloud features through convolutional layers and a self-attention mechanism, and replaces the traditional max pooling operation with hybrid pooling, solving the problem of lost hierarchical information. This allows for the extraction of hierarchical features with distinct geometric structures, improving the preservation of point cloud details. The expression for the Transformer-enhanced hierarchical feature encoder is as follows:

[0094]

[0095]

[0096]

[0097] in, For hierarchical features, Enhance the Transformer for hierarchical features. This refers to the fusion features resulting from splicing and hybrid pooling. To create a mixed pool that combines the maximum and average values, For splicing operations, and These are all point cloud features of different dimensions. Indexes for point cloud features in different dimensions. This is a multilayer perceptron that maps features to a fixed dimension.

[0098] exist Figure 3 In (a) of the Transformer augmented hierarchical feature encoder, the Transformer augmented hierarchical feature encoder includes five convolutional layers, which can obtain point cloud features of five different dimensions. At this time, the expression of the Transformer augmented hierarchical feature encoder can be written as:

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106] in, For hierarchical features, Enhance the Transformer for hierarchical features. This refers to the fusion features resulting from splicing and hybrid pooling. To create a mixed pool that combines the maximum and average values, For splicing operations, and These are all point cloud features of different dimensions. To map features to a fixed-dimensional multilayer perceptron, For multi-scale point clouds with different resolutions. , Indexing point cloud data at different levels;

[0107] In the Transformer-enhanced hierarchical feature encoder, the Hierarchical Feature Enhancement Transformer (HET) has the following structure: Figure 3 As shown in (b) of the Transformer for hierarchical feature enhancement, the fused features are self-attentioned and then residually connected to the original fused features and layer normalized. This transformation produces three feature matrices Q, K, and V. Attention scores are calculated using Q and K, followed by scaling and normalization to obtain weights. The weights are then weighted and summed with V to obtain a new feature representation. The new features are residually connected to the original features and layer normalized to obtain intermediate features. The intermediate features are then fully connected again and residually connected to the original features and layer normalized to obtain the enhanced hierarchical features.

[0108] The expression for the hierarchical feature enhancement Transformer is as follows:

[0109]

[0110]

[0111]

[0112]

[0113] in, For hierarchical features, For layer normalization, It is a fully connected layer. These are intermediate features that have undergone attention enhancement and normalization. For self-attention mechanism, For activation function, To Mapped queries To Mapped keys, To The mapped value, It is the transpose symbol. for Dimensions To incorporate new features of positional encoding, For position encoding, These are the fused features obtained after passing through the convolutional layer.

[0114] In specific embodiments of the present invention, such as Figure 4 The diagram shown is a structure diagram of the Transformer enhanced global feature encoder, which transforms incomplete input point clouds. First, its own features are obtained through dimensional transformation. Then, by performing an edge convolution on its own point cloud data and features through an improved edge convolution layer, it obtains... Features Then and Obtained by downsampling at the farthest point and ,Will and , and The fusion is achieved through edge convolutional layers. of feature And then by using itself and Obtained through edge convolutional layers Features Then and Obtained by downsampling at the farthest point and ,Will and , and The fusion is achieved through edge convolutional layers. of feature And then by using itself and Obtained through edge convolutional layers Features Finally Features After two layers of edge convolution and a global feature enhancement Transformer, global features are obtained. The Transformer-enhanced global feature encoder effectively fuses local structural information by combining graph convolution and attention mechanisms, generating strong semantic global feature representations and enhancing the understanding of the overall macroscopic structure of point clouds. The expression for the Transformer-enhanced global feature encoder is as follows:

[0115]

[0116]

[0117]

[0118] in, As a global feature, Enhance the Transformer for global features. For multi-scale point clouds with different resolutions. All are the first The features obtained by performing edge convolution between the hierarchical point cloud data and the features themselves. The intermediate features are generated by performing edge convolution on the point cloud data and features of its own layer and the point cloud data and features of the next layer. These represent the original features of incomplete point clouds at different resolutions. For different levels of indexing, For edge convolution;

[0119] like Figure 4 As shown in (a) of the Transformer Enhanced Global Feature Encoder, the specific structure of the Transformer Enhanced Global Feature Encoder is as follows:

[0120]

[0121]

[0122]

[0123]

[0124]

[0125]

[0126] in, As a global feature, Enhance the Transformer for global features. , and These are incomplete point clouds at different resolutions. , and All features are obtained by convolving point cloud data at their own level with feature edges. and All of these are intermediate features resulting from convolution of point cloud data and features at their own level with the edge convolution of point cloud data and features at the next level. , and These represent the original features of incomplete point clouds at different resolutions. This is edge convolution.

[0127] In the Transformer augmented global feature encoder The Global Feature Enhancement Transformer (GET) has the following structure: Figure 4 As shown in (b) Global Feature Enhancement Transformer, fused features are obtained through multiple edge convolutional layers, while incomplete input point clouds are obtained through two farthest point downsampling operations. First to Position encoding is performed, then added to the features, resulting in three feature matrices Q, K, and V. Attention scores are calculated using Q and K, followed by scaling and normalization to obtain weights. These weights are then weighted and summed with V to obtain a new feature representation. This new feature is then fully connected and residually connected to the original features. Finally, it is normalized again to obtain the enhanced features combined with the point cloud data. The expression for the global feature enhancement Transformer is as follows:

[0128]

[0129]

[0130]

[0131] in, As a global feature, This is an incomplete point cloud obtained by downsampling from the farthest point twice. This is a fused feature obtained after passing through multiple edge convolutional layers.

[0132] In specific embodiments of the present invention, such as Figure 5 The diagram shows the feature fusion module structure. The lowest-scale point cloud data, after passing through a Transformer enhanced hierarchical feature encoder, is concatenated with its own features. This concatenation is then performed using a multilayer perceptron for dimensional transformation, resulting in the third layer of final hierarchical features. The third layer of final hierarchical features is then concatenated with the hierarchical features obtained from the intermediate-scale point cloud data after passing through a Transformer enhanced hierarchical feature encoder. This concatenation is then performed using a multilayer perceptron for dimensional transformation, resulting in the second layer of final hierarchical features. The second layer of final hierarchical features is then concatenated with the hierarchical features obtained from the highest-scale point cloud data after passing through a Transformer enhanced hierarchical feature encoder. This concatenation is then performed using a multilayer perceptron for dimensional transformation, resulting in the first layer of final hierarchical features. Finally, the first layer of final hierarchical features is concatenated with the global features obtained from a Transformer enhanced global feature encoder. This concatenation is then performed using a multilayer perceptron for dimensional transformation, resulting in the final features.

[0133] The expression for the feature fusion module is as follows:

[0134]

[0135] in, These represent the final hierarchical feature vectors of point clouds at different scales. To enhance the hierarchical feature vectors of point clouds at different scales obtained by the Transformer hierarchical feature encoder. For splicing operations, To map the spliced ​​features to a fixed-dimensional multilayer perceptron.

[0136] In one embodiment of the present invention, in order to address the problem that the transition between incomplete point clouds and missing point clouds in existing point cloud completion technologies is not smooth and the correlation is weak, resulting in distortion of completion results and blurring of details, the present invention inputs the final hierarchical feature vectors of point clouds at different scales into a hierarchical refinement decoder to perform coarse point cloud feature transformation, obtains coarse point cloud global features, guides the coarse point cloud global features with hierarchical features, and combines a cross-attention mechanism to obtain fine missing point clouds at different scales.

[0137] like Figure 6 The diagram shows the hierarchical refinement decoder structure. The final features obtained by the feature fusion module are first transformed linearly to obtain features in three different dimensions, and then transformed in one dimension to obtain the coarse point cloud global features mf. The coarse point cloud global features and the final features are transformed by the global feature enhancement Transformer to obtain the missing point cloud global features. The missing point cloud global features are then transformed by the missing point cloud feature enhancement Transformer. Under the guidance of the final hierarchical features generated by the feature fusion module, fine missing point clouds of different scales are generated layer by layer.

[0138] The hierarchical refinement decoder includes a coarse point cloud feature converter and a missing point cloud feature generator.

[0139] The coarse point cloud feature converter is used to process the final hierarchical feature vectors at different scales through edge convolution and global feature enhancement Transformer to obtain coarse point cloud global features. The expression of the coarse point cloud feature converter is as follows:

[0140]

[0141] in, For coarse point cloud global features, This represents the final layer feature vector of the point cloud at the last scale. To map features to a fixed-dimensional multilayer perceptron, For edge convolution, The global feature enhancement Transformer captures local geometric structures through edge convolution and strengthens feature associations using the global feature enhancement Transformer, achieving a smooth transition from incomplete point clouds to missing point clouds and solving the problem of uneven stitching between the generated point cloud and the original point cloud structure.

[0142] A missing point cloud feature generation Transformer is used to process the global features of the coarse point cloud through a self-attention layer, a cross-attention layer, and a feedforward layer to obtain fine missing point clouds at different scales; in a specific embodiment of the present invention, such as Figure 7As shown, this is the Transformer structure diagram for generating missing point cloud features. In the self-attention layer, the global features of the missing point cloud obtained from the coarse point cloud feature transformer are transformed to generate three feature matrices Q, K, and V, namely Q (query), K (key), and V (value). After performing a linear transformation on Q, K, and V, attention scores are calculated using Q and K. Then, scaling and normalization are performed to obtain weights. The weights are weighted and summed with V to obtain a new feature representation. The new features and the original features are then joined by residual connections and layer normalization to obtain the enhanced features.

[0143] In the cross-attention layer, the enhanced features are transformed into a Q matrix. The final hierarchical features generated by the feature fusion module are transformed to obtain K and V feature matrices. Attention scores are calculated using Q and K, and then scaled and normalized to obtain weights. The weights are weighted and summed with V to obtain a new feature representation. The new features and the original features are residually connected and layer normalized to obtain missing point cloud features that combine hierarchical features.

[0144] In the fully connected layer, the missing point cloud features are mapped to a fixed dimension, then added to the final features after dimensional transformation, and then passed through a multilayer perceptron mapped to a fixed dimension to generate fine missing point clouds at different scales.

[0145] The expression for generating a Transformer from missing point cloud features is as follows:

[0146]

[0147]

[0148]

[0149]

[0150]

[0151]

[0152]

[0153] in, For the generated missing point cloud, For a linear layer of linear transformation, For missing point cloud features, and To map different features to a fixed-dimensional multilayer perceptron, The hierarchical features obtained by the multi-scale feature fusion encoder Enhance the Transformer to include missing point cloud features. for Features obtained after dimensional transformation for Features obtained after dimensional transformation For cross-attention layer and input query The sum of, For the self-attention layer, for Query with its own input The sum of.

[0154] The missing point cloud feature generation Transformer deeply integrates coarse point cloud features with multi-level features through a cross-attention mechanism, achieving a progressive refinement of the missing point cloud from coarse to fine, and restoring a complete result with rich details and geometric coherence.

[0155] In a specific embodiment of the present invention, to address the problem that traditional discriminators struggle to effectively capture the complex 3D structural features of point clouds, resulting in low-quality generated point clouds, the present invention draws upon the idea of ​​adversarial training between the generator and discriminator in Generative Adversarial Networks (GANs). High-precision missing point clouds generated by a hierarchical refinement decoder and the detected real point clouds are input into the discriminator respectively. Based on the incomplete point cloud and finely detailed missing point clouds at different scales, a Transformer-enhanced discriminator is used to generate 3D point cloud data, resulting in a complete point cloud.

[0156] The expression for the Transformer enhanced discriminator is as follows:

[0157]

[0158] in, To enhance the binary classification results output by the discriminator of the Transformer, For fine-grained missing point clouds and true missing point clouds, Enhance the hierarchical feature encoder for Transformer. To map the output dimension 1920 layer by layer to a multilayer perceptron with a dimension of 1; the Transformer enhanced discriminator extracts the hierarchical features of the point cloud through the Transformer architecture, which enhances the ability to distinguish the rationality of the generated point cloud structure and the authenticity of the details, and can effectively guide the generator to output high-quality completion results that are closer to the real distribution.

[0159] In a specific embodiment of the present invention, the present invention designs a loss function for a point cloud completion neural network. The loss function is divided into two parts: one part is the point cloud completion loss function between the coarse point cloud output by the hierarchical refinement decoder and the missing point clouds at three scales and the corresponding real point cloud obtained by sampling the farthest point; the other part is the adversarial loss function generated by the discriminator in the adversarial generative network structure.

[0160] The first part, the point cloud completion loss function, is an indicator for analyzing the difference between the missing point cloud and the real point cloud. However, due to the unordered nature of point clouds, the completion loss function should be permutation-invariant to the point cloud coordinates. The second part, the generative adversarial network's adversarial loss function, is the loss function used to judge the missing point cloud data and the real point cloud data obtained from the hierarchical refinement decoder separately in the discriminator. Therefore, the expression for the loss function of the point cloud completion neural network of this invention is designed as follows:

[0161]

[0162]

[0163]

[0164]

[0165]

[0166] in, The loss function for a point cloud completion system based on hierarchical multi-scale feature fusion is: To complete the loss function The weight, To counteract loss function The weight, To complete the loss function, To counteract the loss function, The chamfer distance between two point cloud datasets. , , To complete point clouds at different scales, For coarse point clouds, For realistic point clouds, and The number of samples that need to be downsampled. For weight parameters, and For two different point cloud data, for The number of points in point cloud data, for The number of points in point cloud data, for Points in point cloud data for Points in point cloud data This represents the total number of samples in the batch. For the sample index of the batch samples, For norm, For the farthest point sampling operation, For multi-scale feature fusion encoder, To refine the decoder in layers, For generator, For discriminator, For fine-grained missing point clouds and true missing point clouds, This is for inputting incomplete point clouds.

[0167] In the completion loss function, the chamfer distance between the missing point cloud and the real point cloud at different scales was calculated. By using the farthest point sampling from The design of multi-scale completion loss increases the proportion of points and can better focus on the features of points in the missing parts, which helps to accurately restore the details of the point cloud. Specifically, it can be set to , This allows the completion loss function and the adversarial loss function to work together, enabling the point cloud completion neural network to be gradually optimized during training.

[0168] The beneficial effects of this invention are as follows:

[0169] This invention solves the problem of insufficient feature extraction in traditional point cloud completion methods by using a multi-scale feature fusion encoder. It utilizes a Transformer-enhanced hierarchical feature encoder and a global feature encoder to capture local details and overall structure of point clouds from different dimensions. Then, the feature fusion module realizes the organic integration of multi-scale features, which significantly improves the richness and robustness of feature representation.

[0170] This invention solves the problems of missing details and discontinuous structure in generated point clouds by using a hierarchical refinement decoder. The coarse point cloud feature transformation module realizes a smooth transition from incomplete regions to complete regions. The missing point cloud feature generation Transformer uses a cross-attention mechanism to enable the generation process to gradually refine from coarse to fine under the guidance of multi-level features, ensuring the geometric continuity and richness of detail of the generated point cloud.

[0171] This invention improves the quality of point cloud completion through the collaborative design of the Transformer enhanced discriminator and the loss function. The Transformer enhanced discriminator, through adversarial training, enables the point cloud completion neural network to learn deeper features, thereby outputting extremely realistic synthetic results. The loss function, through the combination of multi-scale supervision and adversarial training, ensures that the generated point cloud maintains the accuracy of the overall structure while possessing rich detailed information.

[0172] Example 2

[0173] Based on the point cloud completion method based on hierarchical multi-scale feature fusion provided in Example 1, this paper applies it to a specific point cloud completion case. In oil and gas exploration and development, geological modeling is a key step in understanding the structure of underground reservoirs. Due to the limited range of seismic data acquisition and the sparse and uneven distribution of drilling data, the acquired original geological point cloud data often suffers from missing, sparse, or discontinuous problems, affecting the accuracy of reservoir description and subsequent development decisions. Therefore, high-precision completion of geological point cloud data has become an important technical requirement for improving the quality of reservoir modeling. The point cloud completion method based on hierarchical multi-scale feature fusion proposed in this invention is applied to the task of reservoir geological point cloud completion. This network can recover complete and high-resolution three-dimensional geological structures from sparse or incomplete geological point clouds, significantly improving the geometric details and structural integrity of the geological model. Applying the point cloud completion method based on hierarchical multi-scale feature fusion provided in this invention to oil and gas exploration and development can solve the industry problem of insufficient accuracy in underground reservoir characterization caused by sparse and discontinuous point cloud data in geological modeling.

[0174] Workflow:

[0175] Input the original reservoir geological point cloud data, perform farthest point sampling (FPS) on the input point cloud, and generate multiple point cloud subsets with resolutions of 2048 points, 1024 points, and 512 points respectively to capture multi-scale geological features.

[0176] Transformer Enhanced Hierarchical Feature Encoder:

[0177] Point clouds of different resolutions are input into convolutional layers to extract hierarchical features with dimensions of 64, 128, 256, 512, and 1024, respectively. The feature expression capability is enhanced by a self-attention mechanism to capture local details and hierarchical relationships of geological structures.

[0178] Transformer Enhanced Global Feature Encoder: Uses edge convolution to extract local features of point cloud and combines it with the global feature enhancement Transformer module to enhance global semantic information, ensuring that the model can perceive subtle geological changes and grasp the overall structural pattern;

[0179] Feature fusion module: The multi-scale features extracted by the Transformer enhanced hierarchical feature encoder and the Transformer enhanced global feature encoder are fused layer by layer, and integrated step by step from low resolution to high resolution to form a final feature vector with rich semantic information, providing strong guidance for subsequent completion.

[0180] Layered refinement decoder:

[0181] Rough Point Cloud Feature Transformation: Using the final features output by the encoder, a rough missing geological point cloud is generated through linear transformation and MLP. Then, the global feature enhancement Transformer module is used to extract its global features, achieving a smooth transition from incomplete geological points to missing regions. Missing Point Cloud Feature Generation Transformer: Combining a cross-attention mechanism, under the guidance of hierarchical features, refined missing point clouds with high, medium and low resolutions are gradually generated to ensure that the completion result maintains the overall structural consistency while restoring subtle geological features such as faults, fractures, and bedding.

[0182] Transformer Enhanced Discriminator:

[0183] The generated complete geological point cloud and the real point cloud are input into the Transformer augmentation discriminator, and the generation quality is improved through adversarial training.

[0184] Final output: A completed high-resolution geological point cloud that can be directly used for downstream tasks such as 3D geological modeling, reservoir numerical simulation, or drilling trajectory optimization.

[0185] The beneficial effects of this invention are as follows: This invention can recover complete geological features from the original reservoir geological point cloud data with high quality, improve the identification accuracy of key geological structures such as faults, fractures, and unconformities, and provide more reliable geological basis for well location optimization, reserve assessment and development plan formulation, thereby improving the success rate of oil and gas exploration and development while saving operating costs.

Claims

1. A point cloud completion method based on hierarchical multi-scale feature fusion, characterized in that, The point cloud completion neural network is realized by the following steps: An incomplete point cloud is obtained, and multi-scale sampling is performed on the incomplete point cloud to obtain a multi-scale point cloud set with different resolutions; The multi-scale point cloud set with different resolutions is input into a multi-scale feature fusion encoder to extract hierarchical features and global features of each scale of point cloud, and the hierarchical features are fused layer by layer to obtain final hierarchical feature vectors of different scales of point cloud; The final hierarchical feature vectors of different scales of point cloud are input into a hierarchical refinement decoder to convert rough point cloud features to obtain rough point cloud global features, and the hierarchical features are guided to obtain fine missing point clouds of different scales by combining a cross-attention mechanism; Based on the fine missing point cloud and the real missing point cloud, a Transformer enhanced discriminator is used to enhance the effect of the fine missing point cloud to obtain a high-quality missing point cloud; The multi-scale feature fusion encoder includes a Transformer enhanced hierarchical feature encoder, a Transformer enhanced global feature encoder and a feature fusion module; The Transformer enhanced hierarchical feature encoder is configured to process point clouds of different scales by a convolution layer, a hybrid pooling and a hierarchical feature enhancement Transformer to obtain hierarchical features; The Transformer enhanced global feature encoder is configured to process point clouds of different scales by edge convolution and a global feature enhancement Transformer to obtain global features; The feature fusion module is configured to fuse two adjacent features in the hierarchical features and the global features to obtain final hierarchical feature vectors of different scales of point cloud in a form of layer-by-layer fusion from low scale to high scale; The hierarchical refinement decoder includes a rough point cloud feature converter and a missing point cloud feature generation Transformer; The rough point cloud feature converter is configured to process the final hierarchical feature vectors of different scales by edge convolution and a global feature enhancement Transformer to obtain rough point cloud global features; The missing point cloud feature generation Transformer is configured to process the rough point cloud global features by a self-attention layer, a cross-attention layer and a feedforward layer to obtain fine missing point clouds of different scales; The expression of the Transformer enhanced discriminator is as follows: wherein, is a binary classification result of the Transformer enhanced discriminator output, is a fine missing point cloud and a real missing point cloud, is a Transformer enhanced hierarchical feature encoder, is a multi-layer perceptron that maps the output dimension 1920 layer by layer to a dimension of 1.

2. The point cloud completion method based on hierarchical multi-scale feature fusion according to claim 1, characterized in that, The expression of the Transformer enhanced hierarchical feature encoder is as follows: wherein, is a hierarchical feature, is a hierarchical feature enhanced Transformer, is a fused feature after concatenation and mixed pooling, is a mixed pool combining max pool and average pool, is a concatenation operation, and are point cloud features of different dimensions, is an index of point cloud features of different dimensions, is a multi-layer perceptron for mapping features to fixed dimensions; The expression of the Transformer enhanced global feature encoder is as follows: wherein, is a global feature, is a global feature enhanced Transformer, is a multi-scale point cloud of different resolutions, are all the first are features obtained after edge convolution of point cloud data and features of the same level, are intermediate features obtained after edge convolution of point cloud data and features of the same level and point cloud data and features of the next level, are original features of incomplete point clouds of different resolutions, are indexes of different levels, is edge convolution; The expression of the feature fusion module is as follows: wherein, is the final hierarchical feature vector for the different scale point clouds, is the hierarchical feature vector for the different scale point clouds obtained by the Transformer enhanced hierarchical feature encoder, is the concatenation operation, is a multi-layer perceptron that maps the concatenated features to a fixed dimension.

3. The point cloud completion method based on hierarchical multi-scale feature fusion according to claim 2, characterized in that, The expression of the hierarchical feature enhancement Transformer is as follows: wherein, is a hierarchical feature, is a layer normalization, is a fully connected layer, is an intermediate feature enhanced by attention and normalized, is a self-attention mechanism, is an activation function, is a pair of queries mapped, is a pair of keys mapped, is a pair of values mapped, is a transpose symbol, is a dimension, is a new feature combined with position encoding, is a position encoding, is a fusion feature obtained by a convolution layer; The expression of the global feature enhancement Transformer is as follows: wherein, is a global feature, is an incomplete point cloud obtained by twice farthest point downsampling, is a fusion feature obtained by multiple edge convolution layers.

4. The point cloud completion method based on hierarchical multi-scale feature fusion according to claim 1, characterized in that, The expression of the rough point cloud feature converter is as follows: wherein, is a global feature of the coarse point cloud, is a final hierarchical feature vector of the last scale point cloud, is a multi-layer perceptron that maps the features to a fixed dimension, is an edge convolution, is a global feature enhancement Transformer; The expression of the missing point cloud feature generation Transformer is as follows: wherein, is a linear layer for a generated missing point cloud, is a linear layer for a linear transformation, is a missing point cloud feature, and is a different multi-layer perception mapped to a fixed dimension, is a hierarchical feature obtained in a multi-scale feature fusion encoder, is a missing point cloud feature enhancement Transformer, is a feature obtained after a dimension transformation, is a feature obtained after a dimension transformation, is a cross-attention layer with an input query and, is a self-attention layer, is a sum with an input query itself.

5. The point cloud completion method based on hierarchical multi-scale feature fusion according to claim 1, characterized in that, The expression of the loss function of the point cloud completion neural network is as follows: in, The loss function for a point cloud completion system based on hierarchical multi-scale feature fusion is: To complete the loss function The weight, To counteract loss function The weight, To complete the loss function, To counteract the loss function, The chamfer distance between two point cloud datasets. , , To complete point clouds at different scales, For coarse point clouds, For realistic point clouds, and The number of samples that need to be downsampled. For weight parameters, and For two different point cloud data, for The number of points in point cloud data, for The number of points in point cloud data, for Points in point cloud data for Points in point cloud data This represents the total number of samples in the batch. For the sample index of the batch samples, For norm, For the farthest point sampling operation, For multi-scale feature fusion encoder, To refine the decoder in layers, For generator, For discriminator, For fine-grained missing point clouds and true missing point clouds, This is for inputting incomplete point clouds.

Images