Point cloud upsampling method and system fusing self-attention and multi-path graph convolution

Abstract

The application discloses a point cloud up-sampling method and system fusing self-attention and multi-path graph convolution, and relates to the field of point cloud data processing. The method comprises the following steps: obtaining an original point cloud of a target object; inputting the original point cloud into a point cloud up-sampling model to obtain a dense point cloud after up-sampling processing; the point cloud up-sampling model comprises a feature extractor, an up-sampler and a coordinate reconstructor connected in sequence; the feature extractor is used for performing feature extraction on the original point cloud based on a plurality of residual multi-layer graph convolutions and a self-attention module to obtain primary point cloud features; the up-sampler is used for expanding, aggregating and rearranging the primary point cloud features to obtain advanced point cloud features; the coordinate reconstructor is used for regressing the advanced point cloud features to a three-dimensional space to obtain the dense point cloud; and the point cloud up-sampling model is obtained by training a point cloud up-sampling network through a data set. The application improves the quality of point cloud up-sampling.

Description

Technical Field

[0001] This invention relates to the field of point cloud data processing technology, and in particular to a point cloud upsampling method and system that integrates self-attention and multipath graph convolution. Background Technology

[0002] Point clouds are becoming increasingly popular as 3D data, spurring research into theories such as 3D object detection and 3D point cloud completion. They play a crucial role in new applications such as autonomous driving and robot manipulation. Point cloud data acquired through 3D LiDAR sensors provides reliable support for path planning and navigation in autonomous vehicles. However, due to hardware limitations, especially 3D sensor chips, directly acquired point clouds are often low-resolution or even noisy, failing to meet usage requirements. Point cloud upsampling techniques aim to transform sparse, non-uniform point clouds into visually better, more detailed point clouds. Therefore, point cloud upsampling, as a correction operation, has attracted considerable attention from researchers. Traditional point cloud upsampling methods, such as bilaterally regularized sequential depth image upsampling, utilize redundancy between multiple point cloud data points, achieving upsampling only based on a degradation model. Their drawback is that they are only applicable to static scenes, and multi-frame point cloud data relies on precise registration. Point cloud upsampling methods based on single-frame depth images require only the original point cloud data, resulting in high system efficiency. Its drawback is that it is extremely dependent on prior information rather than data-driven, and it is usually difficult to process point cloud data with many features and carrying noise.

[0003] In recent years, with the rapid development of deep learning technology, researchers have increasingly applied it to point cloud upsampling, achieving better results compared to traditional point cloud upsampling methods. However, deep learning-based point cloud upsampling networks largely depend on feature extractors and upsamplers. Currently, deep learning-based point cloud upsampling methods lack multi-scale extraction of local and global features, resulting in excessive outliers and insufficient fine-grained output point clouds. Summary of the Invention

[0004] The purpose of this invention is to provide a point cloud upsampling method and system that integrates self-attention and multipath graph convolution, thereby improving the quality of point cloud upsampling.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A point cloud upsampling method integrating self-attention and multipath graph convolution includes:

[0007] Obtain the original point cloud of the target object;

[0008] The original point cloud is input into a point cloud upsampling model to obtain a dense point cloud after upsampling. The point cloud upsampling model includes a feature extractor, an upsampling unit, and a coordinate reconstructor connected in sequence. The feature extractor is used to extract features from the original point cloud based on multiple residual multi-layer graph convolutions and self-attention modules to obtain primary point cloud features. The upsampling unit is used to expand, aggregate, and rearrange the primary point cloud features to obtain advanced point cloud features. The coordinate reconstructor is used to regress the advanced point cloud features to three-dimensional space to obtain the dense point cloud.

[0009] The point cloud upsampling model is obtained by training the point cloud upsampling network using a dataset.

[0010] Optionally, the feature extractor includes a first k-nearest neighbor algorithm module, a graph convolution module, a first fusion feature extraction module, a second fusion feature extraction module, and a third fusion feature extraction module connected in sequence; the first fusion feature extraction module, the second fusion feature extraction module, and the third fusion feature extraction module have the same structure, and the outputs of the first fusion feature extraction module, the second fusion feature extraction module, and the third fusion feature extraction module are combined as the output of the feature extractor;

[0011] The first fusion feature extraction module includes a bottleneck layer, a second k-nearest neighbor algorithm module, a third k-nearest neighbor algorithm module, a first residual multi-layer graph convolution, a second residual multi-layer graph convolution, a third residual multi-layer graph convolution, a self-attention module, a global max pooling module, and a concatenation module. The output of the bottleneck layer is connected to the inputs of the second k-nearest neighbor algorithm module, the third k-nearest neighbor algorithm module, and the third residual multi-layer graph convolution, respectively. The output of the second k-nearest neighbor algorithm module is connected to the input of the first residual multi-layer graph convolution, the output of the third k-nearest neighbor algorithm module is connected to the input of the second residual multi-layer graph convolution, the output of the third residual multi-layer graph convolution is connected to the input of the self-attention module, and the input of the bottleneck layer is connected to the input of the global max pooling module. The outputs of the first residual multi-layer graph convolution, the second residual multi-layer graph convolution, the self-attention module, and the global max pooling module are all connected to the concatenation module. The output of the concatenation module is combined with the input of the bottleneck layer to obtain the output of the first fusion feature extraction module. The expansion rates of the k-nearest neighbor algorithm in the second k-nearest neighbor algorithm module and the third k-nearest neighbor algorithm module are different.

[0012] Optionally, the first residual multi-layer graph convolution, the second residual multi-layer graph convolution, and the third residual multi-layer graph convolution have the same structure;

[0013] The first residual multi-layer graph convolution includes a fourth k-nearest neighbor algorithm module and multiple graph convolution operation layers connected in sequence. The output of the fourth k-nearest neighbor algorithm module is connected to the input of the first graph convolution operation layer. Each graph convolution operation layer includes a first multi-layer perceptron and a second multi-layer perceptron. Both the first and second multi-layer perceptrons are multi-layer perceptrons with 32 output channels.

[0014] When the graph convolution operation layer is the first graph convolution operation layer, the current graph convolution operation layer is used to: input the center point feature in the input features into the first multilayer perceptron, input the correlation between the neighbor point feature and the center point feature in the input features into the second multilayer perceptron, and then add the outputs of the first multilayer perceptron and the second multilayer perceptron and pass them through the ReLU activation function to obtain the output of the current graph convolution operation layer;

[0015] When the graph convolution operation layer is not the first graph convolution operation layer, the current graph convolution operation layer is used to: input the center point feature in the input features into the first multilayer perceptron, input the correlation between the neighbor point feature and the center point feature in the input features into the second multilayer perceptron, add the outputs of the first multilayer perceptron and the second multilayer perceptron and pass them through the ReLU activation function to obtain the first output result, and add the residual mapping of the output result of the previous graph convolution operation layer to the first output result to obtain the output of the current graph convolution operation layer.

[0016] Optionally, the self-attention module includes a third multilayer perceptron, a fourth multilayer perceptron, a fifth multilayer perceptron, and a sixth multilayer perceptron; the third multilayer perceptron, the fourth multilayer perceptron, the fifth multilayer perceptron, and the sixth multilayer perceptron are all multilayer perceptrons with 32 output channels;

[0017] The self-attention module is used to obtain the Q matrix by passing the input features through a third multilayer perceptron, the K matrix by passing the input features through a fourth multilayer perceptron, and the V matrix by passing the input features through a fifth multilayer perceptron. The transpose of the Q matrix and the K matrix are multiplied to obtain the attention weights. The matrix obtained after normalizing the attention weights through the softmax function is multiplied by the V matrix and then passed through a sixth multilayer perceptron to obtain the output of the self-attention module.

[0018] Optionally, the expansion rate of the k-nearest neighbor algorithm in the second k-nearest neighbor algorithm module is 2, and the expansion rate of the k-nearest neighbor algorithm in the third k-nearest neighbor algorithm module is 3.

[0019] Optionally, the upsampler is used to compress the input features through a bottleneck layer, expand and aggregate the compressed features using edge map convolution, and rearrange the expanded and aggregated features using a shuffle operator to obtain high-level point cloud features.

[0020] Optionally, the coordinate reconstructor includes a seventh multilayer perceptron and an eighth multilayer perceptron connected in sequence.

[0021] This invention also discloses a point cloud upsampling system that integrates self-attention and multipath graph convolution, comprising:

[0022] The original point cloud acquisition module is used to acquire the original point cloud of the target object;

[0023] The original point cloud upsampling module is used to input the original point cloud into the point cloud upsampling model to obtain a dense point cloud after upsampling. The point cloud upsampling model includes a feature extractor, an upsampling unit, and a coordinate reconstructor connected in sequence. The feature extractor is used to extract features from the original point cloud based on multiple residual multi-layer graph convolutions and self-attention modules to obtain primary point cloud features. The upsampling unit is used to expand, aggregate, and rearrange the primary point cloud features to obtain advanced point cloud features. The coordinate reconstructor is used to regress the advanced point cloud features to three-dimensional space to obtain the dense point cloud.

[0024] The point cloud upsampling model is obtained by training the point cloud upsampling network using a dataset.

[0025] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0026] This invention extracts features from the original point cloud through multiple residual multi-layer graph convolutions, integrates more layers of semantic information, and employs a self-attention module to extract global semantic information, making full use of feature information and improving the quality of point cloud upsampling. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of a point cloud upsampling method that integrates self-attention and multipath graph convolution, provided in an embodiment of the present invention.

[0029] Figure 2 This is a schematic diagram of the point cloud upsampling model structure provided in an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of the structure of the first fusion feature extraction module provided in an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of the first residual multilayer graph convolution structure provided in an embodiment of the present invention;

[0032] Figure 5 This is a schematic diagram of the self-attention module structure provided in an embodiment of the present invention;

[0033] Figure 6 This is a schematic diagram illustrating the upsampling results of a point cloud upsampling method that integrates self-attention and multipath graph convolution on the PU-GAN dataset, as provided in an embodiment of the present invention.

[0034] Figure 7 A schematic diagram showing the upsampling comparison results of the method of the present invention and the conventional method on the PU1K dataset, provided in an embodiment of the present invention.

[0035] Figure 8 This is a schematic diagram of a point cloud upsampling system that integrates self-attention and multipath graph convolution, provided as an embodiment of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] The purpose of this invention is to provide a point cloud upsampling method and system that integrates self-attention and multipath graph convolution, thereby improving the quality of point cloud upsampling.

[0038] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] Example 1

[0040] This invention extracts primary point cloud features from the input raw sparse point cloud using a feature extractor that integrates a self-attention module and multipath graph convolution. Then, an upsampling method using graph convolution expands, aggregates, and rearranges these primary point cloud features to obtain higher-level point cloud features. Finally, the higher-level point cloud features are reconstructed into three-dimensional space using a coordinate reconstructor to obtain a dense point cloud.

[0041] This embodiment provides a point cloud upsampling method that integrates self-attention and multipath graph convolution, such as... Figure 1 As shown, the method includes the following steps.

[0042] Step 101: Obtain the original point cloud of the target object.

[0043] Step 102: Input the original point cloud into the point cloud upsampling model to obtain a dense point cloud after upsampling. The point cloud upsampling model includes a feature extractor, an upsampling unit, and a coordinate reconstructor connected in sequence. The feature extractor is used to extract features from the original point cloud based on multiple residual multi-layer graph convolutions and self-attention modules to obtain primary point cloud features. The upsampling unit is used to expand, aggregate, and rearrange the primary point cloud features to obtain advanced point cloud features. The coordinate reconstructor is used to regress the advanced point cloud features to three-dimensional space to obtain the dense point cloud.

[0044] The point cloud upsampling model was obtained by training the point cloud upsampling network using a dataset. The dataset used was a publicly available synthetic dataset consisting of a collection of low-resolution point clouds.

[0045] The datasets used in this embodiment are the large-scale 3D datasets PU-GAN and PU1K, specifically created for point cloud upsampling tasks. The PU-GAN dataset contains 120 training samples and 27 test models, with most point cloud models being simple shapes. The PU1K dataset contains 1147 3D models, including 1020 training sample models and 127 test sample models. The training and testing models include 147 3D models from the PU-GAN dataset and over 1000 3D models from ShapeNetCore. The PU1K dataset covers a wider semantic range, with point cloud models including simple shapes, complex shapes, and even composite shapes, demonstrating its superiority as an experimental dataset in related fields.

[0046] During the training phase of the point cloud upsampling network, each 3D model was cropped into 50 patches, resulting in 51,000 patches from 1020 training samples. Each patch had 256 points as low-resolution input and 1024 points as the ground truth point cloud. During training, the point cloud upsampling network was trained using the low-resolution point cloud with 256 points as input and the ground truth point cloud with 1024 points as output. In the testing phase, the input point cloud consisted of 2048 points, and the ground truth point cloud consisted of 8192 points. The upsampling rate r was set to a fixed value of 4, the batch size was set to 64, and the epoch was set to 100. The upsampling performance of the point cloud upsampling network was evaluated using three metrics: chamfer distance (CD), Hausdorff distance (HD), and point-to-face distance (P2F). Lower metrics indicate better performance.

[0047] The feature extractor includes three fusion feature extraction modules (FEMs), each of which is connected in parallel with three multi-layer graph convolutions with residual design to fuse more layers of semantic information.

[0048] The feature extractor includes a first k-nearest neighbor algorithm module, a graph convolution module, a first fusion feature extraction module, a second fusion feature extraction module, and a third fusion feature extraction module connected in sequence. The first fusion feature extraction module, the second fusion feature extraction module, and the third fusion feature extraction module have the same structure. The outputs of the first fusion feature extraction module, the second fusion feature extraction module, and the third fusion feature extraction module are combined as the output of the feature extractor.

[0049] The first k-nearest neighbor algorithm module is used to obtain the nearest neighbor graph of the original point cloud. The graph convolution module is used to perform feature aggregation on each vertex and its nearest neighbors in the nearest neighbor graph to obtain the multidimensional geometric features of each vertex. The first fusion feature extraction module includes a bottleneck layer, a second k-nearest neighbor algorithm module, a third k-nearest neighbor algorithm module, a first residual multi-layer graph convolution, a second residual multi-layer graph convolution, a third residual multi-layer graph convolution, a self-attention module, a global max pooling and concatenation module. The output of the bottleneck layer is connected to the inputs of the second k-nearest neighbor algorithm module, the third k-nearest neighbor algorithm module, and the third residual multi-layer graph convolution, respectively. The output of the second k-nearest neighbor algorithm module is connected to the inputs of the third k-nearest neighbor algorithm module and the third residual multi-layer graph convolution. The input of a residual multi-layer graph convolution is connected; the output of the third k-nearest neighbor algorithm module is connected to the input of the second residual multi-layer graph convolution; the output of the third residual multi-layer graph convolution is connected to the input of the self-attention module; the input of the bottleneck layer is connected to the input of the global max pooling; the outputs of the first residual multi-layer graph convolution, the second residual multi-layer graph convolution, the self-attention module, and the global max pooling are all connected to the concatenation module; the output of the concatenation module is merged with the input of the bottleneck layer to obtain the output of the first fusion feature extraction module; the expansion rates of the k-nearest neighbor algorithm in the second k-nearest neighbor algorithm module and the third k-nearest neighbor algorithm module are different.

[0050] In the second k-nearest neighbor algorithm module, the expansion rate of the k-nearest neighbor algorithm is 2, and in the third k-nearest neighbor algorithm module, the expansion rate is 3. This invention uses multiple multi-layer graph convolutions with residual designs in parallel (first residual multi-layer graph convolution, second residual multi-layer graph convolution, and third residual multi-layer graph convolution), connecting multiple graph convolutions with different expansion rates to expand the receptive field of the multi-path graph convolution. Adding residual connections to the graph convolution deepens the training layers and better extracts high-level feature information.

[0051] The first residual multi-layer graph convolution, the second residual multi-layer graph convolution, and the third residual multi-layer graph convolution have the same structure.

[0052] like Figure 4As shown, the first residual multilayer graph convolution includes a fourth k-nearest neighbor algorithm module and multiple graph convolution operation layers connected in sequence. The output of the fourth k-nearest neighbor algorithm module is connected to the input of the first graph convolution operation layer. Each graph convolution operation layer includes a first multilayer perceptron and a second multilayer perceptron. Both the first and second multilayer perceptrons are multilayer perceptrons with 32 output channels.

[0053] Figure 3 and Figure 4 In Chinese, KNN stands for k-nearest neighbor algorithm.

[0054] When the graph convolution operation layer is the first graph convolution operation layer, the current graph convolution operation layer is used to: input the center point feature in the input features into the first multilayer perceptron, input the correlation between the neighbor point feature and the center point feature in the input features into the second multilayer perceptron, and then add the outputs of the first multilayer perceptron and the second multilayer perceptron and pass them through the ReLU activation function to obtain the output of the current graph convolution operation layer.

[0055] When the graph convolution operation layer is not the first graph convolution operation layer, the current graph convolution operation layer is used to: input the center point feature in the input features into the first multilayer perceptron, input the correlation between the neighbor point feature and the center point feature in the input features into the second multilayer perceptron, add the outputs of the first multilayer perceptron and the second multilayer perceptron and pass them through the ReLU activation function to obtain the first output result, and add the residual mapping of the output result of the previous graph convolution operation layer to the first output result to obtain the output of the current graph convolution operation layer.

[0056] The self-attention module includes a third multilayer perceptron, a fourth multilayer perceptron, a fifth multilayer perceptron, and a sixth multilayer perceptron; the third multilayer perceptron, the fourth multilayer perceptron, the fifth multilayer perceptron, and the sixth multilayer perceptron are all multilayer perceptrons with 32 output channels.

[0057] The self-attention module is used to process the input features through three multilayer perceptrons with 32 output channels to obtain the Q matrix, K matrix and V matrix. The transpose of the Q matrix and the K matrix are multiplied to obtain the attention weights. The attention weights are normalized by the softmax function, and the resulting matrix is ​​multiplied by the V matrix and then passed through the multilayer perceptron to obtain the output of the self-attention module.

[0058] like Figure 2 As shown, the original point cloud serves as the input point set. N represents the number of points. The k-nearest neighbor algorithm is used to search for the spatial nearest neighbors of each point. The 3D coordinates of the input point cloud are embedded into the latent feature space for subsequent feature learning. For each vertex, its features are mapped to the directed edges of its nearest neighbors, aggregated with the 10 points closest in Euclidean distance, and then passed through a non-linear activation function to obtain the features. F i Let i be the feature corresponding to the center point i, with dimension C.

[0059] The structures of the first fusion feature extraction module, the second fusion feature extraction module, and the third fusion feature extraction module are all as follows: Figure 3 As shown. N×C dimensional geometric feature F i First, a single 1×1 convolutional bottleneck layer is used to reduce computation. The compressed features are then input into three parallel Residual Multi-Graph Convolutional (RESGCN) blocks. These three RESGCN blocks have kernels of the same size. Before the two RESGCN blocks, the dilated k-nearest neighbor algorithm is used to find neighbors, where d represents different dilation rates. Let N(d) be the dilated neighborhood of vertex V. If n1 is the first nearest neighbor of the sequence, then... The features Fl are obtained after graph convolution with different dilation rates and are then input into three parallel multi-layer graph convolutional modules with residual design.

[0060] The global max pooling module of this invention extracts global context information together with multipath graph convolution for different receptive domains. For example... Figure 3 As shown in F c The aggregated features output by the fusion feature extraction module have an N×C shape. F p This represents the output feature of global max pooling. F c It aggregates the output features of three path graph convolutions and a global pooling layer.

[0061] The first residual multi-layer graph convolution, the second residual multi-layer graph convolution, and the third residual multi-layer graph convolution have the same structure, such as... Figure 4 As shown. F l Geometric feature F i The input features of RESGCN are compressed through a 1×1 convolutional bottleneck layer. The feature corresponding to neighbor point j, The characteristics of the center point i.

[0062] The first residual multilayer graph convolution includes multiple cascaded graph convolution operation layers. In each graph convolution operation layer, the centroid feature from the input features is input to a first multilayer perceptron, and the correlation between the neighbor point feature and the centroid feature is input to a second multilayer perceptron. The outputs of the first and second multilayer perceptrons are summed and then activated by the ReLU function, as shown below:

[0063] G l′ =σ{MLP(Δf) ij )+MLP(f l )};

[0064] Where σ represents the ReLU activation function, MLP represents the multilayer perceptron, and Δf ij Δf represents the correlation between the features of neighboring points and the features of the centroid. ij =(f j -f i ), f j f represents the feature corresponding to the neighboring points. i For the feature corresponding to the center point, f l G represents the input features of the current graph convolutional layer. l′ This represents the output of the convolution operation in the l′-th graph convolution operation layer.

[0065] When the graph convolution operation layer is not the first graph convolution operation layer, the residual mapping of the output result of the previous graph convolution operation layer is added to the first output result to obtain the output of the current graph convolution operation layer, which is expressed as:

[0066]

[0067] G l′ The output features of the l′-th graph convolutional layer in the residual multi-layer graph convolution are... For residual mapping, W l′-1 For fixed parameters, G l′-1 The output features of the previous graph convolutional layer are G, and the output of the last graph convolutional layer is G. l The number of graph convolution operation layers in each residual multi-layer graph convolution is 3. Figure 3 The feature G output by the third RESGCN path in the middle l After passing through three 32-output-channel MLPs, the Q matrix, K matrix, and V matrix are obtained. The embedded self-attention module is as follows: Figure 5 As shown, F l This represents the point cloud features output after passing through the self-attention module. The transposes of the Q and K matrices are multiplied, and the resulting matrix is ​​used as the attention weights. After softmax normalization, the resulting matrix is ​​multiplied by the V matrix, and finally, F is obtained through MLP operations.l F l and G l The feature dimensions are the same.

[0068] This invention aggregates the output of the feature extractor into feature F. c The input is compressed through a 1×1 bottleneck layer, and then the aggregated feature F is expanded using edge map convolution. c For N×rC, a periodic shuffle operator is used to rearrange the expanded aggregated features to obtain a high-dimensional point cloud feature F with dimension rN×C'. d C' represents the feature dimension after rearranging the aggregated features N×rC using the periodic shuffling operator. The coordinate reconstructor needs to obtain the high-dimensional point cloud features F d The three-dimensional coordinates are regressed in the mid-range. The three-dimensional coordinates of shape rN×3 are regressed through two sets of MLPs, and the resulting rN×3 is the high-resolution point cloud that needs to be upsampled r times.

[0069] This invention integrates a self-attention mechanism module into multi-path graph convolution to efficiently extract global semantic information and output point cloud features that fully utilize graph structure information, thereby facilitating the generation of high-quality point clouds and solving the problem of lack of correlation between regions.

[0070] The upsampler is used to compress the input features through a bottleneck layer, expand and aggregate the compressed features using edge map convolution, and rearrange the expanded and aggregated features using a shuffle operator to obtain high-level point cloud features.

[0071] The coordinate reconstructor comprises two multilayer perceptrons connected in sequence.

[0072] Traditional point cloud upsampling methods heavily rely on prior information, and are often ill-suited for handling noisy point cloud data with numerous features. Deep learning-based point cloud upsampling networks, however, are data-driven. Low-resolution point cloud data is used as input, and feature extraction maps it to a high-dimensional feature space. The k-nearest neighbor algorithm is used to construct a graph structure for each vertex and its neighborhood points. A fusion feature extraction module performs multi-scale expansion and fusion of the vertex graph structure. The obtained multi-scale point cloud features are then aggregated and arranged using a graph convolution upsampling function to output high-level point cloud features. Finally, a coordinate reconstructor outputs the desired high-resolution, dense point cloud.

[0073] The point cloud upsampling model of this invention first inputs an N×3 original point cloud into the network, which is then fed into a fusion feature extraction module (a first fusion feature extraction module, a second fusion feature extraction module, and a third fusion feature extraction module) with residual structure connections to extract the corresponding N×C dimensional point cloud features. Next, the extracted features are input into an upsampler to obtain high-level point cloud features of rN×C dimension. Finally, the high-level features are passed through a coordinate reconstruction unit to output a dense point cloud of dimension N×rC. Experiments on the synthetic datasets PU-GAN and PU1K show that the method generates dense point clouds with higher granularity and shapes closer to the true values ​​compared to mainstream methods.

[0074] This embodiment was conducted on the TensorFlow platform. The PU-MGS dataset was trained on an Intel Core i9-10900X CPU and an NVIDIA RTX 3090 GPU using the ADAM optimizer with a learning rate of 0.001 and a decay rate of 0.9. The experimental results were reported using an upsampling rate of ×4 (r=4) for easy comparison with the performance of other mainstream point cloud upsampling networks.

[0075] Figure 6 The qualitative performance of this invention on the PU-GAN dataset is shown. Four point cloud models were randomly selected and compared with their low-resolution input point clouds. Since the models included in this dataset are mostly simple-shaped objects with smooth edges, the high-resolution point clouds generated using the method of this invention have a lower cluster distribution and smoother edges that closely resemble the shape of real point clouds. Table 1 reports the quantitative evaluation metrics of 11 randomly selected models of this invention on the PU-GAN dataset; the average performance is above average among mainstream methods.

[0076] Table 2 reports the performance of this invention on the PU1K synthetic dataset compared to the benchmark. Significant improvements are achieved across all three evaluation metrics. On PU-GCN, currently the most competitive dataset overall, this invention achieves improvements of 7.863% in CD and 11.759% in HD. This is because this invention does not use a spatial refiner to adjust the position of the output point cloud to approximate the mesh surface. The best-performing P2F implementation, PU-GACNet, only shows a 3.2% improvement. Due to the inclusion of repulsion loss, this invention achieves a 13.285% improvement compared to PU-GCN. Figure 7 The results show a comparison of upsampling of qualitative point clouds generated on PU1K using the present invention and mainstream methods. Figure 7 Four different types of objects were randomly selected, each corresponding to a different shape level to demonstrate the diversity of the dataset. The selected objects include smooth-surfaced objects (welding torches) in the first row, ordinary-shaped objects (birds) in the second row, objects with complex details (motorcycles) in the third row, and composite objects (human statues) in the fourth row. Figure 7The mid-to-upsampled point cloud output and its magnified view show that the point cloud visualization generated by this invention retains high-quality, fine-grained details, and the degree of restoration is closer to the shape of the real point cloud. See details below. Figure 7 The second and third rows of close-ups (bird's feet, motorcycle handlebars) maintain the quality of the separated structures, rather than tending to merge the original separate structures. For example, the close-up of the objects in the first row (the handle of a welding torch). For the structure in the third row composed of multiple objects (a human statue), Figure 7 Column (a) shows the original point cloud, column (b) shows the result after upsampling using the PU-net point cloud upsampling network, column (c) shows the result after upsampling using the Graph Attention Convolution Network for Point Cloud Upsampling (PU-GACnet), column (d) shows the result after upsampling using the PU-GCN point cloud upsampling network, column (e) shows the result after upsampling using the point cloud upsampling network of this invention, and column (f) shows the real point cloud. The outliers in the output upsampled point cloud of this invention are significantly reduced compared to benchmark methods, and the generation of outliers is significantly reduced compared to advanced networks such as PU-GCN.

[0077] Example 2

[0078] This embodiment provides a point cloud upsampling system that integrates self-attention and multipath graph convolution, such as Figure 8 As shown, the system includes:

[0079] The original point cloud acquisition module 201 is used to acquire the original point cloud of the target object.

[0080] The original point cloud upsampling module 202 is used to input the original point cloud into the point cloud upsampling model to obtain a dense point cloud after upsampling. The point cloud upsampling model includes a feature extractor, an upsampling unit, and a coordinate reconstructor connected in sequence. The feature extractor is used to extract features from the original point cloud based on multiple residual multi-layer graph convolutions and self-attention modules to obtain primary point cloud features. The upsampling unit is used to expand, aggregate, and rearrange the primary point cloud features to obtain advanced point cloud features. The coordinate reconstructor is used to regress the advanced point cloud features to three-dimensional space to obtain the dense point cloud.

[0081] The point cloud upsampling model is obtained by training the point cloud upsampling network using a dataset.

[0082] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0083] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A point cloud upsampling method integrating self-attention and multipath graph convolution, characterized in that, include: Obtain the original point cloud of the target object; The original point cloud is input into the point cloud upsampling model to obtain the upsampled dense point cloud; The point cloud upsampling model includes a feature extractor, an upsampling unit, and a coordinate reconstructor connected in sequence. The feature extractor is used to extract features from the original point cloud based on multiple residual multi-layer graph convolutions and self-attention modules to obtain primary point cloud features. The upsampling unit is used to expand, aggregate, and rearrange the primary point cloud features to obtain advanced point cloud features. The coordinate reconstructor is used to regress the advanced point cloud features to three-dimensional space to obtain the dense point cloud. The point cloud upsampling model is obtained by training the point cloud upsampling network using a dataset; The feature extractor includes a first k-nearest neighbor algorithm module, a graph convolution module, a first fusion feature extraction module, a second fusion feature extraction module, and a third fusion feature extraction module connected in sequence; the first fusion feature extraction module, the second fusion feature extraction module, and the third fusion feature extraction module have the same structure, and the outputs of the first fusion feature extraction module, the second fusion feature extraction module, and the third fusion feature extraction module are combined as the output of the feature extractor; The first fusion feature extraction module includes a bottleneck layer, a second k-nearest neighbor algorithm module, a third k-nearest neighbor algorithm module, a first residual multi-layer graph convolution, a second residual multi-layer graph convolution, a third residual multi-layer graph convolution, a self-attention module, a global max pooling module, and a concatenation module. The output of the bottleneck layer is connected to the inputs of the second k-nearest neighbor algorithm module, the third k-nearest neighbor algorithm module, and the third residual multi-layer graph convolution, respectively. The output of the second k-nearest neighbor algorithm module is connected to the input of the first residual multi-layer graph convolution, the output of the third k-nearest neighbor algorithm module is connected to the input of the second residual multi-layer graph convolution, the output of the third residual multi-layer graph convolution is connected to the input of the self-attention module, and the input of the bottleneck layer is connected to the input of the global max pooling module. The outputs of the first residual multi-layer graph convolution, the second residual multi-layer graph convolution, the self-attention module, and the global max pooling module are all connected to the concatenation module. The output of the concatenation module is combined with the input of the bottleneck layer to obtain the output of the first fusion feature extraction module. The expansion rates of the k-nearest neighbor algorithms in the second k-nearest neighbor algorithm module and the third k-nearest neighbor algorithm module are different. The first residual multi-layer graph convolution, the second residual multi-layer graph convolution, and the third residual multi-layer graph convolution have the same structure; The first residual multi-layer graph convolution includes a fourth k-nearest neighbor algorithm module and multiple graph convolution operation layers connected in sequence. The output of the fourth k-nearest neighbor algorithm module is connected to the input of the first graph convolution operation layer. Each graph convolution operation layer includes a first multi-layer perceptron and a second multi-layer perceptron. Both the first and second multi-layer perceptrons are multi-layer perceptrons with 32 output channels. When the graph convolution operation layer is the first graph convolution operation layer, the current graph convolution operation layer is used to: input the center point feature in the input features into the first multilayer perceptron, input the correlation between the neighbor point feature and the center point feature in the input features into the second multilayer perceptron, and then add the outputs of the first multilayer perceptron and the second multilayer perceptron and pass them through the ReLU activation function to obtain the output of the current graph convolution operation layer; When the graph convolution operation layer is not the first graph convolution operation layer, the current graph convolution operation layer is used to: input the center point feature in the input features into the first multilayer perceptron, input the correlation between the neighbor point feature and the center point feature in the input features into the second multilayer perceptron, add the outputs of the first multilayer perceptron and the second multilayer perceptron and pass them through the ReLU activation function to obtain the first output result, and add the residual mapping of the output result of the previous graph convolution operation layer to the first output result to obtain the output of the current graph convolution operation layer.

2. The point cloud upsampling method fusing self-attention and multipath graph convolution as described in claim 1, characterized in that, The self-attention module includes a third multilayer perceptron, a fourth multilayer perceptron, a fifth multilayer perceptron, and a sixth multilayer perceptron; the third multilayer perceptron, the fourth multilayer perceptron, the fifth multilayer perceptron, and the sixth multilayer perceptron are all multilayer perceptrons with 32 output channels; The self-attention module is used to obtain the Q matrix by passing the input features through a third multilayer perceptron, the K matrix by passing the input features through a fourth multilayer perceptron, and the V matrix by passing the input features through a fifth multilayer perceptron. The transpose of the Q matrix and the K matrix are multiplied to obtain the attention weights. The matrix obtained after normalizing the attention weights through the softmax function is multiplied by the V matrix and then passed through a sixth multilayer perceptron to obtain the output of the self-attention module.

3. The point cloud upsampling method fusing self-attention and multipath graph convolution as described in claim 1, characterized in that, The expansion rate of the k-nearest neighbor algorithm in the second k-nearest neighbor algorithm module is 2, and the expansion rate of the k-nearest neighbor algorithm in the third k-nearest neighbor algorithm module is 3.

4. The point cloud upsampling method fusing self-attention and multipath graph convolution as described in claim 1, characterized in that, The upsampler is used to compress the input features through a bottleneck layer, expand and aggregate the compressed features using edge map convolution, and rearrange the expanded and aggregated features using a shuffle operator to obtain high-level point cloud features.

5. The point cloud upsampling method fusing self-attention and multipath graph convolution as described in claim 1, characterized in that, The coordinate reconstructor includes a seventh multilayer perceptron and an eighth multilayer perceptron connected in sequence.

6. A point cloud upsampling system integrating self-attention and multipath graph convolution, characterized in that, include: The original point cloud acquisition module is used to acquire the original point cloud of the target object; The original point cloud upsampling module is used to input the original point cloud into the point cloud upsampling model to obtain a dense point cloud after upsampling. The point cloud upsampling model includes a feature extractor, an upsampling unit, and a coordinate reconstructor connected in sequence. The feature extractor is used to extract features from the original point cloud based on multiple residual multi-layer graph convolutions and self-attention modules to obtain primary point cloud features. The upsampling unit is used to expand, aggregate, and rearrange the primary point cloud features to obtain advanced point cloud features. The coordinate reconstructor is used to regress the advanced point cloud features to three-dimensional space to obtain the dense point cloud. The point cloud upsampling model is obtained by training the point cloud upsampling network using a dataset; The feature extractor includes a first k-nearest neighbor algorithm module, a graph convolution module, a first fusion feature extraction module, a second fusion feature extraction module, and a third fusion feature extraction module connected in sequence; the first fusion feature extraction module, the second fusion feature extraction module, and the third fusion feature extraction module have the same structure, and the outputs of the first fusion feature extraction module, the second fusion feature extraction module, and the third fusion feature extraction module are combined as the output of the feature extractor; The first fusion feature extraction module includes a bottleneck layer, a second k-nearest neighbor algorithm module, a third k-nearest neighbor algorithm module, a first residual multi-layer graph convolution, a second residual multi-layer graph convolution, a third residual multi-layer graph convolution, a self-attention module, a global max pooling module, and a concatenation module. The output of the bottleneck layer is connected to the inputs of the second k-nearest neighbor algorithm module, the third k-nearest neighbor algorithm module, and the third residual multi-layer graph convolution, respectively. The output of the second k-nearest neighbor algorithm module is connected to the input of the first residual multi-layer graph convolution, the output of the third k-nearest neighbor algorithm module is connected to the input of the second residual multi-layer graph convolution, the output of the third residual multi-layer graph convolution is connected to the input of the self-attention module, and the input of the bottleneck layer is connected to the input of the global max pooling module. The outputs of the first residual multi-layer graph convolution, the second residual multi-layer graph convolution, the self-attention module, and the global max pooling module are all connected to the concatenation module. The output of the concatenation module is combined with the input of the bottleneck layer to obtain the output of the first fusion feature extraction module. The expansion rates of the k-nearest neighbor algorithms in the second k-nearest neighbor algorithm module and the third k-nearest neighbor algorithm module are different. The first residual multi-layer graph convolution, the second residual multi-layer graph convolution, and the third residual multi-layer graph convolution have the same structure; The first residual multi-layer graph convolution includes a fourth k-nearest neighbor algorithm module and multiple graph convolution operation layers connected in sequence. The output of the fourth k-nearest neighbor algorithm module is connected to the input of the first graph convolution operation layer. Each graph convolution operation layer includes a first multi-layer perceptron and a second multi-layer perceptron. Both the first and second multi-layer perceptrons are multi-layer perceptrons with 32 output channels. When the graph convolution operation layer is the first graph convolution operation layer, the current graph convolution operation layer is used to: input the center point feature in the input features into the first multilayer perceptron, input the correlation between the neighbor point feature and the center point feature in the input features into the second multilayer perceptron, and then add the outputs of the first multilayer perceptron and the second multilayer perceptron and pass them through the ReLU activation function to obtain the output of the current graph convolution operation layer; When the graph convolution operation layer is not the first graph convolution operation layer, the current graph convolution operation layer is used to: input the center point feature in the input features into the first multilayer perceptron, input the correlation between the neighbor point feature and the center point feature in the input features into the second multilayer perceptron, add the outputs of the first multilayer perceptron and the second multilayer perceptron and pass them through the ReLU activation function to obtain the first output result, and add the residual mapping of the output result of the previous graph convolution operation layer to the first output result to obtain the output of the current graph convolution operation layer.

Images