A three-dimensional point cloud completion method based on a bidirectional pyramid structure of a mixed attention mechanism

Abstract

The application discloses a three-dimensional point cloud completion method based on a bidirectional pyramid structure of a mixed attention mechanism, comprising the following steps: acquiring a three-dimensional point cloud of a three-dimensional object; inputting the three-dimensional point cloud into a point cloud encoder; obtaining an output feature vector after processing the three-dimensional point cloud through the point cloud encoder; the point cloud encoder comprises a multi-layer perception machine containing an attention mechanism and a full connection layer; inputting the output feature vector F1 into a point cloud decoder; obtaining an output vector after processing through the point cloud decoder; converting the output vector into three-dimensional shapes of different scales respectively; and modifying local geometric details of the three-dimensional shapes respectively to obtain a complete three-dimensional object shape, wherein the bidirectional attention pyramid structure is based on a traditional pyramid structure and additionally comprises a bottom-up path aggregation network, cross-level connection and a mixed attention module. The method can generate missing point clouds well, improves the accuracy, and has better performance under fewer network parameters.

Description

Technical Field

[0001] This invention relates to the field of 3D point cloud completion, and more particularly to a 3D point cloud completion method based on a bidirectional pyramid structure with a hybrid attention mechanism. Background Technology

[0002] 3D reconstruction technology can automatically construct various 3D scenes, thereby reducing manual design costs and shortening design cycles, creating significant economic benefits for society. Point clouds are the most representative 3D data of the digital age, representing the most accurate mapping of the real world to the digital world. Due to its rich spatial details and ease of acquisition, point cloud data has become the preferred representation for understanding related applications in many scenarios.

[0003] In real-world scanning environments, point clouds acquired directly from 3D sensors are often sparse and incomplete due to limitations in sensor resolution or occlusion by objects. The incompleteness of the point cloud model can severely impact subsequent applications. Therefore, completing missing point cloud data has become crucial for 3D model research and applications. Point cloud completion technology can be applied to the following areas:

[0004] (I) Smart City: Using laser scanning and oblique photogrammetry to measure the three-dimensional point cloud data of the city, the point cloud completion technology can visualize and quantify the three-dimensional model of the city more comprehensively, more precisely, more in real time, and more realistically.

[0005] (II) Promotion of cultural relics: Using 3D scanning equipment to collect 3D color point cloud data of ceramics and other works, and obtaining complete and optimized color point cloud information through point cloud completion and other methods, so as to carry out digital promotion and publicity.

[0006] (III) Medical image analysis: Tasks such as completion, classification, segmentation, registration, and target detection based on medical point cloud data can assist doctors in making more accurate diagnoses and treatments. It has important application value in clinical medicine, as well as in the design of personalized medical devices and 3D printing.

[0007] (iv) Robotics & Industrial Automation: After completing, clustering and segmenting 3D point clouds, the results are fused into SLAM (Simultaneous Localization and Mapping) for intelligent navigation.

[0008] Thanks to the rapid development of 3D acquisition technology, 3D sensors, including various types of 3D scanners, LiDAR, and RGB-D cameras, are rapidly becoming more widespread at a lower cost, used to display richer geometric, shape, and scale information. Currently, 3D data is widely used in multiple fields, including but not limited to autonomous driving, robotics, remote sensing, and medicine. In terms of representation, 3D data also exists in various formats such as depth images, point clouds, meshes, and volumetric meshes. Among these, point clouds, as a commonly used format, retain the original geometric information in three-dimensional space without any discretization. Therefore, point cloud data is widely used.

[0009] However, during actual data acquisition, due to factors such as line-of-sight obstruction or improper operation by technicians, the acquired point cloud models may have incomplete shapes. Incomplete point cloud models can severely impact subsequent applications. Therefore, completing missing point cloud data has become crucial for 3D reconstruction and applications such as 3D data recognition and detection.

[0010] In related point cloud completion technologies, the model structure is relatively simple, the extracted point cloud data features are not rich enough and the distinguishability is insufficient. Therefore, the accuracy of point cloud completion for different categories is not high and the detailed information is not complete enough, which cannot meet the needs of high-precision point cloud completion and subsequent applications such as 3D data recognition and detection. Summary of the Invention

[0011] This invention provides a three-dimensional point cloud completion method based on a hybrid attention mechanism for a bidirectional pyramid structure, in order to overcome the above-mentioned technical problems.

[0012] A 3D point cloud completion method based on a hybrid attention mechanism and a bidirectional pyramid structure includes,

[0013] Step 1: Obtain the 3D point cloud of the 3D object;

[0014] Step 2: Input the 3D point cloud into the point cloud encoder. After processing by the point cloud encoder, the 3D point cloud yields an output feature vector.

[0015] The point cloud encoder includes a multilayer perceptron with an attention mechanism, a first fully connected layer, and a second fully connected layer. The multilayer perceptron with an attention mechanism includes an M-layer perceptron, an M-1-layer pooling layer, and a channel attention module. The i-th pooling layer obtains the global features output by the (i+1)-th layer perceptron, where M-1 ≥ i > 0. The M-1 global features are concatenated into a feature vector through a concatenation operation, which involves horizontally superimposing the various global features. The channel attention module extracts feature information from the feature vector and constructs a multi-level feature vector based on the extracted feature information. The second fully connected layer converts the multi-level feature vector into an output feature vector F1.

[0016] Step 3: Input the output feature vector F1 into the point cloud decoder. The point cloud decoder includes a third fully connected layer, a fourth fully connected layer, a fifth fully connected layer, a hybrid attention module, and a bidirectional attention pyramid structure. The output feature vector F1 is processed through the third fully connected layer to obtain feature vector F2, and through the fourth and fifth fully connected layers to obtain feature vector F3. The 3D point cloud is then connected to the output feature vectors F1, F2, and F3, respectively, and represented as Z1, Z2, and Z3. The hybrid attention module then fuses Z1, Z2, and Z3 to obtain the output feature vectors F'1, F'2, and F'3. Finally, the output feature vectors F'1, F'2, and F'3 are input into the bidirectional attention pyramid structure for processing to obtain the output vector. The output vectors are respectively The model is converted into 3D shapes at different scales, and the local geometric details of these 3D shapes are corrected to obtain the complete 3D object shape. The bidirectional attention pyramid structure includes a sixth fully connected layer, a seventh fully connected layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a first hybrid attention module, a second hybrid attention module, a third hybrid attention module, and a fourth hybrid attention module. The sixth fully connected layer extracts the sixth connection output feature of feature vector F'1, the seventh fully connected layer extracts the seventh connection output feature of feature vector F'2, the first convolutional layer extracts the first convolutional output feature of feature vector F'1, the second convolutional layer obtains the second convolutional output feature based on feature vector F'2 and the sixth connection output feature, the third convolutional layer obtains the third convolutional output feature based on feature vector F'3 and the seventh connection output feature, the first hybrid attention module obtains the first hybrid output feature based on the second and first convolutional output features, and the second hybrid attention module obtains the output vector based on the third, second, and first convolutional output features. The third hybrid attention module is used to determine the first hybrid output features and output vector. Obtain the output vector The fourth hybrid attention module is used to output features and output vectors based on the first convolution. Output vector Obtain the output vector

[0017] Preferably, the multilayer perceptron with an attention mechanism includes a first-layer perceptron, a second-layer perceptron, a third-layer perceptron, a fourth-layer perceptron, a fifth-layer perceptron, a first-layer pooling layer, a second-layer pooling layer, a third-layer pooling layer, a fourth-layer pooling layer, and a channel attention module. The first-layer perceptron has 64 neurons, the second-layer perceptron has 128 neurons, the third-layer perceptron has 256 neurons, the fourth-layer perceptron has 512 neurons, and the fifth-layer perceptron has 1024 neurons. A pooling layer is used to obtain the global features output by the second perceptron layer, with a dimension of 128. The second pooling layer is used to obtain the global features output by the third perceptron layer, with a dimension of 256. The third pooling layer is used to obtain the global features output by the fourth perceptron layer, with a dimension of 512. The fourth pooling layer is used to obtain the global features output by the fifth perceptron layer, with a dimension of 1024. The obtained global features are concatenated into a 1920-dimensional feature vector through a concatenation operation. The channel attention module is then used to transform the feature vector into a multi-level feature vector.

[0018] Preferably, the step of fusing Z1, Z2, and Z3 according to the hybrid attention module to obtain output feature vectors F′1, F′2, and F′3 includes,

[0019] S31. The vector Z1 is processed by the channel attention module to obtain the output vector CA. The channel attention module includes a channel average pooling layer, two fully connected layers and a ReLU activation function. The vector CA is obtained according to formula (1).

[0020] CA=W2×RELU(W1×pool(Z1)) (1)

[0021] CA is the output vector of the channel attention module, Z1 has a size of C×L, pool is the channel average pooling layer, W1 and W2 are the parameters of the two fully connected layers, with sizes of C / 16×C and C×C / 16 respectively, and ReLU is the activation function.

[0022] S32. The vector Z1 is processed by a point attention module to obtain the output vector PA. The point attention module includes a point average pooling layer, two fully connected layers, and a ReLU activation function. The vector PA is obtained according to formula (2).

[0023] PA=W′2×RELU(W′1×pool′(Z1)) (2)

[0024] Where PA is the output vector of the point attention module, pool' is the point average pooling layer, and W'1 and W'2 are the parameters of the two fully connected layers, with sizes of L / 16×L and L×L / 16, respectively.

[0025] S33. After processing Z1 according to formula (3), the output feature vector F'1 is obtained.

[0026] F′1=σ(CA×PA)×Z1+Z1 (3)

[0027] Where σ is the Sigmoid function,

[0028] S34. Perform S31-S33 on Z2 and Z3 respectively to obtain the output feature vectors F'2 and F'3.

[0029] Preferably, the multi-level completion loss function of the bidirectional attention pyramid structure is Equation (4).

[0030]

[0031] in, For the output vector, F gt F' gt 、F″ gt For real point clouds of different scales, α is a hyperparameter, calculated according to formula (5). With F gt The chamfer distance between them is calculated according to formula (6). With F' gt The chamfer distance between them is calculated according to formula (7). With F″ gt chamfer distance between them

[0032]

[0033]

[0034]

[0035] d CD The chamfer distance is used to measure the average closest distance between the output vector and the real point cloud.

[0036] A 3D point cloud completion device based on a bidirectional pyramid structure using a hybrid attention mechanism includes,

[0037] The acquisition module is used to acquire the 3D point cloud of a 3D object;

[0038] The feature extraction module is used to input the 3D point cloud into the point cloud encoder. After processing by the point cloud encoder, the 3D point cloud is used to obtain the output feature vector. The point cloud encoder includes a multilayer perceptron with an attention mechanism, a first fully connected layer, and a second fully connected layer. The multilayer perceptron with an attention mechanism includes an M-layer perceptron, an M-1-layer pooling layer, and a channel attention module. The i-th pooling layer obtains the global features output by the (i+1)-th perceptron, where M-1 ≥ i > 0. The obtained M-1 global features are concatenated into a feature vector through a concatenation operation. The concatenation is the horizontal superposition of the various global features. The channel attention module extracts feature information from the feature vector. Based on the extracted feature information, a multi-level feature vector is constructed. The second fully connected layer converts the multi-level feature vector into an output feature vector F1.

[0039] The prediction module inputs the output feature vector F1 into the point cloud decoder. The point cloud decoder includes a third fully connected layer, a fourth fully connected layer, a fifth fully connected layer, a hybrid attention module, and a bidirectional attention pyramid structure. The output feature vector F1 is processed through the third fully connected layer to obtain feature vector F2, and through the fourth and fifth fully connected layers to obtain feature vector F3. The 3D point cloud is then concatenated with the output feature vectors F1, F2, and F3, and denoted as Z1, Z2, and Z3, respectively. The hybrid attention module fuses Z1, Z2, and Z3 to obtain the output feature vectors F'1, F'2, and F'3. Finally, the output feature vectors F'1, F'2, and F'3 are input into the bidirectional attention pyramid structure for processing to obtain the output vector. The output vectors are respectively The model is converted into 3D shapes at different scales, and the local geometric details of these 3D shapes are corrected to obtain the complete 3D object shape. The bidirectional attention pyramid structure includes a sixth fully connected layer, a seventh fully connected layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a first hybrid attention module, a second hybrid attention module, a third hybrid attention module, and a fourth hybrid attention module. The sixth fully connected layer extracts the sixth connection output feature of feature vector F'1, the seventh fully connected layer extracts the seventh connection output feature of feature vector F'2, the first convolutional layer extracts the first convolutional output feature of feature vector F'1, the second convolutional layer obtains the second convolutional output feature based on feature vector F'2 and the sixth connection output feature, the third convolutional layer obtains the third convolutional output feature based on feature vector F'3 and the seventh connection output feature, the first hybrid attention module obtains the first hybrid output feature based on the second and first convolutional output features, and the second hybrid attention module obtains the output vector based on the third, second, and first convolutional output features. The third hybrid attention module is used to determine the first hybrid output features and output vector. Obtain the output vector The fourth hybrid attention module is used to output features and output vectors based on the first convolution. Output vector Obtain the output vector

[0040] This invention provides a 3D point cloud completion method based on a bidirectional pyramid structure with a hybrid attention mechanism. The method generates a point cloud structure with more details through a channel attention module, and uses a hybrid attention module to enable the convolutional neural network to extract features that are helpful for inferring missing point cloud regions from the incomplete point cloud structure, thereby obtaining more effective geometric information. The bidirectional attention pyramid structure can more accurately predict the missing structure of the point cloud, generate missing point clouds better, and achieve better accuracy. At the same time, it has better performance with fewer network parameters. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of the method of the present invention;

[0043] Figure 2 This invention relates to a multilayer perceptron structure containing an attention mechanism;

[0044] Figure 3 This is a structural diagram of the hybrid attention module of the present invention;

[0045] Figure 4 This is the qualitative result of the ShapeNet-55 of this invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Figure 1 This is a flowchart of the method of the present invention, as shown below. Figure 1As shown, the method in this embodiment may include:

[0048] Step 1: Obtain the 3D point cloud of the 3D object;

[0049] Step 2: Input the 3D point cloud into the point cloud encoder. After processing by the point cloud encoder, the 3D point cloud yields the output feature vector.

[0050] The point cloud encoder includes a multilayer perceptron with an attention mechanism, a first fully connected layer, and a second fully connected layer. The multilayer perceptron with an attention mechanism includes an M-layer perceptron, an M-1-layer pooling layer, and a channel attention module. The global features output by the (i+1)-th layer perceptron are obtained through the i-th pooling layer, where M-1 ≥ i > 0. The obtained M-1 global features are concatenated into a feature vector through a concatenation operation, which involves horizontally superimposing the various global features. The feature information extracted from the feature vector by the channel attention module is used to construct a multi-level feature vector based on the extracted feature information. The multi-level feature vector is then converted into an output feature vector F1 through the second fully connected layer.

[0051] The multilayer perceptron with an attention mechanism includes a first-layer perceptron, a second-layer perceptron, a third-layer perceptron, a fourth-layer perceptron, a fifth-layer perceptron, a first-layer pooling layer, a second-layer pooling layer, a third-layer pooling layer, a fourth-layer pooling layer, and a channel attention module. The first-layer perceptron has 64 neurons, the second-layer perceptron has 128 neurons, the third-layer perceptron has 256 neurons, the fourth-layer perceptron has 512 neurons, and the fifth-layer perceptron has 1024 neurons. The first pooling layer... The first layer acquires the global features from the output of the second perceptron, with a dimension of 128. The second pooling layer acquires the global features from the output of the third perceptron, with a dimension of 256. The third pooling layer acquires the global features from the output of the fourth perceptron, with a dimension of 512. The fourth pooling layer acquires the global features from the output of the fifth perceptron, with a dimension of 1024. These acquired global features are then concatenated horizontally through a concatenation operation to form a 1920-dimensional feature vector. A channel attention module then transforms this feature vector into a multi-level feature vector. The different dimensions of the multi-layer perceptron can extract low, medium, and high-level features. Each level of feature contains rich point cloud information. The structure of a multi-layer perceptron with an attention mechanism is as follows: Figure 2 As shown.

[0052] Step 3: Input the output feature vector F1 into the point cloud decoder. The point cloud decoder includes a third fully connected layer, a fourth fully connected layer, a fifth fully connected layer, a hybrid attention module, and a bidirectional attention pyramid structure. The output feature vector F1 is passed through a fully connected layer to obtain feature vector F2, and then through two fully connected layers to obtain feature vector F3. The size of F1 is 1024×3, the size of F2 is 512×3, and the size of F3 is 256×3. Skip connections are used to connect the 3D point cloud with the output feature vectors F1, F2, and F3, respectively, and represent them as Z1, Z2, and Z3. This skip connection not only provides long-range information compensation, making the original incomplete point cloud geometric information still usable in the decoder structure, but also allows residual learning to promote gradient backpropagation, which is beneficial to network convergence.

[0053] The input features Z1, Z2, and Z3 are fused using a hybrid attention module to obtain output feature vectors F′1, F′2, and F′3, respectively. This hybrid attention module is used to extract features beneficial for inferring the missing point cloud region. Learnable weights are introduced into the hybrid attention module to understand the importance of different features. Different input features typically contribute unequally to the output. Features beneficial for generating the missing point cloud structure have larger weights; therefore, a hybrid attention module incorporating channel attention and point attention is used to learn the weights between features. The structure is as follows: Figure 3 As shown.

[0054] The step of fusing Z1, Z2, and Z3 according to the hybrid attention module to obtain output feature vectors F′1, F′2, and F′3 includes...

[0055] S31. The vector Z1 is processed by the channel attention module to obtain the output vector CA. The channel attention module includes a channel average pooling layer, two fully connected layers and a ReLU activation function. The vector CA is obtained according to formula (1).

[0056] CA=W2×RELU(W1×pool(Z1)) (1)

[0057] CA is the output vector of the channel attention module, Z1 has a size of C×L, pool is the channel average pooling layer, which produces a global feature vector of size C×1, W1 and W2 are the parameters of the two fully connected layers, with sizes of C / 16×C and C×C / 16 respectively, and ReLU is the activation function.

[0058] S32. The vector Z1 is processed by a point attention module to obtain the output vector PA. The point attention module includes a point average pooling layer, two fully connected layers, and a ReLU activation function. The vector PA is obtained according to formula (2).

[0059] PA=W′2×RELU(W′1×pool′(Z1)) (2)

[0060] Where PA is the output vector of the point attention module, pool' is the point average pooling layer, which applies a point average pooling layer to obtain a global feature vector of size 1×L, and W′1 and W′2 are the parameters of the two fully connected layers, with sizes of L / 16×L and L×L / 16, respectively.

[0061] S33. After processing Z1 according to formula (3), the output feature vector F'1 is obtained.

[0062] F′1=σ(CA×PA)×Z1+Z1 (3)

[0063] Here, σ is the Sigmoid function, which normalizes the attention weights to the [0,1] interval. Through these operations, the convolutional neural network can extract features beneficial for inferring missing point cloud regions from incomplete point cloud structures, thus obtaining more effective geometric information.

[0064] S34. Perform S31-S33 on Z2 and Z3 respectively to obtain the output feature vectors F'2 and F'3.

[0065] This embodiment, based on the traditional pyramid structure, not only adds a bottom-up path aggregation network but also cross-level connections. By fully fusing high-level features with rich global information and low-level features with geometric details, the network can more accurately predict missing structures in point clouds. Simultaneously, a hybrid attention module is introduced to learn the importance of features at different levels, selectively fusing features to improve point cloud completion accuracy.

[0066] The output feature vectors F'1, F'2, and F'3 are input into the bidirectional attention pyramid structure for processing to obtain the output vector. This refers to the missing point cloud structures predicted by the network at different scales, with sizes of 512×3, 128×3, and 64×3, respectively. The output vectors are then... The shapes are converted into three-dimensional shapes of different scales, and the local geometric details of the three-dimensional shapes are corrected to obtain the complete three-dimensional object shape. The bidirectional attention pyramid structure adds a bottom-up path aggregation network, cross-level connections and hybrid attention modules to the traditional pyramid structure.

[0067] The bidirectional attention pyramid structure includes a sixth fully connected layer, a seventh fully connected layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a first hybrid attention module, a second hybrid attention module, a third hybrid attention module, and a fourth hybrid attention module. The sixth fully connected layer extracts the sixth connection output feature of feature vector F'1, the seventh fully connected layer extracts the seventh connection output feature of feature vector F'2, the first convolutional layer extracts the first convolutional output feature of feature vector F'1, the second convolutional layer obtains the second convolutional output feature based on feature vector F'2 and the sixth connection output feature, the third convolutional layer obtains the third convolutional output feature based on feature vector F'3 and the seventh connection output feature, the first hybrid attention module obtains the first hybrid output feature based on the second and first convolutional output features, and the second hybrid attention module obtains the output vector based on the third, second, and first convolutional output features. The third hybrid attention module is used to determine the first hybrid output features and output vector. Obtain the output vector The fourth hybrid attention module is used to output features and output vectors based on the first convolution. Output vector Obtain the output vector

[0068] The multi-level completion loss function of the bidirectional attention pyramid structure is Equation (4).

[0069]

[0070] in, For the output vector, F gt F' gt F' g ' t The data consists of real point clouds of different scales, with sizes of 512×3, 128×3, and 64×3, respectively. α is a hyperparameter, calculated according to formula (5). With F gt The chamfer distance between them is calculated according to formula (6). With F' gt The chamfer distance between them is calculated according to formula (7). and chamfer distance between them

[0071]

[0072]

[0073]

[0074] d CDThe chamfer distance is used to measure the average closest distance between the output vector and the real point cloud.

[0075] A 3D point cloud completion device based on a bidirectional pyramid structure using a hybrid attention mechanism includes,

[0076] The acquisition module is used to acquire the 3D point cloud of a 3D object;

[0077] The feature extraction module is used to input the 3D point cloud into the point cloud encoder. After processing by the point cloud encoder, the 3D point cloud is used to obtain the output feature vector. The point cloud encoder includes a multilayer perceptron with an attention mechanism, a first fully connected layer, and a second fully connected layer. The multilayer perceptron with an attention mechanism includes an M-layer perceptron, an M-1-layer pooling layer, and a channel attention module. The i-th pooling layer obtains the global features output by the (i+1)-th perceptron, where M-1 ≥ i > 0. The obtained M-1 global features are concatenated into a feature vector through a concatenation operation. The concatenation is the horizontal superposition of the various global features. The channel attention module extracts feature information from the feature vector. Based on the extracted feature information, a multi-level feature vector is constructed. The second fully connected layer converts the multi-level feature vector into an output feature vector F1.

[0078] The prediction module inputs the output feature vector F1 into the point cloud decoder. The point cloud decoder includes a third fully connected layer, a fourth fully connected layer, a fifth fully connected layer, a hybrid attention module, and a bidirectional attention pyramid structure. The output feature vector F1 is processed through the third fully connected layer to obtain feature vector F2, and through the fourth and fifth fully connected layers to obtain feature vector F3. The 3D point cloud is then concatenated with the output feature vectors F1, F2, and F3, and denoted as Z1, Z2, and Z3, respectively. The hybrid attention module fuses Z1, Z2, and Z3 to obtain the output feature vectors F'1, F'2, and F'3. Finally, the output feature vectors F'1, F'2, and F'3 are input into the bidirectional attention pyramid structure for processing to obtain the output vector. The output vectors are respectively The model is converted into 3D shapes at different scales, and the local geometric details of these 3D shapes are corrected to obtain the complete 3D object shape. The bidirectional attention pyramid structure includes a sixth fully connected layer, a seventh fully connected layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a first hybrid attention module, a second hybrid attention module, a third hybrid attention module, and a fourth hybrid attention module. The sixth fully connected layer extracts the sixth connection output feature of feature vector F'1, the seventh fully connected layer extracts the seventh connection output feature of feature vector F'2, the first convolutional layer extracts the first convolutional output feature of feature vector F'1, the second convolutional layer obtains the second convolutional output feature based on feature vector F'2 and the sixth connection output feature, the third convolutional layer obtains the third convolutional output feature based on feature vector F'3 and the seventh connection output feature, the first hybrid attention module obtains the first hybrid output feature based on the second and first convolutional output features, and the second hybrid attention module obtains the output vector based on the third, second, and first convolutional output features. The third hybrid attention module is used to determine the first hybrid output features and output vector. Obtain the output vector The fourth hybrid attention module is used to output features and output vectors based on the first convolution. Output vector Obtain the output vector

[0079] The prediction F-values ​​of the method in this embodiment and other methods were compared on two public datasets, ShapeNet-13 and ShapeNet-55. Pred →Real F GT And the real F GT →Predict F Pred Two types of chamfer distances. The calculation formulas are shown below.

[0080]

[0081]

[0082] Where, d CD (F Pred ,F GT ) represents F Pred With F GT chamfer distance, d CD (F GT ,F Pred ) represents F GT With F Pred The chamfer distance.

[0083] (I) Results on the ShapeNet-13 dataset. The proposed model in this embodiment outperforms other methods in terms of chamfer distance across 13 categories, compared to other methods in the previous embodiment. This demonstrates that the proposed method can generate missing point clouds more effectively with better accuracy.

[0084] Table 1 compares the results of this embodiment with other methods on the ShapeNet-13 dataset.

[0085]

[0086] Paired numbers are chamfer distance × 10 3 (The lower the distance value, the better the completion result).

[0087] (II) Results on the ShapeNet-55 Dataset. The results of the model in this embodiment and PF-Net on the ShapeNet-55 dataset are compared, as shown in Table 2. The dataset is categorized into easy, medium, and difficult levels based on the number of 55 categories. Categories with more than 2500 data points are defined as easy. Categories with fewer than 80 data points are defined as difficult. Categories in between are considered medium. The data in the table shows that the chamfer distance for all three levels and the average chamfer distance for the 55 categories in this embodiment are smaller than those of the PF-Net method. Furthermore, the number of trainable parameters for PF-Net and this embodiment is compared; this embodiment exhibits superior performance with fewer network parameters.

[0088] Table 2 Comparison results of this embodiment and PF-Net on the ShapeNet-55 dataset.

[0089]

[0090] At the same time, it presents qualitative results at three levels in the Shapenet-55 dataset, such as... Figure 4 As shown, three examples are presented for each level. This embodiment can achieve missing point cloud completion with higher accuracy and more detail for various categories.

[0091] Overall beneficial effects:

[0092] This invention provides a 3D point cloud completion method based on a bidirectional pyramid structure with a hybrid attention mechanism. The method generates a point cloud structure with more details through a channel attention module, and uses a hybrid attention module to enable the convolutional neural network to extract features that are helpful for inferring missing point cloud regions from the incomplete point cloud structure, thereby obtaining more effective geometric information. The bidirectional attention pyramid structure can more accurately predict the missing structure of the point cloud, generate missing point clouds better, and achieve better accuracy. At the same time, it has better performance with fewer network parameters.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional point cloud completion method based on a bidirectional pyramid structure of a mixed attention mechanism, characterized in that, Comprising, Step one, obtaining a three-dimensional point cloud of a three-dimensional object; Step two, inputting the three-dimensional point cloud into a point cloud encoder, and obtaining an output feature vector after processing the three-dimensional point cloud by the point cloud encoder, The point cloud encoder comprises a multi-layer perception containing an attention mechanism, a first fully connected layer, and a second fully connected layer, the multi-layer perception containing an attention mechanism comprises M layers of perception, M-1 layers of pooling layer, and a channel attention module, global features output by an i+1 layer of perception are acquired through an i layer of pooling layer respectively, M-1≥i>0, M-1 global features acquired are spliced into a feature vector through a connection operation, the splicing is transversely superimposing each global feature, feature information is extracted from the feature vector through the channel attention module, a multi-level feature vector is constructed according to the extracted feature information, and the multi-level feature vector is converted into an output feature vector through the second fully connected layer ; Step three, output feature vector Input point cloud decoder, the point cloud decoder includes third fully connected layer, fourth fully connected layer, fifth fully connected layer, hybrid attention module, bidirectional attention pyramid structure, output feature vector Obtain feature vector through third fully connected layer Output feature vector Obtain feature vector through fourth fully connected layer and fifth fully connected layer Respectively connect three-dimensional point cloud with output feature vector Feature vector Feature vector Respectively expressed as , , Respectively according to hybrid attention module, the output feature vector , , Fusion obtains output feature vector , , The output feature vector , , Input to bidirectional attention pyramid structure for processing to obtain output vector Respectively convert output vector Into different scale three-dimensional shape, respectively correct the local geometric details of the three-dimensional shape to obtain complete three-dimensional object shape, wherein the bidirectional attention pyramid structure includes sixth fully connected layer, seventh fully connected layer, first convolution layer, second convolution layer, third convolution layer, first hybrid attention module, second hybrid attention module, third hybrid attention module and fourth hybrid attention module, the sixth fully connected layer is used for extracting the sixth connection output feature of feature vector The seventh fully connected layer is used for extracting the seventh connection output feature of feature vector The first convolution layer is used for extracting the first convolution output feature of feature vector The second convolution layer is used for obtaining second convolution output feature according to feature vector The sixth connection output feature, the third convolution layer is used for obtaining third convolution output feature according to feature vector The seventh connection output feature, the first hybrid attention module is used for obtaining first hybrid output feature according to second convolution output feature and first convolution output feature, the second hybrid attention module is used for obtaining output vector According to third convolution output feature, second convolution output feature and first convolution output feature, the third hybrid attention module is used for obtaining output vector According to first hybrid output feature, output vector , the fourth mixed attention module is used to obtain an output vector , the output vector , the output vector ; and a multi-level completion loss function of the bidirectional attention pyramid structure is formula (4), (4) in, For the output vector, , , For real point clouds at different scales, For hyperparameters, calculate according to formula (5). and The chamfer distance between them is calculated according to formula (6). and The chamfer distance between them is calculated according to formula (7). and chamfer distance between them (5) (6) (7) Chamfer distance, used to measure the average nearest distance between the output vector and the ground truth point cloud.

2. The method of claim 1, wherein the method is a three-dimensional point cloud completion method based on a bidirectional pyramid structure of a hybrid attention mechanism. The multi-layer perception with attention mechanism comprises a first layer perception, a second layer perception, a third layer perception, a fourth layer perception, a fifth layer perception, a first layer pooling layer, a second layer pooling layer, a third layer pooling layer, a fourth layer pooling layer, and a channel attention module, the neuron size of the first layer perception is 64, the neuron size of the second layer perception is 128, the neuron size of the third layer perception is 256, the neuron size of the fourth layer perception is 512, the neuron size of the fifth layer perception is 1024, the global feature output by the second layer perception is obtained through the first pooling layer, and the dimension thereof is 128, the global feature output by the third layer perception is obtained through the second pooling layer, and the dimension thereof is 256, the global feature output by the fourth layer perception is obtained through the third pooling layer, and the dimension thereof is 512, the global feature output by the fifth layer perception is obtained through the fourth pooling layer, and the dimension thereof is 1024, the obtained global features are spliced into a 1920-dimensional feature vector through a connection operation, and the feature vector is converted into a multi-level feature vector through the channel attention module.

3. The method of claim 1, wherein, The output feature vector is obtained by fusing the respective outputs of the mixed attention modules , , , , including,​ S31, processing the vector to obtain an output vector through a channel attention module The channel attention module includes a channel average pooling layer, two fully connected layers and an activation function, and a vector CA is obtained according to formula (1) ​​ (1) the output vector of the channel attention module, , is a channel average pooling layer, are parameters of two fully connected layers with sizes , is an activation function,​​​ S32. Apply point attention modules to vectors. The output vector is obtained through processing. The point attention module includes a point average pooling layer, two fully connected layers, and one... The activation function, according to formula (2), yields a vector. , (2) wherein, is the output vector of the point attention module, is a point average pooling layer, and are parameters of two fully connected layers with sizes and , S33, obtaining an output feature vector after processing according to formula (3) ,​ (3) wherein, is a Sigmoid function, S34, respectively, to , perform S31-S33, to obtain the output feature vector , .

4. A three-dimensional point cloud completion device based on a bidirectional pyramid structure of a hybrid attention mechanism, characterized in that, Comprising, The acquisition module is used for acquiring a three-dimensional point cloud of a three-dimensional object; The feature extraction module is configured to input the three-dimensional point cloud into a point cloud encoder, and obtain an output feature vector after processing the three-dimensional point cloud by the point cloud encoder. The point cloud encoder comprises a multi-layer perception network with an attention mechanism, a first full connection layer, and a second full connection layer. The multi-layer perception network with the attention mechanism comprises M-layer perceptrons, M-1-layer pooling layers, and a channel attention module. Global features output by an i+1-layer perceptron are obtained through an i-layer pooling layer, and M-1≥i>0. The obtained M-1 global features are spliced into a feature vector through a connection operation. The splicing is a horizontal superposition of the global features. Feature information is extracted from the feature vector through the channel attention module. A multi-level feature vector is constructed according to the extracted feature information. The multi-level feature vector is converted into an output feature vector through the second full connection layer. ; a prediction module, configured to obtain an output feature vector an input point cloud decoder, comprising a third fully connected layer, a fourth fully connected layer, a fifth fully connected layer, a hybrid attention module, a bidirectional attention pyramid structure, and an output feature vector obtaining a feature vector through the third fully connected layer , an output feature vector obtaining a feature vector through the fourth fully connected layer and the fifth fully connected layer , connecting the three-dimensional point cloud with the output feature vector , the feature vector , and the feature vector respectively and representing them as , , , fusing , , according to the hybrid attention module respectively to obtain an output feature vector , , , inputting the output feature vector , , to the bidirectional attention pyramid structure for processing to obtain an output vector , converting the output vector to a three-dimensional shape of different scales respectively, and modifying local geometric details of the three-dimensional shape respectively to obtain a complete three-dimensional object shape, wherein the bidirectional attention pyramid structure comprises a sixth fully connected layer, a seventh fully connected layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a first hybrid attention module, a second hybrid attention module, a third hybrid attention module, and a fourth hybrid attention module, the sixth fully connected layer is configured to extract a sixth connection output feature of the feature vector , the seventh fully connected layer is configured to extract a seventh connection output feature of the feature vector , the first convolutional layer is configured to extract a first convolutional output feature of the feature vector , the second convolutional layer is configured to obtain a second convolutional output feature according to the feature vector and the sixth connection output feature, the third convolutional layer is configured to obtain a third convolutional output feature according to the feature vector and the seventh connection output feature, the first hybrid attention module is configured to obtain a first hybrid output feature according to the second convolutional output feature and the first convolutional output feature, the second hybrid attention module is configured to obtain an output vector according to the third convolutional output feature, the second convolutional output feature, and the first convolutional output feature, and the third hybrid attention module is configured to obtain an output vector according to the first hybrid output feature and the output vector , the fourth mixed attention module is used for determining the fourth attention value according to ; the multi-level completion loss function of the bidirectional attention pyramid structure is formula (4), (4) wherein is an output vector, , , is a real point cloud of different scales, is a hyper-parameter, calculated according to equation (5) is the chamfer distance between is the chamfer distance between is the chamfer distance between is the chamfer distance between is the chamfer distance between is the chamfer distance between (5) (6) (7) Chamfer distance, used to measure the average nearest distance between the output vector and the ground truth point cloud.

Images