A point cloud completion system and method based on multi-scale self-attention network

Abstract

The present invention discloses a point cloud completion system and method based on a multi-scale self-attention network, the system comprises: a downsampling feature extraction module, a multi-scale self-attention module, and a dual-path dense point cloud generation module, the downsampling feature extraction module is used to downsample the original point cloud data and extract the downsampled point cloud features, the multi-scale self-attention module is used to extract the global features and the point cloud features of the missing part and the sparse point cloud, and the dual-path dense point cloud generation module is used to generate the completed point cloud. The present invention realizes the completion of the point cloud of complex structure and generates the dense point cloud of the complete object.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud processing, and more specifically, to a point cloud completion system and method based on a multi-scale self-attention network. Background Art

[0002] Point cloud is a commonly used data structure for three-dimensional vision. It is a collection of points in three-dimensional space and can be used to express the spatial information of objects. There are two main ways to obtain point clouds: direct acquisition and indirect acquisition. Direct acquisition generally refers to the use of instruments and equipment with relatively precise measurements such as lidar sensors and spectral sensors to integrate the information obtained by various sensors to directly transform real objects into virtual three-dimensional models, thereby obtaining three-dimensional point clouds. Indirect acquisition usually refers to the use of multiple two-dimensional images of the same scene at different angles for three-dimensional reconstruction, and the coordinates of three-dimensional points are calculated by matching feature points between images to form a point cloud. However, for objects with small size or complex shapes and structures, due to factors such as the distance of the object to be measured is too far, the scanning accuracy is too low, and the mutual occlusion of complex structures, point clouds are missing and sparse, especially when there is an unadjustable factor such as mutual occlusion. Completing the missing point cloud can repair the part that cannot be reconstructed by three-dimensional reconstruction, which is of great significance to improving the accuracy and reducing the error of three-dimensional rendering and object phenotype analysis.

[0003] For missing 3D point clouds, the usual solution is to study the shape characteristics of the point cloud, guess the shape of the missing part based on the known part, fill in the missing part, and thus restore a complete 3D point cloud. Existing methods use the specific properties of objects in 3D space to complete, such as the symmetric property, which can detect the plane reflection symmetry, generate a series of initial candidate objects for point clouds, and then use local traversal search to find the optimal object as the final solution; use the prior geometric parameters of specific objects to reconstruct, thereby solving the problem of surface defects of objects; use data retrieval and matching methods to align and match real-time scanned data with the 3D shape database to build the corresponding virtual model. These methods are more effective for completing pure planes, such as using the characteristic that the roof of a car is parallel to the ground to generate a missing roof, but they are not applicable to completing objects with various shapes, and optimization is more difficult.

[0004] At present, point cloud completion mainly uses deep learning methods, and the main idea basically follows the rule of compression first and then restoration, that is, using the encoder-decoder network structure. These methods perform well on public data sets, especially for objects with simple shapes and overall regularity, such as cars, tables, chairs, bookcases and other objects, but in actual application scenarios, the effect on point clouds with complex structures is still insufficient. The prior art discloses a diverse point cloud completion method and device based on self-attention, which relates to the field of computer three-dimensional point cloud completion and deep learning technology, wherein the method includes: obtaining point cloud data, processing the point cloud data, and obtaining an input point proxy sequence; encoding the point proxy sequence, obtaining a point encoding vector, decoding the point encoding vector, and obtaining a predicted point proxy; inputting the predicted point proxy into a multilayer perceptron, obtaining the predicted point center, and restoring the complete point cloud data based on the predicted point center. Thus, the point cloud is processed into a point proxy sequence, and an encoder-decoder is used to construct the long-range relationship between different points of the point cloud to achieve point cloud reconstruction. Although the scheme adopts the self-attention mechanism, it does not consider the problem of single input data size and feature dimension. Summary of the invention

[0005] In order to overcome the defect that the above-mentioned existing point cloud completion methods have poor effects on point clouds with complex structures, the present invention provides a point cloud completion system and method based on a multi-scale self-attention network.

[0006] The primary purpose of the present invention is to solve the above technical problems. The technical solution of the present invention is as follows:

[0007] The first aspect of the present invention provides a point cloud completion system based on a multi-scale self-attention network, comprising:

[0008] A downsampling feature extraction module, a multi-scale self-attention module, and a dual-path dense point cloud generation module. The downsampling feature extraction module is used to downsample the original point cloud data and extract the downsampled point cloud features. The multi-scale self-attention module uses the downsampled point cloud to extract global features and the point cloud features and sparse point clouds of the missing parts. The dual-path dense point cloud generation module uses the original input point cloud and the sparse point cloud of the missing part to generate a completed point cloud.

[0009] Furthermore, the downsampling feature extraction module includes: a farthest distance sampling unit, a neighborhood graph construction unit, a multi-layer perceptron unit, and a maximum pooling layer. The farthest distance sampling unit is used to downsample the input point cloud; the neighborhood graph construction unit uses the k-nearest neighbor algorithm to construct a local neighborhood graph for each point with the downsampled point as the center point, and calculates the edge features composed of the center point and k neighborhood points;

[0010] The multi-layer perceptron unit maps the obtained edge features into features of a preset dimension;

[0011] The maximum pooling layer aggregates the preset dimensional features obtained by mapping and outputs the downsampled point cloud features.

[0012] Furthermore, the multi-scale self-attention module includes: an encoder and a decoder, the encoder is used to extract global features of the input part of the point cloud, and the decoder is used to extract point cloud features of the missing part and the sparse point cloud of the missing part.

[0013] Furthermore, the encoder includes: a first multi-head attention layer and a first feedforward network layer, wherein the output of the first multi-head attention layer is used as the input of the first feedforward network layer, and the downsampled point cloud features are processed by the first multi-head attention layer and the first feedforward network layer in sequence to output the point cloud global features;

[0014] The decoder includes: a second multi-head attention layer, an encoder-decoder attention layer, and a second feedforward network layer. The output of the second multi-head attention layer is used as the input of the encoder-decoder attention layer, and the output of the encoder-decoder attention layer is used as the input of the second feedforward network layer. The global features of the point cloud output by the encoder are processed in sequence by the second multi-head attention layer, the encoder-decoder attention layer, and the second feedforward network layer to obtain the point cloud features of the missing part and the sparse point cloud of the missing part.

[0015] Furthermore, the first multi-head attention layer and the second multi-head attention layer have the same network structure and processing flow, wherein the processing flow is specifically as follows:

[0016] The input point cloud features are filtered using a multi-head attention mechanism, and the filtered point cloud features are sequentially subjected to linear transformation and activation function processing to obtain semantic features;

[0017] The k-nearest neighbor algorithm is used to construct a local neighborhood graph for the input point cloud features and calculate the edge features. Local features of various scales are extracted based on the neighborhood graph and edge features. The extracted local features of various scales are sequentially subjected to linear transformation and activation function processing to obtain the geometric features of each neighborhood, and the geometric features within the local neighborhood are aggregated through the maximum pooling operation.

[0018] The aggregated geometric features and semantic features are connected and processed by linear transformation and activation function in sequence to obtain the residual value of the feature. The residual value is added to the original input point cloud feature to obtain the output of the multi-head attention layer.

[0019] Furthermore, the specific process of filtering the input point cloud features using the multi-head attention mechanism is as follows:

[0020] Use three convolution operations with different parameters to extract the query vector, key vector, and value vector of the input point cloud features respectively;

[0021] Multiply the query vector and the key vector to get the attention weight for each value, and multiply the weight with the corresponding value vector to get the filtered point cloud feature.

[0022] Furthermore, the first feedforward network layer and the second feedforward network layer both include: 2 linear transformation layers and 1 ReLU activation function.

[0023] Furthermore, the processing process of the dual-path dense point cloud generation module is as follows:

[0024] Extract the point cloud coordinate values ​​of the original input part, splice the extracted coordinate values ​​with the 1024-dimensional global features and the 2D grid, pass the spliced ​​features through a multi-layer perceptron to obtain the dense point cloud displacement, and add the point cloud coordinate values ​​of the original input part and the dense point cloud displacement to obtain the dense point cloud sampled on the first path;

[0025] Extract the sparse point cloud coordinate values ​​of the missing part, splice the extracted coordinate values ​​with the 1024-dimensional missing part features and the 2D grid, pass the spliced ​​features through a multi-layer perceptron to obtain the dense point cloud displacement, and add the point cloud coordinate values ​​of the missing part to the dense point cloud displacement to obtain the dense point cloud sampled on the second path;

[0026] The dense point cloud sampled on the first path and the dense point cloud sampled on the second path are merged to obtain a dense point cloud of the complete object.

[0027] A second aspect of the present invention provides a point cloud completion method based on a multi-scale self-attention network, comprising the following steps:

[0028] The down-sampling feature extraction module is used to process the point cloud to be completed to obtain the down-sampled point cloud features;

[0029] The downsampled point cloud features are input into the multi-scale self-attention module to obtain the global features of the point cloud, the point cloud features of the missing parts, and the sparse point cloud;

[0030] The point cloud of the original input part and the sparse point cloud of the missing part are respectively input into the dual-path dense point cloud generation module to generate the dense point cloud of the complete object.

[0031] Furthermore, the specific steps of downsampling feature extraction are:

[0032] Downsample the input point cloud;

[0033] Take the sampled point as the center point and use the k nearest neighbor algorithm to build a local neighborhood graph for each point, and calculate the edge features formed by the center point and k neighboring points;

[0034] Mapping the obtained edge features into features of preset dimensions;

[0035] The preset dimensional features obtained by mapping are aggregated and output to obtain the downsampled point cloud features.

[0036] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0037] The present invention constructs a point cloud completion system based on a multi-scale self-attention network, uses a downsampling feature extraction module to downsample and preliminarily encode the original point cloud, solves the problems of excessive input data size and single feature dimension, and uses a multi-scale self-attention module to obtain a sparse point cloud of the missing part to solve the problem of the lack of inductive preference of the self-attention mechanism; uses a two-way dense point cloud generation module to refine the resolution of the input point cloud and the incomplete point cloud, and merges the two to generate a dense complete object point cloud, which solves the problem caused by the different proportions of points in different structural parts of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a block diagram of a point cloud completion system based on a multi-scale self-attention network according to an embodiment of the present invention.

[0039] Figure 2 This is a principle block diagram of a downsampling feature extraction module according to an embodiment of the present invention.

[0040] Figure 3 This is a principle block diagram of a multi-scale self-attention module according to an embodiment of the present invention.

[0041] Figure 4 This is a flowchart of the multi-head attention layer processing according to an embodiment of the present invention.

[0042] Figure 5 This is a processing flow chart of a dual-path dense point cloud generation module according to an embodiment of the present invention.

[0043] Figure 6 This is a comparison chart of point cloud completion effects according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0045] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0046] Example 1

[0047] like Figure 1As shown, the first aspect of the present invention provides a point cloud completion system based on a multi-scale self-attention network, including: a downsampling feature extraction module, a multi-scale self-attention module, and a dual-path dense point cloud generation module. The downsampling feature extraction module is used to downsample the original point cloud data and extract the downsampled point cloud features. The multi-scale self-attention module uses the downsampled point cloud features to extract global features and the point cloud features and sparse point clouds of the missing parts. The dual-path dense point cloud generation module uses the original input point cloud and the sparse point cloud of the missing part to generate a completed point cloud.

[0048] It should be noted that, in an embodiment of the present invention, the point cloud input to the system is a missing point cloud, i.e., a point cloud to be completed. The downsampling feature extraction module is used to downsample the input original point cloud data and extract the downsampled point cloud features. Through this module, the three-dimensional coordinates of the original point cloud can be mapped to high-dimensional features, and local features can be aggregated to reduce the resolution, which can reduce the computational burden of the network model. The extracted point cloud features are input to the multi-scale self-attention module to obtain global features and point cloud features and sparse point clouds of the missing parts. The multi-scale self-attention module integrates geometric perception features of different scales, which enables the network to have better learning ability for object point clouds. The original point cloud data and the sparse point cloud of the missing part are input to the dual-path dense point cloud generation module to generate the completed point cloud. The dual-path dense point cloud generation module uses a dual-path dense point cloud generation path to refine the resolution of the input point cloud and the incomplete point cloud respectively through a folding-based method, and finally merges the two to generate a dense complete object point cloud.

[0049] Further, such as Figure 2 As shown, the downsampling feature extraction module includes: a farthest distance sampling unit, a neighborhood graph construction unit, a multi-layer perceptron unit, and a maximum pooling layer. The farthest distance sampling unit is used to downsample the input point cloud; the domain graph construction unit uses the k-nearest neighbor algorithm to construct a local neighborhood graph for each point with the point obtained by downsampling as the center point, and calculates the edge features composed of the center point and k neighborhood points;

[0050] The multi-layer perceptron unit maps the obtained edge features into features of a preset dimension;

[0051] The maximum pooling layer aggregates the preset dimensional features obtained by mapping and outputs the downsampled point cloud features.

[0052] It should be noted that in the point cloud, the coordinates of the point cannot represent the local neighborhood information of the location of the point. Directly using the three-dimensional coordinates of the point cloud as the input of the network cannot well extract the global features and local features of the point cloud. On the other hand, if the number of input points is large, the computational burden of the network will be increased, resulting in a long training time or even inability to perform training. The present invention uses a downsampling feature extraction module to aggregate local neighborhood features layer by layer to generate low-resolution, high-dimensional point cloud features. Taking the input number of points as N and the feature dimension as C as an example, the specific process is as follows: first, the farthest distance sampling algorithm is used for downsampling to obtain N / 2 points. The farthest distance sampling algorithm process: specify the sampled area and the area to be sampled, and let the original input point cloud be the area to be sampled; randomly select a point in the area to be sampled as the starting point and move it into the sampled area; in the area to be sampled, iterate cyclically to find the point farthest from the sampled area, and move the point into the sampled point area; iterate cyclically until the number of points in the sampled point area reaches N / 2, and then end the loop. Then, the N / 2 points obtained by sampling are taken as the center points, and the k-nearest neighbor algorithm is used to construct a local neighborhood graph for each point, and the edge features formed by the center point and the k neighboring points are calculated. Edge feature calculation method: Subtract the features of the center point from the features of the neighboring points, and obtain the edge features through a linear transformation. Then, the multi-layer perceptron is used to map to 2C-dimensional features. Finally, the maximum pooling layer is used to aggregate the features to obtain the point cloud features output by this layer. The number of points in the output point cloud is N / 2, and the feature dimension is 2C, which realizes the dual operations of downsampling and feature extraction.

[0053] Further, such as Figure 3 As shown, the multi-scale self-attention module includes: an encoder and a decoder, the encoder is used to extract the global features of the input part of the point cloud, and the decoder is used to extract the point cloud features of the missing part and the sparse point cloud of the missing part.

[0054] It should be noted that in the embodiment of the present invention, the multi-scale self-attention module Transformer is an encoder-decoder structure, whose core component is the self-attention mechanism, which can generate attention weights according to the global context of the input. Similar to natural language processing, a point cloud can be regarded as a sentence, and the points in the point cloud can be regarded as a word. Through Transformer, the global features of the point cloud can be learned and extracted according to the positional relationship of the points. However, the original Transformer lacks some inductive preferences and relies heavily on a large number of data sets for large-scale training, which limits the learning performance of the network. The multi-scale self-attention module proposed in the present invention uses a network based on the improved Transformer. On the original Transformer structure, a multi-scale geometric perception module is used, which combines semantic features and multi-scale geometric features to form a fusion expression containing local information.

[0055] Furthermore, the encoder includes: a first multi-head attention layer and a first feedforward network layer, wherein the output of the first multi-head attention layer is used as the input of the first feedforward network layer, and the downsampled point cloud features are processed by the first multi-head attention layer and the first feedforward network layer in sequence to output the point cloud global features;

[0056] The decoder includes: a second multi-head attention layer, an encoder-decoder attention layer, and a second feedforward network layer. The output of the second multi-head attention layer is used as the input of the encoder-decoder attention layer, and the output of the encoder-decoder attention layer is used as the input of the second feedforward network layer. The global features of the point cloud output by the encoder are processed in sequence by the second multi-head attention layer, the encoder-decoder attention layer, and the second feedforward network layer to obtain the point cloud features of the missing part and the sparse point cloud of the missing part.

[0057] It should be noted that, in a specific embodiment, the encoder predicts a sparse point cloud of the missing part of the object, with a size of m×3, where m is the number of points in the point cloud, and m is set to 400. At the same time, the encoder and decoder respectively extract the global features of the object point cloud and the features of the missing part of the object, and the size of the features is 1×1024.

[0058] It should be noted that the first multi-head attention layer and the second multi-head attention layer receive Q matrix, K matrix, and V matrix as input. The first multi-head attention layer and the second multi-head attention layer use a self-attention mechanism, and its Q matrix, K matrix, and V matrix are derived from the same feature data. As for the encoder-decoder attention layer unique to the decoder, its input K matrix and V matrix are derived from the global features output by the encoder, and the Q matrix is ​​derived from the output of the second multi-head attention layer. The multi-head attention mechanism enables the network to learn related information in different representation subspaces, and can combine the attention weights from different positions. In order to fuse the different features of the point cloud at multiple scales to retain the geometric relationship information layer by layer, the present invention proposes the use of a multi-head attention layer that fuses multi-scale geometric information, which includes a multi-head attention mechanism and a multi-scale geometric information extraction mechanism, and its structure is as follows Figure 4 shown.

[0059] Furthermore, the first multi-head attention layer and the second multi-head attention layer have the same network structure and processing flow, wherein Figure 4 As shown in the figure, the specific processing flow is as follows:

[0060] The input point cloud features are filtered using a multi-head attention mechanism, and the filtered point cloud features are sequentially subjected to linear transformation and activation function processing to obtain semantic features;

[0061] The k-nearest neighbor algorithm is used to construct a local neighborhood graph for the input point cloud features and calculate the edge features. Local features of various scales are extracted based on the neighborhood graph and edge features. The extracted local features of various scales are sequentially subjected to linear transformation and activation function processing to obtain the geometric features of each neighborhood, and the geometric features within the local neighborhood are aggregated through the maximum pooling operation.

[0062] The aggregated geometric features and semantic features are connected and processed by linear transformation and activation function in sequence to obtain the residual value of the feature. The residual value is added to the original input point cloud feature to obtain the output of the multi-head attention layer.

[0063] Furthermore, the specific process of filtering the input point cloud features using the multi-head attention mechanism is as follows:

[0064] Use three convolution operations with different parameters to extract the query vector, key vector, and value vector of the input point cloud features respectively;

[0065] Multiply the query vector and the key vector to get the attention weight for each value, and multiply the weight with the corresponding value vector to get the filtered point cloud feature.

[0066] Furthermore, the first feedforward network layer and the second feedforward network layer both include: 2 linear transformation layers and 1 ReLU activation function.

[0067] Further, such as Figure 5 As shown, the processing process of the dual-path dense point cloud generation module is:

[0068] Extract the point cloud coordinate values ​​of the original input part, splice the extracted coordinate values ​​with the 1024-dimensional global features and the 2D grid, pass the spliced ​​features through a multi-layer perceptron to obtain the dense point cloud displacement, and add the point cloud coordinate values ​​of the original input part and the dense point cloud displacement to obtain the dense point cloud sampled on the first path;

[0069] Extract the sparse point cloud coordinate values ​​of the missing part, splice the extracted coordinate values ​​with the 1024-dimensional missing part features and the 2D grid, pass the spliced ​​features through a multi-layer perceptron to obtain the dense point cloud displacement, and add the point cloud coordinate values ​​of the missing part to the dense point cloud displacement to obtain the dense point cloud sampled on the second path;

[0070] The dense point cloud sampled on the first path and the dense point cloud sampled on the second path are merged to obtain a dense point cloud of the complete object.

[0071] It should be noted that the input point cloud comes from real samples, and its coordinates are relatively stable and accurate, while the predicted missing part of the point cloud is generated by the network and has uncertainty, so the entire sparse point cloud cannot be directly densely upsampled according to the same rule. In order to ensure that the final output point cloud is closer to the real distribution, a dual-path method is used to generate dense upsampling for the missing part of the point cloud and the input point cloud using a folding-based method.

[0072] Example 2

[0073] A second aspect of the present invention provides a point cloud completion method based on a multi-scale self-attention network, comprising the following steps:

[0074] The down-sampling feature extraction module is used to process the point cloud to be completed to obtain the down-sampled point cloud features;

[0075] The downsampled point cloud features are input into the multi-scale self-attention module to obtain the global features, the point cloud features of the missing parts and the sparse point cloud;

[0076] The point cloud of the original input part and the sparse point cloud of the missing part are respectively input into the dual-path dense point cloud generation module to generate the dense point cloud of the complete object.

[0077] It should be noted that in an embodiment of the present invention, the point cloud input to the system is a missing point cloud, i.e., a point cloud to be completed. The downsampling feature extraction module is used to downsample the input original point cloud data and extract the downsampled point cloud features. Through this module, the three-dimensional coordinates of the original point cloud can be mapped to high-dimensional features, and local features can be aggregated to reduce the resolution, which can reduce the computational burden of the network model. The extracted point cloud features are input to the multi-scale self-attention module to obtain global features and features of the missing part and sparse point clouds. The multi-scale self-attention module integrates geometric feature perception modules of different scales, so that the network has better learning ability for object point clouds. The original point cloud data and the coefficient point cloud of the missing part are input to the dual-path dense point cloud generation module to generate the completed point cloud. The dual-path dense point cloud generation module uses a dual-path dense point cloud generation path to refine the resolution of the input point cloud and the incomplete part point cloud respectively through a folding-based method, and merges the two to generate a dense complete object point cloud.

[0078] Furthermore, the downsampling feature extraction module includes: a farthest distance sampling unit, a neighborhood graph construction unit, a multi-layer perceptron unit, and a maximum pooling layer. The farthest distance sampling unit is used to downsample the input point cloud; the neighborhood graph construction unit uses the k-nearest neighbor algorithm to construct a local neighborhood graph for each point with the downsampled point as the center point, and calculates the edge features composed of the center point and k neighborhood points;

[0079] The multi-layer perceptron unit maps the obtained edge features into features of a preset dimension;

[0080] The maximum pooling layer aggregates the preset dimensional features obtained by mapping and outputs the downsampled point cloud features.

[0081] It should be noted that in the point cloud, the coordinates of the point cannot represent the local neighborhood information of the location of the point. Directly using the three-dimensional coordinates of the point cloud as the input of the network cannot well extract the global features and local features of the point cloud. On the other hand, if the number of input points is large, the computational burden of the network will be increased, resulting in a long training time or even inability to perform training. The present invention uses a downsampling feature extraction module to aggregate local neighborhood features layer by layer to generate low-resolution, high-dimensional point cloud features. Taking the input number of points as N and the feature dimension as C as an example, the specific process is as follows: first, the farthest distance sampling algorithm is used for downsampling to obtain N / 2 points. The farthest distance sampling algorithm process: specify the sampled area and the area to be sampled, and let the original input point cloud be the area to be sampled; randomly select a point in the area to be sampled as the starting point and move it into the sampled area; in the area to be sampled, iterate cyclically to find the point farthest from the sampled area, and move the point into the sampled point area; iterate cyclically until the number of points in the sampled point area reaches N / 2, and then end the loop. Then, the N / 2 points obtained by sampling are taken as the center points, and the k-nearest neighbor algorithm is used to construct a local neighborhood graph for each point, and the edge features formed by the center point and the k neighboring points are calculated. Edge feature calculation method: Subtract the features of the center point from the features of the neighboring points, and obtain the edge features through a linear transformation. Then, the multi-layer perceptron is used to map to 2C-dimensional features. Finally, the maximum pooling layer is used to aggregate the features to obtain the point cloud features output by this layer. The number of points in the output point cloud is N / 2, and the feature dimension is 2C, which realizes the dual operations of downsampling and feature extraction.

[0082] Furthermore, the multi-scale self-attention module includes: an encoder and a decoder, the encoder is used to extract global features of the input part of the point cloud, and the decoder is used to extract point cloud features of the missing part and the sparse point cloud of the missing part.

[0083] It should be noted that in the embodiment of the present invention, the multi-scale self-attention module Transformer is an encoder-decoder structure, whose core component is the self-attention mechanism, which can generate attention weights according to the global context of the input. Similar to natural language processing, a point cloud can be regarded as a sentence, and the points in the point cloud can be regarded as a word. Through Transformer, the global features of the point cloud can be learned and extracted according to the positional relationship of the points. However, the original Transformer lacks some inductive preferences and relies heavily on a large number of data sets for large-scale training, which limits the learning performance of the network. The multi-scale self-attention module proposed in the present invention uses a network based on the improved Transformer. On the original Transformer structure, a multi-scale geometric perception module is used, which combines semantic features and multi-scale geometric features to form a fusion expression containing local information.

[0084] Furthermore, the encoder includes: a first multi-head attention layer and a first feedforward network layer, wherein the output of the first multi-head attention layer is used as the input of the first feedforward network layer, and the downsampled point cloud features are processed by the first multi-head attention layer and the first feedforward network layer in sequence to output the point cloud global features;

[0085] The decoder includes: a second multi-head attention layer, an encoder-decoder attention layer, and a second feedforward network layer. The output of the second multi-head attention layer is used as the input of the encoder-decoder attention layer, and the output of the encoder-decoder attention layer is used as the input of the second feedforward network layer. The global features of the point cloud output by the encoder are processed in sequence by the second multi-head attention layer, the encoder-decoder attention layer, and the second feedforward network layer to obtain the point cloud features of the missing part and the sparse point cloud of the missing part.

[0086] It should be noted that, in a specific embodiment, the encoder predicts a sparse point cloud of the missing part of the object, with a size of m×3, where m is the number of points in the point cloud, and m is set to 400. At the same time, the encoder and decoder respectively extract the global features of the object point cloud and the features of the missing part of the object, and the size of the features is 1×1024.

[0087] It should be noted that the first multi-head attention layer and the second multi-head attention layer receive Q matrix, K matrix, and V matrix as input. The first multi-head attention layer and the second multi-head attention layer use a self-attention mechanism, and its Q matrix, K matrix, and V matrix are derived from the same feature data. As for the encoder-decoder attention layer unique to the decoder, its input K matrix and V matrix are derived from the global features output by the encoder, and the Q matrix is ​​derived from the output of the second multi-head attention layer. The multi-head attention mechanism enables the network to learn related information in different representation subspaces, and can combine the attention weights from different positions. In order to fuse the different features of the point cloud at multiple scales to retain the geometric relationship information layer by layer, the present invention proposes the use of a multi-head attention layer that fuses multi-scale geometric information, which includes a multi-head attention mechanism and a multi-scale geometric information extraction mechanism, and its structure is as follows Figure 4 shown.

[0088] Furthermore, the first multi-head attention layer and the second multi-head attention layer have the same network structure and processing flow, wherein the processing flow is specifically as follows:

[0089] The input point cloud features are filtered using a multi-head attention mechanism, and the filtered point cloud features are sequentially subjected to linear transformation and activation function processing to obtain semantic features;

[0090] The k-nearest neighbor algorithm is used to construct a local neighborhood graph for the input point cloud features and calculate the edge features. Local features of various scales are extracted based on the neighborhood graph and edge features. The extracted local features of various scales are sequentially subjected to linear transformation and activation function processing to obtain the geometric features of each neighborhood, and the geometric features within the local neighborhood are aggregated through the maximum pooling operation.

[0091] The aggregated geometric features and semantic features are connected and processed by linear transformation and activation function in sequence to obtain the residual value of the feature. The residual value is added to the original input point cloud feature to obtain the output of the multi-head attention layer.

[0092] Furthermore, the specific process of filtering the input point cloud features using the multi-head attention mechanism is as follows:

[0093] Use three convolution operations with different parameters to extract the query vector, key vector, and value vector of the input point cloud features respectively;

[0094] Multiply the query vector and the key vector to get the attention weight for each value, and multiply the weight with the corresponding value vector to get the filtered point cloud feature.

[0095] Furthermore, the first feedforward network layer and the second feedforward network layer both include: 2 linear transformation layers and 1 ReLU activation function.

[0096] Furthermore, the processing process of the dual-path dense point cloud generation module is as follows:

[0097] Extract the point cloud coordinate values ​​of the original input part, splice the extracted coordinate values ​​with the 1024-dimensional global features and the 2D grid, pass the spliced ​​features through a multi-layer perceptron to obtain the dense point cloud displacement, and add the point cloud coordinate values ​​of the original input part and the dense point cloud displacement to obtain the dense point cloud sampled on the first path;

[0098] Extract the sparse point cloud coordinate values ​​of the missing part, splice the extracted coordinate values ​​with the 1024-dimensional missing part features and the 2D grid, pass the spliced ​​features through a multi-layer perceptron to obtain the dense point cloud displacement, and add the point cloud coordinate values ​​of the missing part to the dense point cloud displacement to obtain the dense point cloud sampled on the second path;

[0099] The dense point cloud sampled on the first path and the dense point cloud sampled on the second path are merged to obtain a dense point cloud of the complete object.

[0100] It should be noted that the input point cloud comes from real samples, and its coordinates are relatively stable and accurate, while the predicted missing part of the point cloud is generated by the network and has uncertainty, so the entire sparse point cloud cannot be directly densely upsampled according to the same rule. In order to ensure that the final output point cloud is closer to the real distribution, a dual-path method is used to generate dense upsampling for the missing part of the point cloud and the input point cloud using a folding-based method.

[0101] Example 3

[0102] This embodiment illustrates the technical effect of the present invention through specific experimental data, and compares the present invention with other point cloud completion methods. This embodiment experiments on the problem of missing agricultural plant point clouds, and compares it with other existing completion methods. The point cloud object used in the experiment is the point cloud of taro seedlings. The experimental data sets are all obtained by low-cost three-dimensional reconstruction methods based on motion recovery structure. Plant point clouds are often missing due to factors such as leaves occluding each other. The completion of agricultural plant point clouds is of great significance to agricultural automation and agricultural phenotypic analysis. The experimental input has missing plant point clouds, and the point cloud is completed by the method provided in the embodiment of this application and other methods, and the chamfer distance and F1 score between the completion result and the standard point cloud data are calculated. The effect of point cloud completion is characterized based on the chamfer distance and F1 score. The experimental results are shown in Table 1:

[0103] Table 1 Comparison of experimental results

[0104]

[0105] In Table 1, the bold font indicates the best effect, and the underlined font indicates the second best effect. Based on Table 1, it can be found that the method provided by the embodiment of the present application is significantly better than other methods. The method of the present invention completes the center and edge of the leaf well, the point cloud is distributed more evenly, the edge of the leaf is clearer, and the details of the original input point cloud are well preserved. The completion effect is visualized as follows: Figure 6 shown.

[0106] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A point cloud completion system based on a multi-scale self-attention network, characterized in that: include: Downsampling feature extraction module, multi-scale self-attention module, dual-path dense point cloud generation module, the downsampling feature extraction module is used to downsample the original point cloud data and extract the downsampled point cloud features, the multi-scale self-attention module uses the downsampled point cloud features to extract the global features of the point cloud and the point cloud features of the missing part and the sparse point cloud, the dual-path dense point cloud generation module uses the original input part of the point cloud and the sparse point cloud of the missing part to generate the completed point cloud; the processing process of the dual-path dense point cloud generation module is: Extract the point cloud coordinate values ​​of the original input part, splice the extracted coordinate values ​​with the 1024-dimensional global features and the 2D grid, pass the spliced ​​features through a multi-layer perceptron to obtain the dense point cloud displacement, and add the point cloud coordinate values ​​of the original input part and the dense point cloud displacement to obtain the dense point cloud sampled on the first path; Extract the sparse point cloud coordinate values ​​of the missing part, splice the extracted coordinate values ​​with the 1024-dimensional missing part features and the 2D grid, pass the spliced ​​features through a multi-layer perceptron to obtain the dense point cloud displacement, and add the point cloud coordinate values ​​of the missing part to the dense point cloud displacement to obtain the dense point cloud sampled on the second path; The dense point cloud sampled on the first path and the dense point cloud sampled on the second path are merged to obtain a dense point cloud of the complete object.

2. A point cloud completion system based on a multi-scale self-attention network according to claim 1, characterized in that: The downsampling feature extraction module includes: a farthest distance sampling unit, a neighborhood graph construction unit, a multi-layer perceptron unit, and a maximum pooling layer. The farthest distance sampling unit is used to downsample the input point cloud; the neighborhood graph construction unit uses the k-nearest neighbor algorithm to construct a local neighborhood graph for each point with the downsampled point as the center point, and calculates the edge features composed of the center point and k neighborhood points; The multi-layer perceptron unit maps the obtained edge features into features of a preset dimension; The maximum pooling layer aggregates the preset dimensional features obtained by mapping and outputs the downsampled point cloud features.

3. The point cloud completion system based on a multi-scale self-attention network according to claim 1, characterized in that: The multi-scale self-attention module includes: an encoder and a decoder, the encoder is used to extract the global features of the input part of the point cloud, and the decoder is used to extract the point cloud features of the missing part and the sparse point cloud of the missing part.

4. A point cloud completion system based on a multi-scale self-attention network according to claim 3, characterized in that: The encoder comprises: a first multi-head attention layer and a first feedforward network layer, wherein the output of the first multi-head attention layer is used as the input of the first feedforward network layer, and the downsampled point cloud features are processed by the first multi-head attention layer and the first feedforward network layer in sequence to output the point cloud global features; The decoder includes: a second multi-head attention layer, an encoder-decoder attention layer, and a second feedforward network layer. The output of the second multi-head attention layer is used as the input of the encoder-decoder attention layer, and the output of the encoder-decoder attention layer is used as the input of the second feedforward network layer. The global features of the point cloud output by the encoder are processed in sequence by the second multi-head attention layer, the encoder-decoder attention layer, and the second feedforward network layer to obtain the point cloud features of the missing part and the sparse point cloud of the missing part.

5. A point cloud completion system based on a multi-scale self-attention network according to claim 4, characterized in that: The first multi-head attention layer and the second multi-head attention layer have the same network structure and processing flow, wherein the processing flow is specifically as follows: The input point cloud features are filtered using a multi-head attention mechanism, and the filtered point cloud features are sequentially subjected to linear transformation and activation function processing to obtain semantic features; The k-nearest neighbor algorithm is used to construct a local neighborhood graph for the input point cloud features and calculate the edge features. Local features of various scales are extracted based on the neighborhood graph and edge features. The extracted local features of various scales are sequentially subjected to linear transformation and activation function processing to obtain the geometric features of each neighborhood, and the geometric features within the local neighborhood are aggregated through the maximum pooling operation. The aggregated geometric features and semantic features are connected and processed by linear transformation and activation function in sequence to obtain the residual value of the feature. The residual value is added to the original input point cloud feature to obtain the output of the multi-head attention layer.

6. A point cloud completion system based on a multi-scale self-attention network according to claim 5, characterized in that: The specific process of filtering the input point cloud features using the multi-head attention mechanism is as follows: using three convolution operations with different parameters to respectively extract the query vector, key vector, and value vector of the input point cloud features; Multiply the query vector and the key vector to get the attention weight for each value, and multiply the weight with the corresponding value vector to get the filtered point cloud feature.

7. The point cloud completion system based on a multi-scale self-attention network according to claim 4, characterized in that: The first feedforward network layer and the second feedforward network layer both include: 2 linear transformation layers and 1 ReLU activation function.

8. A point cloud completion method based on a multi-scale self-attention network point cloud completion system, characterized in that: The following steps are involved: The down-sampling feature extraction module is used to process the point cloud to be completed to obtain the down-sampled point cloud features; The downsampled point cloud features are input into the multi-scale self-attention module to obtain the point cloud features of the missing part and the sparse point cloud of the missing part; The point cloud of the original input part and the sparse point cloud of the missing part are respectively input into the dual-path dense point cloud generation module to generate the dense point cloud of the complete object.

9. The point cloud completion method based on a multi-scale self-attention network point cloud completion system according to claim 8, characterized in that: The specific steps of downsampling feature extraction are: downsampling the input point cloud; Take the sampled point as the center point and use the k nearest neighbor algorithm to build a local neighborhood graph for each point, and calculate the edge features formed by the center point and k neighboring points; Mapping the obtained edge features into features of preset dimensions; The preset dimensional features obtained by mapping are aggregated and output to obtain the downsampled point cloud features.

Images