Point cloud completion method based on multi-head attention mechanism feature encoding and double discrimination decoding

Abstract

The application discloses a point cloud completion method based on a multi-head attention mechanism feature encoding and double-discriminator decoding, wherein, for a local feature perception encoder module, a multi-head attention mechanism and a multi-layer perception machine are combined to perform feature encoding on an existing point cloud, similar point features of an input point cloud are adaptively aggregated, and the perception of local features in a feature extraction process is improved; a double-branch decoder controlled by a discriminator sets the discriminator in a local feature and global feature decoding process, can discriminate the generated skeleton point cloud and fine point cloud based on the input point cloud features, and ensures that the generated point cloud conforms to the global features and detailed features of the existing point cloud. Through the cooperation of the above two modules, the point cloud shape completion effect can be effectively enhanced, the chamfer distance loss of the generated point cloud is reduced, and the fidelity of the generated point cloud is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional reconstruction, in particular to a point cloud completion method based on a multi-head attention mechanism feature encoding and double-discriminative decoding. BACKGROUND

[0002] Three-dimensional point cloud is a direct representation of three-dimensional scenes and objects, and has a wide range of applications in various vision and robot tasks. However, due to factors such as occlusion and measurement missing, the scanned 3D point cloud is usually incomplete, which affects practical applications, and completing incomplete point clouds has become an important task. Therefore, under the condition of limited hardware, point cloud repair and completion based on deep learning is the key and basis for subsequent point cloud related tasks.

[0003] The existing point cloud completion network based on deep learning has the following problems in point cloud completion:

[0004] (1) The local geometric feature analysis effect of the missing area is not good.

[0005] (2) The use of attention mechanism does not combine the data characteristics of three-dimensional geometric models.

[0006] The above problems restrict the completion accuracy of three-dimensional point cloud models and cannot meet the requirements of completing the missing surface features of objects obtained by three-dimensional scanners.

[0007] In 2021, Yao Yongqiang et al. of Tencent Technology (Shenzhen) Co., Ltd. proposed a point cloud data processing method, device, computer equipment and storage medium (authorized announcement number: CN 113327279 B), which encodes the point cloud coordinate features by using a self-correlation weight matrix, obtains the point position self-correlation features corresponding to the point cloud coordinate features, and predicts the missing part in the to-be-completed space model to obtain the missing point part features. The disadvantage is that the method encodes the geometric characteristics of the entire point cloud and does not analyze the geometric feature information of the missing area of the to-be-completed model.

[0008] In 2022, Guoliang et al. of Shandong University proposed a point cloud registration method and system based on point cloud completion (authorized public number: CN 114004871 B), which fused the features of the source point cloud and the target point cloud by using the attention mechanism, so that the semantic information of the two point clouds was mutually completed. The advantage is that it does not need to greatly reduce the original point cloud, and can complete the missing point cloud information to achieve efficient and accurate registration. The disadvantage is that it completes the point cloud by combining the shape information of the existing point cloud, and does not involve obtaining the local geometric features of the missing part and generating unknown shape information.

[0009] In 2020, Lu Jiweng et al. of Tsinghua University proposed a multi-point cloud completion method and device based on self-attention (application announcement number: CN112614071A), which processed point cloud into point agent sequence and used multi-head self-attention layer to construct the relationship between points, and used encoder-decoder to construct the long-range relationship between different points of point cloud to realize point cloud reconstruction. The disadvantage is that the perception effect of local shape information is not good through the way of point agent. In 2022, Gao Ziqi et al. of Guangdong University of Technology proposed a point cloud completion device and method based on adaptive self-attention transformation network (authorized announcement number: CN113486988B), which designed adaptive self-attention transformation module to adaptively fuse the spatial information features of two times of sampling, and then completed the missing point cloud, ensuring the effectiveness of lightweight calculation and multi-feature fusion. The disadvantage is that the attention transformation module used for natural language processing is directly used for point cloud processing, and the completion effect of the point cloud model is not good.

[0010] In summary, to improve the representation, extraction and generation of the geometric features of the missing area of the three-dimensional point cloud to be completed model, it is necessary to improve the point cloud completion network model. SUMMARY

[0011] The purpose of the present application is to overcome the defects of the prior art and provide a point cloud completion method with multi-head attention mechanism feature encoding and double discriminant decoding, which can effectively improve the completion accuracy of the missing area of the three-dimensional model to be completed.

[0012] The purpose of the present application is achieved by a point cloud completion method with multi-head attention mechanism feature encoding and double discriminant decoding, comprising the following steps:

[0013] Step 1) obtaining point cloud data;

[0014] Step 2) local feature encoding based on multi-head attention mechanism;

[0015] Step 2-1) obtaining point cloud features based on multi-layer perception;

[0016] Step 2-2) constructing local feature implicit model based on multi-head attention mechanism;

[0017] Step 3) decoding missing geometric features based on double discriminators;

[0018] Step 4) fusing generated point cloud and existing point cloud.

[0019] As a further limitation of the application, step 1) specifically comprises: acquiring point cloud data by a three-dimensional scanner; for each three-dimensional point cloud model, randomly selecting a viewpoint, and selecting a viewpoint as the center of a sphere, deleting all points on the sphere with R as the radius, and the remaining points are regarded as missing point clouds as network input, and the deleted point cloud is regarded as the true point cloud, wherein the network input is a set of point cloud data

[0020] As a further limitation of the application, step 2-1) specifically comprises: inputting the set of point cloud data given in step 1) Where N represents the MLP operation in the Cartesian coordinate system coordinate system to convert unordered point cloud data into matrix information F1, and the size of F1 is n x a.

[0021] As a further limitation of the application, step 2-2) specifically comprises: first, receiving the acquired feature F1 and setting the downsampling ratio d of the farthest distance point sampling algorithm, obtaining a downsampled point cloud feature matrix F2 with a size of (n / d) x a, then learning the feature F2 using a residual form of multi-head cross attention mechanism, and obtaining a corresponding feature F3:

[0022] F3=f(F2+MultHead(F2,K,V))

[0023] Where: f represents summation and uses LayerNorm for normalization processing, F2 represents the input feature, MultHead(F2,K,V)=Cat(head1,head2,…,head n )W O , the size of feature F3 is (n / d) x 2a;

[0024] The feature F3 is further updated by a feedforward network FFN:

[0025] F4=cat(F3+FFN(F2),F2)

[0026] Where cat() represents a connection operation by torch.nn.cat function; finally, the feature F4 is obtained through a multi-layer perceptron MLP and a maximum pooling Maxpooling to obtain the final feature F5.

[0027] As a further limitation of the application, step 3) comprises:

[0028] Step 3-1) obtains a skeleton point cloud feature based on an encoder;

[0029] Step 3-2) determines the generated skeleton point cloud based on a discriminator;

[0030] Step 3-3) generating fine point cloud features based on skeleton point cloud feature refinement;

[0031] Step 3-4) fusing the skeleton point cloud and the refined point cloud and performing secondary discrimination to obtain the final generated point cloud.

[0032] As a further limitation of the application, the loss geometry feature decoding loss of the discriminator in step 3-2) comprises d CD1 and d CD2 and is weighted using a hyperparameter β:

[0033] L = d CD1 (U1, U'1) + βd CD2 (U2, U'2)

[0034] Where: U1 is the fine point cloud, U2 is the skeleton point cloud, U'1 and U'2 represent the corresponding real point cloud respectively, d CD represents the chamfer distance loss:

[0035]

[0036] The above technical scheme is adopted in the application, compared with the prior art, has the beneficial effects that: in the encoder aspect, the local feature encoder based on the multi-head attention mechanism encodes the features of the existing point cloud, improves the adaptive aggregation of similar point features in the feature extraction process of the multi-layer perception through the multi-head attention mechanism, and can improve the perception of local features; in the decoder aspect, the discriminator is set in the local feature and global feature decoding process, which can discriminate the generated skeleton point cloud and fine point cloud based on the input point cloud features, and ensure that the generated point cloud conforms to the global features and detailed features of the existing point cloud. Thus, based on the improved encoder and double-branch decoder, the chamfer distance loss of the generated point cloud is reduced, and the fidelity of the generated point cloud is improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 Flowchart of the application. DETAILED DESCRIPTION

[0038] As Figure 1 shown, a multi-head attention mechanism feature encoding and double discrimination decoding point cloud completion method comprises the following steps:

[0039] Step 1) obtaining point cloud data;

[0040] In the embodiment, the network is trained by taking the existing data set ShapeNet as an example, the ShapeNet data set contains 8 categories, 30974 three-dimensional point cloud models, 100 models are extracted as a verification set, 150 models are extracted as a test set, and the remaining models are used as a training set. For each three-dimensional point cloud model, a random viewpoint is selected, and a viewpoint is selected as the center of the sphere. All points on the sphere with R as the radius are deleted, and the remaining points are regarded as the missing point cloud as the network input. The deleted point cloud is regarded as the true point cloud, wherein the network input is a set of point cloud data

[0041] Step 2) local feature encoding based on multi-head attention mechanism

[0042] Step 2-1) obtaining point cloud features based on multi-layer perception

[0043] Input step 1) a given set of point cloud data Wherein N represents the Cartesian coordinate system coordinate system, and the unordered point cloud data is converted into a matrix information F1 by performing MLP operation, and the size of F1 is n*a.

[0044] Step 2-2) constructing a local feature implicit model based on a multi-head attention mechanism

[0045] First, the obtained feature F1 is received, and the down-sampling ratio d of the farthest distance point sampling algorithm is set, and the down-sampling point cloud feature matrix F2 with a size of (n / d)*a is obtained. Then, the multi-head cross attention mechanism in the residual form is used to learn the feature F2, and the corresponding feature F3 is obtained:

[0046] F3=f(F2+MultHead(F2,K,V))

[0047] Wherein: f represents summing and using LayerNorm for normalization processing, F2 represents the input feature, MultHead(F2,K,V)=Cat(head1,head2,…,head n )W O In the present application, n=4, and the size of the feature F3 is (n / d)*2a.

[0048] Secondly, the existing feature F2 is further updated through the feedforward network FFN and the connection:

[0049] F4=cat(F3+FFN(F2),F2)

[0050] Wherein, cat() represents the connection operation through the torch.nn.cat function; the feedforward neural network FFN is used to improve the fitting ability of the model

[0051] Finally, the feature F4 is obtained by a multilayer perceptron MLP and a max pooling Maxpooling to obtain the final feature F5.

[0052] Step 3) missing geometry feature decoding based on double discriminators;

[0053] Step 3-1) obtaining the skeleton point cloud feature based on the feature decoding obtained by the encoder;

[0054] Taking the feature F5 extracted by the encoder in step 2) as input, the feature vectors F6 and F7 are obtained through two linear layers, and the refined point cloud Ypre2 corresponding to the skeleton point cloud Ypre1 is obtained by Linear and conv operations on the feature vector F7.

[0055] Step 3-2) determining the generated skeleton point cloud based on the discriminator;

[0056] The loss geometry feature decoding loss of the discriminator includes d CD1 and d CD2 , and is weighted using a hyperparameter β:

[0057] L=d CD1 (U1,U'1)+βd CD2 (U2,U'2)

[0058] Where: U1 is the fine point cloud, U2 is the skeleton point cloud, U'1 and U'2 represent the corresponding real point cloud respectively, d CD represents the chamfer distance loss:

[0059]

[0060] First, taking the skeleton point cloud Ypre1 obtained in step 3-2) as input, performing conv, batchnorm, and relu operations respectively to obtain variable Ypre1_64, and repeating twice to obtain variables Ypre1_128 and Ypre1_256 respectively. Secondly, Maxpooling and squeeze function operations are performed on Ypre1_64, Ypre1_128, and Ypre1_256 and merged to obtain the intermediate feature vector Ym1. Finally, the feature vector Ym1 is sequentially passed through four fully connected layers with output dimensions of 256, 128, 16, and 1, and the last fully connected layer outputs 0 or 1.

[0061] Step 3-3) generating fine point cloud features based on skeleton point cloud feature refinement;

[0062] Taking the feature vector F6 corresponding to the final output result of 1 in step 3-2) as input, the refined point cloud Ypre2 corresponding to the skeleton point cloud Ypre1 is obtained by Linear and conv operations.

[0063] Step 3-4) fuse the skeleton point cloud and the refined point cloud and perform secondary discrimination to obtain the final generated point cloud;

[0064] First, taking the skeleton point cloud Ypre2 obtained in step 3-3) as input, performing conv, batchnorm and relu to obtain variable Ypre2_64, repeating twice to obtain variables Ypre2_128 and Ypre2_256, respectively. Secondly, performing Maxpooling and squeeze operation on Ypre2_64, Ypre2_128 and Ypre2_256 and merging to obtain intermediate feature vector Ym2; finally, sequentially passing the feature vector Ym2 through four fully connected layers with output dimensions of 256, 128, 16 and 1, and outputting 0 or 1 through the last fully connected layer.

[0065] Merging the skeleton point cloud Ypre1 corresponding to the final output result of 1 and the corresponding refined point cloud Ypre2 through torch.nn.cat to obtain the final generated point cloud Ypre.

[0066] Step 4) fuse the generated point cloud and the existing point cloud.

[0067] Step 1) obtain the network input point cloud Corresponding to the existing point cloud, there is a missing part, and the generated point cloud Ypre output in step 3-4) is the missing part of the input point cloud generated by the network, and the completion model is obtained by merging through torch.nn.cat.

[0068] The application provides a point cloud completion method based on multi-head attention mechanism feature encoding and double discrimination decoding, wherein for the local feature perception encoder module, the multi-head attention mechanism and the multi-layer perception machine are combined to encode the feature of the existing point cloud, the similar point features of the input point cloud are adaptively aggregated, and the perception of local features in the feature extraction process is improved; the double branch decoder controlled by the discriminator sets the discriminator in the local feature and global feature decoding process, can discriminate the generated skeleton point cloud and fine point cloud based on the input point cloud features, and ensures that the generated point cloud conforms to the global feature and detail feature of the existing point cloud. Therefore, the two modules jointly act to effectively enhance the point cloud shape completion effect, reduce the chamfer distance loss of the generated point cloud, and improve the fidelity of the generated point cloud.

[0069] The application is not limited to the above embodiments, and based on the technical solutions disclosed in the application, those skilled in the art can make some substitutions and deformations to some technical features disclosed in the application without creative labor, and these substitutions and deformations are within the protection scope of the application.

Claims

1. A point cloud completion method using multi-head attention mechanism feature encoding and dual discriminant decoding, characterized in that, Includes the following steps: Step 1) Obtain point cloud data; Step 2) Local feature encoding based on multi-head attention mechanism; Step 2-1) Obtain point cloud features based on a multilayer perceptron; Step 2-1) specifically includes: inputting a set of point cloud data given in step 1). Where N represents the Cartesian coordinate system, MLP operations are performed in the coordinate system to convert unordered point cloud data into matrix information F1, where the size of F1 is... ; Step 2-2) Construct an implicit model of local features based on a multi-head attention mechanism; Step 2-2) specifically includes: First, receiving the acquired feature F1 and setting the downsampling ratio d for the farthest distance point sampling algorithm, and acquiring a value of... The downsampled point cloud feature matrix F2 is obtained, and then the feature F2 is learned using a residual multi-head cross-attention mechanism to obtain the corresponding feature F3. ； Where: f represents summation using LayerNorm for normalization, and F2 represents the input features. The size of feature F3 is ; Feature F3 is further updated using a feedforward network FFN: ； Here, cat() represents the connection operation performed by the torch.nn.cat function; finally, feature F4 is processed by a multilayer perceptron (MLP) and max pooling to obtain the final feature F5; Step 3) Decoding missing geometric features based on a dual discriminator; Step 3-1) Obtain skeleton point cloud features based on encoder feature decoding; Using the feature F5 extracted by the encoder in step 2) as input, the feature vectors F6 and F7 are obtained through two linear layers. The feature vector F7 is then processed through Linear and conv operations to obtain the skeleton point cloud Ypre1. Step 3-2) Determine the generated skeleton point cloud based on the discriminator; First, using the skeleton point cloud Ypre1 obtained in step 3-2) as input, perform conv, batchnorm, and relu operations to obtain the variable Ypre1_64. Repeat this process twice to obtain the variables Ypre1_128 and Ypre1_256 respectively. Second, perform max pooling and squeeze functions on Ypre1_64, Ypre1_128, and Ypre1_256 and merge them to obtain the intermediate feature vector Ym1. Finally, pass the feature vector Ym1 through four fully connected layers with output dimensions of 256, 128, 16, and 1 in sequence. The last fully connected layer outputs either 0 or 1. Step 3-3) Refine and generate detailed point cloud features based on skeleton point cloud features; Using the feature vector F6 corresponding to the final output result of step 3-2) being 1 as input, the refined point cloud Ypre2 corresponding to the skeleton point cloud Ypre1 is obtained through Linear and conv operations; the obtained network input point cloud corresponds to the existing point cloud. Steps 3-4) Fuse the skeleton point cloud and the refined point cloud and perform secondary discrimination to obtain the final generated point cloud; Step 4) Merge the generated point cloud with the existing point cloud.

2. The point cloud completion method based on multi-head attention mechanism feature encoding and dual discriminant decoding according to claim 1, characterized in that, Step 1) specifically includes: acquiring point cloud data using a 3D scanner; for each 3D point cloud model, randomly selecting a viewpoint and choosing one viewpoint as the center of a sphere, deleting all points on the sphere with radius R, and using the remaining points as the missing point cloud as the network input, with the deleted point cloud as the real point cloud, wherein the network input is a set of point cloud data. .

3. The point cloud completion method based on multi-head attention mechanism feature encoding and dual discriminant decoding according to claim 1, characterized in that, Step 3-2) The loss geometric feature decoding loss of the discriminator includes d CD1 and d CD2 And weighted using the hyperparameter β: ； Where: U1 is the precise point cloud, and U2 is the skeleton point cloud. and These represent the corresponding real point clouds, d and d respectively. CD Indicates chamfer distance loss: 。

Images