A point cloud completion method based on continuous standardized flow and adversarial refinement network

Abstract

A point cloud completion method based on continuous standardization flow and an adversarial refinement network, comprising the following steps: S1: inputting a defective point cloud X i to a point cloud completion network based on a continuous flow model, generating a complete shape of the defective point cloud as a coarse-grained completion result X c ; and S2: inputting the defective point cloud X i and the coarse-grained completion result X c to an adversarial refinement network, further refining local details to generate a fine-grained completion result X f with high-quality local details. The present application gradually generates a point cloud completion result with refined details by proposing two sub-networks, a point cloud completion network based on a flow model and an adversarial refinement network, thereby improving the completion quality.

Description

TECHNICAL FIELD

[0001] The technical field of the present application relates to three-dimensional point cloud completion, continuous normalization flow and generative adversarial network, in particular to a point cloud completion method based on continuous normalization flow and adversarial refinement network. BACKGROUND

[0002] With the rapid development of computer vision field, the processing of three-dimensional data also attracts widespread attention. Compared with planar images, three-dimensional data can record the real scene in all directions, store more information, and avoid the problem of viewpoint occlusion in two-dimensional images, which is convenient for users to observe from multiple angles.

[0003] Point cloud is a data set of points in a coordinate system, which contains rich information such as three-dimensional coordinates, color, intensity value, time, etc. Point cloud retains the original geometric information in three-dimensional space, is the original output of many three-dimensional data acquisition devices, and is very simple and flexible, which is an important representation form of three-dimensional data.

[0004] However, due to limited viewing angle, occlusion, sensor resolution and other factors, point cloud data in the real world is often incomplete. The original point cloud data collected by a three-dimensional laser scanner usually has sparse data points and missing in some areas, which greatly hinders the downstream applications such as autonomous driving, scene understanding and augmented reality. Therefore, it is crucial to complete the incomplete point cloud to obtain high-quality complete point cloud.

[0005] Existing deep learning-based point cloud completion methods tend to complete the overall shape of the point cloud, but lack the ability to complete local details. In addition, existing point cloud completion methods tend to complete the overall shape and lack local details. Generally, existing methods rely on point set distance without probability guarantee as loss function, and common point set distance such as chamfer distance lacks the ability to constrain fine details, which often further leads to the loss of local details. SUMMARY

[0006] The present application provides a point cloud completion method based on continuous normalization flow and adversarial refinement network, which can effectively complete the incomplete point cloud, compensate for the loss of structure, and generate high-quality completion results with fine local details.

[0007] The technical scheme of the present application is as follows:

[0008] 1. A point cloud completion method based on continuous normalization flow and adversarial refinement network, comprising the following steps: S1: inputting an incomplete point cloud X i into a point cloud completion network based on a continuous flow model to generate a complete shape of the incomplete point cloud as a coarse-grained completion result X c ; and S2: inputting the incomplete point cloud X i and the coarse-grained completion result Xc The input is further refined in the adversarial refinement network to generate a fine-grained completion result X with high-quality local details f .

[0009] Preferably, in the above point cloud completion method based on continuous normalization flow and adversarial refinement network, wherein step S1 includes a training phase and a testing phase, wherein: in the training phase, the incomplete point cloud X i and the complete point cloud X gt are respectively encoded by the encoder Q φ and Q′ φ with shared weights to obtain the corresponding shape representations z and z′, and a loss function is introduced to constrain the shape representations of both as similar as possible, and then according to the shape representation of the incomplete point cloud, the continuous normalization flow is used as a generator to learn the mapping of the input point cloud X i to the three-dimensional Gaussian distribution Y, and to calculate the loss function reconstruction likelihood, the formula is as follows:

[0010]

[0011] In the testing phase, the incomplete point cloud X i is encoded by the encoder Q φ to obtain the shape representation z, and then according to the shape representation, the continuous normalization flow is used as a generator to generate the complete point cloud from the three-dimensional Gaussian distribution Y as a coarse completion result X c .

[0012] Preferably, in the above point cloud completion method based on continuous normalization flow and adversarial refinement network, in step S2, the incomplete point cloud X i and the coarse-grained completion result X c are used as inputs of the generator G a , the generator uses a U-shaped network (U-Net) with edge pooling and edge de-pooling layers as the architecture, and uses a residual module as the basic module, and the residual module uses a self-attention mechanism to adaptively extract and aggregate local features of different scales.

[0013] Preferably, in the above point cloud completion method based on continuous normalization flow and adversarial refinement network, wherein in step S2, the adversarial refinement network uses the chamfer distance as the loss function, for the fine-grained completion result X f and the complete point cloud true value X gt , the loss function formula is as follows:

[0014]

[0015] The discriminator D aExtracting fine-grained completion result X f And the complete point cloud true value X gt The corresponding feature of the block, the auxiliary chamfer distance constraint completion result of the adversarial loss function is introduced, and the adversarial loss function formula is as follows

[0016]

[0017]

[0018] According to the technical scheme of the application, the beneficial effects are:

[0019] The point cloud completion method based on continuous standardization flow and adversarial refinement network of the application can gradually generate completion results with fine local details in a coarse-grained to fine-grained manner, thereby improving the quality of the completion results, and the application has certain improvement compared with the benchmark model in the point cloud completion task.

[0020] In order to better understand and illustrate the concept, working principle and effect of the application, the application will be described in detail below by combining the drawings and specific examples. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the specific embodiment of the application or the technical scheme in the prior art, the drawings needed to be used in the specific embodiment or the prior art description will be briefly introduced.

[0022] Figure 1 It is the overall framework diagram of the point cloud completion method based on continuous standardization flow and adversarial refinement network of the application;

[0023] Figure 2 It is the schematic diagram of the first stage point cloud completion network based on flow model of the application;

[0024] Figure 3 It is the schematic diagram of the adversarial refinement network of the application;

[0025] Figure 4 It is the result comparison diagram of the application and other algorithms in the point cloud completion task. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the embodiments of the application more clear, the technical scheme in the embodiments of the application will be described clearly and completely below by combining the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0027] The point cloud completion method based on continuous normalized flow and adversarial fine-grained network of the present invention is a point cloud completion method from coarse-grained to fine-grained, which includes two stages: a point cloud completion network based on a continuous flow model and an adversarial fine-grained network.

[0028] Figure 1 This is a flowchart illustrating the point cloud completion method based on continuous normalized flow and adversarial refined networks of the present invention. See also... Figure 1 The method of the present invention mainly includes the following steps:

[0029] S1: Defective Cloud X i : to remove the defects of cloud X i The input is fed into a point cloud completion network based on a continuous flow model to generate the complete shape of the incomplete point cloud as the coarse-grained completion result X. c This step is the first stage of the method of the present invention, namely, the point cloud completion network stage based on the continuous flow model.

[0030] Figure 2 This is a schematic diagram of the first stage of the point cloud completion network based on the flow model in this invention. See also... Figure 2 The point cloud completion network based on the flow model of this invention mainly includes a testing phase and a training phase. Each phase specifically includes the following steps: In the training phase, the residual point cloud X is... i and complete point cloud X gt Encode (residual cloud X) separately using a weighted encoder. i via encoder Q′ φ The complete point cloud is obtained through encoder Q. φ We obtain the corresponding shape representations z and z′, and introduce the loss function. The shape representations of the two are constrained to be as similar as possible. Then, based on the shape representation of the residual cloud, a continuous normalized flow is used as the generator. Learning Input Point Cloud X i The mapping to the three-dimensional Gaussian distribution Y is performed, and the likelihood is reconstructed using the loss function, as shown in the following formula:

[0031]

[0032] During the testing phase, the defective cloud X will be used. i via encoder Q φ The shape representation z is obtained, and a continuous normalized flow is used as the generator based on the shape representation. The learned mapping quasi-mapping generates a complete point cloud as a coarse completion result X, starting from a 3D Gaussian distribution Y. c .

[0033] S2: The remaining defects will be clouded. i And the coarse-grained completion result X obtained in step S1 (first stage) cThe input is further refined in the adversarial refinement network to generate a fine-grained completion result X with high-quality local details f This step is the second stage of the method of the present application, i.e., the adversarial refinement network stage.

[0034] Figure 3 The schematic diagram of the adversarial refinement network, which is the second stage of the present application, is shown in FIG. 3. Referring to FIG. 3, Figure 3 The adversarial refinement network stage of the present application mainly includes the following steps:

[0035] The incomplete point cloud X i and the coarse-grained completion result X c are input into the generator G a , which uses a U-shaped network (U-Net) with edge pooling and edge de-pooling layers as the architecture and uses a residual module as the basic module. The residual module uses a self-attention mechanism to adaptively extract and aggregate local features of different scales. The network uses chamfer distance as the loss function, and for the fine-grained completion result X f and the complete point cloud true value X gt , the formula is as follows:

[0036]

[0037] The discriminator D a extracts the features of the corresponding blocks of the fine-grained completion result X f and the complete point cloud true value X gt and introduces an adversarial loss to constrain the completion result. The formula of the adversarial loss function is as follows:

[0038]

[0039]

[0040] Figure 4 The result comparison chart of the present application and other algorithms in the point cloud completion task is shown in FIG. 4. As can be seen from FIG. 4, Figure 4 the completion result of the present application can effectively complete the complete shape of the incomplete point cloud and has more accurate local details. The method proposed in the present application achieves better results in objective evaluation indicators compared with existing point cloud completion methods.

[0041] The commonly used objective evaluation indicators in point cloud completion are chamfer distance and F-score.

[0042] Chamfer distance measures the average distance of each point in one point cloud to the nearest neighbor point in another point cloud. For two point clouds X and Y, the formula of chamfer distance is as follows:

[0043]

[0044] wherein x and y represent points in point clouds X and Y, respectively.

[0045] The F-score evaluates the distance between object surfaces, which is defined as the harmonic mean between precision and recall. The precision P(d) measures the accuracy of the reconstruction by calculating the percentage of reconstructed points within a certain distance from the ground truth points. The recall R(d) measures the completeness of the reconstruction by calculating the percentage of ground truth points within a certain distance from the reconstructed points. The strictness of the F-score can be controlled by changing the distance threshold d. For the reconstructed point cloud R and the ground truth point cloud T, the corresponding formulae are as follows:

[0046]

[0047]

[0048]

[0049] wherein r and t in the formulae represent points in the reconstructed point cloud R and the ground truth point cloud T, respectively.

[0050] In the point cloud completion task, the comparison results of the results of the present application and other algorithms on 8 common categories are shown in Table 1. It can be found from Table 1 that the present application achieves better results in the chamfer distance and F-score indicators. In order to more clearly show the results, the best results are marked in bold, and the chamfer distance values are enlarged by 10 4 times. The smaller the chamfer distance indicator is, the better it is, and the larger the F-score indicator is, the better it is.

[0051] Table 1 Comparison results of the results of the present application and other algorithms on 8 common categories

[0052]

[0053] The point cloud completion method based on continuous normalized flow and adversarial refinement network of the present application gradually generates point cloud completion results with refined details through the two sub-networks of the flow model-based point cloud completion network and the adversarial refinement network, thereby improving the completion quality.

[0054] The above description is based on the best embodiments of the concept and working principle of the present application. The above embodiments should not be understood as limiting the scope of protection of the present application, and other embodiments and combinations of implementation modes according to the concept of the present application all belong to the protection scope of the present application.

Claims

1. A point cloud completion method based on continuous normalized flow and adversarial refinement network, characterized in that, comprising the steps of: S1: input the incomplete point cloud into a point cloud completion network based on a continuous flow model, generate a complete shape of the incomplete point cloud as a coarse-grained completion result ;​ and S2: input the incomplete point cloud and the coarse-grained completion result into the adversarial refinement network to further refine the local details and generate a fine-grained completion result with high-quality local details , the adversarial refinement network adopts a chamfer distance as a loss function, for the fine-grained completion result and the complete point cloud real value , the loss function formula is as follows: , the discriminator extracts the features of the corresponding blocks of the fine-grained completion result and the complete point cloud real value , and introduces an adversarial loss to assist in chamfer distance constraint of the completion result, wherein the adversarial loss function formula is as follows: , ; Specifically, this adversarial refined network is based on residual cloud. and coarse-grained completion results As a generator The generator uses a U-Net architecture with side pooling and side depooling layers as input, and a residual module as the base module. The residual module uses a self-attention mechanism to adaptively extract and aggregate local features at different scales.

2. The point cloud completion method based on continuous normalized stream and adversarial refinement network according to claim 1, characterized in that, wherein, Step S1 comprises a training phase and a testing phase, wherein: In the training stage, the incomplete point cloud and the complete point cloud are encoded by the encoder and with shared weights respectively to obtain the corresponding shape representations and , a loss function is introduced to constrain the shape representations of both as similar as possible, and then according to the shape representation of the incomplete point cloud, a continuous normalizing flow is used as the generator to learn the mapping of the input point cloud to a three-dimensional Gaussian distribution , and a loss function reconstruction likelihood is calculated, and the formula is as follows: In a test phase, incomplete point clouds are obtained by an encoder to get a shape representation from which a continuous normalizing flow is used as a generator to learn a mapping from a three-dimensional Gaussian distribution to generate a complete point cloud as a coarse completion result .

Images