A Point Cloud Denoising and Enhancement Method Based on Adversarial Learning

Abstract

The adversarial learning-based point cloud denoising enhancement method of the present invention includes the following steps: during the training of a deep point cloud denoising network, a point cloud noise confidence estimation module is embedded into the original point cloud denoising network; noisy point clouds are input into the deep point cloud denoising network embedded with the point cloud noise confidence estimation module; adversarial loss is calculated based on the point cloud confidence vector estimated by the point cloud noise confidence estimation module and the denoised point cloud result predicted by the deep point cloud denoising network; in each iteration, the input point cloud is perturbed according to the adversarial loss, and adversarial examples are generated and added to the training. The adversarial learning-based point cloud denoising enhancement method of the present invention can dynamically generate adversarial examples that are difficult for the model to handle, improve the data diversity of point cloud denoising training, and the enhanced model of the present invention has certain improvements over the benchmark model in the tasks of generating Gaussian noise and simulating lidar sampling noise denoising.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of 3D point cloud denoising, in particular to a point cloud denoising enhancement method based on adversarial learning. BACKGROUND

[0002] Point cloud is a 3D data format that can retain original geometric information without discretization. Due to this characteristic, point cloud is a data representation suitable for many application scenarios such as robots and autonomous driving. However, point cloud is often disturbed by noise during acquisition, which will affect the performance of downstream tasks related to 3D scenes. Existing point cloud denoising methods can be roughly divided into optimization-based methods and deep learning-based methods. Optimization-based methods usually denoise point cloud according to geometric priors, which is easy to cause over-smoothing phenomenon when dealing with point cloud with large noise. Due to the development of deep learning architecture designed for 3D point cloud, deep learning-based point cloud denoising methods have achieved potential performance. Deep learning-based point cloud denoising methods usually denoise point cloud by predicting the offset of points from the latent manifold. Some methods reconstruct point cloud by learning the latent manifold. Most of the existing learning-based point cloud denoising methods train the model with generated Gaussian noise with a specific standard deviation. However, when the pattern of real noise deviates significantly from the training noise, the denoising performance of the model will be weakened. This lack of training mode diversity makes it difficult for deep learning-based denoising models to handle complex noise, especially real noise.

[0003] The recently emerging adversarial learning technology focuses on the robustness of deep learning methods during the learning process. With the rapid development of this technology, models enhanced based on adversarial learning methods have effectively improved the robustness in various tasks in the field of image processing. The basic process of adversarial learning is to perform adversarial attacks on training data, generate corresponding adversarial samples, and add the generated adversarial samples to the model training process to realize adversarial learning.

[0004] Under the basic adversarial learning framework, the training process of adversarial learning can be formally represented as:

[0005]

[0006] Where θ represents the parameters of the model, (x, y) are the input training sample and the corresponding true value respectively, D is the distribution of the training data set, δ and S represent the perturbation of the adversarial attack and the space allowed by the model respectively. L represents the loss function of model training. The FGSM method searches for the most adversarial data by calculating the gradient of the input data to generate adversarial samples, which can be formally represented as:

[0007]

[0008] where represents the generated adversarial data, and represents the step size of the perturbation. This method generates adversarial data by adding perturbations to the input data in the direction that causes the maximum loss. Similar to the FGSM method, the PGD method perturbs the input data in an iterative manner, which can be formalized as:

[0009]

[0010] where Π x+S represents the projection operation of the adversarial perturbation limited in the allowed perturbation space.

[0011] The problem existing in the prior art described above is that in the framework of adversarial learning, there is a certain trade-off relationship between performance and robustness, and when robustness is pursued, the performance of the model on the training distribution will be attenuated to a certain extent. The small noise of the point cloud is usually close to the training distribution (Gaussian distribution), and simply performing adversarial learning on the point cloud denoising task will affect the denoising performance of the point cloud denoising model in the small noise denoising scene.

[0012] The difficulty of solving the prior art is that the point cloud denoising model based on deep learning is difficult to handle complex noise, especially real noise, due to the lack of diversity of the noise mode of the training sample.

[0013] The significance of solving the problem of the prior art is that the diversity of the training sample noise is improved through the point cloud denoising enhancement method based on adversarial learning, so that the enhanced point cloud denoising model can better adapt to the complex point cloud denoising problem in the real scene. SUMMARY

[0014] The present application provides a point cloud denoising enhancement method based on adversarial learning, which improves the diversity of the denoising model training data by introducing adversarial training, thereby improving the denoising performance of the model under different noise modes.

[0015] The technical scheme of the present application is as follows:

[0016] The point cloud denoising enhancement method based on adversarial learning of the present application comprises the following steps: step one, in the process of training the deep point cloud denoising network, embedding a point cloud noise confidence estimation module into the original point cloud denoising network; step two, inputting the noisy point cloud into the deep point cloud denoising network embedded with the point cloud noise confidence estimation module, calculating the adversarial loss according to the point cloud confidence vector estimated by the point cloud noise confidence estimation module and the denoised point cloud result predicted by the deep point cloud denoising network; and step three, perturbing the input point cloud according to the adversarial loss at each iteration to generate adversarial samples for training.

[0017] Preferably, in the above point cloud denoising enhancement method, in step one, the point cloud noise confidence estimation module is constructed to estimate the noise confidence of the input point cloud, and the operation is differentiable.

[0018] Preferably, in the above point cloud denoising enhancement method, in step two, for the input noisy point cloud, it is input into the deep point cloud denoising network embedded with the point cloud noise confidence estimation module, and the adversarial loss calculated according to the point cloud confidence vector is represented as Where C = (c1,..., c n ) is the point cloud confidence vector estimated by the point cloud noise confidence estimation module, which is used to represent the probability that the points in the point cloud are in a low noise state.

[0019] Preferably, in the above point cloud denoising enhancement method, in step three, the adversarial training process improves the diversity of the training data by dynamically generating adversarial samples that are difficult for the model to handle, thereby enhancing the performance of the deep point cloud denoising network. In the training process, the loss of the original flow and the adversarial flow is used to update the model parameters.

[0020] According to the technical solution of the present application, the beneficial effects are:

[0021] The point cloud denoising enhancement method based on adversarial learning of the present application uses adversarial training to attack the point cloud noise confidence estimation module and the denoising module, dynamically obtains noise patterns that are difficult for the current model to handle as adversarial data, improves the diversity of the denoising model training data, and thus improves the denoising performance of the model under different noise patterns. In the point cloud denoising task of generating Gaussian noise and simulating laser radar sampling noise, the enhanced model of the present application has certain improvement compared with the benchmark model. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and serve to explain the present application, and do not constitute an improper limitation of the present application.

[0023] Figure 1 The flow block diagram of the point cloud denoising enhancement method based on adversarial learning of the present application;

[0024] Figure 2 The overall framework diagram of the point cloud denoising enhancement method based on adversarial learning of the present application; and

[0025] Figure 3 The point cloud denoising result schematic diagram of the present application. DETAILED DESCRIPTION

[0026] In order to better understand and illustrate the concept, working principle and inventive effect of the present application, the present application will be described in detail below by specific embodiments in combination with the drawings: In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0027] The working principle of the present application is: a point cloud noise confidence estimation module C is constructed to evaluate the degree of disturbance of the input noisy point cloud by noise point by point, and the point cloud noise confidence estimation module C is embedded in the reference point cloud denoising network. In the training process, the point cloud noise confidence estimation module and the denoising module are applied to the adversarial attack, the adversarial samples with noise patterns difficult to handle by the current model are generated and added to the training process. The point cloud denoising network is enhanced using this adversarial training method, the noise pattern diversity of the training data is improved, and the denoising effect of the point cloud denoising model under different noise patterns is improved.

[0028] Figure 1 Figure 1 The flow chart of the point cloud denoising enhancement method based on adversarial learning of the present application;

[0029] Referring to Figure 1 The point cloud denoising enhancement method based on adversarial learning of the present application mainly includes the following steps:

[0030] Step one S1: in the process of training the deep point cloud denoising network, the point cloud noise confidence estimation module is embedded in the original point cloud denoising network; wherein the point cloud noise confidence estimation module estimates the degree of disturbance of the noisy point cloud by noise point by point.

[0031] In step one S1, the point cloud noise confidence estimation module is also constructed to estimate the noise confidence of the input point cloud, and its operation is differentiable.

[0032] Step two S2: input the noisy point cloud into the deep point cloud denoising network embedded with the point cloud noise confidence estimation module, and calculate the adversarial loss according to the point cloud confidence vector estimated by the point cloud noise confidence estimation module and the denoised point cloud result predicted by the deep point cloud denoising network. The adversarial loss calculated according to the point cloud confidence vector is represented as C=(c1,…,c n ) is the point cloud confidence vector estimated by the point cloud noise confidence estimation module, which is used to represent the probability that the points in the point cloud may be in a low noise state.

[0033] Step three S3: perturb the input point cloud according to the adversarial loss to generate adversarial samples to join the training. In the training process, the loss of the original stream and the adversarial stream is used to update the model parameters.

[0034] In step three S3, the adversarial training process adds model training by dynamically generating adversarial samples that are difficult for the model to handle, improves the distribution diversity of the training data, and thus enhances the performance of the deep point cloud denoising network. In the training process, the loss of the original stream and the adversarial stream is used to update the model parameters.

[0035] Figure 2 The overall framework diagram of the point cloud denoising enhancement method based on adversarial learning of the present application. Generally, the point cloud denoising model based on deep learning extracts point-by-point features through a feature extraction module, and then sends the point-by-point features to a denoising module to train a deep model according to a denoising loss, thereby realizing point cloud denoising. The point cloud denoising enhancement method based on adversarial learning of the present application inserts a point cloud noise confidence estimation module before the point-by-point features of the point cloud are sent to the denoising module, which is used to estimate the noise confidence of the point cloud. Corresponding Figure 1 to steps 1 and 2; based on the estimated point cloud noise confidence and denoising loss, an adversarial sample is generated by an adversarial attack generator and added to the training, and the loss of the original stream and the adversarial stream, including the denoising loss and the confidence loss, is used to update the model parameters, corresponding Figure 1 to step 3.

[0036] Figure 3 The figure shows the comparison of the denoising results of the enhanced denoising network model of the present application and the benchmark denoising network model. Due to the introduction of the adversarial training of the enhancement method of the present application, the diversity of the noise pattern of the training data is improved, and in addition to improving the performance of generating Gaussian noise, the method can better handle the simulated laser radar sampling noise that has not been seen in training.

[0037] Compared with the existing point cloud denoising method based on deep learning, the method of the present application has better denoising performance and can better handle different modes of noise, and is closer to the needs of the real denoising task.

[0038] Evaluation indicators related to denoising performance in the point cloud denoising task.

[0039] Chamfer Distance is a commonly used point cloud reconstruction evaluation index. By calculating the chamfer distance between the denoised point cloud and the true value point cloud, the denoising performance of the point cloud denoising model can be objectively evaluated. To calculate the chamfer distance between the denoised point cloud and the true value point cloud, first find the nearest point in the true value point cloud for each point in the denoised point cloud and calculate the distance. The minimum distance is accumulated and averaged. Similarly, find the nearest point in the denoised point cloud for each point in the true value point cloud and calculate the distance. The minimum distance is accumulated and averaged. The sum of the two averages is the chamfer distance between the denoised point cloud and the true value point cloud.

[0040] P2M Distance measures the distance between the denoised point cloud and the true value 3D model. By calculating the P2M distance between the denoised point cloud and the true value 3D model, the denoising performance of the point cloud denoising model can be objectively evaluated. To calculate the chamfer distance between the denoised point cloud and the true value 3D model, first need to have the 3D model corresponding to the true value point cloud. By calculating the distance between each point in the denoised point cloud and the corresponding face, the P2M distance between the denoised point cloud and the true value 3D model is obtained.

[0041] In the task of generating Gaussian noise (different standard deviations represent the size of the disturbance) point cloud denoising, the comparison chart of the results of the present application and other algorithms is shown in Table 1. The benchmark models are DMR-Denoise and Score-Denoise, and the enhanced models based on the present application are DMR-AT and Score-AT. From Table 1, it can be found that on the ModelNet-40 and PU data sets, the enhanced models (DMR-AT and Score-AT) of the present application have certain improvement in the chamfer distance and P2M distance, which are the indicators for measuring the point cloud denoising performance, compared with the benchmark models (DMR-Denoise and Score-Denoise).

[0042] Table 1 Comparison of results of the present application and other algorithms in the task of generating Gaussian noise point cloud denoising

[0043]

[0044] In Table 1, under different noise sizes, the chamfer distance and P2M distance indicators of the blackened part using the optimized model of the present application are better than those of the model without using the present application, which can prove that in most cases, the present application has an enhancing effect on the point cloud denoising model based on deep learning.

[0045] Among them, Bilateral and Jet are classical point cloud denoising algorithms based on optimization, and PCNet is a baseline method for point cloud denoising based on deep learning. In order to express simply, the data in the table are multiplied by 10 4 .

[0046] The effectiveness of the present application is also verified through experiments on the task of denoising simulated LiDAR sampling noise point clouds. Table 2 shows the comparison of the denoising performance indicators of the results of the present application and other algorithms on the task of denoising simulated LiDAR sampling noise point clouds, wherein the benchmark models are DMR-Denoise and Score-Denoise, and the enhanced models based on the present application are DMR-AT and Score-AT. As can be seen from Table 2, the enhanced models based on the present application can also obtain certain performance gain on the task of denoising simulated LiDAR sampling noise point clouds.

[0047] Table 2 Comparison of results of the present application and other algorithms on the task of denoising simulated LiDAR sampling noise point clouds

[0048]

[0049] In Table 2, the chamfer distance and P2M distance indicators of the blackened part using the optimized model of the present application are better than those of the model without using the present application under different data sets, which can prove that the present application has an enhancing effect on the point cloud denoising model based on deep learning.

[0050] Among them, Bilateral and Jet are classical point cloud denoising algorithms based on optimization, and PCNet is a point cloud denoising baseline method based on deep learning. In order to express simply, the data in the table are multiplied by 10 4 .

[0051] The above are only preferred embodiments of the present application and are not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0052] English and abbreviation table in the specification

[0053]

[0054]

Claims

1. An adversarial learning-based point cloud denoising enhancement method, characterized in that, The method comprises the following steps: Step one: embedding a point cloud noise confidence estimation module into an original point cloud denoising network in a deep point cloud denoising network training process; and Step two: inputting a noisy point cloud into the deep point cloud denoising network embedded with the point cloud noise confidence estimation module, calculating an adversarial loss according to a point cloud confidence vector estimated by the point cloud noise confidence estimation module and a denoised point cloud result predicted by the deep point cloud denoising network, For the input noisy point cloud, it is input to the depth point cloud denoising network of the embedded point cloud noise confidence estimation module to predict the denoised point cloud result, specifically, the denoised point cloud result predicted by the depth point cloud denoising network is to extract point-by-point features through the feature extraction module, and after the point-by-point features are sent to the denoising module, the depth model is trained according to the denoising loss to realize point cloud denoising; wherein the adversarial loss calculated according to the point cloud confidence vector is represented as , wherein is the point cloud confidence vector estimated by the point cloud noise confidence estimation module, which is used to represent the probability that the points in the point cloud may be in a low noise state; and Step three: generating adversarial samples by perturbing the input point cloud according to the adversarial loss each time to add the adversarial samples to the training.

2. The point cloud denoising enhancement method of claim 1, wherein, In step one, the point cloud noise confidence estimation module is further constructed to estimate the noise confidence of the input point cloud, and the operation thereof is differentiable.

3. The point cloud denoising enhancement method of claim 1, wherein, In step three, the process of generating the adversarial samples to add the adversarial samples to the training is to generate the adversarial samples that are difficult to process by the dynamic generation model to add the adversarial samples to the model training, improve the distribution diversity of the training data, and thus enhance the performance of the deep point cloud denoising network.

Images