Point cloud denoising method and device based on laplace regularization of depth map

Abstract

The application relates to the technical field of computer vision, in particular to a point cloud denoising method and device based on a Laplacian regularization of a depth map. The method divides a noisy point cloud into multiple patches, and each patch comprises multiple patches; the three-dimensional coordinates of each patch are input into a feature extraction neural network to obtain patch features, and the distance between each group of patch pairs is obtained according to the patch features to obtain a subgraph; the similarity of each group of patch pairs is obtained according to the distance, and then the adjacency matrix and the degree matrix of the subgraph are obtained, and then the graph Laplacian matrix is obtained; the denoising point cloud block of each patch is optimized according to the graph Laplacian matrix and the three-dimensional coordinates of all patches; the denoising point cloud blocks of all patches are processed by using a supersampling aggregation method to obtain the final denoising point cloud. Compared with the prior art, the application has the advantages that the robustness based on an optimization method and the strong learning ability of a data-driven method are considered, and overfitting is not prone to occurring under a small data set.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a point cloud denoising method and apparatus based on depth map Laplacian regularization. Background Technology

[0002] In recent years, with the continuous development of 3D scanning technology, acquiring large-scale, high-precision 3D point cloud data has become a reality. The goal of point cloud denoising is to extract the true geometric shape and structure from noisy point cloud data. This not only helps in more accurately analyzing and understanding objects but also has significant application value in fields such as computer vision, autonomous driving, and virtual reality. However, point cloud data acquired from the real world is often accompanied by various types of noise, such as sensor noise, environmental interference, and sampling errors, which greatly affect the quality and accuracy of the data. Therefore, 3D point cloud denoising, as a key step in point cloud data processing, has received widespread research attention.

[0003] Point cloud denoising has received widespread attention and research from scholars over the past few decades. The main research can be broadly categorized into two types: optimization-based methods and machine learning-based methods. However, previous optimization-based point cloud denoising methods heavily rely on geometric priors, often requiring a trade-off between detail preservation and denoising effectiveness, and mostly only work well in low-noise environments. For large amounts of noise or geometric shapes with sharp edges, over-smoothing can easily occur. Recently popular learning-based point cloud denoising methods require thousands of real noisy point clouds and their noise-free versions to be provided to the network for learning, data that is difficult to obtain. With insufficient data, purely data-driven methods are prone to overtraining specific features of the data. Furthermore, due to the large volume of point cloud data, computational costs are enormous. The irregular structure and massive size of point clouds make low-level structural feature extraction difficult. Graph Laplacian Regularization (GLR) relies on the precise features of surface patches, but existing graph-based networks extract features from each individual point, which is unsuitable for graph construction at the patch level. Compared to point-based graphs, patch-based graphs can more comprehensively represent the features of graph vertices. In patch feature extraction, it is difficult to preserve location point information, thus making it impossible to recover the original samples. Furthermore, graph construction involves pairwise similarity measurements, which require extremely high computational complexity for point clouds. Therefore, how to preserve the robustness of optimization methods while also considering the learning capabilities of data-driven methods, and effectively achieve point cloud denoising even with insufficient data, has become a problem that needs to be solved in this field. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art in achieving point cloud denoising, which makes it difficult to balance the robustness of the optimization method and the strong learning ability of the data-driven method. This invention provides a point cloud denoising method and apparatus based on depth map Laplacian regularization.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] According to a first aspect of the present invention, a point cloud denoising method based on depth map Laplacian regularization is provided, comprising the following steps:

[0007] S1, acquire noisy point cloud data, divide the noisy point cloud data into multiple patches, each patch including multiple patches;

[0008] S2, based on each patch, a feature extraction neural network is used to align the three-dimensional coordinate input points of each patch to obtain patch features, and the distance between each pair of patches is obtained according to the patch features to obtain a sub-map;

[0009] S3, based on the distance between each pair of patches, obtain the similarity metric of each pair of patches, and based on the similarity metric, obtain the adjacency matrix and degree matrix of the subgraph;

[0010] S4. Obtain the graph Laplacian matrix based on the adjacency matrix and the degree matrix;

[0011] S5. Based on the graph Laplacian matrix and the three-dimensional coordinates of all patches, optimize the denoised point cloud for each patch.

[0012] S6. Use the oversampling aggregation method to process the denoised point cloud blocks of all patches to obtain the final denoised point cloud.

[0013] As a preferred technical solution, the point-aligned feature extraction neural network is trained and evaluated using a chamfered distance loss function.

[0014] As a preferred technical solution, the chamfer distance loss function is expressed as:

[0015]

[0016] In the formula, loss represents the chamfer distance loss function. Represents the denoised point cloud U of each patch b and corresponding clean point cloud blocks chamfer distance, V represents the noise point cloud patch. b and ground truth The chamfer distance, C t This is a preset constant.

[0017] As a preferred technical solution, S2 specifically includes: the three-dimensional coordinates of each patch are processed by a multilayer perceptron and a max pooling layer to aggregate the information of all points in each patch and output patch features.

[0018] As a preferred technical solution, the distance between each pair of patches is expressed as:

[0019]

[0020] In the formula, d mn f represents the distance between each pair of patches, k represents the k points closest to the patch center in each patch, and f represents the distance between each pair of patches. m and f n These represent the patch characteristics of the m-th patch and the n-th patch, respectively.

[0021] As a preferred technical solution, the similarity metric is expressed as:

[0022]

[0023] In the formula, ω mn ρ represents the similarity measure. m and ρ n Let d represent the degree of the m-th patch and the n-th patch before regularization, respectively. mn This represents the distance between each pair of patches, where r is a preset distance threshold, and r = ∈ C. r , where γ, ∈ and C r All of these are preset constants.

[0024] As a preferred technical solution, the graph Laplacian matrix is ​​obtained by subtracting the degree matrix from the adjacency matrix.

[0025] As a preferred technical solution, the optimization process of the denoised point cloud block is expressed as follows:

[0026]

[0027] In the formula, U * P represents the denoised point cloud block after iterative optimization. Θ Let L represent the 3D coordinates of all patches, L represent the graph Laplacian matrix, V represent the observations, U represent the denoised point cloud variable to be optimized in each iteration, λ is the preset weight parameter and λ≥0, and F is the Frobenius norm.

[0028] As a preferred technical solution, the oversampling aggregation method specifically includes oversampling the edges of each denoised point cloud block, calculating the number of samplings for each point, and averaging the repeatedly sampled points to obtain the final denoised point cloud.

[0029] According to a second aspect of the present invention, a point cloud denoising apparatus based on depth map Laplacian regularization is provided, comprising a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the method described therein.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. This invention embeds Graph Laplacian Regularization (GLR) into a deep learning framework called Graph Laplacian Regularization Network (GLRNet). It leverages the powerful expressiveness of deep neural networks (DNNs) to enhance the prior modeling of GLR. At the same time, it incorporates GLR, a simple and effective denoising prior, into the network, thereby limiting the network's solution space, avoiding overfitting, and obtaining the final denoised point cloud. This GLRNet network has both the robustness of optimization-based methods and the powerful learning ability of data-driven methods, which improves the robustness of denoising performance to small datasets. Even when the amount of data is insufficient, it can still effectively remove noise and outliers from the acquired noisy point cloud and restore a clean point cloud.

[0032] 2. This invention uses a point-aligned feature extraction method to learn accurate graph features. At the same time, it uses adaptive subgraph decomposition and oversampling aggregation to reduce the computational cost of graph construction. It can retain position point information during patch feature extraction, more reliably recover the original sampling, and the performance degradation is negligible. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating the method in an embodiment of the present invention;

[0034] Figure 2 This is a visualization of the comparison between the method and the competing algorithm in the embodiments of the present invention. Detailed Implementation

[0035] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0036] Example

[0037] like Figure 1 As shown, this embodiment provides a point cloud denoising method based on depth map Laplacian regularization, including the following steps:

[0038] Step S1: Obtain noisy point cloud data and divide the noisy point cloud data into multiple patches, each patch comprising multiple subpatches. Specifically:

[0039] For a given noisy point cloud The goal of point cloud denoising is to remove outliers and restore a clean point cloud. To reduce the computational cost of sparse graph processing, I divided the noisy point cloud into B patches. The data is fed into a deep neural network for learning. The low-dimensional manifold model assumes that the point cloud is composed of surface patches located on a low-dimensional manifold. The dimension of the manifold can be used as a regularization term to recover the point cloud. For more efficient computation, the graph Laplacian matrix L is used to approximate the dimension of the manifold.

[0040] make Let v be the coordinate matrix of the point cloud, where v i Let V represent the three-dimensional coordinates of the i-th point. Then the matrix V affected by noise can be simply represented as:

[0041] V = U gt +E (1)

[0042] In the formula, U gt Let E represent the coordinate matrix of the clean point cloud, and let E represent the zero-mean noise matrix.

[0043] For a noisy point cloud V, it is divided into B patches to reduce the computational complexity of sparse graph construction. For each patch, to define the surface patch, M points are first selected using the farthest point sampling method. As the patch center, the k-nearest neighbor algorithm is then used to construct patches, each patch It contains the k points closest to the center of the patch. These patches, sampled from a low-dimensional smooth manifold, are embedded in three-dimensional space. In Chinese, it is called a patch manifold.

[0044] Step S2: Based on each patch, the feature extraction neural network aligns the three-dimensional coordinate input points of each patch to obtain patch features, and the distance between each pair of patches is obtained based on the patch features to obtain a sub-map.

[0045] Specifically:

[0046] To enhance the accuracy of feature extraction, a point-aligned feature extraction network is proposed, which can be applied to the construction of patch maps. It can maintain the correspondence between patch features and point coordinates, so that the recovered point cloud retains the original sampling and avoids information loss.

[0047] Specifically, for each patch, the three-dimensional coordinates of each patch are... Here, Θ∈{x,y,z} represents the three-dimensional coordinates, which serve as the network input. This input passes through three multilayer perceptron layers with 64, 128, and 256 hidden neurons, respectively, to capture point domain information. This information is then fed into a max-pooling layer to aggregate the information from all points in each patch and output the patch features F. * =[f1,…,fM ] T To simplify the implementation of graph construction, the range of graph edges is limited to the K nearest patches around a given patch, instead of all patches, and the distances between these patches are iteratively evaluated. Connecting patches to p... m and p n The distance d between mn Represented as:

[0048]

[0049] In the formula, f m and f n These represent the patch characteristics of the m-th patch and the n-th patch, respectively.

[0050] Step S3: Based on the distance between each pair of patches, obtain the similarity metric for each pair of patches. Based on the similarity metric, obtain the adjacency matrix and degree matrix of the subgraph. Specifically:

[0051] For each given patch pair, the distance d mn It can calculate patch p m and p n Similarity measure between them, i.e., connection patch p m and p n The edge weights of a graph can be defined as:

[0052]

[0053] In the formula, ω mn ρ represents the graph edge weights / similarity measure. m and ρ n Let ρ represent the degree of the m-th patch and the n-th patch before regularization, respectively. m ρ n -1 / γ This is for regularizing the weights, where r is a preset distance threshold, and r = ∈ C. r , where γ, ∈ and C r All of these are preset constants. The graph constructed here is an r-neighborhood graph, meaning that no edge exceeds a distance r.

[0054] Step S4: Obtain the graph Laplacian matrix based on the adjacency matrix and degree matrix. Specifically:

[0055] The subgraph constructed in this embodiment It is defined by each patch as a node and the connections between patches as edges. Given a pair of connected patches p m and p n The edge weight is defined as ω mn This subgraph The adjacency matrix A is an M*M matrix, where the (m,n)th element is ω.mn This subgraph The degree matrix D is a diagonal matrix, and its m-th term is Then, the combined graph Laplace matrix L can be obtained from L = DA.

[0056] Using the aforementioned deep learning network, the Laplacian matrix L is iteratively optimized using features obtained from the learning method.

[0057] Step S5: Optimize the denoised point cloud for each patch based on the graph Laplacian matrix and the 3D coordinates of all patches.

[0058] Let P Θ Let U be the 3D coordinates of all points in all patches, defined as the denoised point cloud. To recover U using graph Laplacian regularization, the optimization formula can be written as a maximum a posteriori problem:

[0059]

[0060] In the formula, U * P represents the denoised point cloud after iterative optimization. The first term on the right side of the equation is the Laplace regularization of the negative logarithmic signal prior graph, and the second term is the fidelity term used to calculate the difference between the observations V and U. λ is a preset weight parameter and λ≥0. Θ Let L represent the 3D coordinates of all patches, L represent the graph Laplacian matrix, V represent the observations, U represent the denoised point cloud variable to be optimized in each iteration, and F is the Frobenius norm.

[0061] Since L depends on the patch in P, the optimization in Equation (4) is non-convex. To obtain an approximate result, in each iteration, L is fixed, U is solved, and then the value of L is updated given U, using the graph construction strategy discussed earlier.

[0062] Step S6: Use the oversampling aggregation method to process the denoised point cloud blocks of all patches to obtain the final denoised point cloud.

[0063] Due to the large-scale sparse graph computation required for GLR optimization, this embodiment proposes a subgraph-based denoising method to overcome the significant computational demands. Specifically, the complete point cloud is divided into B patches. Given a sampling matrix S, a point cloud patch can be defined as V b =S T VS, where b = 1, ..., B, and U b Similarity is defined as point cloud patches after denoising.

[0064] However, subgraph-based methods inevitably introduce errors when constructing the graph. Effective denoising requires constructing a subgraph based on the entire point cloud. This approach ensures comprehensive capture of global structural features while preventing information loss, especially at the boundaries of each subgraph. Therefore, a novel patch aggregation method is proposed to mitigate the impact of boundary discontinuities. Specifically, during the validation process, the noisy cloud patch V... b Oversampling is performed on the edges to improve maximum coverage. Then, the number of samples for each point is calculated, and the samples from the oversampling points are averaged to obtain the final denoised point cloud.

[0065] Furthermore, the aforementioned GLRNet uses chamfer distance as the loss function. Specifically:

[0066] For a given noisy point cloud patch V b Define its corresponding clean point cloud as And its denoised point cloud block is U b For point cloud denoising, chamfer distance (CD) is used as the loss function to evaluate each U. b The denoising result. Definition For U b From any point in the middle Sum of minimum distances:

[0067]

[0068] Similarly, define for From any point in the middle to U b The sum of the minimum distances. Then U b and The chamfer distance between them can be expressed as:

[0069]

[0070] Because it incorporates the GLR prior, the network converges quickly and achieves relatively good initial performance. However, this also limits the scope for network optimization and improvement. Therefore, a new loss function is proposed to enhance the network's learning ability. The chamfer distance between the noisy point cloud and the ground truth is defined as... The final loss function can be expressed as:

[0071]

[0072] In the formula, C t It is a constant.

[0073] Experimental results show that the point cloud denoising method proposed in this embodiment has good performance. Figure 2 The results of this method and competing algorithms are shown in the test results on the PUNet dataset with 2% Gaussian noise. It can be clearly seen that this method is better than the competing algorithms in terms of overall noise removal and is smoother for planes.

[0074] In Table 1, CD represents the chamfer distance. A larger distance indicates a greater difference between the two point cloud sets; a smaller distance indicates better denoising. P2M represents the distance from the point to the mesh. The point cloud is normalized to a unit sphere and tested under Gaussian noise with a quasi-bias of 1%, 2%, and 3% of the boundary sphere radius. As shown in Table 1, our method outperforms the comparison methods in all settings on the PUNet dataset and surpasses other methods in most cases on the ModelNet dataset. Furthermore, our method converges quickly.

[0075] Table 2 compares our method with the state-of-the-art SBD algorithm (fraction-based point cloud denoising algorithm) in a dataset attenuation experiment. Our method used 12.5% ​​of the full dataset (40 samples) as the attenuation rate to create a series of attenuated PUNet datasets, which were then evaluated using Gaussian noise with a standard deviation of 2% and a boundary sphere radius. As shown in Table 2, our method maintains good performance even with a very small dataset, while the SBD method shows a significant performance decline as the dataset size decreases. (Note: The CD and P2M metrics in Tables 1 and 2 are multiplied by 10.) 4 ).

[0076] Table 1 compares this method with related competitive denoising algorithms (using CD and P2M as indicators).

[0077]

[0078] Table 2 compares the performance of our method and the SBD algorithm under dataset decay (using CD as the metric).

[0079]

[0080] Experimental results show that our proposed method outperforms state-of-the-art methods in terms of objective metrics and visual detail preservation. Furthermore, even with limited training data, our method demonstrates strong robustness, significantly outperforming other competing algorithms by 19%.

[0081] Furthermore, this embodiment also provides a point cloud denoising device based on depth map Laplacian regularization, including a memory, a processor, and a program stored in the memory, which implements the aforementioned method when the processor executes the program. The device processor includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from the memory unit into random access memory (RAM). Various programs and data required for device operation can also be stored in the RAM. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus. Multiple components in the device are connected to the I / O interfaces, including: input units, such as a keyboard, mouse, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network interface cards, modems, wireless transceivers, etc. The communication units allow the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks. The processing unit executes the various methods and processes described above, such as the steps in the foregoing embodiments. For example, in some embodiments, the steps in the foregoing embodiments may be implemented as a computer software program tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps described above may be performed. Alternatively, in other embodiments, the CPU may be configured to perform the steps in the foregoing embodiments by any other suitable means (e.g., by means of firmware). The functions described above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SOCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0082] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A point cloud denoising method based on depth map Laplacian regularization, characterized in that, Includes the following steps: S1, acquire noisy point cloud data, divide the noisy point cloud data into multiple patches, each patch including multiple patches; S2, based on each patch, a feature extraction neural network is used to align the three-dimensional coordinate input points of each patch to obtain patch features, and the distance between each pair of patches is obtained according to the patch features to obtain a sub-map; S3, based on the distance between each pair of patches, obtain the similarity metric of each pair of patches, and based on the similarity metric, obtain the adjacency matrix and degree matrix of the subgraph; S4. Obtain the graph Laplacian matrix based on the adjacency matrix and the degree matrix; S5. Based on the graph Laplacian matrix and the three-dimensional coordinates of all patches, optimize the denoised point cloud for each patch. S6. Use the oversampling aggregation method to process the denoised point cloud blocks of all patches to obtain the final denoised point cloud. The point-aligned feature extraction neural network is trained and evaluated using a chamfer distance loss function; The chamfer distance loss function is expressed as: In the formula, This represents the chamfer distance loss function. Denoising cloud of each patch and corresponding ground truth values chamfer distance, Represents noise point cloud patches and ground truth chamfer distance, As a preset constant, For denoising point cloud blocks True value from any point in the middle to the ground The sum of minimum distances ground truth From any point in the cloud to the denoised point The sum of minimum distances.

2. The point cloud denoising method based on depth map Laplacian regularization according to claim 1, characterized in that, Specifically, S2 includes passing the three-dimensional coordinates of each patch through a multilayer perceptron and a max pooling layer, aggregating the information of all points in each patch, and outputting patch features.

3. The point cloud denoising method based on depth map Laplacian regularization according to claim 1, characterized in that, The distance between each pair of patches is represented as: In the formula, d mn This represents the distance between each pair of patches, where k indicates that each patch contains k points closest to its center. and These represent the patch characteristics of the m-th patch and the n-th patch, respectively.

4. The point cloud denoising method based on depth map Laplacian regularization according to claim 1, characterized in that, The similarity metric is expressed as: In the formula, Represents a similarity measure. and Let d represent the degree of the m-th patch and the n-th patch before regularization, respectively. mn This represents the distance between each patch pair, where r is a preset distance threshold. ,in ϵ and All of these are preset constants.

5. The point cloud denoising method based on depth map Laplacian regularization according to claim 1, characterized in that, The graph Laplacian matrix is ​​obtained by subtracting the degree matrix from the adjacency matrix.

6. The point cloud denoising method based on depth map Laplacian regularization according to claim 1, characterized in that, The optimization process for the denoised point cloud blocks is expressed as follows: In the formula, P represents the denoised point cloud block after iterative optimization. Θ This represents the 3D coordinates of all patches, where L represents the graph Laplacian matrix and V represents the observation. Let λ represent the denoised point cloud block variable to be optimized in each iteration, where λ is a preset weight parameter and λ≥0, and F is the Frobenius norm.

7. The point cloud denoising method based on depth map Laplacian regularization according to claim 1, characterized in that, The oversampling aggregation method specifically includes oversampling the edges of each denoised point cloud block, calculating the number of samplings for each point, and averaging the repeatedly sampled points to obtain the final denoised point cloud.

8. A point cloud denoising device based on depth map Laplacian regularization, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-7.

Images