Denoising and complementing method and system for multi-noisy-point single-photon point cloud

Abstract

The invention discloses a multi-noise-point-oriented single photon point cloud denoising and complementing method and system, which is used for converting a single photon original point cloud containing a large amount of noise into clean and complete point cloud data, and comprises the following steps: matching a real point cloud for the single photon original point cloud containing a large amount of noise as a label; performing down-sampling processing on the single-photon original point cloud, extracting features, decoding the features, and gradually generating a complete point cloud by adopting a layered structure from contour to detail; constructing a discriminator to discriminate whether the input complete point cloud is a generation point cloud or a real point cloud; a loss function is constructed, the model is optimized in an adversarial training mode, an optimal model is reserved according to the loss function, and denoising and complementing of the single-photon original point cloud are achieved through the optimal model. According to the invention, based on the characteristic that the number of noisy points of the single-photon original point cloud is large, denoising and complementing of the single-photon point cloud are realized at the same time in an end-to-end manner, and the complementing precision and authenticity are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of single-photon point cloud processing, and more particularly to a single-photon point cloud denoising and completion method and system. BACKGROUND

[0002] Single-photon lidar performs three-dimensional reconstruction of a target by actively emitting a laser pulse to the target object and measuring the time of flight of a single photon. Compared with traditional lidar, single-photon detection has the advantages of high sensitivity, long distance, and low energy consumption, and can still effectively detect in complex environments.

[0003] The high sensitivity of single-photon lidar makes it more sensitive to environmental noise, which leads to problems such as multiple noises and relatively few effective photons when single-photon lidar faces environmental detection in complex scenes. Moreover, even if the single-photon point cloud containing noise is denoised, the obtained point cloud is often incomplete, making it difficult to effectively reconstruct and subsequently detect targets in complex scenes. Therefore, completing the single-photon point cloud is of great significance for subsequent data applications.

[0004] Existing point cloud completion methods mainly have two categories: one is the traditional geometry-based point cloud completion algorithm, which infers the shape of the missing part through the geometric information of the incomplete point cloud. The model is simple and has small computational complexity, but it relies too much on prior information and assumption conditions, and the effect is not good for complex scenes and large missing areas. The other is a deep learning-based point cloud completion algorithm that uses a deep neural network to learn features directly from the point cloud and uses a data-driven approach to achieve end-to-end point cloud completion. However, this method is still prone to overfitting for larger missing areas and produces some noise. Moreover, current methods are mostly proposed and verified for point cloud data obtained by remote sensing, traditional radar, and other devices, and no algorithm has been proposed for single-photon point cloud completion.

[0005] A new grid residual network (GRNet) is used to complete the point cloud (GRNet Gridding Residual Network for Dense Point Cloud Completion). A three-dimensional grid is introduced as an intermediate representation to regularize unordered point clouds. Two differentiable layers, gridding and inverse gridding, are designed to convert between point clouds and 3D grids without losing structural information. A stereoscopic feature sampling layer is proposed to extract features of neighboring points, thereby preserving context information. A 3D convolutional neural network with skip connections is used to complete the missing parts of the point cloud in a coarse-to-fine manner. For point clouds with large missing areas, it is difficult to effectively complete them.

[0006] The PoinTr model (PoinTr Diverse Point Cloud Completion with Geometry-Aware Transformers) is designed based on the Transformer architecture, represents the point cloud as a set of unordered points with position embeddings, converts the point cloud into a point proxy sequence, and converts the point cloud completion task into a set-to-set conversion task. A geometry-aware module is designed to explicitly simulate the local geometric relationship of the point cloud, and an encoder-decoder structure is used to generate the missing point cloud, which consumes a large amount of computing resources and has high computational complexity. SUMMARY

[0007] In view of the defects and needs of the prior art, the present application provides a multi-noise single-photon point cloud denoising and completion method and system, which is applied to a computer device to realize end-to-end single-photon point cloud denoising and completion, and the purpose is to convert a single-photon original point cloud containing a large amount of noise into clean and complete point cloud data.

[0008] The present application is realized by at least one of the following technical solutions.

[0009] A multi-noise single-photon point cloud denoising and completion method comprises the following steps: inputting a single-photon original point cloud to be denoised and completed into a trained optimal denoising and completion model to obtain clean and complete point cloud data. The training of the optimal denoising and completion model comprises the following steps: A data set is constructed, a single-photon original point cloud including three-dimensional coordinates and corresponding intensity values is obtained, and a real point cloud including three-dimensional coordinates is matched for the single-photon original point as a label; The single-photon original point cloud is down-sampled to obtain single-photon point clouds of different resolutions; Features are extracted from the single-photon point clouds of different resolutions, and integrated into a unified global feature vector; The global feature vector is decoded, and a hierarchical structure from outline to detail is used to gradually generate a complete point cloud; A discriminator is constructed to distinguish whether the input point cloud is a generated point cloud or a real point cloud; A loss function is constructed to optimize the model in an adversarial training manner, the optimal model is reserved according to the loss function, and the optimal model is used to realize denoising and completion of the single-photon original point cloud.

[0010] Further, the down-sampling of the single-photon original point cloud comprises: The single-photon original point cloud is down-sampled at different rates in the x and y directions, and an average pooling strategy is used to generate single-photon point clouds of different resolutions.

[0011] Further, the feature extraction comprises: The improved multi-layer perceptron is applied to single-photon point clouds of different resolutions respectively to obtain corresponding one-dimensional implicit feature vectors; The one-dimensional implicit feature vectors are coupled to form a multi-dimensional implicit feature map, and the implicit feature map is mapped to a final global feature vector through linear transformation.

[0012] Further, the hierarchical structure from outline to detail is used to gradually generate a complete point cloud, including: The global feature vector is transmitted layer by layer through three fully connected layers to obtain a deep feature vector, a middle feature vector, and a shallow feature vector in turn; The deep feature vector is projected and decoded to obtain an outline point cloud, the middle feature vector is projected and decoded, and then added to the outline point cloud to obtain a relatively dense point cloud, and the shallow feature vector is projected and decoded, and then added to the relatively dense point cloud to obtain a final complete point cloud with complete density.

[0013] Further, the discriminator includes extracting features of the input point cloud using an improved multi-layer perceptron, and then realizing final classification through three fully connected layers.

[0014] Further, the improved multi-layer perceptron is to apply max pooling to the output of each layer of the original multi-layer perceptron, and connect the pooling results of all layers in the channel dimension to form a feature vector.

[0015] Further, the loss function is composed of a completion loss and an adversarial loss, specifically including: The completion loss is used to measure the shape structure difference between the generated complete point cloud and the real point cloud, and the adversarial loss is used to optimize the generator and the discriminator in an adversarial manner to make the generated point cloud approximate the distribution of the real point cloud, and the final loss function is the weighted sum of the completion loss and the adversarial loss.

[0016] The system for implementing the multi-noise single-photon point cloud denoising and completion method includes: A data set construction module is configured to obtain single-photon original point clouds containing a large amount of noise, and match corresponding real point clouds as labels to construct a data set, wherein the obtained single-photon original point clouds are four-dimensional point cloud data containing three-dimensional coordinates and corresponding intensity values, and the real point clouds are point cloud data containing three-dimensional coordinates; A generation module is configured to gradually generate clean complete point clouds from outline to detail according to the single-photon original point clouds containing a large amount of noise; A discrimination module is configured to discriminate whether the input point cloud is a generated point cloud or a real point cloud, extract features of the input point cloud using an improved multi-layer perceptron, and then realize final classification through three fully connected layers; The training module is configured to construct a loss function, optimize the generator and the discriminator model in an adversarial training manner, and obtain an optimal model according to the loss function.

[0017] Further, the generation module comprises: The down-sampling unit is configured to perform down-sampling processing on the single-photon original point cloud to obtain single-photon point clouds with different resolutions. The feature extraction unit is configured to extract features from the single-photon point clouds with different resolutions and finally integrate the features into a global feature vector. The point generation unit is configured to decode the global feature vector and generate a complete point cloud in stages by using a hierarchical structure from outline to detail.

[0018] The computer device of the present application comprises a memory, a processor and a computer program stored in the memory, and when the computer program is executed on the processor, the method is realized.

[0019] Compared with the prior art, the present application has the following advantages: 1. The technical solution provided by the present application can directly generate a target point cloud from a single-photon original point cloud containing a large amount of noise, and simultaneously realize denoising and completion of the single-photon point cloud in an end-to-end manner.

[0020] 2. The present application extracts features from single-photon point clouds with different resolutions, fully utilizes local information and global information in the single-photon original point cloud, and gradually generates a target point cloud by using a hierarchical structure from outline to detail, thereby achieving high single-photon point cloud completion accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The present application is a kind of multi-noise single-photon point cloud denoising and completion method for the whole flow chart of the embodiment; Figure 2 The network structure diagram of the multi-noise single-photon point cloud denoising and completion method for the embodiment of the present application; Figure 3 The feature extraction unit structure diagram of the embodiment of the present application; Figure 4 The improved multi-layer perceptron structure diagram in the embodiment of the present application; Figure 5 The point generation unit structure diagram in the embodiment of the present application; Figure 6 The discrimination module structure diagram in the embodiment of the present application; Figure 7 The result visualization diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to better understand the above technical solutions, the technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0023] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced without the specific details, other than those described in the specification, and therefore the scope of the present application is not limited to the specific embodiments disclosed in the following description.

[0024] As shown in the figure, a multi-noise point single-photon point cloud denoising and completion method comprises the following steps: Figure 1 Step 1, constructing a data set, obtaining an original single-photon point cloud of letters , including three-dimensional coordinates and corresponding intensity values, matching the original single-photon point cloud data with a real letter point cloud containing three-dimensional coordinates as a label, and dividing the training set, the validation set and the test set in proportion. Wherein is the number of points of the original single-photon point cloud, is the number of points of the real point cloud, represents the point cloud is composed of points, each point is a four-dimensional real value array, represents the point of the point cloud ; represents the point cloud is composed of points, each point is a three-dimensional real value array, represents the point of the point cloud . As an optional embodiment, the definition of the original single-photon point cloud is that for each point satisfies

[0025] , wherein is the number of horizontal pixel points of the single-photon point cloud, is the number of vertical pixel points of the single-photon point cloud, is the number of time windows of the single-photon point cloud, is the intensity resolution of the single-photon point cloud. represents the x coordinate of the point , represents the y coordinate of the point , represents the z coordinate of the point , represents the time coordinate of the point Located in the A time window (equivalent to the z-coordinate) Point The intensity value, Indicates modulus The set of integers, Indicates modulus The set of integers, Indicates modulus The set of integers, Indicates modulus The set of integers.

[0026] In this embodiment, the original point cloud of a single photon The horizontal and vertical pixel counts are 64, the time window is 20, and the number of acquisition frames is 500. for That is, for single-photon primitive point clouds Each point in satisfy .

[0027] Step 2: Construct a generator based on the original single-photon point cloud. Gradually generate clean and complete fine point clouds .

[0028] like Figure 2 As shown, the generator mainly consists of three parts: a downsampling unit, a feature extraction unit, and a point generation unit. In this embodiment, the single-photon point cloud is first... The data is input to the downsampling unit for 2x and 4x downsampling to obtain single-photon point clouds at different resolutions. and ,in Then, single-photon point clouds at different resolutions were analyzed. , and The input is fed into the feature extraction unit for feature extraction, resulting in the final global feature vector. Finally, the global feature vector Point clouds are generated sequentially from outline to detail using point generation units. , and .

[0029] Downsampling unit: Samples the original point cloud of a single photon from the x and y directions. Downsampling is performed. An average pooling strategy is adopted, with the downsampling factor set to [value missing]. The downsampled single-photon point cloud data is then... shaft and The number of pixels on each axis is reduced to the original value. The number of time windows and intensity resolution Keeping the original single-photon point cloud unchanged each The intensity values ​​of all points within the same time window in a region of adjacent pixels are averaged and used to replace the intensity value of that region to obtain the downsampled single-photon point cloud. ;Will each The intensity values ​​of all points within the same time window in a region of adjacent pixels are averaged and used to replace the intensity value of that region to obtain the downsampled single-photon point cloud. The obtained downsampled point cloud, compared to the original single-photon point cloud, has... shaft and The number of pixels on the axis is reduced, while the number of time windows and intensity resolution remain unchanged.

[0030] Feature extraction unit: such as Figure 3 As shown, firstly, the input point cloud is processed... , and By applying the first improved multilayer perceptron (IMLP), the corresponding implicit feature vectors are obtained. , and Then the implicit feature vector , and Concatenate them to form an implicit feature map Finally, the implicit feature maps are processed through a fully connected layer (Linear). Mapped to the final feature vector .

[0031] The structure of the first improved multilayer perceptron is as follows: Figure 4 As shown, max pooling is applied to the output of each layer in the original multilayer perceptron, and the pooling results of all layers are concatenated along the channel dimension to obtain the feature vector. In this embodiment, the feature channel dimensions obtained by the first improved multilayer perceptron are [128-256-512-1024], and therefore, the final concatenation yields the feature vector. .

[0032] Point generation unit: such as Figure 5 As shown, the point generation unit adopts a hierarchical structure from contour to detail, which integrates feature vectors. The shallow feature vectors are obtained by passing the vectors through three fully connected (Linear) layers one by one. Mid-level feature vectors and deep feature vectors Deep feature vectors The contour point cloud is obtained through projection decoding ; middle layer feature vector After projection decoding, the contour point cloud is The relatively dense point cloud is obtained by adding features ; shallow layer feature vector After projection decoding, the relatively dense point cloud is The final complete fine point cloud is obtained by adding features .

[0033] Among them, the projection decoding operation is composed of a fully connected layer (Linear), a linear rectifier function and a convolution layer (Conv) in turn.

[0034] Step three, build a discriminator framework to distinguish whether the input complete point cloud is a generated point cloud (Fake) or a real point cloud (Real).

[0035] As shown in Figure 6 , the discriminator takes the real point cloud The generated fine point cloud as input, uses a second improved multilayer perceptron (IMLP) to extract the features of the input point cloud, and then realizes the final classification through three fully connected layers (Linear). Among them, the feature channel dimension obtained by the MLP in the second improved multilayer perceptron is [64-128-256] in turn, and the feature channel dimension obtained by connecting them together is 448.

[0036] Step four, build a loss function to optimize the model in an adversarial training manner, and save the model with the smallest loss function as the optimal model. The specific training method includes: Use the farthest point sampling to downsample the real point cloud To the same number of point clouds And contour point cloud , respectively And .

[0037] The loss function includes two parts: the completion loss and the adversarial loss , the completion loss is used to measure the shape structure difference between the generated complete point cloud and the real point cloud, and the adversarial loss is used to optimize the generator and discriminator in an adversarial manner, so that the generated point cloud approximates the distribution of the real point cloud, and the final loss function is the weighted sum of the completion loss and the adversarial loss, which is specifically expressed as follows:

[0038]

[0039]

[0040] wherein, and are the weights of the completion loss and the adversarial loss, respectively, and , denotes the surface geometric distance between two point clouds, is a hyper-parameter, is the size of the dataset. denotes the completion loss, denotes the adversarial loss, denotes the output result obtained by inputting the point cloud into the discriminator , denotes the output result obtained by inputting the point cloud into the generator .

[0041] Taking the point cloud as an example, the surface geometric distance is used to represent the geometric shape difference between the underlying surfaces of two point clouds by using the directional distance difference of two point clouds and the same reference point , which is expressed as the weighted sum of the directional distance difference , and the formula is as follows:

[0042]

[0043] wherein, is a hyper-parameter, denotes the norm of a vector, is a generated set of reference points, which is generated in the present practical example by adding Gaussian noise multiple times from the real point cloud , is the number of reference points, is a directional distance function of the reference point to the given point cloud. denotes the generated set of reference points, denotes the surface geometric distance of the point cloud .

[0044] For the point cloud and the reference point , the directional distance function of the two is composed of the distance of the reference point to the point cloud and the direction :

[0045] Let be the reference point in the point cloud , and K-nearest neighbor points of the reference point, distance of the reference point to the point cloud for the reference point with each point weighted sum of distances:

[0046]

[0047] where K represents the number of nearest neighbors, representing the point cloud the Kth nearest neighbor point, representing the reference point the weight of the distance between the reference point and the nearest neighbor point

[0048] the direction is the weighted sum of the vectors from the reference point to each point

[0049] Step five, input the single-photon raw point cloud required to be denoised and completed into the optimal model to obtain clean and complete point cloud data, and the result is shown in Figure 7 .

[0050] In another embodiment, the present embodiment provides a multi-noise single-photon point cloud denoising and completion system, comprising: a data set construction module configured to obtain a single-photon raw point cloud containing a large amount of noise and match a corresponding real point cloud as a label to construct a data set, wherein the obtained single-photon raw point cloud is a four-dimensional point cloud data containing three-dimensional coordinates and corresponding intensity values, and the real point cloud is point cloud data containing three-dimensional coordinates; a generation module configured to generate a clean complete point cloud from the single-photon raw point cloud containing a large amount of noise step by step from outline to detail; a discrimination module configured to discriminate whether an input point cloud is a generated point cloud or a real point cloud, extract features of the input point cloud by using an improved multilayer perceptron, and then realize final classification through three fully connected layers; a training module configured to construct a loss function to optimize the generator and the discriminator model in an adversarial training manner, and obtain an optimal model according to the loss function.

[0051] As an optional implementation manner, the generation module comprises: a downsampling unit configured to perform downsampling processing on the single-photon raw point cloud to obtain single-photon point clouds of different resolutions; a feature extraction unit configured to extract features from the single-photon point clouds of different resolutions and finally integrate the features into a global feature vector; ​​A point generation unit is configured to decode the global feature vector and generate a complete point cloud in stages using a hierarchical structure from outline to detail.

[0052] The above is based on the ideal embodiment of the present application, through the above description, relevant personnel can make various changes and modifications without deviating from the scope of the technical idea of the present application. The technical scope of the present application is not limited to the content of the specification, and must be determined according to the scope of the claims.

Claims

1. A method for denoising and completing multi-noise single-photon point clouds, characterized in that, The process includes the following steps: inputting the original single-photon point cloud that needs to be denoised and completed into the trained optimal denoising and completion model to obtain clean and complete point cloud data; Training the optimal denoising and completion model includes the following steps: Construct a dataset to obtain the original point cloud of a single photon, including its three-dimensional coordinates and corresponding intensity values, and match the original point cloud containing its three-dimensional coordinates as a label for the original single photon. The original single-photon point cloud was downsampled to obtain single-photon point clouds at different resolutions; Features are extracted from single-photon point clouds at different resolutions and integrated into a unified global feature vector. The global feature vector is decoded, and a complete point cloud is generated step by step using a hierarchical structure from outline to detail; Construct a discriminator to determine whether the input point cloud is a generated point cloud or a real point cloud; A loss function is constructed to optimize the model through adversarial training. The optimal model is retained based on the loss function, and the optimal model is used to achieve denoising and completion of the original single-photon point cloud.

2. The method for denoising and completing multi-noise single-photon point clouds according to claim 1, characterized in that, Downsampling of raw single-photon point clouds includes: The original single-photon point cloud was downsampled at different magnifications in the x and y directions, and an average pooling strategy was used to generate single-photon point clouds of different resolutions.

3. The method for denoising and completing multi-noise single-photon point clouds according to claim 1, characterized in that, The extracted features include: An improved multilayer perceptron was applied to single-photon point clouds at different resolutions to obtain corresponding one-dimensional implicit feature vectors. One-dimensional implicit feature vectors are concatenated to form a multi-dimensional implicit feature map, which is then mapped to the final global feature vector through a linear transformation.

4. The method for denoising and completing multi-noise single-photon point clouds according to claim 1, characterized in that, The hierarchical structure, from outline to detail, progressively generates a complete point cloud, including: The global feature vector is passed layer by layer through three fully connected layers to obtain deep feature vector, middle feature vector and shallow feature vector in sequence. The deep feature vectors are projected and decoded to obtain the contour point cloud. The mid-layer feature vectors are projected and decoded, and then summed with the contour point cloud according to their features to obtain a relatively dense point cloud. The shallow feature vectors are projected and decoded, and then summed with the relatively dense point cloud according to their features to obtain the final point cloud with complete density.

5. The method for denoising and completing multi-noise single-photon point clouds according to claim 1, characterized in that, The discriminator includes extracting features from the input point cloud using an improved multilayer perceptron, and then achieving the final classification through three fully connected layers.

6. A method for denoising and completing multi-noise single-photon point clouds according to claim 3 or 5, characterized in that, The improved multilayer perceptron applies max pooling to the output of each layer in the original multilayer perceptron and concatenates the pooling results of all layers along the channel dimension to form a feature vector.

7. The method for denoising and completing multi-noise single-photon point clouds according to claim 1, characterized in that, The loss function consists of two parts: completion loss and adversarial loss, specifically including: The completion loss is used to measure the difference in shape and structure between the generated complete point cloud and the real point cloud, while the adversarial loss is used to optimize the generator and discriminator in an adversarial manner so that the generated point cloud approximates the distribution of the real point cloud. The final loss function is a weighted sum of the completion loss and the adversarial loss.

8. A system for implementing the method for denoising and completing multi-noise single-photon point clouds as described in claim 1, characterized in that, include: The dataset construction module is used to acquire raw single-photon point clouds containing a lot of noise and match them with corresponding real point clouds as labels to construct the dataset. The acquired raw single-photon point clouds are four-dimensional point cloud data containing three-dimensional coordinates and corresponding intensity values, while the real point clouds are point cloud data containing three-dimensional coordinates. The generation module is used to gradually generate a clean, complete point cloud from outline to detail based on the original single-photon point cloud containing a lot of noise. The discrimination module is used to determine whether the input point cloud is a generated point cloud or a real point cloud. It extracts features of the input point cloud using an improved multilayer perceptron and then achieves the final classification through three fully connected layers. The training module is used to construct the loss function to optimize the generator and discriminator models in an adversarial training manner, and obtain the optimal model based on the loss function.

9. The system according to claim 8, characterized in that, The generation module includes: The downsampling unit is used to downsample the raw single-photon point cloud to obtain single-photon point clouds at different resolutions. The feature extraction unit is used to extract features from single-photon point clouds of different resolutions and finally integrate them into a global feature vector. The point generation unit is used to decode the global feature vector and generates a complete point cloud in stages using a hierarchical structure from outline to detail.

10. A computer device, characterized in that, include: A memory and a processor, and a computer program stored in the memory, which, when executed on the processor, implements the method as described in any one of claims 1 to 7.

Images