Diffusion model-based three-dimensional point cloud up-sampling method and system

Abstract

The invention relates to a three-dimensional point cloud up-sampling method and system based on a diffusion model. The method comprises the following steps: constructing and training a three-dimensional point cloud up-sampling network based on the diffusion model; inputting the three-dimensional point cloud into the trained three-dimensional point cloud up-sampling network based on the diffusion model for up-sampling; comprising a forward noise adding process; a reverse denoising process; in the reverse denoising process, position information embedding and input data enhancement are carried out on the noise sample combined with the prompt information, and position embedding information and global enhancement information are obtained; inputting the time step into a time embedding module to obtain time embedding information; then, adding the position embedded information, the global enhancement information and the time embedded information, and inputting the added data into a noise prediction module to realize a preliminary denoising process of a noise-added sample so as to obtain a preliminarily denoised sample; and performing on-channel aggregation on the samples after preliminary denoising to generate a final up-sampling result. According to the invention, a better reconstruction result can be obtained.

Description

Technical Field

[0001] The present invention relates to a three-dimensional point cloud upsampling technology, and in particular to a three-dimensional point cloud upsampling method and system based on a diffusion model, belonging to the technical field of point cloud upsampling. Background Art

[0002] Compared to images, point clouds can directly represent the geometric structure and spatial relationships of objects in three-dimensional space, thus exhibiting unique advantages in many tasks. However, the massive amount of point cloud data places significant pressure on data transmission and storage. Furthermore, inputting high-resolution point clouds into subsequent perception algorithms incurs significant computational costs and places high demands on device performance, limiting their application scenarios. To address these issues, researchers typically downsample 3D point clouds to reduce the performance requirements of computing devices (see C. Chen, H. Yuan, H. Liu, J. Hou, and R. Hamzaoui, “CAS-Net: Cascade attention-based sampling neural network for point cloud simplification,” in Proc. IEEE International Conference on Multimedia and Expo (ICME2023), July 8-14, 2023, Brisbane, Australia). However, for the receiver, upsampling the sparse 3D point cloud to increase its resolution is often necessary to achieve a more comfortable subjective perception experience and more accurate machine cognition results.

[0003] Generally speaking, upsampling methods for 3D point clouds can be divided into two categories: interpolation-based upsampling methods and deep learning-based upsampling methods. Interpolation-based methods generally use interpolation points generated by geometric priors to achieve upsampling (see T.M. Borges, D.C. Garcia, and R.L. de Queiroz, “Fractional super resolution of voxelized point clouds,” IEEE Transactions on Image Processing, vol. 31, pp. 1380-1390, Jan. 2022.). It is generally believed that interpolation-based upsampling methods have relatively low complexity and good performance. However, these methods rely heavily on geometric priors such as normals and curvatures, resulting in poor sampling results in areas with complex structures. Deep learning-based methods, on the other hand, use deep neural networks to directly generate upsampled points (see H.Liu, H.Yuan, J.Hou, R.Hamzaoui, W.Gao, “PUFA-GAN: A frequency-aware generative adversarial network for 3D point cloud upsampling,” IEEE Transactions on Image Processing, vol. 31, pp. 7389-7402, Nov. 2022). Compared with upsampling methods based on interpolation processing, upsampling methods based on deep learning can achieve better upsampling results by leveraging the powerful nonlinear fitting capabilities of neural networks. However, existing upsampling methods usually directly generate the upsampled point cloud representation through a one-step transformation in coordinate space or latent space, which is prone to discontinuities and artifacts, and generally performs poorly in terms of detail preservation. In addition, most existing upsampling methods perform implicit interpolation, which has poor interpretability and is prone to training instability. Therefore, it is necessary to conduct a deeper exploration of the point cloud upsampling task, propose a more stable 3D point cloud upsampling method, and improve the quality and detail fidelity of the upsampled point cloud. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the present invention proposes a three-dimensional point cloud upsampling method based on the diffusion model;

[0005] Unlike existing point cloud upsampling methods, the upsampling method based on the diffusion model proposed in this paper has good interpretability. The diffusion model is based on probability distribution modeling and can explicitly learn the distribution characteristics of point cloud data, thereby generating upsampled point clouds that conform to the real data distribution. This explicit modeling capability makes the diffusion model more stable and reliable in generation tasks. In addition, the generation process of the diffusion model is gradually optimized, which can generate a smoother and more natural point cloud distribution and avoid discontinuities or artifacts. In addition, the core idea of the diffusion model is to generate data through gradual denoising, so it has strong robustness to noise in the input point cloud (which is very beneficial for counteracting the interference of noise introduced during the transmission process). Finally, the diffusion model has a solid theoretical foundation. It is based on theories such as stochastic differential equations (SDE) and variational inference, and has higher credibility and interpretability. Experimental results prove the effectiveness of the proposed scheme.

[0006] The present invention also proposes a three-dimensional point cloud upsampling system based on a diffusion model.

[0007] Explanation of terms:

[0008] 1. PU-Flow (APoint Cloud Upsampling Network With Normalizing Flows): A point cloud upsampling network based on normalizing flows.

[0009] 2. PSEM (Prompt Spatial Expansion Module): Prompt Spatial Expansion Module, which is used to spatially expand the prompt information used in the inverse denoising process to make it the same size as the noisy sample.

[0010] 3. PEM (Positional Embedding Module): Position embedding module, used to embed the position information of each input point during the reverse denoising process.

[0011] 4. GIEM (Global Information Enhancement Module): Global Information Enhancement Module, used to aggregate and enhance input information.

[0012] 5. TEM (Time Embedding Module): Time embedding module, used to embed time step information during the reverse denoising process.

[0013] 6. NPM (Noise Prediction Module): The noise prediction module is the core module of the reverse denoising process, used to predict the noise in the noisy samples.

[0014] 7. CDM (Channel Downsample Module): Channel dimensionality reduction module, used to perform channel dimensionality reduction operation on the output of the noise prediction network to obtain the final upsampling result.

[0015] 8. GELU (Gaussian Error Linear Unit): Gaussian error linear unit activation function, which is a commonly used activation function in deep learning.

[0016] 9. ReLU (Rectified Linear Unit): Linear rectification function, an activation function commonly used in deep learning.

[0017] 10. CD Loss (Chamfer Distance Loss): Chamfer distance loss is a loss function commonly used in deep learning.

[0018] 11. CD (Chamfer Distance): Chamfer distance, a commonly used indicator to measure the similarity between two point clouds. The smaller the value, the more similar the two point clouds are.

[0019] 12. HD (Hausdorff distance): Hausdorff distance is a commonly used indicator to measure the similarity between two point clouds. The smaller the value, the more similar the two point clouds are.

[0020] 13. High Resolution: A term used to describe the clarity of detail in a signal (e.g., an image, point cloud, etc.). Taking point clouds as an example, high-resolution point clouds contain more points per unit volume or surface area, enabling a more detailed description of an object's geometry and surface features.

[0021] 14. Low resolution: A term used to describe the clarity of detail in a signal (e.g., an image, point cloud, etc.). Taking point clouds as an example, low-resolution point clouds contain fewer points per unit volume or surface area and can only describe the general outline or main structure of an object.

[0022] The technical solution of the present invention is:

[0023] A three-dimensional point cloud upsampling method based on a diffusion model, comprising:

[0024] Construct and train a 3D point cloud upsampling network based on the diffusion model; the 3D point cloud upsampling network based on the diffusion model includes a prompt space expansion module (PSEM), a position embedding module (PEM), a global information enhancement module (GIEM), a temporal embedding module (TEM), a noise prediction module (NPM) and a channel dimension reduction module (CDM);

[0025] Input the 3D point cloud into the trained 3D point cloud upsampling network based on the diffusion model for upsampling; including:

[0026] The forward noise addition process includes: gradually adding Gaussian noise to the high-resolution input point cloud to generate a series of noisy samples;

[0027] The reverse denoising process includes: inputting the cue information, i.e., the low-resolution input point cloud, into the cue space expansion module for spatial expansion so that the cue information becomes the same geometric size as the noisy sample;

[0028] The spatially expanded prompt information and the noised samples are connected on the channel, and the connected data are input into the position embedding module and the global information enhancement module respectively, thereby embedding the position information and enhancing the input data, thereby obtaining position embedding information and global enhancement information; at the same time, the time step is input into the time embedding module to obtain time embedding information;

[0029] After that, the position embedding information, global enhancement information and time embedding information are added together and the added data is input into the noise prediction module to implement the preliminary denoising process of the noisy sample and obtain the preliminary denoised sample;

[0030] The samples after preliminary denoising are input into the channel dimensionality reduction module, and the samples after preliminary denoising are aggregated on the channel to generate the final upsampling result.

[0031] According to the preferred embodiment of the present invention, in the forward noise addition process, the high-resolution input point cloud is represented as P I ={(x i ,y i ,z i )|i=1,2,…,N}, where N represents the number of points in the point cloud, (x i ,y i ,z i ) represents the geometric coordinates of a point in space; the high-resolution input point cloud is scaled by a normalization method, and the coordinates of the high-resolution input point cloud are mapped to the range of [0,1] to obtain a normalized point cloud P0;

[0032] After the normalized point cloud P0 undergoes T steps of forward denoising as shown in formula (I), the denoised samples P1, P2, ..., P T ; The specific noise adding process is expressed as:

[0033]

[0034] Among them, β t is the variance of the noise added in the t-th step of the noise addition process, t∈{1,2,…,T}, represents Gaussian distribution, and I is the identity matrix.

[0035] According to the preferred embodiment of the present invention, the noisy sample P at any time t is directly calculated from P0 by the reparameterization technique shown in formula (II). t :

[0036]

[0037] in, is the cumulative noise figure, is standard Gaussian noise.

[0038] According to the preferred embodiment of the present invention, in the reverse denoising process,

[0039] First, the noise sample {P t |t∈{1,2,…,T}} corresponding prompt information, that is, the input point cloud P I Corresponding low-resolution point cloud Get normalized prompt information through normalization method in, Represents the low-resolution point cloud P I low The geometric coordinates of the point in space, r i low Represents the low-resolution point cloud P I low Attribute information, N low Represents the low-resolution point cloud P I low The number of midpoints, N = r·N low , r is the upsampling ratio;

[0040] Then, the normalized prompt information Input into the prompt space expansion module to expand the space and get the prompt information after space expansion and noise samples {P t |t∈{1,2,…,T}} have the same number of points;

[0041] Next, the prompt information after space expansion and the noise sample P t Perform concatenation on the channel to obtain a noise sample combined with prompt information Among them, Concate(·) represents the concatenation operation on the channel;

[0042] Finally, The data is input into the position embedding module and the global information enhancement module to record the position information and enhance the input data, and the position embedding information PE and the global enhancement information GE are obtained.

[0043] According to the preferred embodiment of the present invention, the normalized prompt information Input into the prompt space expansion module to expand the space and get the prompt information after space expansion include:

[0044] The prompt space expansion module normalizes the prompt information Perform zero padding operation; specifically, the normalized prompt information There are N low points, represented by in, Indicates normalized prompt information The geometric coordinates of the point in space, r i low_0 Indicates normalized prompt information Attribute information of

[0045] In the normalized prompt information Each point Insert r-1 all-zero points (0,0,0,0) after the end to get the prompt information after space expansion Expressed as The specific implementation process is shown in formula (III):

[0046]

[0047] Among them, Insert_zero(·,·) represents a zero-filling operation, in which the first input is the point cloud to be expanded, and r is the expansion ratio information.

[0048] According to the present invention, preferably, Input to the position embedding module to record the position information and obtain the position embedding information PE; including:

[0049] The position embedding module consists of two fully connected layers fc1(·), fc2(·) and a GELU activation function layer S Fe (·);

[0050] first, Input a fully connected layer fc1(·) with an input dimension of 7 and an output dimension of 64; then, the output of fc1(·) is input to the GELU activation function layer S Fe (·), introduce nonlinear information; finally, S Fe The output of (·) is input into a fully connected layer fc2(·) with an input dimension of 64 and an output dimension of 256 to obtain the final position embedding PE; the specific implementation process is shown in formula (IV):

[0051]

[0052] According to the present invention, preferably, The global information enhancement module is input to enhance the input data to obtain the global enhanced information GE; including:

[0053] The global information enhancement module (GIEM) consists of three multi-layer perceptrons mlp1(·), mlp2(·), mlp3(·), and the input is The output is GE;

[0054] The first multi-layer perceptron mlp1(·) consists of a one-dimensional convolutional layer with 7 input channels and 64 output channels. A one-dimensional batch normalization layer with 64 input channels and a ReLU activation function layer S re (·); In the first multi-layer perceptron, the input data The number of channels changes from 7 dimensions to 64 dimensions;

[0055] The output of the first multilayer perceptron mlp1(·) is input to the second multilayer perceptron mlp2(·); the second multilayer perceptron mlp2(·) includes a one-dimensional convolutional layer with 64 input channels and 128 output channels. A one-dimensional batch normalization layer with 128 input channels and a ReLU activation function layer S re (·); In the second multi-layer perceptron, the number of channels of the input data is changed from 64 dimensions to 128 dimensions;

[0056] The output of the second multilayer perceptron is input to the third multilayer perceptron; the third multilayer perceptron includes a one-dimensional convolutional layer with 128 input channels and 256 output channels. A one-dimensional batch normalization layer with 128 input channels and a ReLU activation function layer S re (·); In the third multi-layer perceptron, the number of channels of the input data is changed from 128 dimensions to 256 dimensions; the specific implementation process is shown in formula (V):

[0057]

[0058] According to the preferred embodiment of the present invention, in the reverse denoising process,

[0059] Input the current time step t into the time embedding module to obtain the time embedding information TE, After that, the time embedding information TE, the position embedding information PE and the enhanced input data GE are added to obtain the input NP of the denoising network I , NP I =TE+PE+GE;

[0060] NP I Input to the noise prediction module for the current sample P t Noise prediction, get the sample after preliminary denoising

[0061] Will Input into the channel dimensionality reduction module to obtain the final upsampling result

[0062] According to the preferred embodiment of the present invention, the time embedding module includes a sine-cosine time embedding function Scte(·), a fully connected layer fc3(·) and a ReLU activation function layer S re (·);

[0063] The sine-cosine time embedding function Scte(·) maps the time step t to a 256-vector space. The input and output feature dimensions of the fully connected layer fc3(·) are both 256. The fully connected layer fc3(·) performs a linear transformation on the output of the sine-cosine time embedding function. The output of the fully connected layer fc3(·) is input to the ReLU activation function layer S re (·); learn more complex features; the specific implementation process is shown in formula (VI):

[0064] TE=S re (fc3(Scte(t))) (VI).

[0065] According to the preferred embodiment of the present invention, the noise prediction module includes a Transformer decoder; in terms of specific parameter settings, C T =256,N L =4,N H =4, C T Indicates the number of channels of the Transformer decoder, N L Indicates the number of layers of the Transformer decoder, N H represents the number of heads in the multi-head attention module of the Transformer decoder; the specific implementation process is shown in formula (VII):

[0066]

[0067] Here, Tran_d(·) represents the Transformer decoder.

[0068] According to the preferred embodiment of the present invention, the channel dimension reduction module (CDM) includes three one-dimensional convolutional layers Two 1D Batch Normalization layers Two ReLU activation function layers S re (·);

[0069] First, the output of the Transformer decoder Input to the one-dimensional convolution layer In the example, the number of input channels is 256, the number of output channels is 128, and the batch normalization layer is used. and ReLU activation function S re (·);

[0070] Then, the ReLU activation function S re The output of (·) is input to the one-dimensional convolution layer In the example, the number of input channels is 128, the number of output channels is 64, and the batch normalization layer is used. and ReLU activation function layer S re (·);

[0071] Finally, the ReLU activation function layer S re The output of (·) is input to the one-dimensional convolutional layer In the example, the number of input channels is 64, the number of output channels is 3, and the one-dimensional convolution layer is The output is the final upsampling result The specific implementation process is shown in formula (VIII):

[0072]

[0073] According to a preferred embodiment of the present invention, in the three-dimensional point cloud upsampling network based on the diffusion model, the loss function is expressed as shown in formula (IX):

[0074]

[0075] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a three-dimensional point cloud upsampling method based on a diffusion model when executing the computer program.

[0076] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a three-dimensional point cloud upsampling method based on a diffusion model.

[0077] A 3D point cloud upsampling system based on a diffusion model, comprising:

[0078] The module for constructing and training a 3D point cloud upsampling network based on a diffusion model is configured to: construct and train a 3D point cloud upsampling network based on a diffusion model; the 3D point cloud upsampling network based on a diffusion model includes a prompt space expansion module (PSEM), a position embedding module (PEM), a global information enhancement module (GIEM), a temporal embedding module (TEM), a noise prediction module (NPM), and a channel dimension reduction module (CDM);

[0079] The upsampling module is configured to input the 3D point cloud into the trained 3D point cloud upsampling network based on the diffusion model for upsampling; including:

[0080] The forward noise addition process includes: gradually adding Gaussian noise to the high-resolution input point cloud to generate a series of noisy samples;

[0081] The reverse denoising process includes: inputting the cue information, i.e., the low-resolution input point cloud, into the cue space expansion module for spatial expansion so that the cue information becomes the same geometric size as the noisy sample;

[0082] The spatially expanded prompt information and the noised samples are connected on the channel, and the connected data are input into the position embedding module and the global information enhancement module respectively, thereby embedding the position information and enhancing the input data, thereby obtaining position embedding information and global enhancement information; at the same time, the time step is input into the time embedding module to obtain time embedding information;

[0083] After that, the position embedding information, global enhancement information and time embedding information are added together and the added data is input into the noise prediction module to implement the preliminary denoising process of the noisy sample and obtain the preliminary denoised sample;

[0084] The samples after preliminary denoising are input into the channel dimensionality reduction module, and the samples after preliminary denoising are aggregated on the channel to generate the final upsampling result.

[0085] The beneficial effects of the present invention are:

[0086] 1. Unlike existing 3D point cloud geometric upsampling methods that can only use the geometric information of low-resolution point clouds as input, the 3D point cloud upsampling method based on the diffusion model proposed in this invention can input the geometric information and attribute information of low-resolution point clouds into the inverse denoising model as prompt information. The introduction of attribute information effectively improves the accuracy of the upsampling results.

[0087] 2. Compared to the point cloud upsampling network PU-Flow based on normalized flow, the upsampling model proposed in this paper can achieve better reconstruction results. Specifically, when the proposed method is used to upsample the vehicle targets in the KITTI dataset by a factor of 4, the average CD and HD indicators are 0.01556 and 0.04874 respectively. Under the same conditions, when PU-Flow is used to upsample the vehicle targets in the KITTI dataset, the average CD and HD indicators are only 0.01590 and 0.05597 respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 It is a flow chart of the forward denoising process of the three-dimensional point cloud upsampling method based on the diffusion model of the present invention;

[0089] Figure 2 It is a flow chart of the reverse denoising process of the three-dimensional point cloud upsampling method based on the diffusion model of the present invention;

[0090] Figure 3 is a schematic diagram of a spatial extension module (PSEM) used in the present invention;

[0091] Figure 4 is a schematic diagram of a position embedded module (PEM) used in the present invention;

[0092] Figure 5 is a schematic diagram of the Global Information Enhancement Module (GIEM) used in the present invention;

[0093] Figure 6 is a schematic diagram of a time embedding module (TEM) used in the present invention;

[0094] Figure 7 Schematic diagram of the noise prediction module (NPM) used in the present invention;

[0095] Figure 8 Schematic diagram of the channel dimension reduction module (CDM) used in the present invention;

[0096] Figure 9 This is a result image of three-dimensional point cloud upsampling using the method constructed by the present invention. DETAILED DESCRIPTION

[0097] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0098] Example 1

[0099] A three-dimensional point cloud upsampling method based on a diffusion model, comprising:

[0100] Construct and train a 3D point cloud upsampling network based on the diffusion model; the 3D point cloud upsampling network based on the diffusion model includes a prompt space expansion module (PSEM), a position embedding module (PEM), a global information enhancement module (GIEM), a temporal embedding module (TEM), a noise prediction module (NPM) and a channel dimension reduction module (CDM);

[0101] Input the 3D point cloud into the trained 3D point cloud upsampling network based on the diffusion model for upsampling; including:

[0102] The forward noise addition process includes: gradually adding Gaussian noise to the high-resolution input point cloud to generate a series of noisy samples;

[0103] The reverse denoising process includes: inputting the cue information, i.e., the low-resolution input point cloud, into the cue space expansion module for spatial expansion so that the cue information becomes the same geometric size as the noisy sample;

[0104] The spatially expanded prompt information and the noised samples are connected on the channel, and the connected data are input into the position embedding module and the global information enhancement module respectively, thereby embedding the position information and enhancing the input data, thereby obtaining position embedding information and global enhancement information; at the same time, the time step is input into the time embedding module to obtain time embedding information;

[0105] After that, the position embedding information, global enhancement information and time embedding information are added together and the added data is input into the noise prediction module to implement the preliminary denoising process of the noisy sample and obtain the preliminary denoised sample;

[0106] The samples after preliminary denoising are input into the channel dimensionality reduction module, and the samples after preliminary denoising are aggregated on the channel to generate the final upsampling result.

[0107] Example 2

[0108] The difference between the three-dimensional point cloud upsampling method based on the diffusion model described in Example 1 is that:

[0109] In the forward noise addition process, if Figure 1 As shown, the high-resolution input point cloud is represented as P I ={(x i ,y i ,z i )|i=1,2,…,N}, where N represents the number of points in the point cloud. In the present invention, N=120, (x i ,y i ,z i ) represents the geometric coordinates of the point in space; considering the input point cloud P IThe geometric coordinates may be in different ranges. The present invention scales the high-resolution input point cloud through the normalization method from the first step to the fourth step, maps the coordinates of the high-resolution input point cloud to the range of [0,1], and obtains the normalized point cloud P0; in the first step, the center of mass c of the point cloud is calculated using the following formula:

[0110]

[0111] Among them, p i =(x i ,y i ,z i ), is the point cloud P I The i-th point in ;

[0112] In the second step, the input point cloud P is converted into I Every point p in i Subtract the center of mass c to get the translated point cloud P′ I :

[0113] P′ I =Pc;

[0114] Among them, P′ I The i-th point in is denoted as p′ i , which is calculated as: p′ i =p i -c;

[0115] The third step is to calculate P′ using the following formula I Each point p′ i Euclidean distance to the origin and find the maximum radius m:

[0116] m=max i ‖p′ i ‖;

[0117] Among them, ‖p′ i ‖ represents point p′ i Euclidean distance to the origin;

[0118] The fourth step is to use the following formula to translate the translated point cloud P′ I Every point p′ in i Divide by the maximum radius m to get the normalized point cloud P0:

[0119]

[0120] After the normalized point cloud P0 undergoes T steps of forward denoising as shown in formula (I), the denoised samples P1, P2, ..., P T ; In the present invention, T=200. The specific noise adding process is expressed as:

[0121]

[0122] Among them, β t is the variance of the noise added in the t-th step noise adding process, and the present invention takes β t =0.05, t∈{1,2,…,T}, represents Gaussian distribution, and I is the identity matrix.

[0123] By using the reparameterization technique shown in formula (II), the noisy sample P at any time t is calculated directly from P0 t :

[0124]

[0125] in, is the cumulative noise figure, is standard Gaussian noise.

[0126] In the reverse denoising process, if Figure 2 As shown:

[0127] First, the noise sample {P t |t∈{1,2,…,T}} corresponding prompt information, that is, the input point cloud P I Corresponding low-resolution point cloud Get normalized prompt information through normalization method in, Represents the low-resolution point cloud P I low The geometric coordinates of the point in space, r i low Represents the low-resolution point cloud P I low Attribute information, N low Represents the low-resolution point cloud P I low The number of midpoints, the present invention takes N low =30. Generally speaking, N = r·N low , r is the upsampling ratio; in the present invention, r=4.

[0128] Then, the spatial extension module (PSEM) is as follows Figure 3 As shown, the prompt information after normalization Input into the prompt space expansion module to expand the space and get the prompt information after space expansion and noise samples {P t |t∈{1,2,…,T}} have the same number of points;

[0129] Next, the prompt information after space expansion and the noise sample P t Perform concatenation on the channel to obtain a noise sample combined with prompt information Among them, Concate(·) represents the concatenation operation on the channel;

[0130] Finally, The data is input into the position embedding module and the global information enhancement module to record the position information and enhance the input data, and the position embedding information PE and the global enhancement information GE are obtained.

[0131] The prompt information after normalization Input into the prompt space expansion module to expand the space and get the prompt information after space expansion include:

[0132] The prompt space expansion module normalizes the prompt information Perform zero padding operation; specifically, the normalized prompt information There are N low points, represented by in, Indicates normalized prompt information The geometric coordinates of the point in space, r i low_0 Indicates normalized prompt information Attribute information of

[0133] In order to Transform to (r*N low ,4) size, the present invention normalizes the prompt information Each point Insert r-1 all-zero points (0,0,0,0) after the end to get the prompt information after space expansion Expressed as The specific implementation process is shown in formula (III):

[0134]

[0135] Among them, Insert_zero(·,·) represents a zero-filling operation, in which the first input is the point cloud to be expanded, and r is the expansion ratio information.

[0136] Will Input to the position embedding module to record the position information and obtain the position embedding information PE; including:

[0137] Position Embedding Module (PEM) such as Figure 4As shown, the position embedding module consists of two fully connected layers fc1(·), fc2(·) and a GELU activation function layer S Fe (·);

[0138] first, A fully connected layer fc1(·) with an input dimension of 7 and an output dimension of 64 is input; then, the output of fc1(·) is input to the GELU activation function layer S Fe (·), introduce nonlinear information; finally, S Fe The output of (·) is input into a fully connected layer fc2(·) with an input dimension of 64 and an output dimension of 256 to obtain the final position embedding PE; the specific implementation process is shown in formula (IV):

[0139]

[0140] Will The global information enhancement module is input to enhance the input data to obtain the global enhanced information GE; including:

[0141] Global Information Enhancement Module (GIEM) Figure 5 As shown, the global information enhancement module (GIEM) consists of three multi-layer perceptrons mlp1(·), mlp2(·), mlp3(·), and the input is The output is GE;

[0142] The first multi-layer perceptron mlp1(·) consists of a one-dimensional convolutional layer with 7 input channels and 64 output channels. A one-dimensional batch normalization layer with 64 input channels and a ReLU activation function layer S re (·); In the first multi-layer perceptron, the input data The number of channels changes from 7 dimensions to 64 dimensions;

[0143] The output of the first multilayer perceptron mlp1(·) is input to the second multilayer perceptron mlp2(·); the second multilayer perceptron mlp2(·) includes a one-dimensional convolutional layer with 64 input channels and 128 output channels. A one-dimensional batch normalization layer with 128 input channels and a ReLU activation function layer S re (·); In the second multi-layer perceptron, the number of channels of the input data is changed from 64 dimensions to 128 dimensions;

[0144] The output of the second multilayer perceptron is input to the third multilayer perceptron; the third multilayer perceptron includes a one-dimensional convolutional layer with 128 input channels and 256 output channels. A one-dimensional batch normalization layer with 128 input channels and a ReLU activation function layer S re (·); In the third multi-layer perceptron, the number of channels of the input data is changed from 128 dimensions to 256 dimensions; the specific implementation process is shown in formula (V):

[0145]

[0146] In the reverse denoising process,

[0147] In order to record the time information of the diffusion process, the current time step t is input into the time embedding module to obtain the time embedding information TE. After that, the time embedding information TE, the position embedding information PE and the enhanced input data GE are added to obtain the input NP of the denoising network I , NP I =TE+PE+GE;

[0148] NP I Input to the noise prediction module for the current sample P t Noise prediction, get the sample after preliminary denoising

[0149] Will Input into the channel dimensionality reduction module to obtain the final upsampling result In actual use, the noise sample input to the reverse denoising module is generally the sample P after T steps of noise addition. t , that is, random noise, and the time step is t.

[0150] Time embedding module such as Figure 6 As shown, the time embedding module includes a sine-cosine time embedding function Scte(·), a fully connected layer fc3(·) and a ReLU activation function layer S re (·);

[0151] The sine-cosine time embedding function Scte(·) maps the time step t to a 256-vector space so that the model can perceive and process time-related information. The input and output feature dimensions of the fully connected layer fc3(·) are both 256. The fully connected layer fc3(·) performs a linear transformation on the output of the sine-cosine time embedding function; the output of the fully connected layer fc3(·) is input to the ReLU activation function layer S re (·); its role is to introduce nonlinearity, so that the model can learn more complex features; the specific implementation process is shown in formula (VI):

[0152] TE=Sre (fc3(Scte(t))) (VI).

[0153] Noise Prediction Module (NPM) Figure 7 As shown, the noise prediction module includes a Transformer decoder (reference: Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017).); in terms of specific parameter settings, C T =256,N L =4,N H =4, C T Indicates the number of channels of the Transformer decoder, N L Indicates the number of layers of the Transformer decoder, N H represents the number of heads in the multi-head attention module of the Transformer decoder; the specific implementation process is shown in formula (VII):

[0154]

[0155] Here, Tran_d(·) represents the Transformer decoder.

[0156] Channel Dimensionality Reduction Module (CDM) Figure 8 As shown, the channel dimension reduction module (CDM) consists of three one-dimensional convolutional layers Two 1D Batch Normalization layers Two ReLU activation function layers S re (·);

[0157] First, the output of the Transformer decoder Input to the one-dimensional convolution layer In the example, the number of input channels is 256, the number of output channels is 128, and the batch normalization layer is used. and ReLU activation function S re (·);

[0158] Then, the ReLU activation function S re The output of (·) is input to the one-dimensional convolutional layer In the example, the number of input channels is 128, the number of output channels is 64, and the batch normalization layer is used. and ReLU activation function layer S re (·);

[0159] Finally, the ReLU activation function layer Sre The output of (·) is input to the one-dimensional convolutional layer In the example, the number of input channels is 64, the number of output channels is 3, and the one-dimensional convolution layer is The output is the final upsampling result The specific implementation process is shown in formula (VIII):

[0160]

[0161] In the 3D point cloud upsampling network based on the diffusion model, the loss function is expressed as shown in formula (IX):

[0162]

[0163] The effectiveness of the present invention in the 3D point cloud upsampling task and the rationality of the structure used in the present invention are verified through 3D point cloud upsampling experiments and ablation experiments.

[0164] Both experiments are based on vehicle targets in the KITTI target detection dataset, where the low-resolution point cloud is taken from the corresponding vehicle target in the deep feature F4 used in the target detection algorithm VirConv-L (reference: H.Wu, C.Wen, S.Shi, X.Li and C.Wang, "Virtual SparseConvolution for Multimodal 3D Object Detection," 2023IEEE / CVF Conference onComputer Vision and Pattern Recognition(CVPR), Vancouver, BC, Canada, 2023, pp.21653-21662, doi:10.1109 / CVPR52729.2023.02074.), and the high-resolution point cloud is taken from the target detection algorithm VirConv-L (reference: H.Wu, C.Wen, S.Shi, X.Li and C.Wang, "Virtual Sparse Convolution for Multimodal 3D Object Detection," 2023IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR),Vancouver,BC,Canada,2023,pp.21653-21662,doi:10.1109 / CVPR52729.2023.02074.)The corresponding vehicle targets in the multimodal input point cloud used in.

[0165] In order to measure the quality of the upsampled point cloud, the experiment uses chamfer distance (CD) and Hausdorff distance (HD) to measure the similarity between the upsampled point cloud and the high-resolution point cloud. The smaller the chamfer distance (CD) and Hausdorff distance (HD), the higher the quality of the generated point cloud.

[0166] Point cloud upsampling experiment:

[0167] The point cloud upsampling method proposed in this invention is compared with the existing point cloud upsampling network PU-Flow based on normalizing flow (reference: A.Mao, Z.Du, J.Hou, Y.Duan, Y.-J.Liu and Y.He, "PU-Flow: A Point Cloud Upsampling Network With Normalizing Flows," in IEEE Transactions on Visualization and Computer Graphics, vol. 29, no. 12, pp. 4964-4977, Dec. 2023.), and the results are shown in Table 1.

[0168] Table 1 Performance comparison between the 3D point cloud upsampling model proposed in this invention and the currently advanced 3D point cloud upsampling model;

[0169]

[0170] In Table 1, “↓” indicates that the smaller the value, the better the performance. As can be seen from Table 1, the method proposed in the present invention achieves better performance.

[0171] In addition, in order to intuitively demonstrate the effect of the upsampling model proposed in the present invention, Figure 9 A visualization diagram is given. It can be seen that the upsampling model proposed in this invention can obtain better subjective perception results.

[0172] Ablation experiment:

[0173] In order to verify the effectiveness of the network structure used in the present invention, an ablation experiment was conducted, and the results are shown in Table 2.

[0174] Table 2 Ablation experiment results of the present invention;

[0175]

[0176] In Table 2, "↓" indicates that smaller values lead to better performance. As can be seen from Table 2, the model's performance has significantly improved after the introduction of attribute hints. Furthermore, when the diffusion times are too large or too small, model performance degrades. The proposed network structure (Solution 1) achieves optimal performance.

[0177] Example 3

[0178] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the three-dimensional point cloud upsampling method based on the diffusion model described in Example 1 or 2 are implemented.

[0179] Example 4

[0180] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a three-dimensional point cloud upsampling method based on a diffusion model described in embodiment 1 or 2.

[0181] Example 5

[0182] A 3D point cloud upsampling system based on a diffusion model, comprising:

[0183] The module for constructing and training a 3D point cloud upsampling network based on a diffusion model is configured to: construct and train a 3D point cloud upsampling network based on a diffusion model; the 3D point cloud upsampling network based on a diffusion model includes a prompt space expansion module (PSEM), a position embedding module (PEM), a global information enhancement module (GIEM), a temporal embedding module (TEM), a noise prediction module (NPM), and a channel dimension reduction module (CDM);

[0184] The upsampling module is configured to input the 3D point cloud into the trained 3D point cloud upsampling network based on the diffusion model for upsampling; including:

[0185] The forward noise addition process includes: gradually adding Gaussian noise to the high-resolution input point cloud to generate a series of noisy samples;

[0186] The reverse denoising process includes: inputting the cue information, i.e., the low-resolution input point cloud, into the cue space expansion module for spatial expansion so that the cue information becomes the same geometric size as the noisy sample;

[0187] The spatially expanded prompt information and the noised samples are connected on the channel, and the connected data are input into the position embedding module and the global information enhancement module respectively, thereby embedding the position information and enhancing the input data, thereby obtaining position embedding information and global enhancement information; at the same time, the time step is input into the time embedding module to obtain time embedding information;

[0188] After that, the position embedding information, global enhancement information and time embedding information are added together and the added data is input into the noise prediction module to implement the preliminary denoising process of the noisy sample and obtain the preliminary denoised sample;

[0189] The samples after preliminary denoising are input into the channel dimensionality reduction module, and the samples after preliminary denoising are aggregated on the channel to generate the final upsampling result.

Claims

1. A 3D point cloud upsampling method based on a diffusion model, characterized in that: include: Build and train a 3D point cloud upsampling network based on a diffusion model; The 3D point cloud upsampling network based on the diffusion model includes a prompt space expansion module, a position embedding module, a global information enhancement module, a time embedding module, a noise prediction module and a channel dimensionality reduction module; Input the 3D point cloud into the trained 3D point cloud upsampling network based on the diffusion model for upsampling; including: The forward noise addition process includes: gradually adding Gaussian noise to the high-resolution input point cloud to generate a series of noisy samples; The reverse denoising process includes: inputting the cue information, i.e., the low-resolution input point cloud, into the cue space expansion module for spatial expansion so that the cue information becomes the same geometric size as the noisy sample; The spatially expanded prompt information and the noised samples are connected on the channel, and the connected data are input into the position embedding module and the global information enhancement module respectively, thereby embedding the position information and enhancing the input data, thereby obtaining position embedding information and global enhancement information; at the same time, the time step is input into the time embedding module to obtain time embedding information; After that, the position embedding information, global enhancement information and time embedding information are added together and the added data is input into the noise prediction module to implement the preliminary denoising process of the noisy sample and obtain the preliminary denoised sample; The samples after preliminary denoising are input into the channel dimensionality reduction module, and the samples after preliminary denoising are aggregated on the channel to generate the final upsampling result.

2. The three-dimensional point cloud upsampling method based on the diffusion model according to claim 1, characterized in that: In the forward denoising process, the high-resolution input point cloud is represented as P I ={(x i ,y i ,z i )|i=1,2,…,N}, where N represents the number of points in the point cloud, (x i ,y i ,z i ) represents the geometric coordinates of a point in space; the high-resolution input point cloud is scaled by a normalization method, and the coordinates of the high-resolution input point cloud are mapped to the range of [0,1] to obtain a normalized point cloud P0; After the normalized point cloud P0 undergoes T steps of forward denoising as shown in formula (I), the denoised samples P1, P2, ..., P T ; The specific noise adding process is expressed as: Among them, β t is the variance of the noise added in the t-th step of the noise addition process, t∈{1,2,…,T}, N represents the Gaussian distribution, and I is the unit matrix.

3. The three-dimensional point cloud upsampling method based on the diffusion model according to claim 1, characterized in that: By using the reparameterization technique shown in formula (II), the noisy sample P at any time t is calculated directly from P0 t : in, is the cumulative noise figure, is standard Gaussian noise; Further preferably, in the reverse denoising process, First, add the noise sample $P t |t∈{1,2,…,T}% corresponding prompt information, that is, the input point cloud P I Corresponding low-resolution point cloud Get normalized prompt information through normalization method in, Representing low-resolution point clouds The geometric coordinates of a point in space, Representing low-resolution point clouds Attribute information, N low Representing low-resolution point clouds The number of midpoints, N = r·N low , r is the upsampling ratio; Then, the normalized prompt information Input into the prompt space expansion module to expand the space and get the prompt information after space expansion and noise samples $P t |t∈{1,2,…,T}% have the same number of points; Next, the prompt information after space expansion and the noise sample P t Perform concatenation on the channel to obtain a noise sample combined with prompt information Among them, Concate(·) represents the concatenation operation on the channel; Finally, The data is input into the position embedding module and the global information enhancement module to record the position information and enhance the input data, and the position embedding information PE and the global enhancement information GE are obtained.

4. The three-dimensional point cloud upsampling method based on the diffusion model according to claim 1, characterized in that: The prompt information after normalization Input into the prompt space expansion module to expand the space and get the prompt information after space expansion include: The prompt space expansion module normalizes the prompt information Perform zero padding operation; specifically, the normalized prompt information There are N low points, represented by in, Indicates normalized prompt information The geometric coordinates of a point in space, Indicates normalized prompt information Attribute information of In the normalized prompt information Each point Insert r-1 all-zero points (0,0,0,0) after the end to get the prompt information after space expansion Expressed as The specific implementation process is shown in formula (III): Among them, Insert_zero(·,·) represents a zero-filling operation, in which the first input is the point cloud to be expanded, and r is the expansion ratio information.

5. The three-dimensional point cloud upsampling method based on the diffusion model according to claim 3, characterized in that: Will Input to the position embedding module to record the position information and obtain the position embedding information PE; including: The position embedding module consists of two fully connected layers fc1(·), fc2(·) and a GELU activation function layer S ge (·); first, Input a fully connected layer fc1(·) with an input dimension of 7 and an output dimension of 64; then, the output of fc1(·) is input to the GELU activation function layer S ge (·), introduce nonlinear information; finally, S ge The output of (·) is input into a fully connected layer fc2(·) with an input dimension of 64 and an output dimension of 256 to obtain the final position embedding PE; the specific implementation process is shown in formula (IV): More preferably, The global information enhancement module is input to enhance the input data to obtain the global enhanced information GE; including: The global information enhancement module consists of three multi-layer perceptrons mlp1(·), mlp2(·), mlp3(·), and the input is The output is GE; The first multi-layer perceptron mlp1(·) consists of a one-dimensional convolutional layer with 7 input channels and 64 output channels. A one-dimensional batch normalization layer with 64 input channels and a ReLU activation function layer S re (·); In the first multi-layer perceptron, the input data The number of channels changes from 7 dimensions to 64 dimensions; The output of the first multilayer perceptron mlp1(·) is input to the second multilayer perceptron mlp2(·); the second multilayer perceptron mlp2(·) includes a one-dimensional convolutional layer with 64 input channels and 128 output channels. A one-dimensional batch normalization layer with 128 input channels and a ReLU activation function layer S re (·); In the second multi-layer perceptron, the number of channels of the input data is changed from 64 dimensions to 128 dimensions; The output of the second multilayer perceptron is input to the third multilayer perceptron; the third multilayer perceptron includes a one-dimensional convolutional layer with 128 input channels and 256 output channels. A one-dimensional batch normalization layer with 128 input channels and a ReLU activation function layer S re (·); In the third multi-layer perceptron, the number of channels of the input data is changed from 128 dimensions to 256 dimensions; the specific implementation process is shown in formula (V):

6. The three-dimensional point cloud upsampling method based on the diffusion model according to claim 1, characterized in that: In the reverse denoising process, Input the current time step t into the time embedding module to obtain the time embedding information TE, After that, the time embedding information TE, the position embedding information PE and the enhanced input data GE are added to obtain the input NP of the denoising network I , NP I =TE+PE+GE; NP I Input to the noise prediction module for the current sample P t Noise prediction, get the sample after preliminary denoising Will Input into the channel dimensionality reduction module to obtain the final upsampling result Further preferably, the time embedding module includes a sine-cosine time embedding function Scte(·), a fully connected layer fc3(·) and a ReLU activation function layer S re (·); The sine-cosine time embedding function Scte(·) maps the time step t to a 256-vector space. The input and output feature dimensions of the fully connected layer fc3(·) are both 256. The fully connected layer fc3(·) performs a linear transformation on the output of the sine-cosine time embedding function. The output of the fully connected layer fc3(·) is input to the ReLU activation function layer S re (·); Learn more complex features; The specific implementation process is shown in formula (VI): TE=S re (fc3(Scte(t))) (VI).

7. The three-dimensional point cloud upsampling method based on the diffusion model according to claim 1, characterized in that: The noise prediction module includes a Transformer decoder; in terms of specific parameter settings, C T =256,N L =4,N H =4, C T Indicates the number of channels of the Transformer decoder, N L Indicates the number of layers of the Transformer decoder, N H represents the number of heads in the multi-head attention module of the Transformer decoder; the specific implementation process is shown in formula (VII): Where Tran_d(·) represents the Transformer decoder; Further preferably, the channel dimensionality reduction module includes three one-dimensional convolutional layers Two 1D Batch Normalization layers Two ReLU activation function layers S re (·); First, the output of the Transformer decoder Input to the one-dimensional convolutional layer In the example, the number of input channels is 256, the number of output channels is 128, and the batch normalization layer is used. and ReLU activation function S re (·); Then, the ReLU activation function S re The output of (·) is input to the one-dimensional convolutional layer In the example, the number of input channels is 128, the number of output channels is 64, and the batch normalization layer is used. and ReLU activation function layer S re (·); Finally, the ReLU activation function layer S re The output of (·) is input to the one-dimensional convolutional layer In the example, the number of input channels is 64, the number of output channels is 3, and the one-dimensional convolution layer is The output is the final upsampling result The specific implementation process is shown in formula (VIII): Further preferably, in the three-dimensional point cloud upsampling network based on the diffusion model, the loss function is expressed as shown in formula (IX):

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the three-dimensional point cloud upsampling method based on the diffusion model according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a three-dimensional point cloud upsampling method based on a diffusion model are implemented.

10. A 3D point cloud upsampling system based on a diffusion model, characterized in that: include: The module for constructing and training a 3D point cloud upsampling network based on a diffusion model is configured to: construct and train a 3D point cloud upsampling network based on a diffusion model; the 3D point cloud upsampling network based on a diffusion model includes a prompt space expansion module, a position embedding module, a global information enhancement module, a time embedding module, a noise prediction module, and a channel dimensionality reduction module; The upsampling module is configured to input the 3D point cloud into the trained 3D point cloud upsampling network based on the diffusion model for upsampling; including: The forward noise addition process includes: gradually adding Gaussian noise to the high-resolution input point cloud to generate a series of noisy samples; The reverse denoising process includes: inputting the cue information, i.e., the low-resolution input point cloud, into the cue space expansion module for spatial expansion so that the cue information becomes the same geometric size as the noisy sample; The spatially expanded prompt information and the noised samples are connected on the channel, and the connected data are input into the position embedding module and the global information enhancement module respectively, thereby embedding the position information and enhancing the input data, thereby obtaining position embedding information and global enhancement information; at the same time, the time step is input into the time embedding module to obtain time embedding information; After that, the position embedding information, global enhancement information and time embedding information are added together and the added data is input into the noise prediction module to implement the preliminary denoising process of the noisy sample and obtain the preliminary denoised sample; The samples after preliminary denoising are input into the channel dimensionality reduction module, and the samples after preliminary denoising are aggregated on the channel to generate the final upsampling result.

Images