Training method for 3D model completion network, 3D model completion method and device

Abstract

This disclosure provides a training method, a 3D model completion method, and an apparatus for a 3D model completion network, including: pre-constructing a 3D model completion network to be trained; the 3D model completion network to be trained includes a 3D variational autoencoder and a diffusion model; acquiring a 3D model to be trained for network training; inputting the 3D model to be trained into the 3D variational autoencoder, inputting the encoded latent vectors into the diffusion model, performing noise addition and denoising processing on the diffusion model, and then inputting the latent vectors into the decoder to obtain a predicted generated 3D model; based on the predicted generated 3D model, calculating the loss of the 3D variational autoencoder and the diffusion model, and training the 3D model completion network to be trained. The trained 3D model completion network is then used to complete the 3D model to be completed. This ensures the quality of the 3D model while improving production efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of 3D modeling technology, and in particular to a training method for 3D model completion networks, a 3D model completion method, and an apparatus. Background Technology

[0002] 3D (3D) modeling has wide applications in many fields, including animation, game development, architectural design, and industrial manufacturing. In traditional 3D modeling, professional modelers often need to spend a significant amount of time and effort to construct detailed and realistic models. Currently, although scanning modeling technology has solved the modeling efficiency problem to some extent, the output 3D models are usually of poor quality, lack detail, and are not easy to edit and modify later.

[0003] This situation leads to a contradiction: while manual, detailed 3D modeling can guarantee quality, it is extremely inefficient, while fully automated scanning and modeling may sacrifice accuracy. How to ensure both quality and improve efficiency has become an urgent problem to be solved. Summary of the Invention

[0004] This disclosure provides a training method for a 3D model completion network, a 3D model completion method, and an apparatus to address the problem that existing 3D modeling methods cannot guarantee both quality and efficiency.

[0005] To address the aforementioned issues, firstly, a training method for a 3D model completion network is provided, comprising:

[0006] A pre-constructed 3D model completion network is built to be trained; the 3D model completion network includes a 3D Variational Autoencoder and a Diffusion model.

[0007] Obtain the 3D model to be trained for network training;

[0008] The training 3D model is input into the 3D variational autoencoder, the encoded latent vector is input into the diffusion model, and after the diffusion model is processed by adding and removing noise, it is input into the decoder to obtain the predicted generated 3D model.

[0009] Based on the predicted 3D model, loss calculations are performed on the 3D variational autoencoder and the diffusion model, and the 3D model completion network to be trained is trained.

[0010] In any possible implementation of the first aspect, before inputting the 3D model to be trained into the 3D variational autoencoder, the method further includes: scaling the bounding box size of the 3D model to be trained to be as close as possible to a preset value; slicing the sized bounding box into blocks according to preset length, width and height; and inputting the 3D model to be trained into the 3D variational autoencoder, including: inputting valid blocks from the bounding box blocks of the 3D model to be trained into the 3D variational autoencoder.

[0011] In any possible implementation combining the first aspect, the encoded latent vector is input into the diffusion model, subjected to noise addition and denoising processing by the diffusion model, and then input into the decoder, including: in the latent space of the three-dimensional variational autoencoder, adding noise to the encoded latent vector through a noise schedule to obtain a noisy latent vector; performing step-by-step denoising processing on the noisy latent vector to obtain a denoised latent vector, and inputting the denoised latent vector into the decoder; wherein, for each denoising process, the noise latent vector output from the previous stage is input into the three-dimensional U-Net used for denoising at this stage for processing, and the noise latent vector of this stage is output as the input of the next stage of the three-dimensional U-Net, until the last stage of the three-dimensional U-Net processes to obtain the denoised latent vector.

[0012] In any possible implementation combining the first aspect, after the input layer, the encoder of the three-dimensional variational autoencoder further includes a Squeeze-and-Excitation Networks module; the method further includes inputting multiple channel data of the three-dimensional model to be trained during the encoding process into the Squeeze-and-Excitation Networks to process the different channel data and output multi-channel data containing information on the importance of different channels.

[0013] In any possible implementation combining the first aspect, for different representation methods of 3D models, the 3D model completion network to be trained includes a 3D variational autoencoder branch and a shared diffusion model for different 3D model representation methods; inputting the 3D model to be trained into the 3D variational autoencoder, inputting the encoded latent vector into the diffusion model, performing noise addition and denoising processing on the diffusion model, and then inputting it into the decoder to obtain a predicted generated 3D model, includes: determining the representation method of the 3D model to be trained; inputting the 3D model to be trained into the 3D variational autoencoder branch corresponding to the representation method, inputting the encoded latent vector into the shared diffusion model, performing noise addition and denoising processing on the shared diffusion model, and then inputting it into the decoder of the corresponding 3D variational autoencoder branch to obtain a predicted generated 3D model; based on the predicted generated 3D model, performing loss calculation on the 3D variational autoencoder and the diffusion model, and training the 3D model completion network to be trained, includes: based on the predicted generated 3D model, performing loss calculation on the corresponding 3D variational autoencoder branch and the shared diffusion model respectively, and training the 3D model completion network to be trained.

[0014] In any possible implementation of the first aspect, determining the representation method of the three-dimensional model to be trained includes: inputting the three-dimensional model to be trained into a pre-trained three-dimensional model classification network to obtain the representation method of the three-dimensional model to be trained; wherein the three-dimensional model classification network is trained in the following manner: inputting labeled three-dimensional model samples into a pre-constructed classification network model, comparing the output three-dimensional model prediction type with the label carried by the three-dimensional model sample, and training the three-dimensional model classification network based on the comparison result.

[0015] In any possible implementation combining the first aspect, a fully convolutional neural network (FCN) is used for feature extraction in the three-dimensional variational autoencoder; the three-dimensional variational autoencoder includes: a three-dimensional vector quantized variational autoencoder.

[0016] Secondly, a method for completing a 3D model is provided, including:

[0017] Obtain the 3D model to be completed;

[0018] The 3D model to be completed is input into the 3D model completion network to obtain the completed 3D model.

[0019] The three-dimensional model completion network is trained according to the training method of the three-dimensional model completion network as described in the first aspect or any possible implementation in combination with the first aspect; and the three-dimensional model completion network includes a three-dimensional variational autoencoder and a denoising module of a diffusion model.

[0020] Thirdly, a three-dimensional model completion related apparatus is provided, comprising: a three-dimensional model completion network training apparatus that provides functional modules corresponding to the steps of the three-dimensional model completion network training method as described in the first aspect or in any possible embodiment of the first aspect; or a three-dimensional model completion apparatus that provides functional modules corresponding to the steps of the three-dimensional model completion method as described in the second aspect.

[0021] Fourthly, a computer device includes: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, they perform the steps of the three-dimensional model completion network training method as described in the first aspect or in any possible embodiment of the first aspect, or the steps of the three-dimensional model completion method as described in the second aspect.

[0022] The beneficial effects of the embodiments disclosed herein include:

[0023] The 3D model completion network training method, 3D model completion method, and apparatus provided in this disclosure include: pre-constructing a 3D model completion network to be trained; the 3D model completion network to be trained includes a 3D variational autoencoder and a diffusion model; acquiring a 3D model to be trained for network training; inputting the 3D model to be trained into the 3D variational autoencoder, inputting the encoded latent vectors into the diffusion model, performing noise addition and denoising processing on the diffusion model, and then inputting them into the decoder to obtain a predicted generated 3D model; based on the predicted generated 3D model, performing loss calculation on the 3D variational autoencoder and the diffusion model to train the 3D model completion network to be trained; and then using the trained 3D model completion network to complete the 3D model to be completed. The 3D model completion method provided in this disclosure combines manual and artificial intelligence methods. By manually performing some detailed modeling, designers can focus on creating the key parts of the model, and then the trained 3D model completion network automatically identifies and completes the remaining parts of the model, ensuring the quality of the 3D model while improving production efficiency. Attached Figure Description

[0024] Figure 1 A flowchart illustrating a training method for a 3D model completion network provided in this embodiment of the disclosure;

[0025] Figure 2 This is one of the schematic diagrams of the network structure of the three-dimensional model completion network to be trained provided in the embodiments of this disclosure;

[0026] Figure 3 A second schematic diagram of the network structure of the 3D model completion network to be trained provided in an embodiment of this disclosure;

[0027] Figure 4 The third schematic diagram of the network structure of the 3D model completion network to be trained provided in the embodiments of this disclosure;

[0028] Figure 5 Fourth schematic diagram of the network structure of the 3D model completion network to be trained provided in the embodiments of this disclosure;

[0029] Figure 6 A flowchart of a three-dimensional model completion method provided in this disclosure embodiment;

[0030] Figure 7 One of the schematic diagrams of the network structure of the three-dimensional model completion network provided in the embodiments of this disclosure;

[0031] Figure 8 A second schematic diagram of the network structure of the three-dimensional model completion network provided in this embodiment of the disclosure;

[0032] Figure 9 A schematic diagram of a training device for a three-dimensional model completion network provided in an embodiment of this disclosure;

[0033] Figure 10 This is a schematic diagram of a three-dimensional model completion device provided in an embodiment of the present disclosure. Detailed Implementation

[0034] This disclosure provides a training method for a 3D model completion network, a 3D model completion method, and an apparatus. Preferred embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit this disclosure. Furthermore, the embodiments and features described in this application can be combined with each other unless otherwise specified.

[0035] This disclosure provides a training method for a 3D model completion network, such as... Figure 1 As shown, it includes:

[0036] S101. Pre-construct a 3D model completion network to be trained; the 3D model completion network to be trained includes a 3D variational autoencoder (3D VAE) and a diffusion model;

[0037] S102. Obtain the 3D model to be trained for network training;

[0038] S103. Input the 3D model to be trained into a 3D variational autoencoder, input the encoded latent vector into a diffusion model, process the noise and noise reduction of the diffusion model, and then input it into a decoder to obtain the predicted generated 3D model.

[0039] S104. Based on the prediction, generate a 3D model, calculate the loss of the 3D variational autoencoder and the diffusion model, and train the 3D model completion network to be trained.

[0040] In this embodiment, the VAE model is a generative model, and the 3D VAE model can be used to generate 3D models. That is, since the input is a 3D model, the feature extraction modules in the encoder and decoder of the VAE model can be configured to extract features from the 3D model. For example, the input 3D model parameters can be L (length) * H (height) * W (width) * C (number of channels), and a 3D matrix can be used as the convolution kernel to extract features from the 3D model, generating a feature cube. Furthermore, in 3D model processing, each voxel can correspond to a vector.

[0041] Furthermore, in this embodiment of the present disclosure, a Diffusion model is added to the 3D VAE network structure. The Diffusion model is used to add and remove noise to the latent vectors output by the encoder, which can further improve the noise reduction capability of the subsequent 3D model completion network, making the completed 3D model clearer.

[0042] Figure 2 This is a schematic diagram of the network structure of the 3D model completion network to be trained, provided in an embodiment of this disclosure. (See diagram below.) Figure 2 As shown, the 3D model to be trained is input into the input layer of the 3D model, and encoded by the encoder of the 3D VAE network to obtain the latent vector z. The latent vector z is then input into the Diffusion model for noise addition processing to obtain the noisy latent vector Z. T Then Z T The noise reduction process is performed to obtain the noise-reduced latent vector z. The noise-reduced latent vector z is then input into the decoder of the 3D VAE network for decoding. The result is output through the 3D model output layer to obtain the 3D model generated in this prediction.

[0043] Furthermore, the loss can be jointly calculated for the 3D variational autoencoder and the diffusion model. For example, the loss function of the 3D variational autoencoder and the loss function of the diffusion model can be summed to train the 3D model to be trained and complete the network.

[0044] In this embodiment, partial detailed modeling is performed manually: designers can focus on creating key parts of the model, such as detailed surface textures and complex structural components. This allows designers to ensure that the core parts of the model maintain high quality and meet design requirements, while artificial intelligence automatically fills in the unfinished parts with high precision: the trained 3D model completion network automatically identifies and completes the remaining parts of the model. This not only reduces the workload of humans but also ensures the consistency and accuracy of the overall model. By combining human and AI, both the accuracy and quality of 3D modeling are guaranteed, and efficiency is significantly improved, breaking the contradiction of not being able to achieve both efficiency and quality in traditional 3D modeling.

[0045] This disclosure is applicable to various 3D modeling scenarios, including but not limited to film special effects, virtual reality, augmented reality, medical visualization, and product design. Furthermore, the combination of human expertise and AI is not a one-time process, but rather an iterative one. Designers can perform manual fine-tuning after AI automatically fills in the model, and the AI ​​can continue to learn and optimize based on these adjustments. The 3D model completion network training method and 3D model completion method provided in this disclosure can be implemented intermittently during the iterative process of combining human expertise and AI, as needed.

[0046] In another embodiment of this disclosure, a training method for a 3D model completion network is provided, which further includes the following steps before inputting the 3D model to be trained into a 3D variational autoencoder:

[0047] Step 1: By scaling, make the bounding box size of the 3D model to be trained as close as possible to the preset value;

[0048] Step 2: Cut the adjusted bounding box into blocks according to the preset length, width and height;

[0049] Then, in S103, "inputting the three-dimensional model to be trained into the three-dimensional variational autoencoder" can be implemented as follows:

[0050] The effective segments from the bounding box segments of the 3D model to be trained are input into the 3D variational autoencoder.

[0051] In this embodiment of the disclosure, before inputting the 3D model to be trained into the 3D variational autoencoder, preprocessing can be performed on the 3D model to facilitate neural network processing. The bounding box of the 3D model to be trained can be scaled to be as close as possible to a preset value, until no further adjustment is possible, or made equal to a preset value, such as 128*128*128, 1024*1024*1024, etc. However, it should be noted that larger and more detailed data requires higher computing power; the size of the bounding box to be adjusted can be determined according to the actual situation. Then, the size of the segments can be determined, such as 1*1*1, 2*2*2, etc., without limitation.

[0052] Therefore, some of the resulting blocks may be empty blocks that do not contain the 3D model. These empty blocks can be treated as invalid blocks and not input into the model. The valid 3D model blocks to be trained can be input into the 3D variational autoencoder to improve the model training efficiency and accelerate convergence.

[0053] In another embodiment of this disclosure, a training method for a three-dimensional model completion network is provided, which can implement the step S103 of "inputting the encoded latent vector into the diffusion model, performing noise addition and denoising processing on the diffusion model, and then inputting it into the decoder" as follows:

[0054] Step 1: In the latent space of the 3D variational autoencoder, noise is added to the encoded latent vector through noise scheduling to obtain a noisy latent vector;

[0055] Step 2: Perform step-by-step noise reduction on the noisy latent vector to obtain the noise-reduced latent vector, and input the noise-reduced latent vector into the decoder.

[0056] In each noise reduction process, the noise latent vector output from the previous stage is input into the 3D U-Net of the current stage for processing, and the noise latent vector of the current stage is output as the input of the next stage 3D U-Net, until the last stage 3D U-Net process obtains the noise-reduced latent vector.

[0057] In this embodiment, denoising in the Diffusion model can be implemented using 3D U-Net. Each stage of multi-stage denoising can be achieved using 3D U-Net. Except for the first stage, where the noise-added latent vector can be directly input into the first-stage 3D U-Net, each subsequent stage can use the result of the previous stage as input to the current stage's 3D U-Net for processing.

[0058] Furthermore, since 3D models are being processed, 3D U-Net is used to process the three-dimensional data. Three-dimensional data has an additional dimension of information compared to two-dimensional data. During feature extraction, a 3D matrix is ​​used as the convolution kernel to encode the 3D data in the (x, y, z) directions.

[0059] Figure 3 This is another schematic diagram of the network structure of the 3D model completion network to be trained provided in the embodiments of this disclosure. For example... Figure 3 As shown, in the latent space formed by the latent vectors output by the encoder, noise can be added step by step using a noise schedule to obtain the latent vector Z after adding noise. T Then Z TA step-by-step noise reduction process is performed. During noise reduction, multiple stages can be implemented, each stage using 3D U-Net. Figure 3 The latent vector Z in the middle T Hidden vector Z T-1 Taking the first-level noise reduction as an example, Z T and Z T The corresponding noise addition level (e.g., T) is input into the current level of the 3D U-Net, and the output is the noise predicted by the current level of the 3D U-Net, from the latent vector Z. T After removing the noise from the prediction, the next-level latent vector Z is obtained. T-1 The next level Z T-1 To Z T-2 The noise reduction process is similar, and this is repeated step by step until the latent vector z with the last level of noise removed is obtained.

[0060] In another embodiment of this disclosure, a training method for a three-dimensional model completion network is provided. After the input layer, the encoder of the three-dimensional variational autoencoder further includes a Squeeze-and-Excitation Networks (SENet) module.

[0061] The method also includes

[0062] The SENet processes the multi-channel data of the 3D model to be trained during the encoding process, and outputs multi-channel data containing information on the importance of different channels.

[0063] In this embodiment, the 3D model to be trained is typically colored and therefore can have multiple channels (e.g., the three primary color channels (RGB, Red, Green, Blue)). If the importance information of different channels can be determined, the generated 3D model will have higher accuracy. SENet focuses on the relationships between channels and can learn the importance information of each channel. Therefore, an SENet module can be added to the encoder of a 3D variational autoencoder to increase the importance information of multiple channels.

[0064] In implementation, SENet modules can be added between any two layers within the encoder of a 3D variational autoencoder, or immediately after the input layer and adjacent to the encoder. The number of SENet modules can be set according to actual needs and the performance of the device performing neural network training. Figure 4 Taking the addition of an SENet module immediately after the input layer and next to the encoder as an example, this does not limit the scope of this disclosure.

[0065] In another embodiment of this disclosure, a training method for a three-dimensional model completion network is provided. For different representation methods of three-dimensional models, the three-dimensional model completion network to be trained includes a three-dimensional variational autoencoder branch and a shared diffusion model for different representation methods of three-dimensional models.

[0066] Then, “S103, input the 3D model to be trained into a 3D variational autoencoder, input the encoded latent vectors into a diffusion model, perform noise addition and denoising processing on the diffusion model, and then input them into a decoder to obtain the predicted generative 3D model” can be implemented as follows:

[0067] Step 1: Determine the representation method of the 3D model to be trained;

[0068] Step 2: Input the 3D model to be trained into the 3D variational autoencoder branch corresponding to the determined representation method, input the encoded latent vector into the shared diffusion model, after noise addition and noise reduction processing of the shared diffusion model, input it into the decoder of the corresponding 3D variational autoencoder branch to obtain the predicted generated 3D model.

[0069] "S104. Based on the prediction-based generative 3D model, calculate the loss of the 3D variational autoencoder and the diffusion model, and train the completion network of the 3D model to be trained," can be implemented as follows:

[0070] Based on the predicted generative 3D model, the loss is calculated for the corresponding 3D variational autoencoder branch and the shared diffusion model, and the network is trained to complete the 3D model to be trained.

[0071] In this embodiment of the disclosure, the 3D model can be represented in various ways, including point clouds, voxels, meshes, and signed distance functions (SDF). For each representation method, a corresponding 3D variational autoencoder branch can be trained, as well as a shared diffusion model common to various representation methods. Figure 5 The diagram illustrates the structure of a 3D variational autoencoder branch and a shared diffusion model. Assuming there are n ways to represent a 3D model, n 3D variational autoencoder branches are set. Taking branch 1 as an example, the 3D model to be trained using this representation method is input into the 3D model input layer 1. After encoding by encoder 1, the latent variable z1 is obtained. The latent variable z1 is then input into the shared diffusion module for noise addition to obtain the noisy latent variable Z. T 1. For Z T 1. Denoising is performed to obtain the denoised latent variable z1. The latent variable z1 is input into decoder 1 for decoding. The decoded 3D model is output through 3D model output layer 1 to obtain the predicted 3D model of the representation method corresponding to the branch 1 of the three-dimensional variational autoencoder.

[0072] In another embodiment of this disclosure, a training method for a 3D model completion network is provided, which can determine the representation method of the 3D model to be trained by means of the following steps:

[0073] The 3D model to be trained is input into a pre-trained 3D model classification network to obtain the representation of the 3D model to be trained.

[0074] The 3D model classification network is trained as follows: labeled 3D model samples are input into a pre-built classification network model, the predicted type of the output 3D model is compared with the label carried by the 3D model sample, and the 3D model classification network is trained based on the comparison result.

[0075] In this embodiment, a 3D model classification network can be trained to determine the representation method of the 3D model to be trained, thereby determining the 3D variational autoencoder branch that the 3D model to be trained needs to input. Any classification network model can be used to train the 3D model classification network, which will not be elaborated here.

[0076] In another embodiment of this disclosure, a training method for a three-dimensional model completion network is provided, in which a fully convolutional neural network (FCN) is used for feature extraction in a three-dimensional variational autoencoder;

[0077] The three-dimensional variational autoencoder includes a three-dimensional vector quantized variational autoencoder (VQ-VAE).

[0078] In related technologies, feature extraction in 3D variational autoencoders is usually implemented through convolutional neural networks (CNNs), but this limits the size of the input 3D model. To improve 3D variational autoencoders, CNNs in 3D VAEs can be replaced with FCNs.

[0079] Furthermore, in addition to 3D VAE, the three-dimensional variational autoencoder in the embodiments of this disclosure can also be 3D VQ-VAE.

[0080] This disclosure also provides a method for completing a three-dimensional model, such as... Figure 6 As shown, it includes:

[0081] S601. Obtain the 3D model to be completed;

[0082] S602. Input the 3D model to be completed into the 3D model completion network to obtain the completed 3D model.

[0083] The three-dimensional model completion network is trained according to any of the above-mentioned training methods for three-dimensional model completion networks; and the three-dimensional model completion network includes a three-dimensional variational autoencoder and a denoising module of a diffusion model.

[0084] In this embodiment of the disclosure, when the training of the 3D model completion network is completed, the network structure is different from that of the training 3D model completion network. The noise addition process in the diffusion model is removed, so that the latent vector z after VAE encoding can be directly input into the noise reduction module of the diffusion model for noise reduction. Figure 7 This is a schematic diagram of the network structure for completing a running 3D model provided in an embodiment of this disclosure. Figure 7 As shown, the 3D model to be completed can be input into the 3D model input layer of the 3D model completion network. The encoder encodes the latent vector z, and the latent vector z is denoised by the diffusion model to obtain the denoised latent vector z. The denoised latent vector z is input into the decoder for decoding, and the completed 3D model is output through the 3D output model.

[0085] Furthermore, for 3D models to be completed using different representation methods, separate 3D variational autoencoder branches can be constructed, while sharing the denoising processing of the diffusion model, such as... Figure 8 As shown, firstly, the representation method of the 3D model to be completed is determined. Then, the 3D model to be completed is input into the 3D variational autoencoder branch corresponding to its representation method. The encoded latent vectors are input into a shared diffusion model. After denoising processing by the shared diffusion model, the vectors are input into the decoder of the corresponding 3D variational autoencoder branch to obtain the completed 3D model. Alternatively, before inputting the 3D model to be completed into the 3D model completion network, a 3D model classification network can be used to classify the 3D model to be completed to determine the 3D variational autoencoder branch to input it into.

[0086] The 3D model completion method provided in this embodiment combines manual and artificial intelligence methods. It uses manual methods for partial fine modeling: designers can focus on creating the key parts of the model, and then the trained 3D model completion network model can automatically identify and complete the rest of the model, which not only ensures the quality of the 3D model but also improves production efficiency.

[0087] Based on the same disclosed concept, this disclosure also provides a training device for a 3D model completion network and a 3D model completion device. Since the principles by which these devices solve problems are similar to the aforementioned training method for a 3D model completion network and the 3D model completion method, the implementation of these devices can refer to the implementation of the aforementioned methods, and the repeated parts will not be described again.

[0088] This disclosure also provides a three-dimensional model completion related device, including: a three-dimensional model completion network training device that provides functional modules corresponding to the steps of any of the above-described three-dimensional model completion network training method embodiments; or a three-dimensional model completion device that provides functional modules corresponding to the steps of any of the above-described three-dimensional model completion method embodiments.

[0089] This disclosure provides a 3D model completion network training device, such as... Figure 9 As shown, it includes:

[0090] Network construction module 901 is used to pre-build a 3D model completion network to be trained; the 3D model completion network to be trained includes a 3D Variational Autoencoder and a Diffusion model.

[0091] The data acquisition module 902 is used to acquire the 3D model to be trained for network training;

[0092] The training module 903 is used to input the 3D model to be trained into the 3D variational autoencoder, input the encoded latent vector into the diffusion model, process the noise and noise reduction of the diffusion model, and then input it into the decoder to obtain the predicted generated 3D model; and based on the predicted generated 3D model, perform loss calculation on the 3D variational autoencoder and the diffusion model, and train the network to complete the 3D model to be trained.

[0093] In yet another embodiment provided in this disclosure, such as Figure 9 As shown, the device further includes: a data preprocessing module 904;

[0094] The data preprocessing module 904 is used to scale the bounding box size of the 3D model to be trained to be as close as possible to a preset value before inputting the 3D variational autoencoder; and to cut the sized bounding box into blocks according to the preset length, width and height.

[0095] The training module 903 is used to input the effective blocks from the bounding box segments of the 3D model to be trained into the 3D variational autoencoder.

[0096] In yet another embodiment provided in this disclosure, such as Figure 9As shown, the training module 903 is used to add noise to the encoded latent vector in the latent space of the three-dimensional variational autoencoder through a noise schedule to obtain a noisy latent vector; the noisy latent vector is then subjected to step-by-step noise reduction processing to obtain a denoised latent vector, and the denoised latent vector is input into the decoder; wherein, for each level of noise reduction processing, the noise latent vector output from the previous level is input into the three-dimensional U-Net of the current level for processing, and the output noise latent vector of the current level is used as the input of the next level of the three-dimensional U-Net, until the last level of the U-Net network processes and obtains the denoised latent vector.

[0097] In yet another embodiment provided in this disclosure, such as Figure 9 As shown, the training module 903 is also used to input multiple channel data of the 3D model to be trained during the encoding process into the compression and excitation network to process the different channel data and output multi-channel data containing different channel importance information; the compression and excitation network module is located after the input layer and within the encoder of the 3D variational autoencoder.

[0098] In yet another embodiment provided in this disclosure, such as Figure 9 As shown, the training module 903 is further used to determine the representation method of the 3D model to be trained; input the 3D model to be trained into the 3D variational autoencoder branch corresponding to the representation method, input the encoded latent vector into the shared diffusion model, perform noise addition and noise reduction processing on the shared diffusion model, and then input it into the decoder of the corresponding 3D variational autoencoder branch to obtain the predicted generated 3D model; and based on the predicted generated 3D model, perform loss calculations on the corresponding 3D variational autoencoder branch and the shared diffusion model respectively, and train the 3D model completion network to be trained; wherein, for different 3D model representation methods, the 3D model completion network to be trained includes 3D variational autoencoder branches and shared diffusion models for different 3D model representation methods.

[0099] In yet another embodiment provided in this disclosure, such as Figure 9 As shown, the training module 903 is further used to input the three-dimensional model to be trained into a pre-trained three-dimensional model classification network to obtain the representation method of the three-dimensional model to be trained; wherein, the three-dimensional model classification network is trained in the following manner: the three-dimensional model sample with label is input into the pre-constructed classification network model, the predicted type of the output three-dimensional model is compared with the label carried by the three-dimensional model sample, and the three-dimensional model classification network is trained according to the comparison result.

[0100] In another embodiment provided in this disclosure, a fully convolutional neural network (FCN) is used for feature extraction in a three-dimensional variational autoencoder;

[0101] The three-dimensional variational autoencoder includes: a three-dimensional vector quantized variational autoencoder.

[0102] This disclosure provides a three-dimensional model completion device, such as... Figure 10 As shown, it includes:

[0103] Module 1001 is used to acquire the 3D model to be completed.

[0104] The completion module 1002 is used to input the 3D model to be completed into the 3D model completion network to obtain the completed 3D model;

[0105] The three-dimensional model completion network is trained according to any of the above-described training methods for three-dimensional model completion networks; and the three-dimensional model completion network includes a three-dimensional variational autoencoder and a denoising module of a diffusion model.

[0106] This disclosure also provides a computer device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of any of the above-described embodiments of the 3D model completion network training method, or the steps of any of the above-described embodiments of the 3D model completion method.

[0107] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of any of the above-described embodiments of the 3D model completion network training method, or the steps of any of the above-described embodiments of the 3D model completion method.

[0108] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented in hardware or by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0109] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes in the drawings are not necessarily essential for implementing this disclosure.

[0110] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0111] The sequence numbers of the embodiments disclosed above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0112] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the spirit and scope of this disclosure. Thus, if such modifications and variations fall within the scope of this disclosure and its equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A training method for a 3D model completion network, characterized in that, include: Pre-build the network to complete the 3D model to be trained; The training 3D model completion network includes a 3D Variational Autoencoder and a Diffusion model. Obtain the 3D model to be trained for network training; The training 3D model is input into the 3D variational autoencoder, the encoded latent vectors are input into the diffusion model, and after the diffusion model is processed by adding and removing noise, it is input into the decoder to obtain the predicted generated 3D model. Based on the predicted generative 3D model, loss calculation is performed on the 3D variational autoencoder and the diffusion model, and the 3D model completion network to be trained is trained. For different representation methods of 3D models, the 3D model completion network to be trained includes a 3D variational autoencoder branch and a shared diffusion model for different 3D model representation methods; The 3D model to be trained is input into the 3D variational autoencoder, and the encoded latent vectors are input into the diffusion model. After noise addition and denoising processing by the diffusion model, the vectors are input into the decoder to obtain the predicted generative 3D model, including: The 3D model to be trained is input into a pre-trained 3D model classification network to obtain a representation method of the 3D model to be trained; the representation method includes: point cloud, voxel, mesh, and signed distance function; The training 3D model is input into the 3D variational autoencoder branch corresponding to the representation method. The encoded latent vector is input into the shared diffusion model. After noise addition and denoising processing by the shared diffusion model, it is input into the decoder of the corresponding 3D variational autoencoder branch to obtain the predicted generated 3D model.

2. The method as described in claim 1, characterized in that, Before inputting the 3D model to be trained into the 3D variational autoencoder, the method further includes: By scaling, the bounding box size of the 3D model to be trained is made as close as possible to the preset value; According to the preset length, width and height, the adjusted bounding box is cut into pieces; Inputting the 3D model to be trained into the 3D variational autoencoder includes: The effective segments from the bounding box segments of the 3D model to be trained are input into the 3D variational autoencoder.

3. The method as described in claim 1, characterized in that, The encoded latent vector is input into the diffusion model, undergoes noise addition and denoising processing by the diffusion model, and then input into the decoder, including: In the latent space of a three-dimensional variational autoencoder, the encoded latent vector is noise-added using a noise scheduler to obtain a noisy latent vector. The noise latent vector is subjected to step-by-step noise reduction processing to obtain the noise-reduced latent vector, and the noise-reduced latent vector is input into the decoder. In this process, for each level of noise reduction, the noise latent vector output from the previous level is input into the 3D U-Net of the current level for processing, and the noise latent vector of the current level is output as the input of the next level of the 3D U-Net, until the last level of the 3D U-Net processes and obtains the noise-reduced latent vector.

4. The method as described in claim 1, characterized in that, Following the input layer, the encoder of the three-dimensional variational autoencoder also includes a Squeeze-and-Excitation Networks module. The method also includes The compression and excitation network processes the multi-channel data of the 3D model to be trained during the encoding process, and outputs multi-channel data containing information on the importance of different channels.

5. The method according to any one of claims 1-4, characterized in that, Based on the predicted generative 3D model, loss calculations are performed on the 3D variational autoencoder and the diffusion model, and the 3D model completion network to be trained is trained, including: Based on the predicted 3D model, loss calculations are performed on the corresponding 3D variational autoencoder branch and the shared diffusion model, respectively, and the network for completing the 3D model to be trained is trained.

6. The method as described in claim 1, characterized in that, The 3D model classification network is trained using the following method: Labeled 3D model samples are input into a pre-built classification network model. The predicted type of the output 3D model is compared with the label carried by the 3D model sample, and the 3D model classification network is trained based on the comparison result.

7. The method as described in claim 1 or 2, characterized in that, In a 3D variational autoencoder, a fully convolutional neural network (FCN) is used for feature extraction. The three-dimensional variational autoencoder includes: a three-dimensional vector quantized variational autoencoder.

8. A method for completing a three-dimensional model, characterized in that, include: Obtain the 3D model to be completed; The 3D model to be completed is input into the 3D model completion network to obtain the completed 3D model. The three-dimensional model completion network is trained according to the training method of the three-dimensional model completion network as described in any one of claims 1-7; and the three-dimensional model completion network includes a three-dimensional variational autoencoder and a denoising module of a diffusion model.

9. A three-dimensional model completion device, characterized in that, include: A 3D model completion network training device provides functional modules corresponding to the steps of the 3D model completion network training method as described in any one of claims 1 to 7; Alternatively, a three-dimensional model completion device corresponding to the functional module of the three-dimensional model completion method as described in claim 8.

10. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the three-dimensional model completion network training method as described in any one of claims 1 to 7, or the steps of the three-dimensional model completion method as described in claim 8.

Images