A high-resolution three-dimensional reconstruction method fusing a diffusion model

Abstract

The application discloses a kind of high-resolution three-dimensional reconstruction methods of fusion diffusion model, belong to image data processing technical field, including steps: constructing original data set D;Enhanced training set is constructed;Three-dimensional reconstruction network is constructed, including text encoder, renderer, VAE encoder, conditional diffusion model, VAE decoder, MVS module;Condition diffusion model is trained and fine-tuned in three stages, obtain three-dimensional reconstruction model, obtain the image sequence and text instruction of the scene to be reconstructed, with three-dimensional reconstruction model is reconstructed.The application can maintain highly consistent geometry and color restoration under multi-view conditions, significantly reduce the splicing artifact.Through semantic guidance optimization, the texture details and structural consistency of the reconstruction model are greatly improved.Conditional diffusion sampling enables the model to accurately restore local details in complex scenes, improving the stability of real-time rendering.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer modeling, and particularly relates to a high-resolution three-dimensional reconstruction method fusing a diffusion model. BACKGROUND

[0002] With the rapid development of deep learning and diffusion models, three-dimensional reconstruction optimization methods based on diffusion models have shown great potential in three-dimensional modeling, computer vision, virtual reality and other fields. In the reconstruction process, it is crucial to ensure the accuracy of the point cloud, and real-time rendering and simulation also require high resolution and consistency to meet the needs of building high-precision environment models. However, the existing technology still has some shortcomings, such as: the existing method often appears geometric boundary misplacement and color inconsistency when fusing multiple views, resulting in obvious splicing artifacts, especially in complex scenes. The existing method is difficult to guide with high-level semantic information, and only relies on pixel or feature alignment, which is easy to cause local texture blur or inconsistency with the overall structure. Traditional methods based on rendering or MVS are prone to lose details under changes in lighting, occlusion or noise, and the generated results have poor stability, which makes it difficult to meet the needs of real-time or high-resolution reconstruction.

[0003] Glossary:

[0004] MVS (Multi-View Stereo) algorithm is a method of reconstructing a three-dimensional scene using multiple images from different angles. By combining image data from multiple angles, the MVS algorithm can construct a highly detailed 3D model.

[0005] VAE (Variational Auto-Encoder) includes an encoder and a decoder. The encoder encodes the original input (image, speech, text sequence, etc.) into a latent space. At this time, the encoder output is not a single latent representation, but a probability distribution parameter (usually mean and variance) in the latent space. This allows the model to sample in the latent space and generate new data points. Based on the reparameterization technique, the mean and variance are used to generate latent variables z (also known as latent variables). The decoder is symmetric with the encoder and is used to restore the sampled latent variables z (latent space) to data with the same shape as the original input (original data space).

[0006] Diffusion Probabilistic Model (Diffusion model) is a class of generative models based on probabilistic denoising process, whose principle can be decomposed into forward diffusion process and reverse diffusion process. The forward diffusion is a fixed, predefined Markov chain. It starts from a clean original data, and gradually adds a small amount of Gaussian noise to it through a series of T time steps, until the original data is almost completely submerged in noise, and finally approximates a pure Gaussian noise distribution. The reverse process is to train a neural network (conditional denoising network) to reverse the forward noise adding process. That is, given a noisy sample x t , the current time step t and the condition c, the conditional denoising network needs to predict the corresponding previous, slightly noisier sample x t−1 , or predict the noise added when x t−1 becomes x t in the forward process. The reverse process can introduce conditions to guide the denoising network to denoise.

[0007] ‌CLIP (Contrastive Language-Image Pretraining) model is a cross-modal neural network model that maps images and texts to the same semantic space through contrastive learning, realizing the semantic association between images and texts. It adopts a dual-encoder architecture, in which an image encoder converts images into vector representations, and a text encoder‌ encodes text descriptions into vector representations. The two vector representations are then mapped to a shared semantic space through a linear projection layer, and the degree of image-text matching is measured by cosine similarity.

[0008] TSDF (Truncated Signed Distance Function) is a widely used representation method in computer vision and three-dimensional reconstruction fields, especially when using volumetric representation. TSDF can efficiently represent scenes in three-dimensional space, especially in applications that require handling large-scale scenes and real-time interaction. SUMMARY

[0009] The purpose of the present application is to provide a high-resolution three-dimensional reconstruction method that solves the problem of obvious model distortion caused by semantic consistency and detail loss, geometric boundary misplacement and color inconsistency during multi-view fusion, and other problems such as obvious stitching artifacts.

[0010] To achieve the above purpose, the technical scheme adopted by the present application is as follows: a high-resolution three-dimensional reconstruction method integrating diffusion model, comprising the following steps:

[0011] S1, constructing an original data set D;

[0012] Collecting a plurality of groups of samples, each group of samples comprising a real three-dimensional model and text instructions, and constructing an original data set D from all the samples;

[0013] S2, constructing an enhanced training set;

[0014] Dividing the training set D from D in proportion train , data augmentation is performed on the samples in the training set to obtain an enhanced training set , , wherein K is the total number of samples, , T (k) respectively the enhanced real three-dimensional model and the enhanced text instructions of the kth group of samples in the training set;

[0015] S3, constructing a three-dimensional reconstruction network, including a text encoder, a renderer, a VAE encoder, a conditional diffusion model, a VAE decoder, and an MVS module;

[0016] The text encoder is used to input T (k) , and generate a corresponding semantic vector e T ;

[0017] The renderer is used to render into a color image sequence I (k) and a depth map D (k) sequence, , , I (k) is composed of color images of N views, is a color image of view n in I (k) , and is the corresponding depth map; The VAE encoder is used to sample from (k) according to the sampling weight w k of I

[0018] to generate a corresponding latent variable z0; The conditional diffusion model is used to take the latent variable z0 as input and take e T as a condition, and through a forward noise adding and a backward noise removing process, to generate a denoised latent variable

[0019] ;

[0020] The VAE decoder is used to generate a reconstructed image according to , and to construct a reconstructed image sequence from the reconstructed images of N views ;

[0021] ​​The MVS module is used to... Generate reconstructed 3D model ;

[0022] S4, training the 3D reconstruction network;

[0023] Noise prediction loss based on conditional diffusion model, and The conditional diffusion model is trained using reconstruction loss, renderer color consistency loss, and depth consistency loss. Then, image quality and 3D reconstruction constraints are used to fine-tune the conditional diffusion model, resulting in the 3D reconstruction model M. ref ;

[0024] S5, acquire the image sequence and text instructions of the scene to be reconstructed, and process them via M... ref The conditional diffusion model and VAE decoder obtain the corresponding reconstructed graph sequence, which is then processed by the MVS module to obtain the reconstructed 3D model.

[0025] Preferably, the renderer is a differentiable renderer. When rendering as a 2D image, first determine the projection plane based on the perspective of the 2D image. Find the vertex p, obtain its surface normal n(p), and perform Laplacian smoothing according to the following formula to obtain the smoothed surface normal n'(p). Replace n(p) with n'(p).

[0026] ,

[0027] In the formula, N(p) is The set of vertices adjacent to the midpoint p, n(y) is the surface normal of vertex y in N(p), and β is the smoothing coefficient.

[0028] Preferably, the sampling weights are obtained according to the following formula:

[0029] ,

[0030] In the formula, Color image sequence I (k) and reconstructed graph sequence Reconstruction loss between, Indicates to I (k) Find the gradient, where ε is a minimum value to prevent the denominator from being zero.

[0031] Preferably, the conditional diffusion model includes a forward diffusion process and a reverse diffusion process;

[0032] The forward diffusion process is used to add noise to z0 step by step according to the following formula, and generate a noisy data at each time step, where the noisy data at time step t is z t ;

[0033] ,

[0034] wherein α t is a noise schedule coefficient decreasing with time step t, and ϵ is a standard Gaussian noise conforming to standard normal distribution N(0, 1);

[0035] The backward diffusion process is used to predict the noise ϵ t added in the forward diffusion process from time step t-1 to t in the process according to the following formula: θ (z t ,t,e T ), and predict the noise-added data z t-1 at time step t-1;

[0036] ,

[0037] wherein, is a noise standard deviation coefficient of the backward diffusion process, is a second noise sampling.

[0038] As preferred, S4 is divided into phase one, phase two and phase three;

[0039] Phase one, training the conditional diffusion model;

[0040] The noise prediction loss L diff based on the conditional diffusion model, the reconstruction loss L recon between , the color consistency loss L color of the renderer and the depth consistency loss L depth are used to construct a first loss function L1, so as to minimize L1 to adjust the network parameters of the conditional diffusion model to the convergence of the three-dimensional reconstruction network, and the converged three-dimensional reconstruction network is marked as M S1 ;

[0041] Phase two, image quality guided conditional diffusion model fine-tuning;

[0042] On the basis of L1, local loss L local and VGG perceptual loss L perc are introduced for color images and reconstructed images, and a second loss function L2 is constructed to minimize L2 to adjust the network parameters of the conditional diffusion model to the convergence of M S1 , and the converged M S1 is marked as M S2 ;

[0043] Phase three, three-dimensional reconstruction constraint guided conditional diffusion model fine-tuning;

[0044] Based on L2, a photometric consistency error E is introduced for MVS module reconstruction. photo Construct a second loss function L3 to minimize the L3-adjusted conditional diffusion model network parameters to M. S2 Convergence yields the 3D reconstruction model M. ref .

[0045] Preferably, in S4, stage one includes steps Sa1~Sa2:

[0046] Sa1, construct the first loss function L1;

[0047] ,

[0048] In the formula, L diff For the noise prediction loss of the conditional diffusion model, L recon L represents the reconstruction loss between the color image sequence and the reconstructed image sequence. color For the color consistency loss of the renderer, L depth λ1, λ2, and λ3 are the depth consistency loss of the renderer, and the weights of the corresponding multiplication terms in L1 are respectively.

[0049] Sa2, from training set D train A batch of samples is fed into a 3D reconstruction network. L1 is calculated, and the network parameters of the conditional diffusion model are adjusted by minimizing L1 until the 3D reconstruction network converges. The converged 3D reconstruction network is then labeled as M. S1 .

[0050] Preferably, the color consistency loss L color and depth consistency loss L depth Calculate according to the following formulas respectively:

[0051] ,

[0052] In the formula, g is The total number of mid vertices, for The g-th vertex p g I i (π i (p g )) is p g Projection position π in the color map of viewpoint i i (p g The color of ) I j (π j (p g )) is p g The projection position π in the color image at viewpoint j j (p g The color of )

[0053] ,

[0054] D i (u,v) is a depth value of a coordinate (u,v) in the depth map of the view i, is a reprojected coordinate of (u,v) in the depth map of the view j, is a depth value of a coordinate in the depth map of the view j.

[0055] As preferred, in S4, the second stage comprises steps Sb1-Sb2:

[0056] Sb1, constructing a second loss function L2;

[0057] ,

[0058] wherein L local is a local loss, L perc is a VGG perceptual loss, ~ are weights of the corresponding multiplication terms in L2, respectively;

[0059] Sb2, training M train using the training set D S1 , and adjusting the network parameters of the conditional diffusion model in M S1 to convergence with the minimization of L1, to obtain M S2 .

[0060] As preferred, the local loss L local and the VGG perceptual loss L perc are obtained according to the following formulas, respectively:

[0061] ,

[0062] is a detail region of , I n ( ) is a pixel value of a pixel point in , and I is a pixel value of a pixel point in .

[0063] ,

[0064] wherein, ϕ(⋅) is a feature extraction function, G SR (⋅) is a super-resolution network, is a square of L2 norm.

[0065] As preferred, the photometric consistency error E photo is calculated according to the following formula:

[0066] ,

[0067] In the formula, for M new Inner vertex p, I i (π i (p) represents the projection position π of p in the color image at viewpoint i. i The color of (p), I j (π j (p) represents the projection position π of p in the color image at viewpoint j. j The color of (p); It is an L1 norm;

[0068] The MVS module is used to... Generate reconstructed 3D model The method is as follows;

[0069] The MVS module generates the initial reconstruction model M new Then M new and The TSDF values ​​are fused according to the fusion weights to form the reconstructed 3D model. ;

[0070] ,

[0071] ,

[0072] In the formula, for M new Let p be an interior vertex, and x be the nearest observed point corresponding to p. for The TSDF value of vertex p. , They are respectively and M new The TSDF value of vertex p, w gt w new They are respectively and M new The weight of M, δ is the control M new The velocity decay factor of the weight, exp(∙) is the exp function.

[0073] Regarding the dataset: After constructing the original dataset D, this invention divides it into a training set and performs data augmentation to obtain the augmented training set. Used for subsequent model training to improve the robustness of the 3D reconstruction network.

[0074] Regarding renderers: Renderers are used to... When rendering a 2D image, The surface normal n(p) of the vertex p is subjected to Laplacian smoothing to obtain a smoothed surface normal n'(p) to replace n(p), thereby reducing artifacts introduced by the discrete mesh and rendering, evenly distributing the normal direction, and making the light transition more natural.

[0075] Regarding the conditional diffusion model: the latent variable z0 is taken as input, and e T is taken as condition, and the conditional diffusion model is focused on training, which is divided into three stages.

[0076] The first stage is used to train the conditional diffusion model, at this time the MVS module in the three-dimensional reconstruction network does not participate, and the text instruction T (k) in the sample is taken as condition, and the text encoder generates a semantic vector, and the renderer renders into a color image sequence I (k) and a depth map D (k) sequence, and then the VAE encoder is used to sample the color images in I (k) , add noise to the conditional diffusion model, denoise, and decode the VAE decoder to generate the reconstructed image corresponding to the color image, in the loss function part, in addition to the noise prediction loss L diff of the conditional diffusion model itself, the reconstruction loss L between I and D recon , and the color consistency loss L color and the depth consistency loss L depth across views are also considered, and the result is obtained , and the network parameters of the conditional diffusion model are updated using an optimization algorithm until the conditional diffusion model converges. This stage mainly learns to generate multi-view reconstruction image sequences that meet the text conditions from noise.

[0077] The second stage is to fine-tune the conditional diffusion model based on image quality. Based on the completion of the first stage, local loss L local and VGG perceptual loss L perc are also introduced, L local ensures small area details (such as edges and textures); L perc uses a pre-trained network (such as VGG19) to extract high-level features to ensure overall semantics and clarity. The MVS module does not participate in this stage, and the quality of the reconstructed image is mainly fine-tuned to obtain a higher quality reconstructed image sequence.

[0078] The third stage is to fine-tune the conditional diffusion model based on three-dimensional reconstruction constraints. Based on the completion of the second stage, the MVS module participates at this time, and the MVS module performs three-dimensional reconstruction on the reconstructed image sequence output by the second stage to output an initial reconstruction model M new , and the photometric consistency error E photo is introduced to constrain the consistency of different view projections during three-dimensional reconstruction, thereby ensuring Mnew The accuracy of the geometry reconstruction, and in M new With Fusion weights are introduced between the two to smooth the transition and retain the advantages of both.

[0079] Compared with the prior art, the present application has the advantages that:

[0080] (1) The present application introduces semantic guidance in multi-view condition diffusion, naturally fills in the missing areas of geometry and texture, and significantly improves the semantic consistency and detail fidelity of the reconstruction results. Thus, the texture details and structural consistency of the reconstructed three-dimensional model are greatly improved.

[0081] (2) Condition diffusion sampling enables the model to accurately restore local details in complex scenes, improving the stability of real-time rendering. On the basis of the condition diffusion model, color consistency loss L color , depth consistency loss L depth are introduced for the color image and the reconstructed image, local loss L local and VGG perception loss L perc are introduced, so that highly consistent geometry and color restoration can be maintained under multi-view conditions, significantly reducing the splicing artifacts.

[0082] (3) The present application divides the training into three stages, through coarse, medium and fine reconstruction strategies, combined with TSDF weighted fusion and local super-resolution fine-tuning, not only smooth and continuous in the overall geometry, but also accurately correct early errors and restore high-frequency details.

[0083] In summary, the present application greatly improves the quality and application value of multi-view three-dimensional reconstruction by introducing semantic guidance and condition diffusion sampling, providing an innovative solution for computer vision and virtual reality. By solving the problems of semantic consistency and detail loss, this method provides a new technical path for high-precision three-dimensional modeling, especially in real-time digital twinning, which has wide application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0084] Figure 1 is a schematic diagram of the three-dimensional reconstruction network of the present application;

[0085] Figure 2 is a flowchart of the present application. DETAILED DESCRIPTION

[0086] The present application will be further described below in conjunction with examples and drawings.

[0087] Example 1: Referring to Figure 1 and Figure 2 , a high-resolution three-dimensional reconstruction method of a fusion diffusion model, comprising the following steps;

[0088] S1, construct the original dataset D;

[0089] Collect multiple sets of samples, each set of samples including a real 3D model and text instructions, and combine all samples to form the original dataset D;

[0090] S2, constructing an enhanced training set;

[0091] Divide the training set D proportionally from D. train Data augmentation is performed on samples within the training set to obtain an augmented training set. , Where K is Total number of internal samples T (k) They are respectively Enhanced realistic 3D model and enhanced text commands for the k-th sample group;

[0092] S3 constructs a 3D reconstruction network, including a text encoder, renderer, VAE encoder, conditional diffusion model, VAE decoder, and MVS module;

[0093] The text encoder is used to input T. (k) Generate the corresponding semantic vector e T ;

[0094] The renderer is used to... Rendered as a color image sequence I (k) and depth map D (k) sequence, , I (k) Composed of color images from N perspectives, For I (k) A color image from a mid-angle perspective n. for The corresponding depth map;

[0095] The VAE encoder is used to press I (k) sampling weight w k from The corresponding latent variable z0 is generated by sampling in the middle;

[0096] The conditional diffusion model is used to take the latent variable z0 as input and e T As a condition, a denoising latent variable is generated through a forward noise addition and reverse denoising process. ;

[0097] The VAE decoder is used to... generate Reconstruction graph And reconstruct the graph from N perspectives. Constructing a sequence of reconstructed graphs ;

[0098] The MVS module is used to generate a reconstructed three-dimensional model according to ; ;

[0099] S4, training a three-dimensional reconstruction network;

[0100] Stage one, training a conditional diffusion model;

[0101] The noise prediction loss L of the conditional diffusion model diff , and the reconstruction loss L between recon , the color consistency loss L of the renderer color and the depth consistency loss L depth Construct the first loss function L1 to minimize L1 to adjust the network parameters of the conditional diffusion model to the convergence of the three-dimensional reconstruction network, and mark the converged three-dimensional reconstruction network as M S1 ;

[0102] Stage two, image quality guided conditional diffusion model fine-tuning;

[0103] On the basis of L1, for color images and reconstructed images, introduce local loss L local and VGG perceptual loss L perc , construct the second loss function L2 to minimize L2 to adjust the network parameters of the conditional diffusion model to M S1 converge, and mark the converged M S1 as M S2 ;

[0104] Stage three, three-dimensional reconstruction constraint guided conditional diffusion model fine-tuning;

[0105] On the basis of L2, for the MVS module reconstruction, introduce the photometric consistency error E photo Construct the second loss function L3 to minimize L3 to adjust the network parameters of the conditional diffusion model to M S2 converge, and get the three-dimensional reconstruction model M ref ;

[0106] S5, obtain the image sequence and text instruction of the scene to be reconstructed, get the corresponding reconstructed image sequence through the conditional diffusion model M ref , VAE decoder, and then get the reconstructed three-dimensional model through the MVS module.

[0107] Regarding the text instruction, it can be set as a text describing the real three-dimensional model, for example:

[0108] Text instruction 1, a wooden chair with a backrest, the color is brown;

[0109] Text instruction 2: a red two-door sports car with smooth lines;

[0110] Text instruction 3: a white modern office building with a glass curtain wall.

[0111] Based on the method of the embodiment, during training, according to I (k) , the sampling weight w k is sampled from as the latent variable z0, and z t is obtained by adding noise step by step. T The conditional diffusion model to be learned is how to restore noise step by step into under the given condition e T , and the graph obtained by restoration is the reconstructed graph.

[0112] During inference, the conditional diffusion model starts from a pure noise vector , and gradually reverses the de-noising to obtain the reconstructed graph. At this time, the noise sampling does not depend on , but is freely sampled, and the guidance of the generation process depends on T (k) corresponding to the semantic vector e T .

[0113] Embodiment 2: see Figures 1 to 2 , on the basis of embodiment 1, more specifically, the renderer is a differentiable renderer, and when rendering into a two-dimensional graph, first determine the projection plane according to the perspective of the two-dimensional graph, for the vertex p of , obtain its surface normal n(p), and perform Laplacian smoothing according to the following formula to obtain the smoothed surface normal n'(p), and replace n(p) with n'(p);

[0114] ,

[0115] In the formula, N(p) is the vertex set adjacent to the vertex p in , n(y) is the surface normal of the vertex y in N(p), and β is a smoothing coefficient.

[0116] The sampling weight is obtained according to the following formula:

[0117] ,

[0118] In the formula, is the reconstruction loss between the color graph sequence I (k) and the reconstructed graph sequence , represents the gradient of I (k) , and ε is a minimum value to prevent the denominator from being zero.

[0119] The conditional diffusion model includes a forward diffusion process and a reverse diffusion process;

[0120] The forward diffusion process is used to add noise to z0 at each time step t according to the following formula, and to generate a noisy data z t ;

[0121] ,

[0122] wherein α t is a noise schedule coefficient decreasing with time step t, and ϵ is a standard Gaussian noise conforming to a standard normal distribution N(0, 1);

[0123] The backward diffusion process is used to predict the noise ϵ t added in the forward diffusion process from time step t-1 to t according to the following formula, and to predict the noisy data z θ (z t ,t-1, e T ) at time step t-1. t-1 ;

[0124] ,

[0125] wherein, is a noise standard deviation coefficient of the backward diffusion process, is a second noise sample.

[0126] Embodiment 3: see Figure 1 , Figure 2 On the basis of embodiment 1, we give the specific implementation steps of S4, and the rest are the same as embodiment 1 or embodiment 2. In S4, stage one includes steps Sa1-Sa2:

[0127] Sa1, construct a first loss function L1;

[0128] ,

[0129] wherein, L diff is a noise prediction loss of the conditional diffusion model, L recon is a reconstruction loss between the color image sequence and the reconstructed image sequence, L color is a color consistency loss of the renderer, L depth is a depth consistency loss of the renderer, and λ1, λ2, λ3 are weights of the corresponding multiplication terms in L1, respectively;

[0130] The color consistency loss L color and the depth consistency loss L depth are calculated according to the following formulas, respectively:

[0131] ,

[0132] wherein, g is the total number of vertices, p the gth vertex in the interior, p g , I i the color of the projection location p i (p g ) in the color map of view i, I g the color of the projection location p i (p g ) in the color map of view j; j j g g the color of the projection location p j (p g ) in the color map of view j;

[0133] ,

[0134] D i (u,v) is the depth value of the coordinate (u,v) in the depth map of view i, is the re-projection coordinate of (u,v) in the depth map of view j, is the depth value of the coordinate in the depth map of view j;

[0135] Sa2, a batch of samples are obtained from the training set D train and input into the three-dimensional reconstruction network, L1 is calculated, and the network parameters of the conditional diffusion model are adjusted to minimize L1 until the three-dimensional reconstruction network converges, and the converged three-dimensional reconstruction network is marked as M S1 .

[0136] Phase two includes steps Sb1-Sb2:

[0137] Sb1, a second loss function L2 is constructed;

[0138] ,

[0139] wherein L local is a local loss, L perc is a VGG perception loss, ~ are weights of corresponding multiplication terms in L2, respectively;

[0140] The local loss L local and the VGG perception loss L perc are obtained according to the following formulas, respectively:

[0141] ,

[0142] is a detail area of , I n ​​​( ) is the pixel value of the pixel point in the image, is the pixel value of the pixel point in the image;

[0143] ,

[0144] In the formula, ϕ(·) is a feature extraction function, G SR (·) is a super-resolution network, is the square of the L2 norm;

[0145] Sb2, the training set D train is used to train M S1 , and M S1 is adjusted to convergence by minimizing L1, to obtain M S2 .

[0146] The third stage is specifically that, on the basis of L2 having contained L diff , L recon , L depth , L local , L perc , a light consistency error E photo is added to obtain L3, and the respective weights of the six parts in L3 are redistributed, and the network parameters of the conditional diffusion model are adjusted to M S2 by minimizing L3 during training, to obtain the three-dimensional reconstruction model M ref .

[0147] The light consistency error E photo is calculated according to the following formula;

[0148] ,

[0149] In the formula, for the vertex p in M new , I i (π i (p)) is the color of the projection position π i (p) of p in the color image at the viewing angle i, and I j (π j (p)) is the color of the projection position π j (p) of p in the color image at the viewing angle j; is the L1 norm;

[0150] In addition, during model training, the MVS module is used to generate a reconstructed three-dimensional model according to , and the specific method is as follows:

[0151] ​The MVS module generates the initial reconstruction model M new Then M new and The TSDF values ​​are fused according to the fusion weights to form the reconstructed 3D model. ;

[0152] ,

[0153] ,

[0154] In the formula, for M new Let p be an interior vertex, and x be the nearest observed point corresponding to p. for The TSDF value of vertex p. , They are respectively and M new The TSDF value of vertex p, w gt w new They are respectively and M new The weight of M, δ is the control M new The velocity decay factor of the weight, exp(∙) is the exp function.

[0155] During model inference, no Then there is no need to merge.

[0156] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A high-resolution three-dimensional reconstruction method of fusion diffusion model, characterized in that, The method comprises the following steps; S1, constructing an original data set D; Collecting a plurality of groups of samples, each group of samples comprising a real three-dimensional model and text instructions, and all samples constituting the original data set D; S2, constructing an enhanced training set; Divide the training set D from D in proportion train Data augmentation is performed on the samples in the training set to obtain an augmented training set , wherein K is the total number of samples in the training set, , T (k) are respectively the augmented real three-dimensional model and the augmented text instruction of the kth group of samples in the training set S3, constructing a three-dimensional reconstruction network comprising a text encoder, a renderer, a VAE encoder, a conditional diffusion model, a VAE decoder and an MVS module; The text encoder is used for inputting T (k) , to generate a corresponding semantic vector e T ; The renderer is used to... Rendered as a color image sequence I (k) and depth map D (k) sequence, , I (k) Composed of color images from N perspectives, For I (k) A color image from a mid-angle perspective n. for The corresponding depth map; The VAE encoder is configured to encode the input image I (k) with a sampling weight w k from the latent variable z0 is generated by sampling from the latent variable distribution p(z0|I ; The conditional diffusion model is configured to take the latent variable z0 as input, and e T As a condition, by the forward noise adding and reverse denoising process, the denoised latent variable z0 is generated ; The VAE decoder is configured to decode the latent representation to generate a reconstructed image ;​​​​ The MVS module is configured to generate a reconstructed three-dimensional model based on the generated depth map ; S4, training the three-dimensional reconstruction network; a reconstruction loss between the noise prediction loss based on the conditional diffusion model, and and a color consistency loss and a depth consistency loss of a renderer, train the conditional diffusion model, and respectively guide fine-tuning of the conditional diffusion model with image quality and three-dimensional reconstruction constraints to obtain a three-dimensional reconstruction model M ref ; S5, obtaining the image sequence and text instructions of the scene to be reconstructed, and inputting the image sequence and text instructions into the M ref The corresponding reconstructed image sequence is obtained through the conditional diffusion model and the VAE decoder, and the reconstructed three-dimensional model is obtained through the MVS module.

2. The method of claim 1, wherein, The renderer is a differentiable renderer, which When rendered as a two-dimensional graph, first determine a projection plane according to the perspective of the two-dimensional graph, and then The vertex p of the surface, obtain the surface normal n(p), and perform Laplacian smoothing according to the following formula to obtain the smoothed surface normal n'(p), and replace n(p) with n'(p); ， In the formula, N(p) is N(p) is a set of vertices adjacent to the vertex p, n(y) is a surface normal of the vertex y in N(p), and β is a smoothing coefficient.

3. The method of claim 1, wherein the method further comprises: The sampling weight is obtained according to the following formula: ， wherein is a sequence of color images I (k) and a sequence of reconstructed images is a reconstruction loss between I denotes the gradient of I (k) with respect to I, and ε is a small value to prevent the denominator from being zero.

4. The method of claim 1, wherein, The conditional diffusion model comprises a forward diffusion process and a reverse diffusion process; The forward diffusion process is used to add noise to z0at each time step according to the following equation, where the noisy data at time step t is z t ; ， In the formula, α t is a noise scheduling coefficient that decreases with time step t, and ε is a standard Gaussian noise conforming to a standard normal distribution N(0, 1). The backward diffusion process is used to predict the noise e t added in the forward diffusion process from time step t-1 to t in the forward diffusion process from z θ (z t ,t,e T ) according to the following equation and predict the noisy data z t-1 at time step t-1; ， wherein is the coefficient of the standard deviation of the noise for the reverse diffusion process, is the second noise sample.

5. The method of claim 1, wherein the method further comprises: S4 is divided into phase one, phase two and phase three; Phase one, training the conditional diffusion model; reconstruction loss L between the conditionally diffused model and diff , and reconstruction loss L recon , color consistency loss L of the renderer color , and depth consistency loss L depth constructing a first loss function L1 to minimize L1 to adjust network parameters of the conditionally diffused model to convergence of the three-dimensional reconstruction network, and marking the converged three-dimensional reconstruction network as M S1 ; Phase two, image quality guiding conditional diffusion model fine-tuning; On the basis of L1, aiming at color image and reconstructed image, local loss L is introduced local and VGG perception loss L perc , the second loss function L2 is constructed to minimize L2 to adjust the conditional diffusion model network parameters to M S1 converge, and the converged M S1 is marked as M S2 ; Phase three, three-dimensional reconstruction constraint guiding conditional diffusion model fine-tuning; introducing a photometric consistency error E for the MVS module reconstruction on L2 basis photo constructing a second loss function L3 to minimize L3 adjusting the conditional diffusion model network parameters to M S2 converging to a three-dimensional reconstruction model M ref .

6. The method of claim 5, wherein the high-resolution 3D reconstruction of a fusion diffusion model is based on a diffusion tensor imaging (DTI) model. In S4, phase one comprises steps Sa1-Sa2: Sa1, constructing a first loss function L1; ， In the formula, L diff L is the noise prediction loss of the conditional diffusion model recon L is the reconstruction loss between the color image sequence and the reconstructed image sequence color L is the color consistency loss of the renderer depth L is the depth consistency loss of the renderer, and λ1, λ2, and λ3 are weights of corresponding multiplication terms in L1. Sa2, from the training set D train A batch of samples is input into the three-dimensional reconstruction network, L1 is calculated, and the network parameters of the conditional diffusion model are adjusted to minimize L1 until the three-dimensional reconstruction network converges, and the converged three-dimensional reconstruction network is marked as M S1 .

7. The method of claim 6, wherein the high-resolution 3D reconstruction of a fusion diffusion model is based on a diffusion tensor imaging (DTI) model. the color consistency loss L color and the depth consistency loss L depth respectively according to the following equations: ， In the formula, g is The total number of mid vertices, for The g-th vertex p g I i (π i (p g )) is p g Projection position π in the color map of viewpoint i i (p g The color of ) I j (π j (p g )) is p g The projection position π in the color image at viewpoint j j (p g The color of ) ， D i (u,v) is the depth value of the coordinate (u,v) in the depth map of view i, is the re-projected coordinate of (u,v) in the depth map of view j, is the depth value of the coordinate (u,v) in the depth map of view j.

8. The method of claim 6, wherein the method further comprises: In S4, phase two comprises steps Sb1-Sb2: Sb1, constructing a second loss function L2; ， where L local is the local loss, L perc is the VGG perceptual loss, ~ are the weights of the corresponding multiplication terms in L2, respectively. Sb2, with the training set D train Train M S1 And adjust M with minimizing L1 S1 The network parameters of the conditional diffusion model in M to convergence, get M S2 .

9. The method of claim 8, wherein, local loss L local VGG perceptual loss L perc respectively according to the following equations: ， For the detail region, I n ( ) is the pixel value of the pixel point in the image I ; and is the pixel value of the pixel point in the image I . ， In the formula, ϕ(·) is a feature extraction function, G SR (·) is a super-resolution network, is the square of the L2 norm.

10. The method of claim 5, wherein the method further comprises: Luminosity uniformity error E photo is calculated according to the following formula; ， wherein, for M new the inner vertex p, i i (p) is the color of the projection location π i (p) in the color map of view i i (p) is the color of the projection location π j (p) in the color map of view j j (p) in the color map of view j j is the L1 norm;​ The MVS module is configured to generate a reconstructed three-dimensional model from a method for generating a reconstructed three-dimensional model is; The MVS module generates an initial reconstruction model M new M is fused with the TSDF value of the point cloud new with the TSDF value of the point cloud into a reconstructed three-dimensional model according to a fusion weight ; ， ， where M new the inner vertex p, x is the closest observation point corresponding to p, is the TSDF value of vertex p, , are and M new the TSDF value of vertex p, w gt , w new are and M new the weight of M new a speed decay factor controlling M new the exp(·) function.

Images