Method and system for generating 3D scene from text, medium and program product
Through the trajectory diffusion converter and Gaussian-driven multi-view latent variable diffusion model, combined with SDS++ loss, the visual quality and speed problems of complex scene-level 3D generation in the existing technology are solved, and the generation and generalization of real-world 3D scenes are realized. Improve capability.
Patent Information
- Application Number
- CN202410867945.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-01
- Publication Date
- 2024-11-26
AI Technical Summary
When generating complex scene-level 3D scenes, the visual quality, generation speed and generalization capabilities are insufficient, making it difficult to generate real-world 3D scenes.
The trajectory diffusion converter is used to generate dense view camera trajectories, and the visual quality of 3D Gaussian images is optimized through Gaussian-driven multi-view latent variable diffusion model and SDS++ loss. Combining sparse view subsets and pixel-aligned 3D Gaussian representations to achieve 3D scenes generate.
It improves the visual quality and generation speed of 3D scenes, can generate complex scene-level 3D scenes in the real world, and enhances the generalization ability of the model.
Smart Images

Figure CN119027576A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and graphics, and in particular to a method, system, medium and program product for generating a 3D scene from text. Background Art
[0002] The technology of generating 3D scenes from text has great potential in fields such as games, robotics, and VR / AR. Early methods optimized 3D representations through fractional distillation sampling, but this process was lengthy and unstable. In contrast, newer methods use feed-forward networks, diffusion models, and reconstruction models to generate 3D representations directly from text or text-guided multi-view images, significantly improving the generation speed. In addition, 3D Gaussian technology further speeds up training and rendering, driving the next wave of progress in text-to-3D generation. However, most current methods only focus on object-level 3D generation. Although some recent studies have addressed the problem of scene-level 3D generation, existing methods still lack in visual quality, generation speed, and generalization ability due to relying only on 2D priors or a limited number of 3D datasets.
[0003] The Chinese patent application with publication number CN117953180A proposes an efficient and consistent text-to-3D dual-mode multi-view latent variable diffusion generation method, which uses a two-dimensional latent variable diffusion model to jointly train a dual-mode multi-view latent variable diffusion model; obtains the text of the three-dimensional object to be generated, and based on the efficiency and consistency trade-off strategy, uses the dual-mode multi-view latent variable diffusion model to infer the text to generate a first three-dimensional object; finally, the surface texture of the first three-dimensional object is optimized to generate a second three-dimensional object. However, this method can only generate object-level 3D products well, and is not suitable for complex scene-level 3D generation. Summary of the invention
[0004] In view of the problems existing in the prior art, the purpose of the present invention is to provide a method, system, medium and program product for generating a 3D scene from text, which can generate a 3D scene of the real world.
[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:
[0006] A method for generating a 3D scene from text, the method comprising the following steps:
[0007] S1, input text and noisy camera trajectory, and use the trained trajectory diffusion transformer to generate dense view camera trajectory;
[0008] S2, performing sparse processing on the dense view camera trajectory obtained in step S1 to obtain a sparse view subset of the camera trajectory, inputting the text and the sparse view subset of the camera trajectory, and using a Gaussian-driven multi-view latent variable diffusion model to generate a pixel-aligned 3D Gaussian as an intermediate 3D representation;
[0009] S3. For the 3D Gaussian image obtained in step S2, use SDS++ loss to back-propagate the loss of the camera-rendered image obtained by random interpolation in the corresponding trajectory to enhance the visual quality of the 3D Gaussian image.
[0010] In step S1, the step of training the trajectory diffusion converter specifically includes:
[0011] S1.1、Set the camera trajectory Represented as a set of camera parameters c i ={r1,t i ,f i ,p i}; where r i represents the i-th camera c i The corresponding rotation matrix; t i represents the i-th camera c i The corresponding offset matrix; f i represents the i-th camera c i Corresponding focal length; p i represents the i-th camera c i The corresponding reference point;
[0012] S1.2, using the architecture of the diffusion transformer module to generate camera trajectories;
[0013] S1.3, the trajectory diffusion transformer is trained to minimize the following loss objective:
[0014]
[0015] in, represents the trace diffusion converter loss, It's expectation. represents the camera trajectory, t represents the time step, is a parameterized trajectory diffusion transformer, is the noisy camera trajectory, y is the text condition;
[0016] S1.4. Training loss Until the trajectory diffusion converter converges;
[0017] S1.5. After training, input a random noise camera trajectory sampled from a Gaussian distribution According to the text information, the corresponding dense view camera trajectory can be obtained through the trajectory diffusion transformer
[0018] In step S2, the Gaussian-driven multi-view latent variable diffusion model training step specifically includes:
[0019] S2.1. Dense view camera trajectory obtained in step S1 Perform sparse uniform sampling to obtain a sparse view subset of the camera trajectory Sparse view subsets for text and camera tracks As input to the Gaussian-driven multi-view latent variable diffusion model;
[0020] S2.2. Processing a Sparse View Subset of the Camera Trajectory via a Frozen Image Latent Encoder ε Corresponding sparse view images, obtain multi-view latent variables Where N≤M, N represents the number of sparsely uniformly sampled views, M represents the number of dense views in step S1, c represents the number of channels, h represents the height, and w represents the width; then the multi-view latent variable Add noise to generate noise latent variables Noisy Multi-view Latent Variables The sparse subset of views corresponding to the camera trajectory Parallel input to the potential denoising network in the form of a U-shaped structure In the above example, the denoised multi-view latent variables are y represents the text condition, t represents the time step;
[0021] S2.3. Sparse View Subsets of Camera Trajectories By combining the ray graph (o×d,d) with the noisy multi-view latent variables Combined and integrated into the potential denoising network , where o and d represent the origin and direction of the pixel alignment ray, respectively;
[0022] S2.4. Using multi-view latent diffusion targets to denoise multi-view latent variables To conduct supervision:
[0023]
[0024] in Indicates that the loss is a two-dimensional loss, It's expectation. represents the real image, c represents the camera, y represents the text condition, and t represents the time step;
[0025] S2.5. To generate rendering-based denoised 3D Gaussian, the denoised multi-view latent variables Input to Gaussian decoder In the output, Gaussian features Among them, τ i ,q i ,s i ,α i ,c i represent the depth, rotation quaternion, scale, opacity and spherical harmonic coefficients of the 3D Gaussian for the corresponding view i;
[0026] S2.6, Gaussian decoder Using the original image latent decoder The decoded predicted depth is then converted to a pixel-aligned Gaussian position μ i =o i +τ i d i , where o i is the origin of the pixel alignment ray corresponding to view i, d i is the direction of the pixel alignment ray corresponding to view i, τ i is the depth value predicted for the pixel of the corresponding view i;
[0027] S2.7, Multi-view Figure 3 D Gauss Jointly represent the 3D scene, randomly sample views from dense view camera trajectories to supervise the predicted 3D Gaussian in image space, ensuring the consistency and accuracy of the 3D scene representation:
[0028]
[0029] in It's expectation. is the rendering function, e(·,·) is the reconstruction loss used to penalize the difference between images, is the real data of the image and camera pair in the dense view, y is the text condition, and t is the time step;
[0030] S2.8. The total training loss is simply the sum of the above losses:
[0031] S2.9. Total training loss Until the Gaussian-driven multi-view latent variable diffusion model converges;
[0032] S2.10. After completing the training of the Gaussian-driven multi-view latent variable diffusion model, the camera trajectory is obtained using the trained trajectory diffusion transformer. and randomly sampled noisy multi-view latent variables Input the trained latent denoising network And according to the corresponding text condition y, the denoised multi-view latent variable is obtained Where t is the current denoising time step;
[0033] S2.11. Denoised multi-view latent variables After Gaussian decoder Get the pixel-aligned 3D Gaussian, and pass the 3D Gaussian with the camera parameter C ′ Rendering and encoding ε to obtain the Gaussian-driven denoising latent variable:
[0034]
[0035] in The 3D Gaussian decoded after denoising is based on the corresponding camera parameters Rendered image;
[0036] S2.12. Noisy Multi-view Latent Variables Denoised latent variables obtained via Gaussian-driven multi-view latent variable diffusion model Can be transformed into noisy multi-view latent variables for the next step Iterate until t = 1, and the 3D Gaussian of the last step is directly used as the final 3D result to participate in the optimization of step S3;
[0037] In step S3, the step of optimizing the 3D Gaussian image with SDS++ specifically includes:
[0038] S3.1, use the trained Gaussian-driven multi-view latent variable diffusion model to infer to time step t = 1, the last step of 3D Gaussian As the initial 3D scene;
[0039] S3.2. Rendering 3D Gaussian using randomly continuously sampled cameras C from dense view camera trajectories Generate image The image x is encoded into a latent space ε(x) = z and then perturbed with randomly sampled noise ∈ and time step t to produce latent variables z t ;
[0040] S3.3, 2D diffusion model∈ θ According to the latent variable z t , text conditional y and learnable text embeddings And the scene-independent optimized text φ, predicting the denoised latent variable
[0041] Introducing a learnable distribution for source prediction For adaptive estimation of the current distribution:
[0042]
[0043] in, Using a learnable text embedding To efficiently estimate the current distribution;
[0044] Object Prediction without Classifier Guidance to Improve Text Alignment:
[0045]
[0046] Among them, ω cfg is the proportion without classifier guidance;
[0047] S3.4. Use the following combination prediction
[0048]
[0049] S3.5, the proposed SDS++ loss is expressed by the following formula:
[0050]
[0051] in, is the expectation, t is the time step, c is the camera parameter, y is the text condition, λ z and λ x are the weights of the latent space and image space targets, respectively, is the latent variable for prediction, is the latent variable for prediction Through the decoder The resulting image, w(t) is the time-dependent noise intensity;
[0052] S3.6. The calculated SDS++ loss will be passed to the 3D Gaussian through back propagation to update the 3D scene. The updated 3D Gaussian will be used as the initial scene for the next update. It will be iterated continuously until the preset number of updates is completed to obtain the final optimized 3D scene.
[0053] A system for generating a 3D scene from text, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of a method for generating a 3D scene from text as described in any one of claims 1 to 4.
[0054] A computer-readable storage medium having a computer program / instruction stored thereon, wherein the computer program / instruction, when executed by a processor, implements the steps of a method for generating a 3D scene from text as described in any one of claims 1 to 4.
[0055] A computer program product comprises a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the steps of a method for generating a 3D scene from text as claimed in any one of claims 1 to 4 are implemented.
[0056] After adopting the above scheme, the present invention proposes an architecture for generating 3D scenes from text, which combines three collaborative processes similar to the roles in filmmaking to solve this problem. The trajectory diffusion transformer as a photography director models the distribution of dense view camera trajectories. For image sequences, directly modeling the distribution of dense views is complex and resource-intensive. To address this problem, we use a multi-view latent diffusion model driven by Gaussians as a decorator to model the image distribution through a sparse subset of dense views. The model uses pixel-aligned 3D Gaussians as intermediate representations. Finally, to improve the visual quality of the generated 3D scene, we adopt a novel SDS++ loss as a detail optimizer to optimize the 3D Gaussians through dense camera interpolation rendering. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 This is the overall flow chart of the present invention architecture;
[0058] Figure 2 is a processing flow chart of the trajectory diffusion converter in the present invention;
[0059] Figure 3 The Gaussian-driven multi-view latent diffusion model processing flow (left) and SDS++loss optimization process (right) of the present invention. DETAILED DESCRIPTION
[0060] like Figure 1 As shown, the present invention proposes a method for generating a 3D scene from text, which comprises the following steps:
[0061] S1. Input text and noisy camera trajectory, and use the trained trajectory diffusion transformer to generate dense view camera trajectory.
[0062] Reference Figure 2 As shown, the steps of training the trajectory diffusion converter specifically include:
[0063] S1.1、Set the camera trajectory Represented as a set of camera parameters c i = {r i ,t i ,f i ,p i}. The rotation matrix of the camera position is r~SO(3), where SO(3) represents a three-dimensional rotation set, which is used to describe all possible rotations in three-dimensional space. i represents the i-th camera c iThe corresponding rotation matrix, ~ indicates sampling from the distribution; the offset matrix of the camera pose represents a set of three-dimensional real vectors, ~ represents sampling from a distribution, t i represents the i-th camera c i Corresponding offset matrix; focal length represents a two-dimensional positive real vector set, ~ represents sampling from a distribution, and f i represents the i-th camera c i Corresponding focal length; reference point represents a set of two-dimensional real vectors, ~ represents sampling from a distribution, and p i represents the i-th camera c i The corresponding reference point;
[0064] S1.2. The camera trajectory is generated using the architecture of the diffusion transformer module (DiT). Each diffusion transformer module includes a cross-attention layer for extracting information from the text input and fusing the information into the input noisy camera trajectory; a self-attention layer for internally correlating the output obtained from the cross-attention layer to improve the understanding of the data; and a multi-layer perceptron layer for nonlinearly transforming the output obtained from the self-attention layer to obtain an updated noisy camera trajectory; the above DiT module is repeated n times, and the present invention sets n to 8 to obtain a noisy camera trajectory that completes one denoising. In addition, the time step t participates in the input of each layer, allowing the model to effectively learn temporal dynamics.
[0065] S1.3, the trajectory diffusion transformer is trained to minimize the following loss objective:
[0066]
[0067] in, represents the trace diffusion converter loss, It's expectation. represents the camera trajectory, t represents the time step, is a parameterized trajectory diffusion transformer, is the noisy camera trajectory and y is the text condition.
[0068] S1.4. Training loss Until the trajectory diffusion converter converges.
[0069] S1.5. After training, input a random noise camera trajectory sampled from a Gaussian distribution According to the text information, the corresponding dense view camera trajectory can be obtained through the trajectory diffusion transformer
[0070] S2. After sparse processing of the dense view camera trajectory obtained in step S1, a sparse view subset of the camera trajectory is obtained, the sparse view subset of the input text and the camera trajectory is obtained, and a Gaussian-driven multi-view latent variable diffusion model is used to generate a pixel-aligned 3D Gaussian as an intermediate 3D representation.
[0071] Reference Figure 3 As shown in Figure 1, the training steps of the Gaussian-driven multi-view latent variable diffusion model specifically include:
[0072] S2.1. For more efficient training, the dense view camera trajectory obtained in step S1 Perform sparse uniform sampling to obtain a sparse view subset of the camera trajectory The sparse uniform sampling method is to select N views with the largest interval from M views at equal intervals. Sparse view subsets of text and camera trajectory As input to the Gaussian-driven multi-view latent variable diffusion model.
[0073] S2.2. Processing a Sparse View Subset of the Camera Trajectory via a Frozen Image Latent Encoder ε Corresponding sparse view images, obtain multi-view latent variables Where N≤M, N represents the number of sparsely uniformly sampled views, M represents the number of dense views in step S1, c represents the number of channels, h represents the height, and w represents the width. Then, the multi-view latent variable Add noise to generate noise latent variables Noisy Multi-view Latent Variables The sparse subset of views corresponding to the camera trajectory Parallel input to the potential denoising network in the form of a U-shaped structure In the above example, the denoised multi-view latent variables are y represents the text condition and t represents the time step.
[0074] S2.3. Sparse View Subsets of Camera Trajectories By combining the ray graph (o×d,d) with the noisy multi-view latent variables Combined and integrated into the potential denoising network where o and d represent the origin and direction of the pixel alignment ray, respectively.
[0075] S2.4. Using multi-view latent diffusion targets to denoise multi-view latent variables To conduct supervision:
[0076]
[0077] in Indicates that the loss is a two-dimensional loss, It's expectation. represents the real image, c represents the camera, y represents the text condition, and t is the time step.
[0078] S2.5. To generate rendering-based denoised 3D Gaussian, the denoised multi-view latent variables Input to Gaussian decoder In the output, Gaussian features Among them, τ i ,q i ,s i ,α i ,c i represent the depth, rotation quaternion, scale, opacity and spherical harmonics coefficients of the 3D Gaussian for the corresponding view i.
[0079] S2.6, Gaussian decoder Using the original image latent decoder The decoded predicted depth is then converted to a pixel-aligned Gaussian position μ i =o i +τ i d i , where o i is the origin of the pixel alignment ray corresponding to view i, d i is the direction of the pixel alignment ray corresponding to view i, τ i is the predicted depth value for the pixel corresponding to view i.
[0080] S2.7, Multi-view Figure 3 D Gauss Jointly represent the 3D scene, randomly sample views from dense view camera trajectories to supervise the predicted 3D Gaussian in image space, ensuring the consistency and accuracy of the 3D scene representation:
[0081]
[0082] in It's expectation. is the rendering function, e(·,·) is the reconstruction loss used to penalize the difference between images, is the ground truth data of the image and camera pair in the dense view, y is the text condition, and t is the time step.
[0083] S2.8. The total training loss is simply the sum of the above losses:
[0084] S2.9. Total training loss Until the Gaussian-driven multi-view latent variable diffusion model converges.
[0085] S2.10. After completing the training of the Gaussian-driven multi-view latent variable diffusion model, the camera trajectory is obtained using the trained trajectory diffusion transformer. and randomly sampled noisy multi-view latent variables Input the trained latent denoising network And according to the corresponding text condition y, the denoised multi-view latent variable is obtained Here, t is the current denoising time step.
[0086] S2.11. Denoised multi-view latent variables After Gaussian decoder Get the pixel-aligned 3D Gaussian, and pass the 3D Gaussian with the camera parameter C ′ Rendering and encoding ε to obtain the Gaussian-driven denoising latent variable:
[0087]
[0088] in The 3D Gaussian decoded after denoising is based on the corresponding camera parameters The rendered image.
[0089] S2.12. Noisy Multi-view Latent Variables Denoised latent variables obtained via Gaussian-driven multi-view latent variable diffusion model Can be transformed into noisy multi-view latent variables for the next step Iterate until t=1, and the 3D Gaussian of the last step is directly used as the final 3D result to participate in the optimization of step S3.
[0090] S3. For the 3D Gaussian image obtained in step S2, use SDS++ loss to back-propagate the loss of the camera-rendered image obtained by random interpolation in the corresponding trajectory to enhance the visual quality of the 3D Gaussian image.
[0091] Reference Figure 3 As shown in the figure, the steps of SDS++ optimization of 3D Gaussian images include:
[0092] S3.1, use the trained Gaussian-driven multi-view latent variable diffusion model to infer to time step t = 1, the last step of 3D Gaussian As the initial 3D scene.
[0093] S3.2. Rendering 3D Gaussian using randomly continuously sampled cameras C from dense view camera trajectories Generate image The image x is encoded into a latent space ε(x) = z and then perturbed with randomly sampled noise ∈ and time step t to produce latent variables z t .
[0094] S3.3, 2D diffusion model∈ θ According to the latent variable z t , text conditional y and learnable text embeddings And the scene-independent optimized text φ, predicting the denoised latent variable
[0095] Different from the standard SDS loss setting The present invention introduces a learnable distribution source prediction For adaptive estimation of the current distribution:
[0096]
[0097] in, Using a learnable text embedding to efficiently estimate the current distribution.
[0098] This method utilizes the original latent denoising network and minimizes Training is performed together with the optimization process. Target prediction Improving text alignment with no classifier guidance:
[0099]
[0100] Among them, ω cfg is the proportion without classifier guidance.
[0101] S3.4. Use the following combination prediction
[0102]
[0103] S3.5, the proposed SDS++ loss is expressed by the following formula:
[0104]
[0105] in is the expectation, t is the time step, c is the camera parameter, y is the text condition, λ z and λ x are the weights of the latent space and image space targets, respectively, is the latent variable for prediction, is the latent variable for prediction Through the decoder The resulting image, w(t) is the time-dependent noise intensity.
[0106] S3.6. The calculated SDS++ loss will be passed to the 3D Gaussian through back propagation to update the 3D scene. The updated 3D Gaussian will be used as the initial scene for the next update. It will be iterated continuously until the preset number of updates is completed to obtain the final optimized 3D scene.
[0107] In the above, the present invention uses a trajectory diffusion transformer as a photography guide to model the distribution of camera trajectories according to text descriptions; next, a multi-view latent variable diffusion model driven by Gaussians is used as a decorator to model the distribution of image sequences according to camera trajectories and text. This model is fine-tuned from a 2D diffusion model and can directly generate pixel-aligned 3D Gaussians as a direct 3D scene representation for 3D consistent denoising; finally, these 3D Gaussians are further refined as detail optimizers to obtain realistic 3D scenes through a novel SDS++ loss that incorporates prior knowledge of the 2D diffusion model.
[0108] The present invention also provides a system for generating a 3D scene from text, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method for generating a 3D scene from text as described above.
[0109] The present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.
[0110] The computer-readable medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiments.
[0111] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0112] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0113] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.
[0114] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A method for generating a 3D scene from text, characterized in that: The method comprises the following steps: S1, input text and noisy camera trajectory, and use the trained trajectory diffusion transformer to generate dense view camera trajectory; S2, performing sparse processing on the dense view camera trajectory obtained in step S1 to obtain a sparse view subset of the camera trajectory, inputting the text and the sparse view subset of the camera trajectory, and using a Gaussian-driven multi-view latent variable diffusion model to generate a pixel-aligned 3D Gaussian as an intermediate 3D representation; S3. For the 3D Gaussian image obtained in step S2, use SDS++ loss to back-propagate the loss of the camera-rendered image obtained by random interpolation in the corresponding trajectory to enhance the visual quality of the 3D Gaussian image.
2. The method for generating a 3D scene from text according to claim 1, characterized in that: In step S1, the step of training the trajectory diffusion converter specifically includes: S1.1、Set the camera trajectory Represented as a set of camera parameters c i = {r i ,t i ,f i ,p i }; where r i represents the i-th camera c i The corresponding rotation matrix; t i represents the i-th camera c i The corresponding offset matrix; f i represents the i-th camera c i Corresponding focal length; p i represents the i-th camera c i The corresponding reference point; S1.2, using the architecture of the diffusion transformer module to generate camera trajectories; S1.3, the trajectory diffusion transformer is trained to minimize the following loss objective: in, represents the trace diffusion converter loss, It's expectation. represents the camera trajectory, t represents the time step, is a parameterized trajectory diffusion transformer, is the noisy camera trajectory, y is the text condition; S1.
4. Training loss Until the trajectory diffusion converter converges; S1.
5. After training, input a random noise camera trajectory sampled from a Gaussian distribution According to the text information, the corresponding dense view camera trajectory can be obtained through the trajectory diffusion transformer 3. The method for generating a 3D scene from text according to claim 2, characterized in that: In step S2, the Gaussian-driven multi-view latent variable diffusion model training step specifically includes: S2.
1. Dense view camera trajectory obtained in step S1 Perform sparse uniform sampling to obtain a sparse view subset of the camera trajectory Sparse view subsets for text and camera tracks As input to the Gaussian-driven multi-view latent variable diffusion model; S2.
2. Processing a Sparse View Subset of the Camera Trajectory via a Frozen Image Latent Encoder ε Corresponding sparse view images, obtain multi-view latent variables Where N≤M, N represents the number of sparsely uniformly sampled views, M represents the number of dense views in step S1, c represents the number of channels, h represents the height, and w represents the width; then the multi-view latent variable Add noise to generate noise latent variables Noisy Multi-view Latent Variables The sparse subset of views corresponding to the camera trajectory Parallel input to the potential denoising network in the form of a U-shaped structure In the above example, the denoised multi-view latent variables are y represents the text condition, t represents the time step; S2.
3. Sparse View Subsets of Camera Trajectories By combining the ray graph (o×d,d) with the noisy multi-view latent variables Combined and integrated into the potential denoising network , where o and d represent the origin and direction of the pixel alignment ray, respectively; S2.
4. Using multi-view latent diffusion targets to denoise multi-view latent variables To conduct supervision: in Indicates that the loss is a two-dimensional loss, It's expectation. represents the real image, c represents the camera, y represents the text condition, and t represents the time step; S2.
5. To generate rendering-based denoised 3D Gaussian, the denoised multi-view latent variables Input to Gaussian decoder In the output, Gaussian features Among them, τ i ,q i ,s i ,α i ,c i represent the depth, rotation quaternion, scale, opacity and spherical harmonic coefficients of the 3D Gaussian for the corresponding view i; S2.6, Gaussian decoder Using the original image latent decoder The decoded predicted depth is then converted to a pixel-aligned Gaussian position μ i =o i +τ i d i , where o i is the origin of the pixel alignment ray corresponding to view i, d i is the direction of the pixel alignment ray corresponding to view i, τ i is the depth value predicted for the pixel of the corresponding view i; S2.7, Multi-view 3D Gaussian Jointly represent the 3D scene, randomly sample views from dense view camera trajectories to supervise the predicted 3D Gaussian in image space, ensuring the consistency and accuracy of the 3D scene representation: in It's expectation. is the rendering function, l(·,·) is the reconstruction loss used to penalize the difference between images, is the real data of the image and camera pair in the dense view, y is the text condition, and t is the time step; S2.
8. The total training loss is simply the sum of the above losses: S2.
9. Total training loss Until the Gaussian-driven multi-view latent variable diffusion model converges; S2.
10. After completing the training of the Gaussian-driven multi-view latent variable diffusion model, the camera trajectory is obtained using the trained trajectory diffusion transformer. and randomly sampled noisy multi-view latent variables Input the trained latent denoising network And according to the corresponding text condition y, the denoised multi-view latent variable is obtained Where t is the current denoising time step; S2.
11. Denoised multi-view latent variables After Gaussian decoder Get the pixel-aligned 3D Gaussian, render it with the camera parameter C′ and encode ε to get the Gaussian-driven denoising latent variable: in The 3D Gaussian decoded after denoising is based on the corresponding camera parameters Rendered image; S2.
12. Noisy Multi-view Latent Variables Denoised latent variables obtained via Gaussian-driven multi-view latent variable diffusion model Can be transformed into noisy multi-view latent variables for the next step Iterate until t=1, and the 3D Gaussian of the last step is directly used as the final 3D result to participate in the optimization of step S3.
4. The method for generating a 3D scene from text according to claim 3, characterized in that: In step S3, the step of optimizing the 3D Gaussian image with SDS++ specifically includes: S3.1, use the trained Gaussian-driven multi-view latent variable diffusion model to infer to time step t = 1, the last step of 3D Gaussian As the initial 3D scene; S3.
2. Rendering 3D Gaussian using randomly continuously sampled cameras C from dense view camera trajectories Generate image The image x is encoded into a latent space ε(x) = z and then perturbed with randomly sampled noise ∈ and time step t to produce latent variables z t ; S3.3, 2D diffusion model∈ θ According to the latent variable z t , text conditional y and learnable text embeddings And the scene-independent optimized text φ, predicting the denoised latent variable Introducing a learnable distribution for source prediction For adaptive estimation of the current distribution: in, Using a learnable text embedding To efficiently estimate the current distribution; Object Prediction without Classifier Guidance to Improve Text Alignment: Among them, ω cfg is the proportion without classifier guidance; S3.
4. Use the following combination prediction S3.5, the proposed SDS++ loss is expressed by the following formula: in is the expectation, t is the time step, c is the camera parameter, y is the text condition, λ z and λ x are the weights of the latent space and image space targets, respectively, is the latent variable for prediction, is the latent variable for prediction Through the decoder The resulting image, w(t) is the time-dependent noise intensity; S3.
6. The calculated SDS++ loss will be passed to the 3D Gaussian through back propagation to update the 3D scene. The updated 3D Gaussian will be used as the initial scene for the next update. It will be iterated continuously until the preset number of updates is completed to obtain the final optimized 3D scene.
5. A system for generating a 3D scene from text, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of a method for generating a 3D scene from text as described in any one of claims 1 to 4.
6. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of a method for generating a 3D scene from text as described in any one of claims 1 to 4 are implemented.
7. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of a method for generating a 3D scene from text as described in any one of claims 1 to 4 are implemented.
Citation Information
Cited By
Absolute scale camera track image-to-video generation method suitable for real scene
CN119919496A
Indoor scene layout generation method and device and electronic equipment
CN121147440A
Three-dimensional content generation method and device based on retrieval enhancement and personalized generation
CN121392164A
Method and apparatus for generating three-dimensional content based on search enhancement and personalization generation
CN121392164B
Three-dimensional scene generation method based on video model
CN122115715A
