A method for multi-modal to 3D object generation by mixed diffusion supervision

Abstract

The application discloses a kind of generation methods of multimodal to 3D object by mixed diffusion supervision, comprising the following steps: S1, using modal similarity loss alignment modal prompt and the encoding of rendering image, the semantic information of multiple modal is better injected into the generation process of 3D object, for making the 3D object generated more conform to the requirement of each modal prompt;S2, using mixed diffusion supervision utilizes the prior knowledge of 2D diffusion model and 3D diffusion model, by combining the advantages of two kinds of pre-training diffusion model, for improving the generation effect and consistency of 3D object: S3, using three-stage optimization method, by combining mixed diffusion supervision in each stage, 3D object is generated;The method can effectively improve the generation quality of 3D object by using modal similarity loss, mixed diffusion supervision and three-stage optimization method, has significant practical value and application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer processing, and particularly relates to a multi-modal-to-3D object generation method through mixed diffusion supervision. BACKGROUND

[0002] In recent years, the field of multi-modal modeling has made significant breakthroughs and progress, and the pace of technological and research progress has accelerated. Some research explores the joint training of multi-modal in supervised and self-supervised environments, aiming to use a single model to build a unified representation of various modalities to achieve more comprehensive cross-modal understanding. Visual Transformer (ViT) has been widely used in downstream tasks such as visual question answering and image caption generation due to its diverse model architecture and training methods. Multi-modal encoders have also achieved significant success in visual-language, video-audio, and video-speech-language fields. Aligning data from different modalities is an active research area and has shown great potential in cross-modal retrieval and building a unified multi-modal representation.

[0003] IMAGEBIND is a method that learns a shared representation space by utilizing various types of image-paired data. It does not require a dataset where all modalities co-occur, and using image-paired data alone is sufficient to combine these modalities together. IMAGEBIND can leverage the latest large-scale visual-language models and extend their zero-shot capabilities to new modalities through natural pairing of images with other modalities. This feature enables IMAGEBIND to perform rich combined multi-modal tasks between different modalities. However, these tasks are currently limited to two-dimensional fields, and how to apply IMAGEBIND to three-dimensional fields has not been explored.

[0004] Diffusion models learn data distribution by denoising and restoring the original data. With the help of pre-trained models, diffusion models have made significant progress in image, video, speech, and 3D generation, revolutionizing the field of computer vision. Pioneering works such as Stable Diffusion and DeepFloyd generate high-quality images from text prompts by learning prior knowledge from large-scale datasets. Subsequent work further fine-tuned the text-to-image diffusion model to better adapt to downstream tasks. For example, Stable unCLIP can accept CLIP image embeddings as well as text prompts, enabling image variations. In addition, some research has developed diffusion models that achieve 3D perception by generating images rendered by objects. Diffusion models are powerful tools for modeling and generating complex data, and due to their robust performance, they have broad application prospects. In the field of automated 3D content generation, they are often used to guide the generation of 3D objects with their strong prior knowledge.

[0005] Consistency Distillation Sampling (CDS) is a method of distilling the prior knowledge of the diffusion model into the 3D generation field, which explores the deterministic sampling prior of ordinary differential equation (ODE) for 3D generation. Specifically, given the image rendered by the 3D model, the target 3D score function is first estimated using the pre-trained 2D diffusion model, and an ODE is constructed for trajectory sampling. Then two adjacent samples are sampled along the ODE trajectory, and the sample with less noise is used to guide the sample with more noise, so as to distill the deterministic prior into the 3D model. The CDS loss effectively solves the stochastic differential equation (SDE) uncertainty problem in the existing and widely used score distillation sampling (SDS) loss, and provides a more reliable and consistent framework to guide the 3D generation process. However, due to the inherent limitations of the diffusion model itself, the 2D diffusion model enhances the imagination and generalization ability of 3D generation, but since it is trained on a large-scale 2D dataset, it performs poorly in 3D consistency and determinacy; in contrast, the 3D diffusion model ensures more accurate 3D geometric structure and consistency, but due to the limited size of the 3D training dataset, it lacks generalization, often resulting in overly smooth 3D object surfaces.

[0006] Recent progress in 3D object generation has benefited from the robust prior provided by diffusion models. However, current models are task-specific and can only serve one modality at a time, requiring retraining when switching modalities. For the case of 3D-from-image models and text prompts, a simple approach is to convert the text prompt into an image and then use a 3D-from-image model for generation. However, this method is time-consuming and labor-intensive, and information loss is inevitable due to the modal conversion. SUMMARY

[0007] To solve the above problems, the present application proposes a method for generating 3D objects from multiple modalities through hybrid diffusion supervision.

[0008] To achieve the above purpose, the present application adopts the following technical solutions:

[0009] A method for generating 3D objects from multiple modalities through hybrid diffusion supervision, comprising the following steps:

[0010] S1, using a modality similarity loss to align the encoding of the modality prompt and the rendered image, so as to better inject semantic information of multiple modalities into the generation process of 3D objects, so as to make the generated 3D objects more in line with the requirements of each modality prompt;

[0011] S2, using mixed diffusion supervision, using the prior knowledge of 2D diffusion model and 3D diffusion model, combining the advantages of the two pre-training diffusion models to improve the generation effect and consistency of 3D objects:

[0012] S3, using a three-stage optimization method, combining mixed diffusion supervision in each stage to generate 3D objects.

[0013] Preferably, step S1 enhances the correct guidance of each modality in the 3D object generation process by aligning the embedding generated by the multi-modal alignment encoder with the CLIP embedding of the image rendered from the 3D object, and the specific process is as follows:

[0014] S11, using a differentiable renderer to render a set of images x from the corresponding NeRF model or Mesh model under a given camera perspective p;

[0015] S12, data augmentation is performed on the image x to obtain detailed geometric structure and texture; the data augmentation includes global augmentation, local augmentation and normalization; wherein the global augmentation is used to perform random perspective transformation on the rendered image, and the local augmentation is used to perform random cropping and perspective transformation on the rendered image;

[0016] S13, input the data augmented rendered image into the CLIP image encoder to obtain its embedding, and then calculate the modality similarity loss between the embedding of the input modality prompt and the embedding of the rendered image, the calculation formula is:

[0017]

[0018]

[0019]

[0020]

[0021] wherein, represents the final overall modality similarity loss; represents the modality similarity loss after global augmentation on the rendered image; represents the modality similarity loss after local augmentation on the rendered image; represents the modality similarity loss after normalization on the rendered image; ω g , ω l and ω z are weighting parameters; i represents the number of data augmentations; n represents the number of data augmentations in each iteration; W irepresents the similarity weight during the i-th enhancement; avg[·] represents the calculation of the average similarity between the input modal embedding and all the rendered image embeddings; cos(a, b) represents the cosine similarity between a and b; C m represents the modal cue embedding encoded by the multi-modal alignment encoder; ε represents the CLIP image encoder; G(x) represents the global enhancement operation on the rendered image x; L(x) represents the local enhancement operation on the rendered image x; Z(x) represents the normalization operation on the rendered image x.

[0022] Preferably, the mixed diffusion supervision in step S2 includes pixel-level plane supervision and spatial-level stereo supervision.

[0023] Preferably, the 2D diffusion model and the modal similarity loss are used together as pixel-level plane supervision in step S2 to guide the model to generate 3D in the 3D object generation process with any modal as the condition, and the corresponding supervision loss is:

[0024] Consistency distillation sampling loss: supervised by consistency distillation sampling loss, consistency distillation sampling uses the deterministic sampling prior of ordinary differential equation to generate 3D, and the calculation formula of consistency distillation sampling loss is:

[0025]

[0026] wherein, represents the expected value; λ(t2) represents the loss weight; represents the square of the two-norm; D Φ (·) represents a pre-trained 2D diffusion model; represents the noise latent vector; t1 and t2 represent two adjacent diffusion time steps, and t1>t2; C m represents the modal cue embedding encoded by the multi-modal alignment encoder; sg(·) represents the stop gradient operator; represents the less noisy latent vector obtained by performing a deterministic sampling of one discretization step through a numerical ordinary differential equation solver;

[0027] Enhanced 2D score distillation sampling loss: supervised by the score distillation sampling loss in Dreamfusion, the score distillation sampling distills the 2D prior of the pre-trained diffusion model into the 3D model, the 3D model is represented by parameters θ, and the score distillation sampling loss is extended from the latent space to the image space, and the final gradient calculation is:

[0028]

[0029] wherein, ​​denotes the gradient of the parameter θ; θ denotes the parameter of the 3D model; denotes the loss function for extending the score distillation sampling loss from the latent space to the image space; denotes the expectation; λ(t) denotes the loss weight; z denotes the latent vector; denotes the estimation of the latent vector z using the denoising function; ω img is a weighted parameter; x denotes the image rendered from the 3D object; denotes the estimation image obtained by decoding the prediction of the diffusion model denotes the estimation image obtained by decoding the prediction of the diffusion model

[0030] A regularization method is adopted The calculation formula of the enhanced 2D score distillation sampling loss is as follows:

[0031]

[0032] wherein, denotes the enhanced 2D score distillation sampling loss; ω* denotes the loss function weight; denotes the loss function for extending the score distillation sampling loss from the latent space to the image space; ω nz denotes the loss function weight; denotes the regularization loss;

[0033] The complete pixel-level plane supervision includes the modality similarity loss, the consistency distillation sampling loss and the enhanced 2D score distillation sampling loss, and the specific formula of the pixel-level plane supervision is as follows:

[0034]

[0035] wherein, denotes the pixel-level plane supervision; ω ms denotes the loss function weight; denotes the modality similarity loss; ω cds denotes the loss function weight; denotes the consistency distillation sampling loss; denotes the enhanced 2D score distillation sampling loss; denotes that only one component is used as supervision in different stages of the XBind optimization.

[0036] Preferably, the 3D diffusion model and the 2D diffusion model are used together as the space-level stereo supervision in step S2, and the corresponding supervision loss is as follows:

[0037] 3D score distillation sampling loss: the 3D score distillation sampling loss is used to supervise the generation of the 3D object, and the specific gradient of the 3D score distillation sampling loss is as follows:

[0038]

[0039] where, denotes the gradient of parameter θ; θ denotes the parameters of 3D model; denotes 3D score distillation sampling loss; denotes expectation; λ(t) denotes loss weight; denotes denoising function; z t denotes noise latent vector; c(x r , R, T) denotes embedding of reference view and relative camera extrinsic; x r denotes reference view; denotes relative camera rotation of target view angle; denotes relative camera translation of target view angle; t denotes time step; ∈ is standard Gaussian noise; z denotes latent vector;

[0040] Reference view loss: the reference view loss is used to improve the quality of 3D object generated from reference view angle, and the calculation formula of the loss function is:

[0041]

[0042] where, denotes reference view loss; ω rgb denotes loss weight; denotes square of two-norm; m denotes reference view mask; ⊙ denotes Hadamard product; x r denotes reference view; is image rendered from reference view angle; ω mask denotes loss weight; M(·) denotes foreground mask;

[0043] 3D score distillation sampling loss and reference view loss constitute spatial level stereo supervision together, and the calculation formula of the loss function is:

[0044]

[0045] where, denotes spatial level stereo supervision loss; ω 3d denotes loss weight; denotes reference view loss;

[0046] Pixel level plane supervision and spatial level stereo supervision constitute hybrid diffusion supervision in three-stage optimization process, and the calculation formula is:

[0047]

[0048] where, denotes hybrid diffusion supervision loss; represents a pixel-level plane supervision; represents a spatial-level stereo supervision loss.

[0049] Preferably, the 2D diffusion model in step S2 adopts a Stable Diffusion v2-1-unclip model, and the 3D diffusion model adopts a Zero-1-to-3 model.

[0050] Preferably, the specific process of step S3 is:

[0051] S31, in the first stage, the modal similarity loss in the pixel-level plane supervision and the enhanced 2D score distillation sampling loss are used, and the spatial-level stereo supervision and the normal regularization are combined to update the neural radiance field model until convergence, which is used to learn the rough texture and 3D geometry aligned with the modal prompt, and a low-resolution NeRF model is used for 3D representation;

[0052] S32, in the second stage, the rendering normal map of the 3D object is used as the input of the 2D and 3D diffusion models, the modal similarity loss in the pixel-level plane supervision and the consistency distillation sampling loss are used, and the spatial-level stereo supervision, the normal consistency loss and the Laplacian smoothing loss are used to optimize the differential differentiable voxel grid embedding body model, which is used to enhance the geometric details of the 3D object, and a high-resolution differential differentiable voxel grid embedding body model is used as the 3D representation, the 3D representation is converted from the implicit neural radiance field model to the explicit differential differentiable voxel grid embedding body model, and the geometry and texture of the 3D object are optimized separately;

[0053] S33, the color image rendered by the differential differentiable voxel grid embedding body model is input into the pre-trained 2D diffusion model and 3D perception diffusion model, and by using the modal similarity loss in the pixel-level plane supervision and the enhanced 2D score distillation sampling loss, and the spatial-level stereo supervision, a high-fidelity 3D object with rich texture details is generated.

[0054] The method has the following beneficial effects: the method for generating a 3D object from multi-modal to multi-modal through mixed diffusion supervision (XBind) combines a multi-modal alignment encoder and a pre-trained diffusion model, and can generate a 3D object from text, image and audio modalities. The method further introduces a new loss function, namely a modality similarity (MS) loss, which aligns the encoding of the modality prompt and the rendered image, and better injects semantic information of the multi-modal into the generation process of the 3D object, so that the generated 3D object is more consistent with the requirements of the modal prompt. In addition, the mixed diffusion supervision fully utilizes the prior knowledge of the 2D diffusion model and the 3D diffusion model, and combines the advantages of the two pre-trained diffusion models, so as to ensure the generation effect and consistency of the 3D object. Finally, the three-stage optimization framework combines the mixed diffusion supervision, and uses the step-by-step optimization of the three stages to further improve the generation quality of the 3D object. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a generation result of a multi-modal prompt of the method;

[0056] Figure 2 is a model architecture diagram of the method;

[0057] Figure 3 is a 3D object generation result of the XBind;

[0058] Figure 4 is a qualitative comparison diagram of generation results of the XBind and a baseline method;

[0059] Figure 5 is a qualitative comparison diagram of generation results of the XBind and other advanced methods in the field of text-to-3D;

[0060] Figure 6 is a display diagram of ablation research of each module in the XBind. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to examples. It should be understood that the specific examples described herein are only used to explain the present application, and are not used to limit the present application.

[0062] As shown in Figures 1 to 6 , a method for generating a 3D object from multi-modal to multi-modal through mixed diffusion supervision includes the following steps:

[0063] S1, adopt the modal similarity loss to align the encoding of the modal prompt and the rendered image, and better inject the semantic information of multiple modalities into the generation process of the 3D object, so as to make the generated 3D object more in line with the requirements of each modal prompt;

[0064] Step S1 enhances the correct guidance of each modality in the 3D object generation process by aligning the embedding generated by the multi-modal alignment encoder with the CLIP embedding of the image rendered from the 3D object, and the specific process is as follows:

[0065] S11, using a differentiable renderer, render a set of images x from the corresponding NeRF model or Mesh model under a given camera perspective p;

[0066] S12, data augmentation is performed on the image x to obtain detailed geometric structure and texture; the data augmentation includes global enhancement, local enhancement and normalization; wherein the global enhancement is used to perform random perspective transformation on the rendered image, and the local enhancement is used to perform random cropping and perspective transformation on the rendered image;

[0067] S13, input the rendered image after data augmentation into the CLIP image encoder to obtain its embedding, and then calculate the modal similarity loss between the embedding of the input modal prompt and the embedding of the rendered image, the calculation formula is as follows:

[0068]

[0069]

[0070]

[0071]

[0072] wherein, represents the final overall modal similarity loss; represents the modal similarity loss after global enhancement on the rendered image; represents the modal similarity loss after local enhancement on the rendered image; represents the modal similarity loss after normalization on the rendered image; ω g , ω l and ω z are weighting parameters; i represents the number of data augmentations; n represents the number of data augmentations in each iteration; W i represents the similarity weight during the i-th enhancement; avg[·] represents the calculation of the average similarity between the input modal embedding and all rendered image embeddings; cos(a, b) represents the cosine similarity between a and b; C mrepresents the modality cue embedding encoded by the multi-modal alignment encoder; ε represents the CLIP image encoder; G(x) represents a global enhancement operation on the rendered image x; L(x) represents a local enhancement operation on the rendered image x; Z(x) represents a normalization operation on the rendered image x.

[0073] S2, the prior knowledge of the 2D diffusion model and the 3D diffusion model is used to improve the generation effect and consistency of the 3D object by combining the advantages of the two pre-training diffusion models.

[0074] The mixed diffusion supervision in step S2 includes pixel-level plane supervision and spatial-level stereo supervision.

[0075] In step S2, the 2D diffusion model and the modality similarity loss are used as pixel-level plane supervision, and the model is guided to generate 3D in any modality during the 3D object generation process. The corresponding supervision loss is:

[0076] Consistency distillation sampling loss: the consistency distillation sampling loss is used for supervision. The consistency distillation sampling uses the deterministic sampling prior of ordinary differential equation to generate 3D. The calculation formula of the consistency distillation sampling loss is:

[0077]

[0078] Among them, represents the expected value; λ(t2) represents the loss weight; represents the square of the two-norm; D Φ (·) represents a pre-trained 2D diffusion model; represents a noise latent vector; t1 and t2 represent two adjacent diffusion time steps, and t1>t2; C m represents the modality cue embedding encoded by the multi-modal alignment encoder; sg(·) represents the stop gradient operator; represents a less noisy latent vector obtained by executing a deterministic sampling of one discrete step through a numerical ordinary differential equation solver from ;

[0079] Enhanced 2D score distillation sampling loss: the score distillation sampling loss in Dreamfusion is used for supervision. The score distillation sampling distills the 2D prior of the pre-training diffusion model into the 3D model. The 3D model uses parameters θ to represent, and the score distillation sampling loss is extended from the latent space to the image space. The final gradient calculation is:

[0080]

[0081] Among them, ​gradient of parameter θ; θ represents a parameter of a 3D model; loss function representing extending the score distillation sampling loss from latent space to image space; represents expectation; λ(t) represents loss weight; z represents latent vector; represents an estimate of latent vector z using a denoising function; ω img is a weighted parameter; x represents an image rendered from a 3D object; represents an estimated image obtained by decoding a diffusion model prediction

[0082] a regularization method is adopted minimizing the variance of the sampled z coordinates along the NeRF ray distribution, and the calculation formula of the enhanced 2D score distillation sampling loss is:

[0083]

[0084] wherein, represents enhanced 2D score distillation sampling loss; ω* represents loss function weight; loss function representing extending the score distillation sampling loss from latent space to image space; ω nz represents loss function weight; represents regularization loss;

[0085] The complete pixel-level plane supervision includes modal similarity loss, consistency distillation sampling loss and enhanced 2D score distillation sampling loss, and the specific formula of the pixel-level plane supervision is:

[0086]

[0087] wherein, represents pixel-level plane supervision; ω ms represents loss function weight; represents modal similarity loss; ω cds represents loss function weight; represents consistency distillation sampling loss; represents enhanced 2D score distillation sampling loss; represents using only one component as supervision at different stages of XBind optimization.

[0088] The 3D diffusion model and the 2D diffusion model in step S2 are taken together as spatial-level stereo supervision, and the corresponding supervision loss is:

[0089] 3D score distillation sampling loss: the 3D score distillation sampling loss is used to supervise the generation of 3D objects, and the specific gradient of the 3D score distillation sampling loss is: ​

[0090]

[0091] where, denotes the gradient of parameter θ; θ denotes the parameters of 3D model; denotes 3D score distillation sampling loss; denotes expectation; λ(t) denotes loss weight; denotes denoising function; z t denotes noise latent vector; c(x r denotes the embedding of reference view and relative camera extrinsic; x r denotes reference view; denotes relative camera rotation of target view; denotes relative camera translation of target view; t denotes time step; ∈ is standard Gaussian noise; z denotes latent vector;

[0092] Reference view loss: the reference view loss is used to improve the quality of 3D object generated from the reference view, and the calculation formula of the loss function is:

[0093]

[0094] where, denotes reference view loss; ω rgb denotes loss weight; denotes square of two-norm; m denotes reference view mask; ⊙ denotes Hadamard product; x r denotes reference view; is the image rendered from the reference view; ω mask denotes loss weight; M(·) denotes foreground mask;

[0095] 3D score distillation sampling loss and reference view loss constitute spatial level stereo supervision together, and the calculation formula of the loss function is:

[0096]

[0097] where, denotes inter-level stereo supervision loss; ω 3d denotes loss weight; denotes reference view loss;

[0098] Pixel level plane supervision and spatial level stereo supervision constitute hybrid diffusion supervision in three-stage optimization process, and the calculation formula is:

[0099]

[0100] where, denotes hybrid diffusion supervision loss; represents pixel-level plane supervision; represents spatial-level stereo supervision loss.

[0101] The 2D diffusion model in step S2 adopts a Stable Diffusion v2-1-unclip model, and the 3D diffusion model adopts a Zero-1-to-3 model.

[0102] S3, using a three-stage optimization method, generating a 3D object by combining mixed diffusion supervision at each stage;

[0103] The specific process of step S3 is as follows:

[0104] S31, in the first stage, using the modal similarity loss in the pixel-level plane supervision and the enhanced 2D score distillation sampling loss, while combining the spatial-level stereo supervision and the normal vector regularization to update the neural radiance field model until convergence, for learning the rough texture and 3D geometry aligned with the modal prompt, and using a low-resolution NeRF model for 3D representation;

[0105] S32, in the second stage, taking the rendered normal map of the 3D object as the input of the 2D and 3D diffusion models, using the modal similarity loss in the pixel-level plane supervision and the consistency distillation sampling loss, while using the spatial-level stereo supervision, the normal consistency loss and the Laplacian smoothing loss to optimize the differential differentiable voxel grid embedding body model, for enhancing the geometric details of the 3D object, and using a high-resolution differential differentiable voxel grid embedding body model as the 3D representation, converting the 3D representation from the implicit neural radiance field model to the explicit differential differentiable voxel grid embedding body model, and optimizing the geometry and texture of the 3D object separately;

[0106] S33, inputting the color image rendered by the differential differentiable voxel grid embedding body model into the pre-trained 2D diffusion model and 3D perception diffusion model, and generating a high-fidelity 3D object with rich texture details by using the modal similarity loss in the pixel-level plane supervision and the enhanced 2D score distillation sampling loss, and the spatial-level stereo supervision.

[0107] Specific experimental results:

[0108] Figure 3 The results generated by the multi-modal to 3D object generation method (XBind) of the application through mixed diffusion supervision are shown. The first row represents text-to-3D, the second row represents image-to-3D, and the image prompt input is located in the lower left corner of each generated result, and the third row represents audio-to-3D. Specifically, for image input, the focus of the application is generation rather than reconstruction, aiming to generate a 3D object similar in style to the input image, rather than a 1:1 reconstruction. From Figure 3As can be seen from Table 1, regardless of the type of modal prompt, the method for generating multi-modal to 3D objects by mixed diffusion supervision (XBind) of the present application can generate high-fidelity 3D objects with detailed geometry and texture, while maintaining excellent 3D consistency. This proves the effectiveness of the present application.

[0109] Figure 4 A qualitative comparison of the generation results of the method for generating multi-modal to 3D objects by mixed diffusion supervision (XBind) of the present application and the baseline method is shown. The first row represents text-to-3D, the second row represents image-to-3D, and the third row represents audio-to-3D. As a pioneering work of 3D generation with arbitrary modal prompts, the present application developed three baseline models for comparative analysis. These baseline models use different diffusion models and loss functions to perform three-stage 3D object optimization. First, ZeroStableCDS-3D uses Zero-1-to-3 and Stable unCLIP as 3D and 2D diffusion priors, respectively, and uses CDS loss for supervision in all stages. Second, StableSDS-3D only uses Stable unCLIP as a 2D diffusion prior, and uses SDS loss for supervision in all stages. Finally, ZeroSDS-3D only uses Zero-1-to-3 as a 3D diffusion prior, and uses SDS loss as supervision in all stages. From Figure 4 As can be seen from Table 1, the method for generating multi-modal to 3D objects by mixed diffusion supervision (XBind) of the present application surpasses other baseline methods by generating 3D consistent objects with high fidelity and rich details, highlighting its excellent performance, which is due to the mixed diffusion supervision and different loss functions used in the three stages of the present application.

[0110] Figure 5 A qualitative comparison of the generation results of the method for generating multi-modal to 3D objects by mixed diffusion supervision (XBind) of the present application and the generation results of other advanced methods in the field of text-to-3D is shown. As shown in Figure 5 In this comparison, the present application includes four different advanced text-to-3D methods: DreamFusion, Magic3D, Fantasia3D, and ProlificDreamer. In order to maintain fairness in the comparison, Figure 5 The results of the four advanced methods shown in Table 2 were obtained using the implementation of the open source library threestudio. As shown in Figure 5 As can be seen from Table 2, the method for generating multi-modal to 3D objects by mixed diffusion supervision (XBind) of the present application is superior to these four advanced methods in terms of both geometric structure and texture quality, highlighting the effectiveness of the modules in XBind and demonstrating its ability to generate high-fidelity textured meshes.

[0111] Figure 6 The ablation study of the generation method of multi-modal to 3D object (XBind) by mixed diffusion supervision is shown. The first row represents text-to-3D, the second row represents image-to-3D, and the third row represents audio-to-3D. The present application conducts an ablation study on each module in the generation method of multi-modal to 3D object (XBind) by mixed diffusion supervision to evaluate their effectiveness. The results are shown in Figure 6 (a) lack of spatial level stereo supervision, the model lacks spatial prior, resulting in incorrect geometry and texture, and obvious 3D inconsistency; (b) lack of pixel level plane supervision, the model is difficult to generalize, and the texture and geometry generated from non-reference view are poor in quality; (c) relying only on the first stage for 3D optimization will cause the geometry and texture to be blurred, and lack of realism; (d) lack of MS loss, the model shows 3D inconsistency and lack of details, highlighting the role of MS loss in ensuring 3D consistency and high quality geometry and texture; (e) in the case of integration of all modules, the generation method of multi-modal to 3D object (XBind) by mixed diffusion supervision can generate high quality texture mesh.

[0112] The above is only the preferred embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for multi-modal-to-3D object generation by mixed diffusion supervision, the method comprising: The method comprises the following steps: S1, using a modal similarity loss to align the encoding of the modal prompt and the rendered image, and better injecting semantic information of multiple modalities into the generation process of the 3D object, so as to make the generated 3D object more in line with the requirements of each modal prompt; Step S1 aligns the embedding generated by the multi-modal alignment encoder with the CLIP embedding of the image rendered from the 3D object, enhances the correct guidance of each modality in the 3D object generation process, and the specific process is: S11, using a differentiable renderer to render a set of images x from the corresponding NeRF model or Mesh model under a given camera perspective p; S12, data augmentation is performed on the image x to obtain detailed geometric structure and texture; the data augmentation includes global enhancement, local enhancement and normalization; wherein the global enhancement is used for random perspective transformation of the rendered image, and the local enhancement is used for random cropping and perspective transformation of the rendered image; S13, input the rendered image after data augmentation into a CLIP image encoder to obtain its embedding, and then calculate the modal similarity loss between the embedding of the input modal prompt and the embedding of the rendered image, and the calculation formula is: wherein, represents the final overall modal similarity loss; represents the modal similarity loss after global augmentation on the rendered images; represents the modal similarity loss after local augmentation on the rendered images; represents the modal similarity loss after normalization on the rendered images; ω g , ω l , and ω z are weighting parameters; i represents the number of data augmentations; n represents the number of data augmentations in each iteration; W i represents the similarity weight during the i-th augmentation; avg[·] represents the calculation of the average similarity between the input modal embedding and all rendered image embeddings; cos(a, b) represents the cosine similarity between a and b; C m represents the modal cue embedding encoded by the multi-modal alignment encoder; ε represents the CLIP image encoder; G(x) represents the global augmentation operation on the rendered image x; L(x) represents the local augmentation operation on the rendered image x; Z(x) represents the normalization operation on the rendered image x; S2, using hybrid diffusion supervision to utilize the prior knowledge of 2D diffusion model and 3D diffusion model, and combining the advantages of the two pre-trained diffusion models to improve the generation effect and consistency of the 3D object: S3, using a three-stage optimization method, combining hybrid diffusion supervision in each stage to generate a 3D object; The specific process of step S3 is: S31, in the first stage, the modal similarity loss in the pixel-level plane supervision and the enhanced 2D score distillation sampling loss are used, and the spatial-level stereo supervision and normal regularization are combined to update the neural radiation field model until convergence, which is used to learn the rough texture and 3D geometric shape aligned with the modal prompt, and a low-resolution NeRF model is used for 3D representation; S32, in the second stage, the rendered normal map of the 3D object is used as the input of the 2D and 3D diffusion models, the modal similarity loss in the pixel-level plane supervision and the consistency distillation sampling loss are used, and the spatial-level stereo supervision, normal consistency loss and Laplace smoothing loss are used to optimize the differential differentiable voxel grid embedding body model, which is used to enhance the geometric details of the 3D object, and a high-resolution differential differentiable voxel grid embedding body model is used as the 3D representation, the 3D representation is converted from the implicit neural radiation field model to the explicit differential differentiable voxel grid embedding body model, and the geometry and texture of the 3D object are optimized separately; S33, input the color image rendered by the differential differentiable voxel grid embedding body model into the pre-trained 2D diffusion model and 3D perception diffusion model, and generate a high-fidelity 3D object with rich texture details by using the modal similarity loss in the pixel-level plane supervision and the enhanced 2D score distillation sampling loss in the pixel-level plane supervision, and the spatial-level stereo supervision.

2. A method for multi-modal to 3D object generation by mixed diffusion supervision as claimed in claim 1, wherein, The hybrid diffusion supervision in step S2 includes pixel-level plane supervision and spatial-level stereo supervision.

3. A method for multi-modal to 3D object generation by mixed diffusion supervision as claimed in claim 2, wherein, The 2D diffusion model and the modality similarity loss are used together as pixel-level plane supervision in step S2, guiding the model to generate 3D in the process of 3D object generation with any modality as a condition, and the corresponding supervision loss is: Consistency distillation sampling loss: supervised by consistency distillation sampling loss, consistency distillation sampling uses ordinary differential equation deterministic sampling prior for 3D generation, and the calculation formula of consistency distillation sampling loss is: ​ wherein, denotes an expected value; l(t2) denotes a loss weight; denotes a squared two-norm; D Φ (·) denotes a pre-trained 2D diffusion model; denotes a noisy latent vector; t1 and t2 denote two adjacent diffusion time steps, with t1 > t2; C m represents a modality cue embedding encoded by a multi-modal alignment encoder; sg(·) denotes a stop-gradient operator; denotes a less noisy latent vector resulting from a deterministic sampling of a discretization step; denotes a less noisy latent vector resulting from a deterministic sampling of a discretization step; Enhanced 2D score distillation sampling loss: supervised by the score distillation sampling loss in Dreamfusion, the score distillation sampling distills the 2D prior of the pre-trained diffusion model into the 3D model, the 3D model is represented by parameters θ, and the score distillation sampling loss is extended from the latent space to the image space, and the final gradient calculation is: wherein, denotes the gradient of the parameter θ; θ denotes a parameter of the 3D model; denotes a loss function that extends the score distillation sampling loss from the latent space to the image space; denotes the expectation; λ(t) denotes a loss weight; z denotes a latent vector; denotes an estimate of the latent vector z using a denoising function; ω img is a weighting parameter; x denotes an image rendered from the 3D object; denotes an estimate image obtained by decoding the diffusion model prediction denotes an estimate image obtained by decoding the diffusion model prediction Using a regularization method Minimizing the variance of the sampled z coordinates along the NeRF ray distribution, the calculation formula of the enhanced 2D score distillation sampling loss is: wherein, represents an enhanced 2D score distillation sampling loss; ω * represents a loss function weight; represents a loss function that extends the score distillation sampling loss from the latent space to the image space; ω nz represents a loss function weight; represents a regularization loss; The complete pixel-level plane supervision includes the modality similarity loss, the consistency distillation sampling loss and the enhanced 2D score distillation sampling loss, and the specific formula of the pixel-level plane supervision is: where, represents pixel-level plane supervision; ω ms represents loss function weight; represents modality similarity loss; ω cds represents loss function weight; represents consistency distillation sampling loss; represents enhanced 2D score distillation sampling loss; represents using only one component as supervision at different stages of XBind optimization.

4. The method of generating multi-modal to 3D objects by mixed diffusion supervision of claim 3, wherein, The 3D diffusion model and the 2D diffusion model are used together as spatial-level stereo supervision in step S2, and the corresponding supervision loss is: 3D score distillation sampling loss: using the 3D score distillation sampling loss to supervise the generation of 3D objects, and the specific gradient of the 3D score distillation sampling loss is: wherein, denotes the gradient of the parameter θ; θ denotes a parameter of the 3D model; denotes the 3D score distillation sampling loss; denotes the expectation; λ(t) denotes the loss weight; denotes the denoising function; z t denotes the noise latent vector; c(x r denotes the embedding of the reference view and the relative camera extrinsics; x r denotes the reference view; denotes the relative camera rotation of the target view; denotes the relative camera translation of the target view; t denotes the time step; ∈ is a standard Gaussian noise; z denotes the latent vector; Reference view loss: using the reference view loss to improve the quality of the 3D object generated from the reference view angle, and the calculation formula of the loss function is: wherein, denotes the reference view loss; ω rgb denotes the loss weight; denotes the squared two-norm; m denotes the reference view mask; denotes the Hadamard product; x r denotes the reference view; is the image rendered from the reference perspective; ω mask denotes the loss weight; M(·) denotes the foreground mask; The 3D score distillation sampling loss and the reference view loss constitute the spatial-level stereo supervision, and the calculation formula of the loss function is: wherein, denotes the spatial level stereo supervision loss; ω 3d denotes the loss weight; denotes the reference view loss; The pixel-level plane supervision and the spatial-level stereo supervision constitute the mixed diffusion supervision in the three-stage optimization process, and the calculation formula is: wherein, represents the mixed diffusion supervision loss; represents the pixel-level plane supervision; represents the spatial-level stereo supervision loss.

5. A method for multi-modal to 3D object generation by mixed diffusion supervision as claimed in claim 4, wherein, The 2D diffusion model in step S2 adopts the Stable Diffusion v2-1-unclip model, and the 3D diffusion model adopts the Zero-1-to-3 model.

Images