Multi-modal to 3D object generation method through mixed diffusion supervision

Abstract

The invention discloses a method for generating a multi-modal 3D object through mixed diffusion supervision, and the method comprises the following steps: S1, employing modal similarity loss to align modal prompt and rendering image coding, better injecting semantic information of multiple modals into the generation process of the 3D object, the 3D object generation module is used for enabling a generated 3D object to better meet the requirements of each modal prompt; s2, mixed diffusion supervision is adopted, priori knowledge of a 2D diffusion model and a 3D diffusion model is utilized, and the advantages of the two pre-training diffusion models are combined, so that the generation effect and consistency of a 3D object are improved; S3, a three-stage optimization method is utilized, mixed diffusion supervision is combined in each stage, and the 3D object is generated; according to the method, by adopting modal similarity loss, mixed diffusion supervision and three-stage optimization methods, the generation quality of the 3D object can be effectively improved, and the method has remarkable practical value and application prospect.

Description

Technical Field

[0001] The present invention relates to the field of computer processing technology, and in particular to a method for generating a 3D object from a multi-modality through mixed diffusion supervision. Background Art

[0002] In recent years, the field of multimodal modeling has achieved remarkable breakthroughs and progress, and the pace of technological and research progress has accelerated. Some studies have explored the joint training of multimodality in supervised and self-supervised settings, aiming to build a unified representation of various modalities using a single model to achieve a more comprehensive cross-modal understanding. Visual Transformer (ViT) has been widely used in downstream tasks such as visual question answering and image captioning due to its diverse model architectures and training methods. Multimodal encoders have also achieved remarkable success in the fields of vision-language, video-audio, and video-speech-language. Aligning data of different modalities is an active research area, showing great potential in cross-modal retrieval and building unified multimodal representations.

[0003] IMAGEBIND is a method for learning a shared representation space by leveraging various types of image pairing data. It does not require a dataset where all modalities co-occur, and using only image pairing data is sufficient to bring these modalities together. IMAGEBIND can leverage the latest large-scale visual-language models and extend its zero-shot capabilities to new modalities through natural pairing of other modalities with images. This property enables IMAGEBIND to perform rich combinatorial multimodal tasks between different modalities. However, these tasks are currently limited to the two-dimensional domain, and how to apply IMAGEBIND to the three-dimensional domain remains unexplored.

[0004] Diffusion models learn data distribution by denoising and recovering the original data. With the help of pre-trained models, diffusion models have made significant progress in image, video, speech and 3D generation, revolutionizing the field of computer vision. Pioneering works such as Stable Diffusion and DeepFloyd generate high-quality images from text prompts by learning prior knowledge from large-scale datasets. Subsequent works further fine-tuned text-to-image diffusion models to better adapt to downstream tasks. For example, Stable unCLIP can accept CLIP image embeddings as well as text prompts to achieve image variants. In addition, some studies have developed diffusion models that achieve 3D perception by generating object-rendered images. Diffusion models are powerful tools for complex data modeling and generation, and have broad application prospects due to their robust performance. In the field of automatic 3D content generation, their strong prior knowledge is often used to guide the generation of 3D objects.

[0005] Consistency Distillation Sampling (CDS) is a method to distill the prior knowledge of the diffusion model into the field of 3D generation, which explores the deterministic sampling prior of the ordinary differential equation (ODE) for 3D generation. Specifically, in each training iteration, given an image rendered by a 3D model, the target 3D score function is first estimated using a pre-trained 2D diffusion model, and an ODE is constructed for trajectory sampling. Then two adjacent samples are sampled along the ODE trajectory, and the samples with less noise are used to guide the samples with more noise, thereby distilling the deterministic prior into the 3D model. The CDS loss effectively solves the uncertainty problem of the stochastic differential equation (SDE) in the existing and widely used score distillation sampling (SDS) loss, providing a more reliable and consistent framework to guide the 3D generation process. However, due to the inherent limitations of the diffusion model itself, the 2D diffusion model enhances the imagination and generalization ability of 3D generation, but because it is trained on a large-scale 2D dataset, it performs poorly in terms of 3D consistency and certainty; in contrast, the 3D diffusion model ensures more accurate 3D geometric structure and consistency, but due to the limited size of the 3D training dataset and the lack of generalization, it often results in the generated 3D object surface being too smooth.

[0006] Recent progress in 3D object generation has benefited from the robust priors provided by diffusion models. However, current models are task-specific and can only serve one modality at a time, requiring retraining when switching modalities. Given an image-generated 3D model and textual prompts, a simple approach is to convert the textual prompts into images and then use the image-generated 3D model for generation. However, this approach is time-consuming and laborious, and inevitably leads to information loss due to modality switching. Summary of the invention

[0007] To solve the above problems, the present invention proposes a method for generating multi-modal 3D objects through mixed diffusion supervision.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A method for generating multimodal 3D objects through mixed diffusion supervision, comprising the following steps:

[0010] S1. Use modal similarity loss to align the encoding of modal prompts and rendered images, and better inject the semantic information of multiple modalities into the generation process of 3D objects, so as to make the generated 3D objects more in line with the requirements of each modal prompt;

[0011] S2. Hybrid diffusion supervision uses the prior knowledge of 2D diffusion model and 3D diffusion model to improve the generation effect and consistency of 3D objects by combining the advantages of two pre-trained diffusion models:

[0012] S3. Generate 3D objects using a three-stage optimization approach by incorporating mixed diffusion supervision at each stage.

[0013] Preferably, step S1 enhances the correct guidance of each modality in the 3D object generation process by aligning the embedding generated by the multimodal alignment encoder with the CLIP embedding of the image rendered from the 3D object, and the specific process is:

[0014] S11. Render a set of images x at a given camera view p from the corresponding NeRF model or Mesh model using a differentiable renderer;

[0015] S12, performing data enhancement on the image x to obtain detailed geometric structure and texture; the data enhancement includes global enhancement, local enhancement and normalization; wherein the global enhancement is used to perform random perspective transformation on the rendered image, and the local enhancement is used to perform random cropping and perspective transformation on the rendered image;

[0016] S13, input the data-augmented rendered image into the CLIP image encoder to obtain its embedding, and then calculate the modality similarity loss between the embedding of the input modality prompt and the embedding of the rendered image, and the calculation formula is:

[0017]

[0018]

[0019]

[0020]

[0021] in, Represents the final overall modality similarity loss; represents the modality similarity loss after global enhancement of the rendered image; Represents the modality similarity loss after local enhancement of the rendered image; represents the modal similarity loss after normalizing the rendered image; ω g ,ω l and ω z is a weighting parameter; i represents the number of data enhancements; n represents the number of data enhancements in each iteration; W irepresents the similarity weight during the i-th enhancement; avg[·] represents the calculation of the average similarity between the input modality embedding and all rendered image embeddings; cos(a,b) represents the cosine similarity between a and b; C m represents the modality hint embedding encoded by the multimodal alignment encoder; ε represents the CLIP image encoder; G(x) represents the global enhancement operation on the rendered image x; L(x) represents the local enhancement operation on the rendered image x; Z(x) represents the normalization operation on the rendered image x.

[0022] Preferably, the hybrid diffusion supervision in step S2 includes pixel-level planar supervision and space-level stereo supervision.

[0023] Preferably, in step S2, the 2D diffusion model and the modality similarity loss are used together as pixel-level plane supervision to guide the model of 3D generation with any modality as a condition during the 3D object generation process, and the corresponding supervision loss is:

[0024] Consistent distillation sampling loss: Consistent distillation sampling loss is used for supervision. Consistent distillation sampling uses ordinary differential equation deterministic sampling priors for 3D generation. Consistent distillation sampling loss The calculation formula is:

[0025]

[0026] in, represents the expected value; λ(t2) represents the loss weight; represents the square of the two-norm; D Φ (·) represents a pre-trained 2D diffusion model; represents the noise potential vector; t1 and t2 represent two adjacent diffusion time steps, and t1>t2; C m represents the modality cue embedding encoded by the multimodal alignment encoder; sg(·) denotes the stop gradient operator; Represents the solution of the numerical ordinary differential equation from A latent vector with less noise obtained by deterministic sampling with one discretization step;

[0027] Enhanced 2D score distillation sampling loss: The score distillation sampling loss in Dreamfusion is used for supervision. The score distillation sampling distills the 2D prior of the pre-trained diffusion model into the 3D model. The 3D model is represented by parameter θ and the score distillation sampling loss is extended from the latent space to the image space. The final gradient calculation is:

[0028]

[0029] in, represents the gradient of parameter θ; θ represents the parameter of the 3D model; represents the loss function that extends the score distillation sampling loss from the latent space to the image space; represents the expectation; λ(t) represents the loss weight; z represents the potential vector; represents the estimation of the latent vector z using the denoising function; ω img is a weighting parameter; x represents the image rendered from the 3D object; represents the predicted by the diffusion model by decoding The estimated image obtained;

[0030] Using regularization method Minimizing the variance of the z-coordinates of the samples along the NeRF ray distribution, the enhanced 2D score distillation sampling loss is calculated as:

[0031]

[0032] in, represents the enhanced 2D score distillation sampling loss; ω* represents the loss function weight; represents the loss function that extends the score distillation sampling loss from the latent space to the image space; ω nz Represents the weight of the loss function; represents the regularization loss;

[0033] The complete pixel-level plane supervision includes modality similarity loss, consistency distillation sampling loss and enhanced 2D score distillation sampling loss. The specific formula of pixel-level plane supervision is:

[0034]

[0035] in, represents pixel-level planar supervision; ω ms Represents the weight of the loss function; represents the modality similarity loss; ω cds Represents the weight of the loss function; represents the consistency distillation sampling loss; represents the enhanced 2D score distillation sampling loss; Indicates that only one of the components is used as supervision at different stages of XBind optimization.

[0036] Preferably, in step S2, the 3D diffusion model and the 2D diffusion model are used together as spatial-level stereo supervision, and the corresponding supervision loss is:

[0037] 3D score distillation sampling loss: 3D object generation is supervised using the 3D score distillation sampling loss. The specific gradient of the 3D score distillation sampling loss is:

[0038]

[0039] in, represents the gradient of parameter θ; θ represents the parameter of the 3D model; represents the 3D score distillation sampling loss; represents expectation; λ(t) represents loss weight; represents the denoising function; z t represents the noise potential vector; c(x r , R, T) represents the embedding of the reference view and the relative camera extrinsic parameters; x r represents a reference view; Relative camera rotation representing the target viewpoint; represents the relative camera translation of the target view; t represents the time step; ∈ is standard Gaussian noise; z represents the latent vector;

[0040] Reference view loss: Reference view loss is used to improve the quality of 3D objects generated from a reference view. The loss function is calculated as:

[0041]

[0042] in, represents the reference view loss; ω rgb represents the loss weight; represents the square of the two-norm; m represents the reference view mask; ⊙ represents the Hadamard product; x r represents a reference view; is the image rendered from the reference viewpoint; ω mask represents the loss weight; M(·) represents the foreground mask;

[0043] The two supervision methods, 3D score distillation sampling loss and reference view loss, together constitute spatial-level stereo supervision. The loss function is calculated as:

[0044]

[0045] in, represents the spatial-level stereo supervision loss; ω 3d represents the loss weight; represents the reference view loss;

[0046] Pixel-level planar supervision and spatial-level stereo supervision together constitute the hybrid diffusion supervision in the three-stage optimization process, and the calculation formula is:

[0047]

[0048] in, represents the mixed diffusion supervision loss; represents pixel-level planar supervision; represents the spatial-level stereo supervision loss.

[0049] Preferably, the 2D diffusion model in step S2 adopts the Stable Diffusion v2-1-unclip model, and the 3D diffusion model adopts the Zero-1-to-3 model.

[0050] Preferably, the specific process of step S3 is:

[0051] S31. In the first stage, the modality similarity loss in pixel-level plane supervision and the enhanced 2D score distillation sampling loss are used, while spatial-level stereo supervision and normal vector regularization are combined to update the neural radiation field model until convergence, which is used to learn the rough texture and 3D geometry aligned with the modality hint, and use the low-resolution NeRF model for 3D representation;

[0052] S32. In the second stage, the rendered normal map of the 3D object is used as the input of the 2D and 3D diffusion models. The modal similarity loss and the consistency distillation sampling loss in the pixel-level plane supervision are used. At the same time, the spatial-level stereo supervision, the normal consistency loss and the Laplace smoothness loss are used to optimize the differentially differentiable voxel grid embedding model to enhance the geometric details of the 3D object. The high-resolution differentially differentiable voxel grid embedding model is used as the 3D representation. The 3D representation is converted from the implicit neural radiation field model to the explicit differentially differentiable voxel grid embedding model, and the geometry and texture of the 3D object are optimized separately.

[0053] S33. The color image rendered by the differentially differentiable voxel grid embedded model is input into the pre-trained 2D diffusion model and the 3D-aware diffusion model, and high-fidelity 3D objects with rich texture details are generated by utilizing the modality similarity loss in pixel-level planar supervision and the enhanced 2D score distillation sampling loss, as well as spatial-level stereo supervision.

[0054] After adopting the above technical solution, the present invention has the following beneficial effects: A method (XBind) for generating multimodal to 3D objects through hybrid diffusion supervision of the present invention combines a multimodal alignment encoder with a pre-trained diffusion model, and can generate 3D objects from multiple modalities such as text, images, and audio. Among them, the present invention also introduces a new loss function, called Modality Similarity (MS) loss, which aligns the encoding of modal prompts and rendered images, and better injects the semantic information of multiple modalities into the generation process of 3D objects, ensuring that the generated 3D objects are more in line with the requirements of each modal prompt. In addition, hybrid diffusion supervision makes full use of the prior knowledge of 2D diffusion models and 3D diffusion models, and by combining the advantages of two pre-trained diffusion models, the generation effect and consistency of 3D objects are guaranteed. Finally, the three-stage optimization framework further improves the generation quality of 3D objects by combining hybrid diffusion supervision and utilizing three-stage step-by-step optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A diagram showing the generation result of the multimodal prompt of the present invention;

[0056] Figure 2 This is a model architecture diagram of the present invention;

[0057] Figure 3 Generate a result graph for the 3D object of XBind proposed by the present invention;

[0058] Figure 4 A qualitative comparison of the results generated by the proposed XBind and baseline methods;

[0059] Figure 5 A qualitative comparison diagram of the generation results of the Vincent 3D task performed by XBind proposed in the present invention and other advanced methods in the Vincent 3D field;

[0060] Figure 6 This is a diagram showing the ablation study of each module in the XBind proposed in the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0062] like Figures 1 to 6 As shown, a method for generating a multi-modal 3D object through mixed diffusion supervision includes the following steps:

[0063] S1. Use modal similarity loss to align the encoding of modal prompts and rendered images, and better inject the semantic information of multiple modalities into the generation process of 3D objects, so as to make the generated 3D objects more in line with the requirements of each modal prompt;

[0064] Step S1 enhances the correct guidance of each modality in the 3D object generation process by aligning the embedding generated by the multimodal alignment encoder with the CLIP embedding of the image rendered from the 3D object. The specific process is:

[0065] S11. Render a set of images x at a given camera view p from the corresponding NeRF model or Mesh model using a differentiable renderer;

[0066] S12, performing data enhancement on the image x to obtain detailed geometric structure and texture; the data enhancement includes global enhancement, local enhancement and normalization; wherein the global enhancement is used to perform random perspective transformation on the rendered image, and the local enhancement is used to perform random cropping and perspective transformation on the rendered image;

[0067] S13, input the data-augmented rendered image into the CLIP image encoder to obtain its embedding, and then calculate the modality similarity loss between the embedding of the input modality prompt and the embedding of the rendered image, and the calculation formula is:

[0068]

[0069]

[0070]

[0071]

[0072] in, Represents the final overall modality similarity loss; represents the modality similarity loss after global enhancement of the rendered image; Represents the modality similarity loss after local enhancement of the rendered image; represents the modal similarity loss after normalizing the rendered image; ω g ,ω l and ω z is a weighting parameter; i represents the number of data enhancements; n represents the number of data enhancements in each iteration; W i represents the similarity weight during the i-th enhancement; avg[·] represents the calculation of the average similarity between the input modality embedding and all rendered image embeddings; cos(a, b) represents the cosine similarity between a and b; C mrepresents the modality hint embedding encoded by the multimodal alignment encoder; ε represents the CLIP image encoder; G(x) represents the global enhancement operation on the rendered image x; L(x) represents the local enhancement operation on the rendered image x; Z(x) represents the normalization operation on the rendered image x.

[0073] S2. Hybrid diffusion supervision uses the prior knowledge of 2D diffusion model and 3D diffusion model to improve the generation effect and consistency of 3D objects by combining the advantages of two pre-trained diffusion models:

[0074] The hybrid diffusion supervision described in step S2 includes pixel-level planar supervision and space-level stereo supervision.

[0075] In step S2, the 2D diffusion model and the modality similarity loss are used together as pixel-level plane supervision to guide the model of 3D generation with any modality as a condition during the 3D object generation process. The corresponding supervision loss is:

[0076] Consistent distillation sampling loss: Consistent distillation sampling loss is used for supervision. Consistent distillation sampling uses ordinary differential equation deterministic sampling priors for 3D generation. Consistent distillation sampling loss The calculation formula is:

[0077]

[0078] in, represents the expected value; λ(t2) represents the loss weight; represents the square of the two-norm; D Φ (·) represents a pre-trained 2D diffusion model; represents the noise potential vector; t1 and t2 represent two adjacent diffusion time steps, and t1>t2; C m represents the modality cue embedding encoded by the multimodal alignment encoder; sg(·) denotes the stop gradient operator; Represents the solution of the numerical ordinary differential equation from A latent vector with less noise obtained by deterministic sampling with one discretization step;

[0079] Enhanced 2D score distillation sampling loss: The score distillation sampling loss in Dreamfusion is used for supervision. The score distillation sampling distills the 2D prior of the pre-trained diffusion model into the 3D model. The 3D model is represented by parameter θ and the score distillation sampling loss is extended from the latent space to the image space. The final gradient calculation is:

[0080]

[0081] in, represents the gradient of parameter θ; θ represents the parameter of the 3D model; represents the loss function that extends the score distillation sampling loss from the latent space to the image space; represents the expectation; λ(t) represents the loss weight; z represents the potential vector; represents the estimation of the latent vector z using the denoising function; ω img is a weighting parameter; x represents the image rendered from the 3D object; represents the predicted by the diffusion model by decoding The estimated image obtained;

[0082] Using regularization method Minimizing the variance of the z-coordinates of the samples along the NeRF ray distribution, the enhanced 2D score distillation sampling loss is calculated as:

[0083]

[0084] in, represents the enhanced 2D score distillation sampling loss; ω* represents the loss function weight; represents the loss function that extends the score distillation sampling loss from the latent space to the image space; ω nz Represents the weight of the loss function; represents the regularization loss;

[0085] The complete pixel-level plane supervision includes modality similarity loss, consistency distillation sampling loss and enhanced 2D score distillation sampling loss. The specific formula of pixel-level plane supervision is:

[0086]

[0087] in, represents pixel-level planar supervision; ω ms Represents the weight of the loss function; represents the modality similarity loss; ω cds Represents the weight of the loss function; represents the consistency distillation sampling loss; represents the enhanced 2D score distillation sampling loss; Indicates that only one of the components is used as supervision at different stages of XBind optimization.

[0088] In step S2, the 3D diffusion model and the 2D diffusion model are used together as spatial-level stereo supervision, and the corresponding supervision loss is:

[0089] 3D score distillation sampling loss: 3D object generation is supervised using the 3D score distillation sampling loss. The specific gradient of the 3D score distillation sampling loss is:

[0090]

[0091] in, represents the gradient of parameter θ; θ represents the parameter of the 3D model; represents the 3D score distillation sampling loss; represents expectation; λ(t) represents loss weight; represents the denoising function; z t represents the noise potential vector; c(x r , R, T) represents the embedding of the reference view and the relative camera extrinsic parameters; x r represents a reference view; Relative camera rotation representing the target viewpoint; represents the relative camera translation of the target view; t represents the time step; ∈ is standard Gaussian noise; z represents the latent vector;

[0092] Reference view loss: Reference view loss is used to improve the quality of 3D objects generated from a reference view. The loss function is calculated as:

[0093]

[0094] in, represents the reference view loss; ω rgb represents the loss weight; represents the square of the two-norm; m represents the reference view mask; ⊙ represents the Hadamard product; x r represents a reference view; is the image rendered from the reference viewpoint; ω mask represents the loss weight; M(·) represents the foreground mask;

[0095] The two supervision methods, 3D score distillation sampling loss and reference view loss, together constitute spatial-level stereo supervision. The loss function is calculated as:

[0096]

[0097] in, represents the inter-level stereo supervision loss; ω 3d represents the loss weight; represents the reference view loss;

[0098] Pixel-level planar supervision and spatial-level stereo supervision together constitute the hybrid diffusion supervision in the three-stage optimization process, and the calculation formula is:

[0099]

[0100] in, represents the mixed diffusion supervision loss; represents pixel-level planar supervision; represents the spatial-level stereo supervision loss.

[0101] The 2D diffusion model in step S2 adopts the StableDiffusionv2-1-unclip model, and the 3D diffusion model adopts the Zero-1-to-3 model.

[0102] S3, using a three-stage optimization approach to generate 3D objects by combining mixed diffusion supervision at each stage;

[0103] The specific process of step S3 is:

[0104] S31. In the first stage, the modality similarity loss in pixel-level plane supervision and the enhanced 2D score distillation sampling loss are used, while spatial-level stereo supervision and normal vector regularization are combined to update the neural radiation field model until convergence, which is used to learn the rough texture and 3D geometry aligned with the modality hint, and use the low-resolution NeRF model for 3D representation;

[0105] S32. In the second stage, the rendered normal map of the 3D object is used as the input of the 2D and 3D diffusion models. The modal similarity loss and the consistency distillation sampling loss in the pixel-level plane supervision are used. At the same time, the spatial-level stereo supervision, the normal consistency loss and the Laplace smoothness loss are used to optimize the differentially differentiable voxel grid embedding model to enhance the geometric details of the 3D object. The high-resolution differentially differentiable voxel grid embedding model is used as the 3D representation. The 3D representation is converted from the implicit neural radiation field model to the explicit differentially differentiable voxel grid embedding model, and the geometry and texture of the 3D object are optimized separately.

[0106] S33. The color image rendered by the differentially differentiable voxel grid embedded model is input into the pre-trained 2D diffusion model and the 3D-aware diffusion model, and high-fidelity 3D objects with rich texture details are generated by utilizing the modality similarity loss in pixel-level planar supervision and the enhanced 2D score distillation sampling loss, as well as spatial-level stereo supervision.

[0107] Specific experimental results:

[0108] Figure 3 The results generated by the invention's method (XBind) of generating 3D objects from multiple modalities through mixed diffusion supervision are shown. The first row represents text-generated 3D, the second row represents image-generated 3D, where the image prompt input is located in the lower left corner of each generated result, and the third row represents audio-generated 3D. Specifically, for image input, the focus of the invention is generation rather than reconstruction, aiming to generate 3D objects similar in style to the input image, rather than 1:1 reconstruction. Figure 3It can be seen that regardless of the type of modality cue, the method for generating 3D objects from multiple modalities through hybrid diffusion supervision (XBind) of the present invention is able to generate high-fidelity 3D objects with detailed geometry and texture while maintaining excellent 3D consistency. This proves the effectiveness of the present invention.

[0109] Figure 4 A qualitative comparison of the generation results of a multimodal 3D object generation method (XBind) through hybrid diffusion supervision and baseline methods is shown. The first row represents text-generated 3D, the second row represents image-generated 3D, and the third row represents audio-generated 3D. As a pioneering work in 3D generation with arbitrary modalities as prompts, the present invention developed three baseline models for comparative analysis. These baseline models use different diffusion models and loss functions for three-stage 3D object optimization. First, ZeroStableCDS-3D uses Zero-1-to-3 and Stable unCLIP as 3D and 2D diffusion priors, respectively, and uses CDS loss for supervision in all stages. Secondly, StableSDS-3D only uses Stable unCLIP as 2D diffusion prior, and uses SDS loss for supervision in all stages. Finally, ZeroSDS-3D only uses Zero-1-to-3 as 3D diffusion prior, and uses SDS loss as supervision in all stages. From Figure 4 It can be seen that the method of generating multimodal 3D objects through hybrid diffusion supervision (XBind) of the present invention surpasses other baseline methods by generating 3D consistent objects with high fidelity and rich details, highlighting its excellent performance, which is due to the hybrid diffusion supervision and different loss functions adopted by the present invention in three stages.

[0110] Figure 5 The qualitative comparison of the generation results of a method for generating 3D objects from multimodal objects through hybrid diffusion supervision (XBind) in the present invention and other advanced methods in the field of 3D objects is shown. Figure 5 As shown, in this comparison, the present invention includes four different advanced methods of text to 3D: DreamFusion, Magic3D, Fantasia3D and ProlificDreamer. In order to keep the comparison fair, Figure 5 The results of the four advanced methods presented in this paper are obtained using the open source library threestudio. Figure 5 As shown, the proposed method for generating 3D objects from multimodal objects via hybrid diffusion supervision (XBind) outperforms the four advanced methods in terms of both geometric structure and texture quality, highlighting the effectiveness of each module in XBind and demonstrating its ability to generate high-fidelity textured meshes.

[0111] Figure 6 The ablation study of the method for generating 3D objects from multiple modalities through hybrid diffusion supervision (XBind) of the present invention is shown. The first row represents text-generated 3D, the second row represents image-generated 3D, and the third row represents audio-generated 3D. The present invention performs an ablation study on each module in the method for generating 3D objects from multiple modalities through hybrid diffusion supervision (XBind) to evaluate their effectiveness. The results are shown in Figure 6 As shown in the figure: (a) Lacking spatial-level stereo supervision, the model lacks spatial priors, resulting in incorrect geometry and texture, and obvious 3D inconsistency; (b) Lacking pixel-level planar supervision, the model is difficult to generalize, and the texture and geometry generated from non-reference perspectives are of poor quality; (c) Relying solely on the first stage for 3D optimization leads to blurred geometry and texture, and lack of realism; (d) Lacking MS loss, the model exhibits 3D inconsistency and insufficient details, highlighting the role of MS loss in ensuring 3D consistency and high-quality geometry and texture; (e) When all modules are integrated, the present invention's method for generating multimodal 3D objects through mixed diffusion supervision (XBind) is able to generate high-quality textured meshes.

[0112] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for generating multimodal 3D objects through mixed diffusion supervision, characterized in that: The following steps are involved: S1. Use modal similarity loss to align the encoding of modal prompts and rendered images, and better inject the semantic information of multiple modalities into the generation process of 3D objects, so as to make the generated 3D objects more in line with the requirements of each modal prompt; S2. Hybrid diffusion supervision uses the prior knowledge of 2D diffusion model and 3D diffusion model to improve the generation effect and consistency of 3D objects by combining the advantages of two pre-trained diffusion models: S3. Generate 3D objects using a three-stage optimization approach by incorporating mixed diffusion supervision at each stage.

2. A method for generating a multi-modal 3D object through mixed diffusion supervision as claimed in claim 1, characterized in that: Step S1 enhances the correct guidance of each modality in the 3D object generation process by aligning the embedding generated by the multimodal alignment encoder with the CLIP embedding of the image rendered from the 3D object. The specific process is: S11. Render a set of images x at a given camera view p from the corresponding NeRF model or Mesh model using a differentiable renderer; S12, performing data enhancement on the image x to obtain detailed geometric structure and texture; the data enhancement includes global enhancement, local enhancement and normalization; wherein the global enhancement is used to perform random perspective transformation on the rendered image, and the local enhancement is used to perform random cropping and perspective transformation on the rendered image; S13, input the data-augmented rendered image into the CLIP image encoder to obtain its embedding, and then calculate the modality similarity loss between the embedding of the input modality prompt and the embedding of the rendered image, and the calculation formula is: in, Represents the final overall modality similarity loss; represents the modality similarity loss after global enhancement of the rendered image; Represents the modality similarity loss after local enhancement of the rendered image; represents the modal similarity loss after normalizing the rendered image; ω g ,ω l and ω z is a weighting parameter; i represents the number of data enhancements; n represents the number of data enhancements in each iteration; W i represents the similarity weight during the i-th enhancement; avg[·] represents the calculation of the average similarity between the input modality embedding and all rendered image embeddings; cos(a,b) represents the cosine similarity between a and b; C m represents the modality hint embedding encoded by the multimodal alignment encoder; ε represents the CLIP image encoder; G(x) represents the global enhancement operation on the rendered image x; L(x) represents the local enhancement operation on the rendered image x; Z(x) represents the normalization operation on the rendered image x.

3. The method for generating a multi-modal 3D object by hybrid diffusion supervision as claimed in claim 1, characterized in that: The hybrid diffusion supervision described in step S2 includes pixel-level planar supervision and space-level stereo supervision.

4. A method for generating a multi-modal 3D object through mixed diffusion supervision as claimed in claim 3, characterized in that: In step S2, the 2D diffusion model and the modality similarity loss are used together as pixel-level plane supervision to guide the model of 3D generation with any modality as a condition during the 3D object generation process. The corresponding supervision loss is: Consistent distillation sampling loss: Consistent distillation sampling loss is used for supervision. Consistent distillation sampling uses ordinary differential equation deterministic sampling priors for 3D generation. Consistent distillation sampling loss The calculation formula is: in, represents the expected value; λ(t2) represents the loss weight; represents the square of the two-norm; D Φ (·) represents a pre-trained 2D diffusion model; represents the noise potential vector; t1 and t2 represent two adjacent diffusion time steps, and t1>t2; C m represents the modality cue embedding encoded by the multimodal alignment encoder; sg(·) denotes the stop gradient operator; Represents the solution of the numerical ordinary differential equation from A latent vector with less noise obtained by deterministic sampling with one discretization step; Enhanced 2D score distillation sampling loss: The score distillation sampling loss in Dreamfusion is used for supervision. The score distillation sampling distills the 2D prior of the pre-trained diffusion model into the 3D model. The 3D model is represented by parameter θ and the score distillation sampling loss is extended from the latent space to the image space. The final gradient calculation is: in, represents the gradient of parameter θ; θ represents the parameter of the 3D model; represents the loss function that extends the score distillation sampling loss from the latent space to the image space; represents the expectation; λ(t) represents the loss weight; z represents the potential vector; represents the estimation of the latent vector z using the denoising function; ω img is a weighting parameter; x represents the image rendered from the 3D object; represents the predicted by the diffusion model by decoding The estimated image obtained; Using regularization method Minimizing the variance of the z-coordinates of the samples along the NeRF ray distribution, the enhanced 2D score distillation sampling loss is calculated as: in, represents the enhanced 2D score distillation sampling loss; ω * Represents the weight of the loss function; represents the loss function that extends the score distillation sampling loss from the latent space to the image space; ω nz Represents the weight of the loss function; represents the regularization loss; The complete pixel-level plane supervision includes modality similarity loss, consistency distillation sampling loss and enhanced 2D score distillation sampling loss. The specific formula of pixel-level plane supervision is: in, represents pixel-level planar supervision; ω ms Represents the weight of the loss function; represents the modality similarity loss; ω cds Represents the weight of the loss function; represents the consistency distillation sampling loss; represents the enhanced 2D score distillation sampling loss; Indicates that only one of the components is used as supervision at different stages of XBind optimization.

5. A method for generating a multi-modal 3D object through mixed diffusion supervision as claimed in claim 4, characterized in that: In step S2, the 3D diffusion model and the 2D diffusion model are used together as spatial-level stereo supervision, and the corresponding supervision loss is: 3D score distillation sampling loss: 3D object generation is supervised using the 3D score distillation sampling loss. The specific gradient of the 3D score distillation sampling loss is: in, represents the gradient of parameter θ; θ represents the parameter of the 3D model; represents the 3D score distillation sampling loss; represents expectation; λ(t) represents loss weight; represents the denoising function; z t represents the noise potential vector; c(x r ,R,T) represents the embedding of the reference view and the relative camera extrinsic parameters; x r represents a reference view; Relative camera rotation representing the target viewpoint; represents the relative camera translation of the target view; t represents the time step; ∈ is standard Gaussian noise; z represents the latent vector; Reference view loss: Reference view loss is used to improve the quality of 3D objects generated from a reference view. The loss function is calculated as: in, represents the reference view loss; ω rgb represents the loss weight; represents the square of the two-norm; m represents the reference view mask; ⊙ represents the Hadamard product; x r represents a reference view; is the image rendered from the reference viewpoint; ω mask represents the loss weight; M(·) represents the foreground mask; The two supervision methods, 3D score distillation sampling loss and reference view loss, together constitute spatial-level stereo supervision. The loss function is calculated as: in, represents the spatial-level stereo supervision loss; ω 3d represents the loss weight; represents the reference view loss; Pixel-level planar supervision and spatial-level stereo supervision together constitute the hybrid diffusion supervision in the three-stage optimization process, and the calculation formula is: in, represents the mixed diffusion supervision loss; represents pixel-level planar supervision; represents the spatial-level stereo supervision loss.

6. A method for generating a multi-modal 3D object through mixed diffusion supervision as claimed in claim 5, characterized in that: The 2D diffusion model in step S2 adopts the Stable Diffusion v2-1-unclip model, and the 3D diffusion model adopts the Zero-1-to-3 model.

7. A method for generating a multi-modal 3D object through mixed diffusion supervision as claimed in claim 5, characterized in that: The specific process of step S3 is: S31. In the first stage, the modality similarity loss in pixel-level plane supervision and the enhanced 2D score distillation sampling loss are used, while spatial-level stereo supervision and normal vector regularization are combined to update the neural radiation field model until convergence, which is used to learn the rough texture and 3D geometry aligned with the modality hint, and use the low-resolution NeRF model for 3D representation; S32. In the second stage, the rendered normal map of the 3D object is used as the input of the 2D and 3D diffusion models. The modal similarity loss and the consistency distillation sampling loss in the pixel-level plane supervision are used. At the same time, the spatial-level stereo supervision, the normal consistency loss and the Laplace smoothness loss are used to optimize the differentially differentiable voxel grid embedding model to enhance the geometric details of the 3D object. The high-resolution differentially differentiable voxel grid embedding model is used as the 3D representation. The 3D representation is converted from the implicit neural radiation field model to the explicit differentially differentiable voxel grid embedding model, and the geometry and texture of the 3D object are optimized separately. S33. The color image rendered by the differentially differentiable voxel grid embedded model is input into the pre-trained 2D diffusion model and the 3D-aware diffusion model, and high-fidelity 3D objects with rich texture details are generated by utilizing the modality similarity loss in pixel-level planar supervision and the enhanced 2D score distillation sampling loss, as well as spatial-level stereo supervision.

Images