A method for generating a three-dimensional model driven by multi-modal features guided by text

Abstract

This invention discloses a multimodal feature-driven 3D model generation method guided by images and text, belonging to the field of image data processing technology. The method includes constructing a dataset; pre-training an image encoding model; pre-training a latent diffusion model; constructing a 3D generation model including the image encoding model, latent diffusion model, backprojection layer, and merging layer; acquiring multi-view RGB images or text prompts of the object to be modeled; and generating a 3D Gaussian set of the object to be modeled using the 3D generation model. This invention introduces a 12-channel attribute map and proposes a generation method combining attribute maps, diffusion models, and 3D Gaussian splashing, greatly improving the quality and application value of 3D reconstruction and rendering, and providing an innovative solution for virtual reality, digital twins, and intelligent manufacturing. By solving the consistency and efficiency problems of existing technologies, it provides a new technical path for high-precision 3D modeling, with particularly broad application prospects in industrial inspection and immersive interaction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, and particularly relates to a kind of text-guided multi-modal feature driven three-dimensional model generation method. BACKGROUND

[0002] With the rapid development of three-dimensional generation and rendering technology, three-dimensional representation methods based on diffusion model have shown great potential in virtual reality, digital twin, and intelligent manufacturing. As a generation model, diffusion generation method based on three-dimensional Gaussian splash introduces the joint mechanism of stable diffusion model and variational autoencoder, which not only significantly improves the quality and efficiency of three-dimensional reconstruction, but also ensures multi-view consistency and geometric accuracy, thereby promoting the development of three-dimensional modeling and rendering. In three-dimensional scene generation, ensuring the quality of multi-view attribute map and generated three-dimensional Gaussian set is crucial for performing high-precision modeling and realistic rendering. Practical application requirements often require cross-view consistency, detail restoration, and complex material expression to meet the needs of virtual reality and industrial detection. However, existing technologies generally have the problem of insufficient utilization of multi-view attribute information, often failing to model multiple attributes such as color, depth, scale, and spatial distribution uniformly in the generation stage, resulting in insufficient three-dimensional representation in cross-view consistency and geometric accuracy. At the same time, existing methods often rely on complex three-dimensional optimization or repeated training process in the three-dimensional modeling process, resulting in low inference efficiency and difficulty in achieving high efficiency while ensuring generation quality. Therefore, existing technologies cannot balance geometric consistency and rendering efficiency.

[0003] Glossary:

[0004] Patch Embedding is usually translated as "patch embedding" or "image patch embedding". In visual tasks, it first divides the image into multiple image patches, then flattens each image patch into a long vector, introduces a learnable matrix, linearly projects the long vector into the model hidden space dimension, and then appends classification tokens and position encoding, thereby converting each image patch into a vector representation, and then concatenating the vector sequence. This vector sequence is the input sequence of the Transformer encoder.

[0005] Diffusion Model is a model that learns the distribution of data through a forward process of gradually adding noise and a backward process of gradually removing noise. The forward process gradually adds Gaussian noise to the original data (such as images) until it approaches a pure noise distribution; the backward process predicts the noise at each step by training a neural network (usually UNet), thereby reconstructing the data in reverse.

[0006] LDMs (Latent Diffusion Models) are improvements on diffusion models, with the core being the transfer of the diffusion process from pixel space to latent space. Therefore, their operation involves a three-stage process: compression, diffusion, and restoration. Compression and restoration can be implemented using VAEs (Visual Algorithms).

[0007] Unlike traditional autoencoders, VAEs (Variational Autoencoders) learn not only an effective representation of the data but also its probability distribution. They consist of an encoder and a decoder. The encoder aims to map the input data into a latent space, assuming that the variables in this space follow a certain distribution (usually Gaussian). Then, by sampling from this distribution, a latent code z is generated. The decoder then reconstructs the input data from z.

[0008] GELU (Gaussian Error Linear Unit) is an activation function widely used in deep learning. Summary of the Invention

[0009] The purpose of this invention is to provide a text-guided, multimodal feature-driven 3D model generation method that solves the above-mentioned problems, improves the consistency and geometric accuracy of 3D representation across viewpoints, and enhances rendering efficiency.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows: a method for generating 3D models guided by image and text multimodal features, comprising the following steps:

[0011] S1, construct dataset D;

[0012] S11, Obtain N 3D models, where the nth model is labeled M. n The default attribute map shape is 12×H×W, where 12, H, and W are the number of attribute channels, the height and width of the map, respectively. The 12 channels are divided into 3 color channels, 1 opacity channel, 3 ellipsoidal principal axis channels, 4 rotation quaternion channels, and 1 depth channel.

[0013] S12, for M n Preset V in Each viewpoint renders the RGB image of viewpoint v. Surface normal diagram World coordinate map spliced ​​into a 9-channel tensor , Then render the attribute graph of the viewpoint v. M n Input and tags ;

[0014] S13, obtain M n The text prompt is encoded into a text vector C. n , will (X n ,Y n C n As the nth sample in dataset D, each 3D model is processed sequentially to obtain the dataset. ;

[0015] S2, construct an image coding network, and use... For input, Train an image coding network for the labels to obtain the image coding model;

[0016] S3, a pre-trained latent diffusion model, including S31~S33;

[0017] S31, Construct a potential diffusion model, including a VAE and a conditional diffusion model, wherein the VAE includes a VAE encoder and a VAE decoder;

[0018] S32, Training the VAE, the VAE encoder will generate attribute maps Encoding into the latent space yields the original latent code z0. The VAE decoder is then used to decode z0 back into the pixel space to obtain the predicted attribute map. ;

[0019] S33, training a conditional diffusion model in the latent space, using z0 as the data input during training, C n As conditional inputs, the predicted latent code is output through a forward noise addition process and a reverse noise reduction process. ;

[0020] S4, construct a 3D generative model, including an image coding model, a latent diffusion model, a backprojection layer, and a merging layer. The backprojection layer is used to... Each pixel is back-projected into 3D space to generate Gaussian primitives corresponding to each pixel, and all Gaussian primitives of viewpoint v are combined to form a Gaussian set. The merging layer is used to combine V in All Gaussian sets of the viewpoint are merged into M. n The three-dimensional Gaussian set G n ;

[0021] S5, acquire multi-view RGB images or text prompts of the object to be modeled, and generate a 3D Gaussian set G of the object to be modeled using the 3D generative model. md .

[0022] Preferably, in S11, each 3D model includes a geometric mesh and a texture map; in S12, the RGB image of the viewpoint v... , surface normal map , world coordinate map , attribute map are all generated by the IBender renderer, , , .

[0023] As a preference, the image encoding network comprises an embedding layer, an L-layer stack of improved Transformer layers, and a decoding layer, wherein the l-th layer of the improved Transformer layer is denoted as TF (l) , 1≤l≤L;

[0024] The embedding layer is the embedding layer of ViT, which is used for input to generate an initial vector sequence ;

[0025] The TF (l) includes a pre-normalization layer, a multi-head attention layer, an addition layer, four multi-perception machines MLP1-MLP4, and a multiplication layer;

[0026] The output feature of TF (l-1) is sequentially subjected to the pre-normalization layer and the multi-head attention layer to obtain the attention feature ; ; and are element-wise added by the addition layer to obtain the addition feature , and then the first MLP feature is obtained by MLP1 ; is divided into two paths, one of which outputs the second MLP feature by MLP2 and GELU layer, and the other of which obtains the third MLP feature by MLP3; and are element-wise added by the addition layer, and then the output feature of TF (l) is obtained by MLP4 ;

[0027] The decoding layer is used to map each element in to a 12-dimensional feature, and then output the first predicted attribute map by shape adjustment and splicing.

[0028] During training, the L1 loss or L2 loss of and is minimized.

[0029] As a preference, the training of VAE in S32 comprises Sa1-Sa2;

[0030] Sa1, constructs the VAE loss LVAE L VAE =L rec +βL KL In the formula, L rec For reconstruction loss, L KL Let β be the KL divergence regularization term, and L be the L. KL The weights;

[0031] Sa2, to minimize L VAE Train the VAE encoder and VAE decoder, and combine the trained VAE encoder and VAE decoder to form a VAE model.

[0032] As a preferred option, L rec L KL We obtain them respectively from the following formulas:

[0033] ,

[0034] ,

[0035] In the formula, μ is the square of the L2 norm. i Let σ be the i-th component of the mean μ of z0. i Let σ be the i-th component of the standard deviation σ of z0.

[0036] As a preferred option, in S4, for The pixel p with coordinates (u,v) uv The backprojection layer generates its Gaussian primitive g. uv The methods include S41~S42;

[0037] S41, based on the camera parameters and depth of the viewpoint v, p uv Back-projecting into three-dimensional space yields the corresponding world coordinates. ;

[0038] S42, from Get p from uv color Opacity principal axis of the ellipsoid Rotation Quaternions , constitutes the Gaussian primitive g uv , .

[0039] As a preferred option, in S5:

[0040] Generate G from multi-view RGB images of the object to be modeled md The method is as follows: RGB images from each viewpoint are sequentially processed through an image coding model, a latent diffusion model, and a backprojection layer to obtain the corresponding Gaussian set for that viewpoint; then, a merging layer is applied to obtain G.md ;

[0041] Generate based on text prompts for the object to be modeled The method is to encode the text prompt as a text vector C. md For each viewpoint, the text vector is used as a conditional input, and noise is gradually predicted from Gaussian noise through a reverse denoising process to generate the prediction latent code for each viewpoint. Then, after passing through a VAE decoder and a backprojection layer, a Gaussian set for each viewpoint is generated, and finally, a merging layer is used to obtain G. md .

[0042] Compared with the prior art, the advantages of the present invention are as follows:

[0043] (1) The present invention introduces a 12-channel attribute map, wherein the attribute map organizes the 12-channel parameters of each view into a regular two-dimensional attribute map (including color, opacity, ellipsoidal principal axis, rotation quaternion, and depth features), and performs three-dimensional reconstruction based on the attribute map. This can simultaneously ensure the consistency between multi-view attributes and three-dimensional geometry, ensure the alignment of cross-view geometry and appearance, improve the accuracy of subsequent modeling, improve the overall quality of three-dimensional reconstruction, and provide stronger data support for virtual reality and digital twins.

[0044] (2) Construct a three-dimensional generative model based on the attribute graph, wherein the image coding model is used as the input for the 9-channel image. This invention improves the image coding model by structurally refining it. It can simultaneously model local texture features and global relationships across channels and spatial locations under multi-channel input conditions, thereby enhancing the ability to model the coupling relationships between multimodal attributes such as color, normals, and spatial coordinates. The latent diffusion model includes a VAE and a conditional diffusion model. To adapt to the 12-channel attribute map, only the number of input / output channels in the first and last convolutional layers of the existing VAE needs to be changed from 3 to 12. This allows the original RGB latent space prior to be directly transferred to the 12-channel image task, and subsequent minor fine-tuning is sufficient for convergence. It features lower training / inference computational power, more stable convergence, and less data dependency. Therefore, the latent diffusion model of this invention not only focuses on the integrity of the global structure but also on local details such as normals, depth, and rotation. It also features high generation efficiency and stable training and inference, significantly improving the detail quality of 3D reconstruction.

[0045] (3) Potential diffusion model output predicted attribute map Then, it is directly generated through the back projection layer. The Gaussian primitives corresponding to each pixel, and the Gaussian primitive g uv , ,in Color is obtained by back-projection from the depth channel and the camera's intrinsic and extrinsic parameters. Opacity principal axis of the ellipsoid Rotation Quaternions It comes directly from the predicted attribute map and is naturally stitched together into a 3D Gaussian set across different viewpoints; it has the advantages of explicitly utilizing camera imaging geometry and cross-viewpoint consistency supervision, and the 3D Gaussian set is naturally connected to the high-speed rendering and editing ecosystem of Gaussian sputtering GS; compared with point cloud reprocessing, it has the characteristics of short path, low latency and strong interactivity.

[0046] In summary, this invention introduces a 12-channel property map and proposes a generation method that combines the property map, diffusion model, and 3D Gaussian splashing. This significantly improves the quality and application value of 3D reconstruction and rendering, providing an innovative solution for virtual reality, digital twins, and intelligent manufacturing. By addressing the consistency and efficiency issues of existing technologies, it offers a new technical path for high-precision 3D modeling, with particularly broad application prospects in industrial inspection and immersive interaction. Attached Figure Description

[0047] Figure 1 This is a 3D generated model structure diagram;

[0048] Figure 2 This is a diagram of the image coding model structure.

[0049] Figure 3 Flowchart for training a potential diffusion model.

[0050] Figure 4 A flowchart for generating a 3D Gaussian set based on multi-view RGB images.

[0051] Figure 5 This is a flowchart for generating a 3D Gaussian set based on text prompts. Detailed Implementation

[0052] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0053] Example 1: See Figures 1 to 5 A multimodal feature-driven 3D model generation method guided by images and text includes the following steps:

[0054] S1, construct dataset D;

[0055] S11, Obtain N 3D models, where the nth model is labeled M. n The default attribute map shape is 12×H×W, where 12, H, and W are the number of attribute channels, the height and width of the map, respectively. The 12 channels are divided into 3 color channels, 1 opacity channel, 3 ellipsoidal principal axis channels, 4 rotation quaternion channels, and 1 depth channel.

[0056] S12, for Mn Preset V in Each viewpoint renders the RGB image of viewpoint v. Surface normal diagram World coordinate map spliced ​​into a 9-channel tensor , Then render the attribute graph of the viewpoint v. M n Input and tags ;

[0057] S13, obtain M n The text prompt is encoded into a text vector C. n , will (X n ,Y n C n As the nth sample in dataset D, each 3D model is processed sequentially to obtain the dataset. ;

[0058] S2, construct an image coding network, and use... For input, Train an image coding network for the labels to obtain the image coding model;

[0059] S3, a pre-trained latent diffusion model, including S31~S33;

[0060] S31, Construct a potential diffusion model, including a VAE and a conditional diffusion model, wherein the VAE includes a VAE encoder and a VAE decoder;

[0061] S32, Training the VAE, the VAE encoder will generate attribute maps Encoding into the latent space yields the original latent code z0. The VAE decoder is then used to decode z0 back into the pixel space to obtain the predicted attribute map. ;

[0062] S33, training a conditional diffusion model in the latent space, using z0 as the data input during training, C n As conditional inputs, the predicted latent code is output through a forward noise addition process and a reverse noise reduction process. ;

[0063] S4, construct a 3D generative model, including an image coding model, a latent diffusion model, a backprojection layer, and a merging layer. The backprojection layer is used to... Each pixel is back-projected into 3D space to generate Gaussian primitives corresponding to each pixel, and all Gaussian primitives of viewpoint v are combined to form a Gaussian set. The merging layer is used to combine V in All Gaussian sets of the viewpoint are merged into M.n The three-dimensional Gaussian set G n ;

[0064] S5, acquire multi-view RGB images or text prompts of the object to be modeled, and generate a 3D Gaussian set G of the object to be modeled using the 3D generative model. md .

[0065] Example 2: See Figures 1 to 5 More specifically, based on Example 1,

[0066] In S11, each 3D model contains a geometric mesh and a texture map;

[0067] In S12, the RGB image of the viewpoint v Surface normal diagram World coordinate map Attribute diagram All were generated by the lBender renderer. , , .

[0068] The image coding network includes an embedding layer, L stacked improved Transformer layers, and a decoding layer, wherein the l-th improved Transformer layer is labeled TF. (l) , 1≤l≤L;

[0069] The embedding layer is a ViT embedding layer, used for input. Generate an initial vector sequence ;

[0070] The TF (l) It includes a pre-normalization layer, a multi-head attention layer, an addition layer, four multi-perceptron MLP1~MLP4 and a multiplication layer;

[0071] TF (l-1) Output characteristics Attention features are obtained by sequentially passing the pre-normalization layer and the multi-head attention layer. ; and Additive features are obtained by adding elements one by one through the additive layers. The first MLP feature is then obtained through MLP1. ; It is divided into two paths, one of which outputs the second MLP feature through the MLP2 and GELU layers. The third MLP feature is obtained through MLP3. ; and TF is obtained by adding elements one by one through an additive layer and then passing it through MLP4. (l) Output characteristics ;

[0072] The decoding layer is used to... Each element is mapped to a 12-dimensional feature map, and then the first predicted attribute map is output through shape adjustment and splicing. ;

[0073] During training, minimize and L1 loss or L2 loss.

[0074] The training VAE in S32 includes Sa1~Sa2;

[0075] Sa1, constructing VAE loss L VAE L VAE =L rec +βL KL In the formula, L rec For reconstruction loss, L KL Let β be the KL divergence regularization term, and L be the L. KL The weights;

[0076] Sa2, to minimize L VAE Train the VAE encoder and VAE decoder, and combine the trained VAE encoder and VAE decoder to form a VAE model.

[0077] L rec L KL We obtain them respectively from the following formulas:

[0078] ,

[0079] ,

[0080] In the formula, μ is the square of the L2 norm. i Let σ be the i-th component of the mean μ of z0. i Let σ be the i-th component of the standard deviation σ of z0.

[0081] In S4, for The pixel p with coordinates (u,v) uv The backprojection layer generates its Gaussian primitive g. uv The methods include S41~S42;

[0082] S41, based on the camera parameters and depth of the viewpoint v, p uv Back-projecting into three-dimensional space yields the corresponding world coordinates. ;

[0083] S42, from Get p from uv color Opacity principal axis of the ellipsoid Rotation Quaternions , constitutes the Gaussian primitive g uv , .

[0084] In S5:

[0085] Generate G from multi-view RGB images of the object to be modeled md The method is as follows: RGB images from each viewpoint are sequentially processed through an image coding model, a latent diffusion model, and a backprojection layer to obtain the corresponding Gaussian set for that viewpoint; then, a merging layer is applied to obtain G. md ;

[0086] Generate based on text prompts for the object to be modeled The method is to encode the text prompt as a text vector C. md For each viewpoint, the text vector is used as a conditional input, and noise is gradually predicted from Gaussian noise through a reverse denoising process to generate the prediction latent code for each viewpoint. Then, after passing through a VAE decoder and a backprojection layer, a Gaussian set for each viewpoint is generated, and finally, a merging layer is used to obtain G. md .

[0087] Example 3: See Figures 1 to 5 Based on Example 1 or Example 2, the more specific implementation steps are as follows:

[0088] Regarding the RGB diagram of the viewpoint v in step S12 Surface normal diagram World coordinate map : , Each of these is a three-channel tensor; concatenating the three results in a nine-channel tensor. . Each pixel stores the brightness values ​​of the three channels r, g, and b, and the brightness values ​​range from [0,1]. Each pixel stores the normal vector (n) of that pixel on the x, y, and z axes. x ,n y ,n z The value range of each normal vector is [-1, 1], and it has been normalized. Each pixel stores the corresponding three-dimensional world coordinates (X, Y, Z).

[0089] Regarding the attribute diagram of viewpoint v in step S12 It contains a total of 12 channels, including 3 color channels with values ​​ranging from [0,1] for each channel; 1 channel for opacity; and ellipsoidal principal axis s=(s x ,sy ,s z )3 channels; Rotation quaternion q=(q w ,q x ,q y ,q z 4 channels, normalized to unit length. Depth 1 channel, range [0,1].

[0090] Regarding the S2 image coding network, an improved Transformer layer is included, consisting of an embedding layer and an L-layer stack. The embedding layer workflow is as follows:

[0091] Sa1, will Divided into There are 1 image block, each image block having a size of P×P;

[0092] Sa2 flattens each image patch into a long vector, where the long vector of the p-th image patch flattened is represented as... , ;

[0093] Sa3, then Mapped to the latent space dimension of the model via linear projection. Then add the classification token and location code to get vector representation The present invention takes The value is 768. After this step, the vector representations corresponding to the 1st to Nth image patches are obtained. ~ ;

[0094] Sa4 concatenates the vector representations of all image patches into an initial vector sequence. : .

[0095] Regarding improvements to the Transformer layer, using the l-th layer TF (l) For example, the input is The output is , When l=1, TF (1) Input is When l=L, TF (L) The output is , can be represented as , ~ These are the 1st to Nth image patches corresponding to TF. (L) Vector representation within.

[0096] Regarding the decoding layer, obtain the Lth layer TF. (L) Output Using linear layers Mapping to a 12-channel feature space yields the mapped features. , In the formula W out b out These are the weights and biases of the linear layer, 1 ≤ p ≤ N; then N are... Following the order of segmentation, shape adjustments and splicing are performed to obtain the desired scale. Same first predicted attribute graph Each viewpoint corresponds to a first predicted attribute map.

[0097] Regarding step S3, pre-training a potential diffusion model, including S31~S33;

[0098] S32 is the training VAE. The VAE includes a VAE encoder and a VAE decoder. The input to the VAE encoder is a 12-channel attribute map. First, Feature extraction is performed to estimate the mean μ and standard deviation σ, and then reparameter sampling is performed to generate the original latent code z0. ,in For element-wise multiplication, It is random noise and follows a standard normal distribution. The VAE decoder decodes z0 into a predicted property map. During training, the loss function used is Li. VAE =L rec +βL KL Includes reconstruction loss L rec and KL divergence regularization term L KL L rec Used to measure reconstruction results With input The gap, L KL It is used to measure the difference between the coding distribution and the standard normal distribution, and to avoid latent space collapse.

[0099] S33 represents training the conditional diffusion model in the latent space. The conditional diffusion model includes a forward noise addition process and a backward noise reduction process;

[0100] Forward noise addition process: Noise is added to z0 step by step, and the noise-added data corresponding to time step t is z t Reverse denoising process: combining conditional input C n The added noise is predicted step by step over time, and the predicted noise is gradually subtracted to obtain the predicted latent code. The training objective is to make the predicted noise approximate the actual noise.

[0101] Regarding S5, it can obtain multi-view RGB images or text prompts of the object to be modeled, and then generate a 3D Gaussian set G of the object to be modeled through a 3D generative model. mdThis is the reasoning process for the 3D generative model, applicable to scenarios with or without real images.

[0102] If real images are available, acquire multi-view RGB images of the object to be modeled. For each viewpoint, pass the RGB image through an image encoding model, a latent diffusion model, and a backprojection layer to obtain the corresponding Gaussian set. Then, pass the Gaussian set through a merging layer to obtain G. md That's all.

[0103] If no real image is available, obtain text prompts for the object to be modeled, and for each viewpoint, use random Gaussian noise. Initially, noise is predicted and denoised step by step to obtain the prediction latent code for each viewpoint. Then, after passing through a VAE decoder and a backprojection layer, a Gaussian set for each viewpoint is generated, and finally, a merging layer is used to obtain G. md .

[0104] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating 3D models guided by text and images and driven by multimodal features, characterized in that, Includes the following steps: S1, construct dataset D; S11, Obtain N 3D models, where the nth model is labeled M. n The default attribute map shape is 12×H×W, where 12, H, and W are the number of attribute channels, the height and width of the map, respectively. The 12 channels are divided into 3 color channels, 1 opacity channel, 3 ellipsoidal principal axis channels, 4 rotation quaternion channels, and 1 depth channel. S12, for M n Preset V in Each viewpoint renders the RGB image of viewpoint v. Surface normal diagram World coordinate map spliced ​​into a 9-channel tensor , Then render the attribute graph of the viewpoint v. M n Input and tags ; S13, obtain M n The text prompt is encoded into a text vector C. n , will (X n ,Y n C n As the nth sample in dataset D, each 3D model is processed sequentially to obtain the dataset. ; S2, construct an image coding network, and use... For input, Train an image coding network for the labels to obtain the image coding model; S3, a pre-trained latent diffusion model, including S31~S33; S31, Construct a potential diffusion model, including a VAE and a conditional diffusion model, wherein the VAE includes a VAE encoder and a VAE decoder; S32, Training the VAE, the VAE encoder will generate the attribute map Encoding into the latent space yields the original latent code z0. The VAE decoder is then used to decode z0 back into the pixel space to obtain the predicted attribute map. ; S33, training a conditional diffusion model in the latent space, using z0 as the data input during training, C n As conditional inputs, the predicted latent code is output through a forward noise addition process and a reverse noise reduction process. ; S4, construct a 3D generative model, including an image coding model, a latent diffusion model, a backprojection layer, and a merging layer. The backprojection layer is used to... Each pixel is back-projected into 3D space to generate Gaussian primitives corresponding to each pixel, and all Gaussian primitives of viewpoint v are combined to form a Gaussian set. The merging layer is used to combine V in All Gaussian sets of the viewpoint are merged into M. n The three-dimensional Gaussian set G n ; S5: Obtain multi-view RGB images or text prompts of the object to be modeled, and generate a 3D Gaussian set G of the object to be modeled using the 3D generative model. md ; The image coding network includes an embedding layer, L stacked improved Transformer layers, and a decoding layer, wherein the l-th improved Transformer layer is labeled TF. (l) , 1≤l≤L; The embedding layer is a ViT embedding layer, used for input. Generate an initial vector sequence ; The TF (l) It includes a pre-normalization layer, a multi-head attention layer, an addition layer, four multi-perceptron MLP1~MLP4 and a multiplication layer; TF (l-1) Output characteristics Attention features are obtained by sequentially passing the pre-normalization layer and the multi-head attention layer. ; and Additive features are obtained by adding elements one by one through the additive layers. The first MLP feature is then obtained through MLP1. ; It is divided into two paths, one of which outputs the second MLP feature through the MLP2 and GELU layers. The third MLP feature is obtained through MLP3. ; and TF is obtained by adding elements one by one through an additive layer and then passing it through MLP4. (l) Output characteristics ; The decoding layer is used to... Each element is mapped to a 12-dimensional feature map, and then the first predicted attribute map is output through shape adjustment and splicing. ; During training, minimize and L1 loss or L2 loss.

2. The method for generating a 3D model guided by image and text using multimodal features according to claim 1, characterized in that, In S11, each 3D model contains a geometric mesh and a texture map; in S12, the RGB image of the viewpoint v. Surface normal diagram World coordinate map Attribute diagram All were generated by the lBender renderer. .

3. The method for generating a 3D model guided by image and text using multimodal features according to claim 1, characterized in that, The training VAE in S32 includes Sa1~Sa2; Sa1, constructing VAE loss L VAE L VAE =L rec +βL KL In the formula, L rec For reconstruction loss, L KL Let β be the KL divergence regularization term, and L be the L. KL The weights; Sa2, to minimize L VAE Train the VAE encoder and VAE decoder, and combine the trained VAE encoder and VAE decoder to form a VAE model.

4. The method for generating a 3D model guided by image and text using multimodal features, as described in claim 3, is characterized in that... L rec L KL We obtain them respectively from the following formulas: ， ， In the formula, μ is the square of the L2 norm. i Let σ be the i-th component of the mean μ of z0. i Let σ be the i-th component of the standard deviation σ of z0.

5. The method for generating a 3D model guided by image and text using multimodal features according to claim 1, characterized in that, In S4, for The pixel p with coordinates (u,v) uv The backprojection layer generates its Gaussian primitive g. uv The methods include S41~S42; S41, based on the camera parameters and depth of the viewpoint v, p uv Back-projecting into three-dimensional space yields the corresponding world coordinates. ; S42, from Get p from uv color Opacity principal axis of the ellipsoid Rotation Quaternions , constitutes the Gaussian primitive g uv , .

6. The method for generating a 3D model guided by image and text using multimodal features according to claim 1, characterized in that, In S5: Generate G from multi-view RGB images of the object to be modeled md The method is as follows: RGB images from each viewpoint are sequentially processed through an image coding model, a latent diffusion model, and a backprojection layer to obtain the corresponding Gaussian set for that viewpoint; then, a merging layer is applied to obtain G. md ; Generate G based on text prompts for the object to be modeled md The method is to encode the text prompt as a text vector C. md For each viewpoint, the text vector is used as a conditional input, and noise is gradually predicted from Gaussian noise through a reverse denoising process to generate the prediction latent code for each viewpoint. Then, after passing through a VAE decoder and a backprojection layer, a Gaussian set for each viewpoint is generated, and finally, a merging layer is used to obtain G. md .

Images