Text-to-three-dimensional Gaussian shape generation method and device, equipment and medium

Abstract

The invention discloses a text-to-three-dimensional Gaussian shape generation method and device, equipment and a medium, and the method comprises the steps: obtaining an initialized three-dimensional Gaussian model through a pre-trained three-dimensional generator according to the text input for generating a target task; performing multi-view projection to obtain a key view image and a group of auxiliary view images; performing feature extraction on the auxiliary view image, and fusing the extracted features with the features of the key view image through an attention mechanism; performing multi-view joint geometric optimization on the initialized three-dimensional Gaussian model through fractional distillation loss by using the fused features and text input injection diffusion model to obtain a three-dimensional Gaussian model subjected to geometric optimization; and a layered Gaussian constraint mechanism is implemented, reinforced constraint is applied to Gaussian points generated earlier, higher degree of freedom is given to Gaussian points newly added in the later period, and a final three-dimensional Gaussian shape is obtained. According to the method, the geometric semantic consistency and the overall fidelity and fineness of the generated result under different visual angles are improved.

Description

Technical Field

[0001] The present invention relates to the interdisciplinary technical field of computer graphics and natural language, and more particularly to a method, apparatus, device, and medium for generating three-dimensional Gaussian shapes from text; more particularly, to a method, apparatus, device, and medium for generating three-dimensional Gaussian shapes from text based on a multi-perspective guidance and layered Gaussian constraint mechanism. Background Art

[0002] The generation of three-dimensional Gaussian geometric shapes guided by text is a task at the intersection of natural language and computer graphics, and has important application prospects in three-dimensional content creation and animation generation.

[0003] Traditional 3D shape generation methods based on meshes, point clouds, etc., have difficulty accurately depicting the details and textures of complex shapes. For example, meshes require a large number of triangular facets to represent complex geometric details, and point clouds are often limited by point density and distribution. Although methods such as neural radiation fields based on implicit 3D representation can effectively render complex scenes, they suffer from slow processing speeds and difficulty controlling local areas. Therefore, some scholars have proposed a 3D shape generation method based on Gaussian splashing. 3D shapes are modeled using 3D Gaussian functions. Each Gaussian point contains attributes such as position, opacity, and spherical harmonic coefficients. This can more efficiently represent complex scenes and can be combined with text guidance to flexibly control the generation of 3D shapes through natural language.

[0004] However, the multi-face problem of generated shapes is a relatively common phenomenon. This mainly refers to the appearance of multiple unnatural and incoherent faces on the generated 3D model. These faces have problems such as dislocation and overlap in position, angle or shape that are obviously inconsistent with real logic. Some areas may have additional, unrealistic faces. These faces are semantically or geometrically inconsistent with other parts of the model, destroying the integrity and consistency of the model. The generated 3D shape may have multiple overlapping faces in local areas. These faces interfere with each other, affecting the realism and visual effect of the generated results. For example, when a child sees a horse, no matter from which perspective, he should be able to recognize that it is the shape of a horse. However, some existing methods have difficulty achieving geometric consistency under different perspectives, affecting the observer's accurate understanding and cognition of the 3D shape. This limits the application of 3D shape generation methods in the field of high-quality content generation.

[0005] To date, existing 3D Gaussian shape generation technologies have shown improvements in generation quality and diversity, but still face a series of key challenges, such as the common "multi-face problem", that is, when the generated 3D objects are observed from different perspectives, inconsistent or even contradictory geometric structures may appear.

[0006] Existing techniques typically use a single rendered image as a guide for optimizing 3D geometry using diffusion models. This single-view supervision can easily lead to the so-called "multi-face problem" in generated 3D objects. This problem occurs when a generated object appears visually appealing from certain viewpoints, but exhibits distorted geometry and even multiple, inconsistent appearances from other viewpoints. This phenomenon stems from the lack of effective multi-viewpoint geometric information during the generation process, making it difficult to effectively constrain and ensure the global geometric consistency of the generated 3D object.

[0007] Some methods attempt to limit the perspective by adding text orientation hints, such as adding vague instructions such as "front" and "side". However, this type of text hint is inherently indirect and vague and cannot provide precise geometric constraints. Therefore, it is still difficult to completely solve the perspective consistency problem.

[0008] In addition, some studies have used additional geometric auxiliary information such as depth maps and normal maps to pre-train 2D diffusion models to enhance their perception of 3D structure. However, these methods rely on complex pre-training processes and additional geometric supervision signals, making them difficult to generalize effectively in real-world scenarios. They also fail to fully utilize the joint information from multiple perspectives, often resulting in a significantly increased computational burden and failing to fundamentally guarantee multi-perspective consistency.

[0009] In traditional 3D generation processes, 3D Gaussian points generated at different stages are typically subject to the same optimization intensity constraints. This results in either early geometric structures being too free and unstable during detail generation, or later details being insufficient to flexibly express complex structures. This is particularly true when using Gaussian distributions to explicitly express 3D structures. Due to the extreme variability of Gaussian points themselves, the random loss imposed by each optimization step can lead to drastic changes in the global structure, ultimately making it difficult for the model to converge stably and severely limiting its ability to express detail. Many works directly optimize the overall 3D shape parameters based on random loss during the detail optimization phase, treating all regions equally. This can easily lead to the stable geometric structure after early optimization being "destroyed" or blurred during later detail optimization.

[0010] In addition, some technologies introduce normal maps or reflectance maps to guide appearance optimization, making the generated texture more consistent with the real geometric structure. This approach eliminates texture artifacts caused by lighting to a certain extent and improves the geometric rationality of the generated appearance. However, the normal map mainly provides geometric direction information, and the reflectance map only represents the basic reflection color, which lacks sufficient description capabilities for complex textures. This approach requires extracting high-quality normal maps and reflectance maps from 3D models or multi-view images in advance. The preprocessing process itself may introduce noise and errors. In addition, in order for the diffusion model to understand the normal map and reflectance map, they need to be jointly trained, which increases the complexity and resource consumption of model training.

[0011] Therefore, technical personnel in this field urgently need to solve the problem of how to circumvent the defects of the above-mentioned existing methods and deal with the problem of inconsistent multi-view geometry of generated results. Summary of the Invention

[0012] In view of this, the present invention provides a method, device, equipment and medium for generating three-dimensional Gaussian shapes from text, which at least partially circumvents the defects of the current methods and can solve the problem of being unable to handle multi-perspective geometric inconsistencies in the generated results.

[0013] In order to achieve the above object, the present invention adopts the following technical solutions:

[0014] In a first aspect, an embodiment of the present invention provides a method for generating a three-dimensional Gaussian shape from text, comprising the following steps:

[0015] S10, obtaining an initialized 3D Gaussian model through a pre-trained 3D generator according to the text input of the target task;

[0016] S20, performing multi-view projection on the initialized three-dimensional Gaussian model to obtain a key view image and a set of auxiliary view images;

[0017] S30, extracting features from the auxiliary view image, and fusing the extracted features with the features of the key view image through an attention mechanism;

[0018] S40, using the fused features and text input to inject the diffusion model, and performing multi-view joint geometric optimization on the initialized three-dimensional Gaussian model through fractional distillation loss to obtain a geometrically optimized three-dimensional Gaussian model;

[0019] S50. Implement a hierarchical Gaussian constraint mechanism on the geometrically optimized three-dimensional Gaussian model, impose stronger constraints on the Gaussian points generated earlier, and grant higher degrees of freedom to the Gaussian points added later, to obtain the final three-dimensional Gaussian shape.

[0020] Furthermore, the step S10 includes:

[0021] Inputting a text prompt for generating a target task into a pre-trained 3D generator; the pre-trained 3D generator includes: a pre-processed text-guided image generation 2D diffusion model and a point cloud generation model;

[0022] The image generation two-dimensional diffusion model generates a two-dimensional image that matches the text semantics based on the input text prompt;

[0023] Inputting the generated two-dimensional image into the point cloud generation model to generate a rough three-dimensional point cloud;

[0024] Each point cloud is converted into a parametric representation of a 3D Gaussian sphere to generate an initialized 3D Gaussian model.

[0025] Furthermore, the initialized three-dimensional Gaussian model is expressed as: N represents the total number of point clouds, i represents the index, i∈(1,N); the parameter of the three-dimensional Gaussian ball of each point cloud represents G i =(μ i ,∑ i , α i , c i );

[0026] in, Indicates the center position of the Gaussian distribution; Represents the covariance matrix, which describes the shape and direction of the Gaussian distribution in three-dimensional space; α i ∈[0, 1] represents the opacity of Gaussian distribution; Represents the color information of Gaussian distribution.

[0027] Furthermore, the step S30 includes:

[0028] The auxiliary view image is input into the image encoder, and the multi-view features are dynamically fused through a zero-initialized convolutional layer; the weights and biases of the convolutional layer are initially zero; the weights are gradually updated during the training process to achieve progressive injection of auxiliary view features;

[0029] The multi-view features are fused with the key view image skip connection features through the cross attention mechanism.

[0030] Furthermore, the multi-view features are fused with the key view image skip connection features through a cross attention mechanism; including:

[0031] The key view image skip connection features are used as the query vector Q; the multi-view features are used as the key K and value V vectors; the features are calculated by cross attention:

[0032]

[0033] where MultiHeadAttn(·) is the cross attention function; R is a set of real numbers; b is the batch size, h, w are the height and width of the image resolution, and d is the channel dimension;

[0034] The cross-attention feature output is linearly mapped and reshaped back to the skip connection space to obtain the fused feature:

[0035]

[0036] Among them, P out is a linear mapping layer, R(·) is a dimension conversion function; x SC is the skip connection feature.

[0037] Furthermore, in step S40, the fractional distillation loss L SDS1 for:

[0038]

[0039] Where z t is the noisy image at time step t, t is the diffusion time step, ∈ is the real noise added to the disturbance, ∈ θ is the noise value predicted by the diffusion model, T is the input text prompt, x c It is the guiding characteristic condition of the auxiliary perspective.

[0040] Furthermore, the step S50 includes:

[0041] For a set of geometrically optimized three-dimensional Gaussian models, obtain a corresponding perspective projection view;

[0042] Calculate fractional distillation loss;

[0043] Apply hierarchical constraints, at time step t i Divide the Gaussian points into layers and calculate the anchor loss:

[0044]

[0045] Among them, P i =(μ i ,∑ i , α i , c i ) is the Gaussian parameter state of the current time step;

[0046] is the anchor point attribute state at the initial time step; λ(t i ) corresponds to the tth i The weight function of the layer, λ(t i )=λ max ·exp(-γ·t i), λ max is the maximum constraint weight of the earliest Gaussian ball, γ is the hyperparameter that controls the weight decay speed, t i is the generation time step number of the i-th Gaussian point; N is the number of Gaussian points;

[0047] Combined with the total loss function, the local details are optimized to obtain the final three-dimensional Gaussian shape; the total loss function includes: fractional distillation loss L SDS2 and anchor loss; among them, the fractional distillation loss L SDS2 for:

[0048]

[0049] Among them, z t is the representation of the image obtained by 3D Gaussian rendering after adding noise in the diffusion step t, ∈ is the real noise vector, ∈ θ is the noise predicted by the diffusion model under the guidance of the text condition T, where T is the input text guidance condition.

[0050] In a second aspect, an embodiment of the present invention further provides a device for generating a three-dimensional Gaussian shape from text, using the method for generating a three-dimensional Gaussian shape from text as described in any one of the first aspects, the device comprising:

[0051] The initialization module is used to obtain an initialized 3D Gaussian model through a pre-trained 3D generator based on the text input of the target task;

[0052] The geometry generation module is used to perform multi-view projection on the initialized 3D Gaussian model to obtain a key view image and a set of auxiliary view images; feature extraction is performed on the auxiliary view images and the extracted features are fused with the features of the key view image through an attention mechanism; the fused features and text input are injected into the diffusion model, and the initialized 3D Gaussian model is subjected to multi-view joint geometric optimization using fractional distillation loss to obtain a geometrically optimized 3D Gaussian model;

[0053] The detail optimization module implements a hierarchical Gaussian constraint mechanism on the geometrically optimized 3D Gaussian model, imposes stronger constraints on the Gaussian points generated earlier, and grants higher degrees of freedom to the Gaussian points added later, to obtain the final 3D Gaussian shape.

[0054] In a third aspect, an embodiment of the present invention provides a device for generating a three-dimensional Gaussian shape from text, comprising:

[0055] at least one processor; and a memory communicatively coupled to the at least one processor;

[0056] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods for generating three-dimensional Gaussian shapes from text in the first aspect.

[0057] In a fourth aspect, an embodiment of the present invention further provides a storage medium, wherein the storage medium stores instructions, and when the instructions are executed on a terminal, any method for generating a three-dimensional Gaussian shape from text as described in the first aspect can be implemented.

[0058] The descriptions of the second to fourth aspects of the present invention can refer to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second to fourth aspects can refer to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0059] It can be seen from the above technical solutions that, compared with the prior art, the present invention has the following technical effects:

[0060] 1. A multi-view guided geometry optimization strategy is used to address the geometric structure inconsistency problem in traditional methods. This strategy integrates guidance information from multiple viewpoints to enhance the model's global understanding of the 3D structure, thereby improving the geometric semantic consistency of the generated results from different viewpoints.

[0061] 2. A Gaussian layered constraint mechanism was introduced. Early-generated Gaussian points are given stronger constraints to ensure the stability of the initial geometric structure; while later-generated Gaussian points are given more freedom to fully exploit their potential for optimizing local details. This strategy not only significantly improves the geometric stability of the generated 3D model, but also ensures that later-generated models can more freely depict intricate 3D details, effectively enhancing the fidelity and detail of the overall generated model. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0063] Figure 1 Flowchart of the method for generating three-dimensional Gaussian shapes from text provided by the present invention.

[0064] Figure 2 This is a diagram of the network architecture for the text-to-3D Gaussian initialization stage provided by the present invention.

[0065] Figure 3This is the overall architecture diagram of the Gaussian geometry generation stage provided by the present invention.

[0066] Figure 4 This is a flowchart of the Gaussian detail optimization stage provided by the present invention.

[0067] Figure 5 The text-to-3D Gaussian generation results provided by the present invention include renderings of car, food, and doll categories.

[0068] Figure 6 A block diagram of the device for generating three-dimensional Gaussian shapes from text provided by the present invention.

[0069] Figure 7 A structural diagram of the device for generating three-dimensional Gaussian shapes from text provided by the present invention. DETAILED DESCRIPTION

[0070] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0071] Example 1:

[0072] The embodiment of the present invention discloses a method for generating a three-dimensional Gaussian shape from text, referring to Figure 1 As shown, the following steps are included:

[0073] S10, obtaining an initialized 3D Gaussian model through a pre-trained 3D generator according to the text input of the target task;

[0074] S20, performing multi-view projection on the initialized three-dimensional Gaussian model to obtain a key view image and a set of auxiliary view images;

[0075] S30, extracting features from the auxiliary view image, and fusing the extracted features with the features of the key view image through an attention mechanism;

[0076] S40, using the fused features and text input to inject the diffusion model, and performing multi-view joint geometric optimization on the initialized three-dimensional Gaussian model through fractional distillation loss to obtain a geometrically optimized three-dimensional Gaussian model;

[0077] S50. Implement a hierarchical Gaussian constraint mechanism on the geometrically optimized three-dimensional Gaussian model, impose stronger constraints on the Gaussian points generated earlier, and grant higher degrees of freedom to the Gaussian points added later, to obtain the final three-dimensional Gaussian shape.

[0078] The present invention realizes the generation of three-dimensional Gaussian shapes from text based on multi-perspective guidance and hierarchical Gaussian constraint mechanism. For the task of generating three-dimensional Gaussian geometric shapes from text, the rough geometric structure of the three-dimensional shape is first generated in the geometry generation stage, and then the local details are refined and optimized in the detail optimization stage to improve the richness of the overall generation effect of the model.

[0079] Specifically, the method consists of two key stages: geometry generation stage and detail optimization stage.

[0080] During the geometry generation phase, this embodiment uses a pre-trained 2D image diffusion model combined with a fractional distillation sampling loss to guide the initial geometric construction of 3D Gaussian shapes. To address the "multi-face problem" caused by traditional single-view optimization, this paper designs a multi-view guidance strategy: in addition to the primary view rendered image, multiple view rendered images are introduced as joint conditions. A pre-trained encoding network is used to extract multi-view image features and integrate them into the diffusion model optimization process, thereby improving the consistency and stability of the geometric structure across different viewpoints.

[0081] In the detail optimization stage, a hierarchical Gaussian constraint mechanism is proposed. Specifically, the Gaussian points in the generation process are divided into different generations in chronological order, and strong constraints are imposed on the Gaussian points generated earlier to maintain the stability of their initial geometric structure; higher degrees of freedom are given to the Gaussian points added later to enhance the ability to express local details. This generational optimization strategy can adaptively improve the model's modeling capabilities for complex areas such as high-frequency textures and boundary details while maintaining overall structural consistency, thereby achieving high-quality, clear-structured, and rich-looking three-dimensional Gaussian shape generation. In summary, this method improves global consistency by introducing multi-view information, and achieves balanced optimization of details and structures through a hierarchical Gaussian constraint mechanism, significantly alleviating the problems of poor geometric consistency and insufficient detail expression in existing text-to-3D generation methods.

[0082] The following is a detailed description of each of the above steps:

[0083] like Figure 2 Figure 1 shows the network architecture of the text-to-3D Gaussian initialization stage. The pre-trained 3D generator includes a pre-processed text-guided image generation 2D diffusion model and a point cloud generation model.

[0084] In step S10, user-provided text input is first received. A pre-trained 3D generator then generates a 2D image that semantically matches the text based on a 2D diffusion model generated from the image. The generated image is then fed into a point cloud generation model to generate a coarse 3D point cloud. Finally, each point is converted into a parametric representation of a 3D Gaussian sphere, generating an initialized 3D Gaussian model.

[0085] Specifically, in the text-to-3D Gaussian shape generation task, the text prompt T is mapped into a set of 3D Gaussian distributions N represents the total number of point clouds, i represents the index, i∈(1,N); each Gaussian distribution G i Determined by some parameters: position Represents the center position of the Gaussian distribution, R represents the set of real numbers; the covariance matrix Describe the shape, direction, and color of a Gaussian distribution in three-dimensional space Represents the color information of Gaussian distribution, opacity α i ∈[0, 1] controls the transparency of the Gaussian distribution.

[0086] exist Figure 2 In [1], we initialize a 3D Gaussian distribution with basic geometric structure based on the input text prompt. First, by receiving the text input T provided by the user, this text is fed into a pre-trained 3D generator, which contains a pre-trained text-guided image generation 2D diffusion model, thereby generating a 2D image to ensure that the generated image matches the text semantics, I = f txt2img (T), where f txt2img It is a two-dimensional image diffusion model, and the output image size is H and W represent the width and height of the image respectively. Then, the point cloud is sampled based on the generated two-dimensional image by reverse diffusion. where f img2pc It is a point cloud diffusion model. The number of output point clouds is N. Finally, each point p i Converted to the parameter representation of three-dimensional Gaussian sphere G i =(μ i ,∑ i , α i , c i ), these Gaussian points serve as the starting input for the geometric optimization stage, providing a structural basis for subsequent multi-view consistency training and detail refinement.

[0087] like Figure 3The figure shows the overall architecture of the Gaussian geometry generation stage. In steps S20-S40, multi-view projection is first performed on the 3D Gaussian generated in the initialization phase. The projected key view serves as the input to the pre-trained 2D diffusion model. To improve the consistency of the overall Gaussian geometry during each iterative parameter update, in addition to the text-guided condition, a set of auxiliary views are rendered. These auxiliary view images provide geometric information about the object from different viewpoints, helping to avoid geometric inconsistencies across multiple views. For the auxiliary views, the feature extractor uses an image encoder to extract features. After passing through a convolutional layer, the output is additively fused with the key view features and then fed into an additional pre-trained network architecture with the same structure as the encoding layer of the pre-trained 2D diffusion model. The skip connections of the pre-trained 2D diffusion model are transformed by normalization and linear mapping. Then, an attention mechanism is used to interact the keys and values ​​of the auxiliary views with the key view query values ​​of the skip connections. This allows the feature information of other auxiliary views to be effectively injected into the geometry optimization process of the current key view. Finally, the output is added to the skip connection through linear mapping and fed into the decoder layer of the pre-trained 2D diffusion model.

[0088] Specifically, if Figure 3 The geometric optimization stage shown (corresponding to steps S20 to S40) is to initialize the Gaussian set To optimize, first perform multi-view projection and render the key view image I key and a set of auxiliary view images They observe the same object from different camera poses. In order to introduce cross-view constraints, we use the auxiliary image The input image encoder ImgEnc, in order to prevent the auxiliary view from interfering with the optimization path of the pre-trained two-dimensional diffusion model in the early stage of training, we introduced a zero convolution layer ZeroConv in its feature path, that is, a convolution operator with weights and biases initialized to zero. This layer ensures that the auxiliary features are not effectively injected in the initial stage, and only relies on the main view and text guidance for optimization. As the training progresses, the weights of the convolution layer are gradually updated, and the auxiliary view features gradually participate in the guidance, thereby stabilizing convergence and improving multi-view consistency. Next is the module AssisEnc with the same structure as the encoding layer and intermediate layer of the pre-trained two-dimensional diffusion model, which extracts conditional features and expands them into a set of tensors in the spatial dimension:

[0089]

[0090] Where b is the batch size, h, w are the height and width of the image resolution, M is the number of auxiliary views (e.g. between 2 and 6), and c is the channel dimension. The key view image generates skip connection features in the main diffusion model After normalization Norm, linear mapping Liner layer and dimension transformation Reshape layer, The superscript number 1 indicates the number of key views, and then the key view features are used as queries and the multi-view features xc are used as keys and values ​​for attention calculation:

[0091] Q=P Q (x′ SC ), K=P K (x c ), V=P V (x c )

[0092] In the attention mechanism, P Q 、P K 、P V These are three learnable linear transformation matrices that project input features into query, key, and value spaces.

[0093] get:

[0094]

[0095] d represents the channel dimension;

[0096] The final output is returned to the skip connection space after linear mapping and Reshape operation:

[0097]

[0098] Among them, P out is a linear mapping layer, and R is a dimension conversion. Combined with the auxiliary feature x c With the text hint T, we adopt the fractional distillation loss SDS1 to perform guided optimization on Gaussian:

[0099]

[0100] Among them, z t is the noisy image at time step t, t is the diffusion time step, ∈ is the real noise added to the disturbance, ∈ θ is the noise value predicted by the diffusion model, T is the input text prompt, x c It is the guiding characteristic condition of the auxiliary perspective.

[0101] like Figure 4As shown, a flowchart of the Gaussian detail optimization stage is shown. In step S50, for the result of the geometric generation stage, its projection view is obtained and input into a pre-trained two-dimensional diffusion model together with the text. The core is to guide the detail optimization of the three-dimensional Gaussian through the model, and gradually optimize the output of the model by calculating the noise difference between the generated image and the real image. Specifically, the loss value measures the difference between the two-dimensional rendered image generated by the diffusion model and the original image, and adjusts the parameters of the three-dimensional Gaussian through back propagation, thereby gradually improving the detail performance of the model. The optimization objective function consists of two parts. One part is based on the fractional distillation loss L SDS2 Optimize a 3D Gaussian:

[0102]

[0103] Among them, z t is the representation of the image obtained by 3D Gaussian rendering after adding noise in the diffusion step t, ∈ is the real noise vector, ∈ θ is the noise predicted by the diffusion model under the guidance of the text condition T, where T is the input text guidance condition. This loss continuously compares the noise difference between the rendered image and the image generated by the diffusion model to achieve reverse gradient updates of the 3D Gaussian, thereby optimizing its appearance details.

[0104] In addition, in order to ensure the stability of the geometric structure generated in the early stage, we adopted a hierarchical Gaussian constraint mechanism, which optimizes the Gaussian points generated at different time points by assigning different constraint strengths. For Gaussian points generated earlier, the system imposes stronger constraints to maintain the stability of the geometric shape generated in the early stage; while for Gaussian points generated later, it gives greater freedom to optimize the performance of the details. This mechanism effectively balances the stability of the global shape with the optimization of local details, thereby improving the visual effect and geometric accuracy of the generated object. Suppose the i-th Gaussian point belongs to the t-th i Layers (sorted by generation time), we define the anchor properties is the state at the initial time step, and its offset is constrained using the following anchor loss:

[0105]

[0106] Among them, P i =(μ i ,∑ i , α i , c i ) is the Gaussian parameter state of the current time step, λ(t i ) corresponds to the tth i The weight function of the layer, λ(t i )=λ max ·exp(-γ·t i), λ max is the maximum constraint weight of the earliest Gaussian ball, γ is the hyperparameter that controls the weight decay speed, t i is the time step number of the i-th Gaussian point. This mechanism allows the structure points generated early to be subject to strong constraints to maintain a stable shape, while the detail points added later have greater freedom to adjust the details.

[0107] In general, the total loss function of the detail optimization stage is:

[0108]

[0109] Among them, λ SDS2 and λ anchor are the traditional distillation fraction loss L SDS2 and anchor loss L anchor The weight coefficient of .

[0110] Figure 5 The results of generating 3D Gaussian from text are shown, including categories such as cars, food, and dolls. It can be seen that these 3D Gaussian models contain the geometric structure and texture information of objects in the corresponding categories. For example, if the text "a furry toy" is input, based on the method of the present invention, an initialized 3D dinosaur doll Gaussian model will be generated by the pre-trained 3D generator first. Then, in the geometry stage, the initialized dinosaur Gaussian model will be subjected to multi-view joint geometry optimization by rendering the key view and auxiliary view. Finally, in the detail optimization stage, based on the hierarchical Gaussian constraints, the details are optimized and enriched to obtain the final doll model (such as Figure 5 The last subgraph in .

[0111] Example 2:

[0112] Based on the same inventive concept, the embodiment of the present invention also provides a device for generating a three-dimensional Gaussian shape from text, using the method for generating a three-dimensional Gaussian shape from text as in Example 1, referring to Figure 6 As shown, the device includes:

[0113] The initialization module is used to obtain an initialized 3D Gaussian model through a pre-trained 3D generator based on the text input of the target task;

[0114] The geometry generation module is used to perform multi-view projection on the initialized 3D Gaussian model to obtain a key view image and a set of auxiliary view images; feature extraction is performed on the auxiliary view images and the extracted features are fused with the features of the key view image through an attention mechanism; the fused features and text input are injected into the diffusion model, and the initialized 3D Gaussian model is subjected to multi-view joint geometric optimization using fractional distillation loss to obtain a geometrically optimized 3D Gaussian model;

[0115] The detail optimization module implements a hierarchical Gaussian constraint mechanism on the geometrically optimized 3D Gaussian model, imposes stronger constraints on the Gaussian points generated earlier, and grants higher degrees of freedom to the Gaussian points added later, to obtain the final 3D Gaussian shape.

[0116] Example 3:

[0117] An embodiment of the present invention further provides a device for generating a three-dimensional Gaussian shape from text, comprising:

[0118] at least one processor; and a memory communicatively coupled to the at least one processor;

[0119] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating a three-dimensional Gaussian shape from text in Example 1.

[0120] Reference Figure 7 As shown, the device may include: a processor 71, a communications interface 72, a memory 73, and a communication bus 74, wherein the processor 71, the communications interface 72, and the memory 73 communicate with each other via the communication bus 74. The processor 71 may call logic instructions in the memory 73 to execute a method for generating a three-dimensional Gaussian shape from text, the method comprising:

[0121] S10, obtaining an initialized 3D Gaussian model through a pre-trained 3D generator according to the text input of the target task;

[0122] S20, performing multi-view projection on the initialized three-dimensional Gaussian model to obtain a key view image and a set of auxiliary view images;

[0123] S30, extracting features from the auxiliary view image, and fusing the extracted features with the features of the key view image through an attention mechanism;

[0124] S40, using the fused features and text input to inject the diffusion model, and performing multi-view joint geometric optimization on the initialized three-dimensional Gaussian model through fractional distillation loss to obtain a geometrically optimized three-dimensional Gaussian model;

[0125] S50. Implement a hierarchical Gaussian constraint mechanism on the geometrically optimized three-dimensional Gaussian model, impose stronger constraints on the Gaussian points generated earlier, and grant higher degrees of freedom to the Gaussian points added later, to obtain the final three-dimensional Gaussian shape.

[0126] Example 4:

[0127] An embodiment of the present invention further provides a storage medium, in which instructions are stored. When the instructions are executed on a terminal, a method for generating a three-dimensional Gaussian shape from text can be implemented as in Example 1.

[0128] Computer readable storage media may be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a register, a hard disk, an optical fiber, a portable

[0129] Compact Disc Read-Only Memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above, or any other form of computer-readable storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application-specific integrated circuit (ASIC). In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, apparatus, or device.

[0130] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0131] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a three-dimensional Gaussian shape from text, characterized in that: The following steps are involved: S10, obtaining an initialized 3D Gaussian model through a pre-trained 3D generator according to the text input of the target task; S20, performing multi-view projection on the initialized three-dimensional Gaussian model to obtain a key view image and a set of auxiliary view images; S30, extracting features from the auxiliary view image, and fusing the extracted features with the features of the key view image through an attention mechanism; S40, using the fused features and text input to inject the diffusion model, and performing multi-view joint geometric optimization on the initialized three-dimensional Gaussian model through fractional distillation loss to obtain a geometrically optimized three-dimensional Gaussian model; S50. Implement a hierarchical Gaussian constraint mechanism on the geometrically optimized three-dimensional Gaussian model, impose strong constraints on the Gaussian points generated earlier, and grant higher degrees of freedom to the Gaussian points added later, to obtain the final three-dimensional Gaussian shape.

2. The method for generating a three-dimensional Gaussian shape from text according to claim 1, characterized in that: The step S10 includes: Inputting a text prompt for generating a target task into a pre-trained 3D generator; the pre-trained 3D generator includes: a pre-processed text-guided image generation 2D diffusion model and a point cloud generation model; The image generation two-dimensional diffusion model generates a two-dimensional image that matches the text semantics based on the input text prompt; Inputting the generated two-dimensional image into the point cloud generation model to generate a rough three-dimensional point cloud; Each point cloud is converted into a parametric representation of a 3D Gaussian sphere to generate an initialized 3D Gaussian model.

3. The method for generating a three-dimensional Gaussian shape from text according to claim 2, characterized in that: The initialized three-dimensional Gaussian model is expressed as: N represents the total number of point clouds, i represents the index, i∈(1,N); the parameter of the three-dimensional Gaussian ball of each point cloud represents G i =(μ i ,∑ i , α i , c i ); in, Indicates the center position of the Gaussian distribution; Represents the covariance matrix, which describes the shape and direction of the Gaussian distribution in three-dimensional space; α i ∈[0, 1] represents the opacity of the Gaussian distribution; Represents the color information of Gaussian distribution.

4. The method for generating a three-dimensional Gaussian shape from text according to claim 1, wherein: The step S30 includes: The auxiliary view image is input into the image encoder, and the multi-view features are dynamically fused through a zero-initialized convolutional layer; the weights and biases of the convolutional layer are initially zero; the weights are gradually updated during the training process to achieve progressive injection of auxiliary view features; The multi-view features are fused with the key view image skip connection features through the cross attention mechanism.

5. The method for generating a three-dimensional Gaussian shape from text according to claim 4, characterized in that: The multi-view features are fused with the key view image jump connection features through the cross attention mechanism; including: The key view image skip connection features are used as the query vector Q; the multi-view features are used as the key K and value V vectors; the features are calculated by cross attention: where MultiHeadAttn(·) is the cross attention function; R is a set of real numbers; b is the batch size, h, w are the height and width of the image resolution, and d is the channel dimension; The cross-attention feature output is linearly mapped and reshaped back to the skip connection space to obtain the fused feature: Among them, P out is a linear mapping layer, R(·) is a dimension conversion function; x SC is the skip connection feature.

6. The method for generating a three-dimensional Gaussian shape from text according to claim 1, characterized in that: In step S40, the fractional distillation loss L SDS1 for: Where z t is the noisy image at time step t, t is the diffusion time step, ∈ is the real noise added to the disturbance, ∈ θ is the noise value predicted by the diffusion model, T is the input text prompt, x c It is the guiding characteristic condition of the auxiliary perspective.

7. The method for generating a three-dimensional Gaussian shape from text according to claim 2, characterized in that: The step S50 includes: For a set of geometrically optimized three-dimensional Gaussian models, obtain a corresponding perspective projection view; Calculate fractional distillation loss; Apply hierarchical constraints, at time step t i Divide the Gaussian points into layers and calculate the anchor loss: Among them, P i =(μ i ,∑ i , α i , c i ) is the Gaussian parameter state of the current time step; is the anchor point attribute state at the initial time step; λ(t i ) corresponds to the tth i The weight function of the layer, λ(t i )=λ max ·exp(-γ·t i ), λ max is the maximum constraint weight of the earliest Gaussian ball, γ is the hyperparameter that controls the weight decay speed, t i is the generation time step number of the i-th Gaussian point; N is the number of Gaussian points; Combined with the total loss function, the local details are optimized to obtain the final three-dimensional Gaussian shape; the total loss function includes: fractional distillation loss L SDS2 and anchor loss; among them, the fractional distillation loss L SDS2 for: Among them, z t is the representation of the image obtained by 3D Gaussian rendering after adding noise in the diffusion step t, ∈ is the real noise vector, ∈ θ is the noise predicted by the diffusion model under the guidance of the text condition T, where T is the input text guidance condition.

8. A device for generating a three-dimensional Gaussian shape from text, characterized in that: Using the method for generating a three-dimensional Gaussian shape from text as described in any one of claims 1 to 7, the device comprises: The initialization module is used to obtain an initialized 3D Gaussian model through a pre-trained 3D generator based on the text input of the target task; The geometry generation module is used to perform multi-view projection on the initialized 3D Gaussian model to obtain a key view image and a set of auxiliary view images; feature extraction is performed on the auxiliary view images and the extracted features are fused with the features of the key view image through an attention mechanism; the fused features and text input are injected into the diffusion model, and the initialized 3D Gaussian model is subjected to multi-view joint geometric optimization using fractional distillation loss to obtain a geometrically optimized 3D Gaussian model; The detail optimization module implements a hierarchical Gaussian constraint mechanism on the geometrically optimized 3D Gaussian model, imposes stronger constraints on the Gaussian points generated earlier, and grants higher degrees of freedom to the Gaussian points added later, to obtain the final 3D Gaussian shape.

9. A device for generating three-dimensional Gaussian shapes from text, characterized in that include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods for generating a three-dimensional Gaussian shape from text according to claims 1-7.

10. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions are executed on the terminal, the method for generating a three-dimensional Gaussian shape from text as claimed in any one of claims 1 to 7 can be implemented.

Images