A three-dimensional gaussian-based low-memory text generation three-dimensional model method and system

Abstract

The present application relates to the field of three-dimensional generation and computer vision, and particularly to a low-memory text-based three-dimensional model generation method and system based on three-dimensional Gauss. The method comprises the following steps: initializing a Gauss body set through a three-dimensional point cloud diffusion model, fusing position information and time step information to generate a feature sequence, alternately performing local attention and global attention calculation on the feature sequence to output geometric prediction, calculating score distillation sampling loss using a two-dimensional image diffusion model, combining three-dimensional point cloud diffusion loss to update geometric parameters, generating a binary label based on loss gradient norm, constructing opacity regularization loss to differentiate and constrain the opacity of the Gauss body, and iteratively optimizing the combined loss and removing low-opacity Gauss bodies. The present application reduces the computational complexity through an alternating attention mechanism, realizes active pruning through opacity regularization, significantly reduces the memory occupation, and realizes efficient deployment on resource-constrained devices.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional generation and computer vision, in particular to a low-memory text-to-3D model method and system based on three-dimensional Gauss. BACKGROUND

[0002] With the development of generative artificial intelligence, text-to-3D model technology has shown great potential in virtual reality, game development and metaverse content construction. Three-dimensional Gauss gradually becomes the mainstream representation method for text-to-3D model task due to its explicit scene representation and real-time rendering capability.

[0003] However, the existing text-to-3D method based on three-dimensional Gauss faces the problem of high memory occupation during training, which limits its deployment on ordinary consumer-level hardware. The main reasons are as follows: some frameworks use Transformer structure to model Gauss body features, and the computational complexity of its self-attention mechanism has a quadratic relationship with the number of Gauss bodies , as the number of Gauss bodies increases during training, the memory consumption grows explosively; cross-view rendering, SDS loss calculation and multi-round gradient backpropagation lead to rapid expansion of the number of Gauss bodies; some redundant Gauss bodies with low contribution occupy storage space but are not cleaned up in time. Therefore, the existing technology cannot balance generation quality and memory occupation under limited computing resources, restricting the application of the technology on resource-constrained devices. SUMMARY

[0004] The present application provides a low-memory text-to-3D model method and system based on three-dimensional Gauss, aiming to solve the problem of high memory occupation of the existing text-to-3D model method based on three-dimensional Gauss, and realize low-memory efficient deployment.

[0005] To achieve the above purpose, the present application provides the following technical solution:

[0006] The present application provides a low-memory text-to-3D model method based on three-dimensional Gauss, comprising:

[0007] S100: receiving a text input, extracting text semantic features through a pre-trained text encoder, generating an initial three-dimensional point cloud through a text input three-dimensional point cloud diffusion model, and initializing a three-dimensional Gauss body set based on the initial three-dimensional point cloud;

[0008] S200: fusing the position information and time step information of the three-dimensional Gauss body set to generate a Gauss body feature sequence;

[0009] S300: alternately performing local attention calculation and global attention calculation on the Gauss body feature sequence, and outputting geometry prediction;

[0010] S400: render the geometric prediction as a two-dimensional image, calculate a score distillation sampling loss by using a pre-trained two-dimensional image diffusion model, and update geometric parameters of the three-dimensional Gaussian body set in combination with a three-dimensional point cloud diffusion loss;

[0011] S500: calculate a loss gradient norm of each Gaussian body position in the three-dimensional Gaussian body set, and generate a binary label for the corresponding Gaussian body according to comparison of the loss gradient norm with a preset threshold;

[0012] S600: construct an opacity regularization loss based on the binary label, and constrain the opacity of the Gaussian body with a label of 1 to tend to 1 and the opacity of the Gaussian body with a label of 0 to tend to 0;

[0013] S700: combine the score distillation sampling loss, an image mean square error loss, and the opacity regularization loss to iteratively optimize the three-dimensional Gaussian body set, remove Gaussian bodies with an opacity lower than a pruning threshold, and output a three-dimensional Gaussian model.

[0014] As a preferred technical solution of the present application, the fusion of the position information and the time step information of the three-dimensional Gaussian body set comprises:

[0015] Linearly projecting the position information of the three-dimensional Gaussian body set;

[0016] Embedding coding is performed on the time step information;

[0017] The projected position information and the embedded coded time step information are fused and subjected to layer normalization to obtain the Gaussian body feature sequence.

[0018] As a preferred technical solution of the present application, the steps of alternately performing local attention calculation and global attention calculation on the Gaussian body feature sequence comprise:

[0019] According to the total number of current Gaussian bodies, the Gaussian body feature sequence is divided into a plurality of non-overlapping local regions, and multi-head attention calculation is independently performed in each local region;

[0020] A semantic neighbor graph is constructed in a feature space by using a k-nearest neighbor algorithm on the Gaussian body feature sequence, and feature aggregation is performed on the semantic neighbor graph;

[0021] The multi-head attention calculation in the local region and the feature aggregation based on the semantic neighbor graph are alternately and sequentially performed.

[0022] As a preferred technical solution of the present application, the division of the Gaussian body feature sequence into a plurality of non-overlapping local regions according to the total number of current Gaussian bodies comprises:

[0023] Divide the current total number of Gaussian bodies by the Gaussian body number threshold of a single local region to obtain a local region number;

[0024] Uniformly divide the Gaussian body feature sequence according to the local region number.

[0025] As a preferred technical solution of the present application, the feature aggregation of the semantic neighbor graph comprises:

[0026] Linearly projecting the Gaussian body feature sequence to generate a query vector, a key vector and a value vector;

[0027] Using the k-nearest neighbor algorithm on the value vector to construct a semantic neighbor graph in the feature space and perform a first round of feature aggregation;

[0028] Based on the features after the first round of aggregation, the neighbor relationship is recalculated for a second round of feature aggregation to obtain enhanced features;

[0029] Perform multi-head attention calculation on the enhanced features, the query vector and the key vector.

[0030] As a preferred technical solution of the present application, the number of neighbors when calculating the neighbor relationship in the second round of feature aggregation is less than the number of neighbors in the first round of feature aggregation.

[0031] As a preferred technical solution of the present application, the opacity regularization loss based on the binary label comprises:

[0032] For each Gaussian body in the set of three-dimensional Gaussian bodies, calculate the binary cross-entropy loss between its binary label and its opacity;

[0033] Average the binary cross-entropy loss of all Gaussian bodies to obtain the opacity regularization loss.

[0034] As a preferred technical solution of the present application, combining the score distillation sampling loss and the opacity regularization loss to iteratively optimize the set of three-dimensional Gaussian bodies comprises:

[0035] Set weight coefficients for the score distillation sampling loss, the image mean square error loss and the opacity regularization loss, respectively;

[0036] Multiply each loss by the corresponding weight coefficient and sum to obtain a total loss;

[0037] Based on the total loss, update the position, scaling, rotation, opacity and color parameters of the set of three-dimensional Gaussian bodies by back propagation.

[0038] The present application also proposes a low-memory text-based three-dimensional model generation system based on three-dimensional Gaussian, comprising:

[0039] An initialization module is used to receive text input, extract text semantic features through a pre-trained text encoder, generate an initial three-dimensional point cloud from the text input three-dimensional point cloud diffusion model, and initialize a three-dimensional Gaussian volume set based on the initial three-dimensional point cloud.

[0040] The feature fusion module is used to fuse the position information and time step information of the three-dimensional Gaussian volume set to generate a Gaussian volume feature sequence;

[0041] An alternating attention module is used to alternately perform local attention calculations and global attention calculations on the Gaussian feature sequence and output geometric predictions.

[0042] The geometry optimization module is used to render the geometric prediction as a two-dimensional image, calculate the fractional distillation sampling loss using a pre-trained two-dimensional image diffusion model, and update the geometric parameters of the three-dimensional Gaussian body set by combining the three-dimensional point cloud diffusion loss.

[0043] The label generation module is used to calculate the loss gradient norm of each Gaussian body position in the three-dimensional Gaussian body set, and generate binary labels for the corresponding Gaussian bodies based on the comparison between the loss gradient norm and a preset threshold.

[0044] The regularization module is used to construct an opacity regularization loss based on the binary labels, constraining the opacity of the Gaussian body with label 1 to tend to 1, and constraining the opacity of the Gaussian body with label 0 to tend to 0.

[0045] The iterative optimization module is used to combine the fractional distillation sampling loss, the image mean square error loss, and the opacity regularization loss to iteratively optimize the three-dimensional Gaussian volume set, remove Gaussian volumes with opacity below the pruning threshold, and output a three-dimensional Gaussian model.

[0046] The beneficial effects of this invention are:

[0047] 1. This invention reduces computational complexity from... Reduce to Local attention reduces single-time memory usage, while global attention maintains cross-region consistency through the k-nearest neighbor semantic graph. This mechanism significantly reduces peak memory usage while ensuring the semantic coherence and geometric stability of the generative model.

[0048] 2. This invention utilizes a gradient norm-based self-supervised label generation mechanism combined with binary cross-entropy opacity regularization loss to achieve differentiated constraints, actively compressing the opacity of redundant Gaussian bodies and periodically pruning them. This innovative combined strategy effectively suppresses the disorderly expansion of the number of Gaussian bodies, significantly reducing memory usage and shortening training time.

[0049] 3. This invention organically combines efficient modeling of alternating attention with active pruning using opacity regularization to construct a complete low-memory optimization system. This system achieves stable operation on consumer-grade devices while ensuring generation quality, breaking the dependence on high-end professional hardware. Attached Figure Description

[0050] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0051] Figure 1 This is a flowchart illustrating a low-memory text-to-3D model generation method based on 3D Gaussian.

[0052] Figure 2 This is a flowchart illustrating the structure of the AttTransformer module of this invention;

[0053] Figure 3 This is the flowchart of the alternating attention method of the present invention;

[0054] Figure 4 This is a schematic diagram of the structure of a low-memory text generation 3D model system based on 3D Gaussian in this invention;

[0055] Figure 5 This is a rendering of a 3D model generated by the present invention based on the text description "a brown, fluffy rabbit";

[0056] Figure 6 This is a rendering of a 3D model generated by the present invention based on the text description "a delicious hamburger";

[0057] Figure 7 This is a rendering of a 3D model generated by the present invention based on the text description "a blue tulip";

[0058] Figure 8 This is a rendering of a 3D model generated by the present invention based on the text description "an old car covered with vines". Detailed Implementation

[0059] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0060] Example 1: As Figure 1 As shown, the present invention provides a low-memory text generation method for 3D modeling based on 3D Gaussian, comprising:

[0061] S100: Receive text input, extract text semantic features through a pre-trained text encoder, generate an initial three-dimensional point cloud from the text input three-dimensional point cloud diffusion model, and initialize a three-dimensional Gaussian volume set based on the initial three-dimensional point cloud.

[0062] Specifically, it first receives a text description of the target 3D model input by the user. The text description, written in natural language, describes the semantic content of the 3D model to be generated, such as "a brown, fluffy rabbit." The pre-trained CLIP text encoder is used to process the text description. The system performs encoding processing to extract its high-dimensional semantic vector, which is used for subsequent cross-modal semantic alignment.

[0063] To accelerate model convergence and provide a reasonable initial geometry, the text description... The pre-trained text is input into the Point-E point cloud diffusion model for processing. The Point-E model, based on a diffusion denoising mechanism, can generate a coarse 3D point cloud corresponding to the text semantics. After multi-step denoising and sampling by the diffusion model, a coarse 3D point cloud containing... Initial 3D point cloud of each point ,in This indicates the number of points in the point cloud. Each point contains three-dimensional spatial coordinate information.

[0064] Based on the generated initial 3D point cloud Initialize the set of three-dimensional Gaussian volumes Specifically, the point cloud The three-dimensional coordinates of each point are directly used as the initial position of the corresponding Gaussian body, that is, let the first... The position of the Gaussian body ,in The index is a Gaussian volume index, with values ​​ranging from 1 to... Simultaneously, initialize other geometric and appearance parameters for each Gaussian body: scaling parameters. The rotation parameter is used to control the size of the Gaussian volume, with the initial value set to the reciprocal of the local neighborhood density of the point cloud to ensure that the size of the Gaussian volume is adapted to the point cloud distribution density; Used to control the orientation of the Gaussian body, initialized with a unit quaternion (1, 0, 0, 0) to represent a state without rotation; opacity Used to control the transparency of the Gaussian body, the initial value is set to 0.5, in a semi-transparent state for subsequent optimization and adjustment; color coefficient. The color information used to control the Gaussian volume is represented by spherical harmonic function coefficients, initialized as a zero vector, representing neutral gray. Through the above process, the entire process from text input to the initialization of the 3D Gaussian volume set is completed, providing a semantically sound and parameter-appropriate initial structure for subsequent geometric optimization and appearance refinement stages.

[0065] S200: The position information and time step information of the three-dimensional Gaussian volume set are fused to generate a Gaussian volume feature sequence;

[0066] Furthermore, fusing the positional information and time step information of the three-dimensional Gaussian volume set includes:

[0067] The positional information of the three-dimensional Gaussian volume set is linearly projected;

[0068] The time step information is embedded and encoded;

[0069] The projected position information is fused with the embedded encoded time step information and then normalized to obtain the Gaussian body feature sequence.

[0070] Specifically, the AttTransformer module receives data input during the geometry optimization phase. The AttTransformer module receives the Gaussian body position information at the current time step and the time step of the diffusion model as input. The Gaussian body position information originates from the state of the Point-E initialization position after the diffusion process in step S100, and is represented as... ,in The current number of Gaussian volumes, where 3 represents the three-dimensional spatial coordinates; time step. This represents the current denoising time step of the diffusion model.

[0071] First, the position information of the Gaussian body Perform linear projection processing to map it from three-dimensional spatial coordinates to the feature dimension. ,in The feature dimension D is a preset parameter. The value of the feature dimension D should be between 128 and 512, and in a preferred embodiment, it is set to 256 dimensions. This dimension can fully express the spatial semantic information of the Gaussian volume while maintaining a moderate computational cost. The linear projection is implemented through a learnable linear transformation layer, denoted as... .

[0072] At the same time, for time step information Embedding encoding is performed to convert discrete time step values ​​into continuous high-dimensional vector representations, denoted as... The temporal embedding can be implemented using a sinusoidal positional encoding mechanism or a learnable embedding layer, with the output dimension and the positional projection feature dimension being... To maintain consistency, when using sinusoidal position coding, a coding vector is generated consisting of multiple sine and cosine components of different frequencies, covering multiple orders of magnitude, ensuring a discriminative representation of diffusion processes at different time scales.

[0073] Projected position features With embedded encoded time step features Element-wise addition and fusion are performed, and then the fusion result is subjected to layer normalization to obtain the Gaussian body feature sequence. The process is represented as:

[0074] ;

[0075] in, The representative layer normalization operation is used to stabilize the feature distribution and accelerate model convergence. Through the above feature fusion process, the spatial geometric information and diffusion time information of the Gaussian body are effectively combined to generate a Gaussian body feature sequence. It incorporates both the prior position of the current Gaussian body and the temporal information of the diffusion model, providing a unified feature representation for subsequent alternating attention calculations.

[0076] S300: Alternately perform local attention calculation and global attention calculation on the Gaussian feature sequence, and output geometric prediction;

[0077] Furthermore, the step of alternately performing local attention calculation and global attention calculation on the Gaussian body feature sequence includes:

[0078] The Gaussian feature sequence is divided into multiple non-overlapping local regions based on the current total number of Gaussian bodies, and multi-head attention calculation is performed independently in each local region.

[0079] A semantic nearest neighbor graph is constructed in the feature space for the Gaussian body feature sequence using the k-nearest neighbor algorithm, and feature aggregation is performed on the semantic nearest neighbor graph;

[0080] The multi-head attention computation within the local region is alternately and sequentially performed in series with feature aggregation based on semantic nearest neighbor graphs.

[0081] Furthermore, dividing the Gaussian body feature sequence into multiple non-overlapping local regions based on the current total number of Gaussian bodies includes:

[0082] Divide the total number of Gaussian volumes by the threshold for the number of Gaussian volumes in a single local region to obtain the number of local regions.

[0083] The Gaussian body feature sequence is divided evenly according to the number of local regions.

[0084] Furthermore, feature aggregation of the semantic nearest neighbor graph includes:

[0085] Linear projection is performed on the Gaussian feature sequence to generate query vector, key vector, and value vector;

[0086] The k-nearest neighbor algorithm is used to construct a semantic nearest neighbor graph in the feature space for the value vector and to perform the first round of feature aggregation;

[0087] Based on the features obtained from the first round of aggregation, the nearest neighbor relationships are recalculated for the second round of feature aggregation to obtain enhanced features.

[0088] The enhanced features are then subjected to multi-head attention computation with the query vector and key vector.

[0089] Furthermore, the number of neighbors calculated in the second round of feature aggregation is less than the number of neighbors in the first round of feature aggregation.

[0090] Specifically, in this S300, an alternating attention mechanism is used to reduce the computational complexity of the traditional self-attention mechanism while maintaining semantic consistency across regions. The alternating attention mechanism includes two parts: local attention computation and global attention computation, which are executed alternately and in series.

[0091] like Figure 2 As shown, the AttTransformer module's processing flow consists of three core stages: region partitioning, local attention computation, and global attention computation. The input Gaussian feature sequence is first partitioned into regions, decomposing the large-scale Gaussian set into multiple local regions. Multi-head attention computation is performed independently within each local region, reducing the memory usage of a single computation. Simultaneously, a semantic nearest neighbor graph is constructed in the feature space through a graph enhancement module, enabling global information exchange across regions. Local and global attention are executed alternately in series, and the geometric prediction result is output after weighted aggregation. This alternating attention architecture effectively reduces computational complexity while maintaining semantic consistency.

[0092] To reduce the memory footprint of large Gaussian sets in attention processing, a block-based local attention computation strategy is adopted, based on the total number of Gaussian volumes. Adaptive determination of the number of local regions Specifically, the total number of Gaussian bodies currently... Divide by the Gaussian volume threshold for a single local region and round up the result. The calculation formula is as follows:

[0093] ;

[0094] in The threshold for the number of Gaussian bodies that can be stably processed in a single local region is determined based on the GPU memory capacity and the efficiency of multi-head attention computation. In a preferred embodiment, it is set to 512. For GPUs with larger memory, this threshold can be appropriately increased to 1024, and for devices with limited memory, it can be reduced to 256. Less than hour, A value of 1 indicates that no region division is performed, and all Gaussian volumes undergo attention calculation within a single region. The Gaussian volume feature sequence H generated in step S200 is then processed according to the number of local regions. Divide evenly to obtain Non-overlapping local regions For the former Each region contains [number] areas. Gaussian body characteristics; for the last region It includes the remaining Gaussian body features, ensuring that all Gaussian bodies are assigned to the appropriate regions.

[0095] A multi-head attention mechanism is independently applied to perform feature interaction computation within each local region. The number of heads in the multi-head attention mechanism can be set between 4 and 16, with 8 heads in a preferred embodiment. Each head captures different feature subspace information in parallel. The computation process of multi-head attention includes: linearly projecting the input features into a query matrix. Key matrix Sum matrix Calculate the attention weight matrix and the value matrix. Weighted aggregation is performed, and finally, the multi-head results are integrated through the output projection layer. After performing the multi-head attention calculation in parallel on each local region, the outputs of all local regions are concatenated in their original order to obtain the local attention features. This process is represented as follows:

[0096] ;

[0097] in, This represents standard multi-head attention computation. This indicates a concatenation operation. By using localized block processing, the original computational complexity is reduced from... Reduce to This significantly reduces video memory usage.

[0098] To compensate for the semantic incoherence caused by the limited field of view of local attention, a global attention computation based on the nearest neighbor in the feature space is adopted. First, the input feature sequence is linearly projected to generate the query vector. Key vector Sum value vector All three dimensions are .

[0099] For the value vector The input graph enhancement sub-network is used for processing. Specifically, convolutional layers are first used to perform feature transformation on the value vectors, and then the k-nearest neighbor algorithm is used to construct a semantic nearest neighbor graph in the feature space. The number of neighbors for the first round of feature aggregation is set. In order to capture a wide range of semantic associations, in a preferred embodiment The value should be between 16 and 64, preferably 32. The k-nearest neighbor algorithm uses Euclidean distance or cosine similarity as a similarity metric to calculate the similarity of each Gaussian feature. Find the nearest neighbors, construct the nearest neighbor connection relationship, and then aggregate the feature information of the neighboring nodes for each node to complete the first round of feature aggregation.

[0100] Based on the features from the first round of aggregation, convolutional layers are used to further transform the features, and then the nearest neighbor relationships are recalculated for a second round of feature aggregation. In the second round of aggregation, the number of neighbors... It should be less than the number of neighbors in the first round. In the preferred embodiment Values The features are 0.3 to 0.7 times, preferably 0.5 times. Two rounds of progressive nearest neighbor graph aggregation allow the first round to capture broad semantic associations, while the second round further refines local consistency features by reducing the number of neighbors, resulting in enhanced features. .

[0101] Enhanced features Compared with the original query vector and key vector The input multi-head attention module performs weighted aggregation calculations, thereby enabling long-range information interaction across regions without significantly increasing the computational load, while maintaining global semantic consistency.

[0102] like Figure 3 As shown, the execution flow of alternating attention includes a local attention stage, which involves region partitioning and independent multi-head attention computation within each region, and a global attention stage, which constructs a semantic nearest neighbor graph and aggregates features through a graph enhancement module. These two stages are executed alternately, forming a cyclic iterative structure. Specifically, the local attention stage computes in parallel for each partitioned region, outputting local features; the global attention stage establishes cross-regional semantic connections using the k-nearest neighbor algorithm, aggregating global information. After multiple alternating iterations, the system determines whether the convergence condition is met. If not, it continues to execute alternating attention computation; if so, it outputs the final geometric prediction result. This flow design achieves a balance between local refinement and global consistency.

[0103] In the AttTransformer module, local attention and global attention computations are executed alternately and in series. In each round of processing, local attention is first executed to capture fine features within the region, and then global attention is executed to establish semantic connections across regions. This process is repeated repeatedly, enabling the model to efficiently process large-scale Gaussian sets while maintaining global geometric and semantic consistency. After multiple layers of alternating attention processing, the AttTransformer module outputs the geometric prediction value of the Gaussian volume, including updates to geometric parameters such as position, scaling, and rotation.

[0104] S400: Render the geometric prediction as a two-dimensional image, calculate the fractional distillation sampling loss using a pre-trained two-dimensional image diffusion model, and update the geometric parameters of the three-dimensional Gaussian body set by combining the three-dimensional point cloud diffusion loss.

[0105] Specifically, firstly, based on the geometric prediction values ​​output by the AttTransformer module in step S300, the three-dimensional Gaussian volume set is... The geometric parameters of each Gaussian body are updated. Specifically, the predicted position offset, scaling adjustment, and rotation adjustment are superimposed on the current parameters of the corresponding Gaussian body.

[0106] Then, differentiable rendering technology is used to render the updated 3D Gaussian volume set into 2D images from multiple preset camera perspectives. The number of camera perspectives can be set to 2 to 8; in a preferred embodiment, 4 camera perspectives are used, evenly distributed around the 3D model. The azimuth angles of the camera positions are evenly distributed, and the elevation angle can be set between 15° and 45°. The distance between the camera and the model center is adaptively adjusted according to the model's bounding box size to ensure complete coverage of the model's different perspectives. The rendering resolution can be set between 256×256 and 1024×1024 pixels; in a preferred embodiment, it is set to 512×512 pixels to balance rendering quality and computational efficiency. The differentiable rendering process uses a point-based... The blending method projects each 3D Gaussian volume onto a 2D image plane, based on its position. Scaling and rotation The projection distribution on the image plane is calculated through perspective projection transformation; for each pixel in the image, the Gaussian volumes affecting that pixel are sorted from front to back according to depth, and then the color of each Gaussian volume is accumulated sequentially. and opacity Contribute to generating rendered images .

[0107] The rendered 2D image The input is processed using a pre-trained two-dimensional image diffusion model. This two-dimensional image diffusion model can employ pre-trained models such as the Stable Diffusion series. This model is based on the UNet architecture and is pre-trained on large-scale image-text pair datasets, possessing powerful text-to-image semantic alignment capabilities. The input to the diffusion model includes: a rendered image. The text description extracted by the text encoder in step S100 after normalization processing The conditional embedding vector and the diffusion time step of random sampling The diffusion time step Sampling is performed within a preset range to avoid extreme noise conditions in the diffusion process. In a preferred embodiment, the time step sampling range is [0.02, 0.98]. The fractional distillation sampling loss is calculated using this diffusion model. ,in The set of parameters representing a three-dimensional Gaussian volume.

[0108] Simultaneously, the AttTransformer module predicts the target position of the Gaussian body based on a 3D point cloud diffusion model during processing. The mean square error between the current Gaussian body position and the target position predicted by AttTransformer is calculated to obtain the 3D point cloud diffusion loss. This 3D point cloud diffusion loss constrains the geometric distribution of the Gaussian body in 3D space, ensuring the rationality of the overall structure and semantic consistency.

[0109] Fractional distillation sampling loss gradient of the two-dimensional image diffusion model A weighted combination with the 3D point cloud diffusion loss is obtained to yield a comprehensive geometric optimization loss. Backpropagation is then performed based on this comprehensive loss to calculate the position of each Gaussian volume in the 3D Gaussian volume set. Scaling Rotation The gradients of the geometric parameters with respect to the loss are calculated. The Adam optimizer is used to iteratively update each geometric parameter based on the calculated gradients.

[0110] S500: Calculate the loss gradient norm at each Gaussian body position in the set of three-dimensional Gaussian bodies, and generate binary labels for the corresponding Gaussian bodies based on the comparison between the loss gradient norm and a preset threshold.

[0111] Specifically, after completing the geometry optimization stage, the appearance refinement stage begins. In each iteration of the appearance refinement, the current total loss is first calculated. The total loss mentioned above. This includes a weighted combination of multiple losses, such as the fractional distillation sampling loss in step S400 and the mean square error loss between the rendered image and the reference image.

[0112] Then, for the set of three-dimensional Gaussian volumes For each Gaussian body, calculate the total loss. Relative to the position of the Gaussian body gradient The gradient Let be a three-dimensional vector, representing the partial derivatives of the total loss with respect to the Gaussian volume position in the three spatial dimensions. The gradient at all Gaussian volume positions can be efficiently calculated using the backpropagation algorithm.

[0113] gradient vector at each Gaussian body position Calculate its L2 norm to obtain the gradient norm. The gradient norm reflects the degree of influence of the Gaussian body's position on the final loss: a larger gradient norm indicates that the Gaussian body contributes significantly to the rendering result or its position has not yet converged and needs further optimization; a smaller gradient norm indicates that the Gaussian body contributes less to the rendering result and may be a redundant Gaussian body or a Gaussian body with overlapping positions.

[0114] Based on the calculated gradient norm With preset threshold The comparison dynamically generates binary supervision labels for each Gaussian body. The tag generation rules are as follows:

[0115] ;

[0116] in, This is a preset gradient threshold used to distinguish between important and redundant Gaussian bodies. When the gradient norm is greater than the threshold... At that time, mark This indicates that the Gaussian volume is an important Gaussian volume, contributing significantly to the rendering result or that its location has not fully converged; when the gradient norm is less than or equal to the threshold... At that time, mark This indicates that the Gaussian volume is redundant and contributes very little to the final rendering result. The gradient threshold... It can be set to to Within the scope, in the preferred embodiment, it is set as follows: The threshold should be selected to effectively distinguish between important and redundant Gaussian bodies in the gradient norm distribution. The specific value can be adjusted according to the complexity of the scene and the number of Gaussian bodies.

[0117] The aforementioned gradient norm-based self-supervised label generation mechanism automatically identifies redundant Gaussian bodies in the scene without manual annotation, providing a reliable supervisory signal for the opacity regularization in step S600. This mechanism can dynamically adjust labels during the training process, accurately reflecting the importance of each Gaussian body in the current optimization stage.

[0118] S600: Construct an opacity regularization loss based on the binary labels, constraining the opacity of the Gaussian body with a label of 1 to tend to 1, and constraining the opacity of the Gaussian body with a label of 0 to tend to 0;

[0119] Furthermore, constructing the opacity regularization loss based on the binary labels includes:

[0120] For each Gaussian body in the set of three-dimensional Gaussian bodies, calculate the binary cross-entropy loss between its binary label and its opacity;

[0121] The opacity regularization loss is obtained by averaging the binary cross-entropy loss over all Gaussian bodies.

[0122] Specifically, based on the binary label generated in step S500 Construct opacity regularization loss The opacity distribution of Gaussian volumes is actively constrained. The opacity regularization loss employs a binary cross-entropy form for the set of three-dimensional Gaussian volumes. For each Gaussian body, calculate its binary label. rather than opacity The binary cross-entropy loss between them.

[0123] Specifically, the loss of opacity regularization The calculation formula is:

[0124] ;

[0125] in, This represents the total number of 3D Gaussian volumes in the current scene; The index represents the Gaussian body, with values ​​ranging from 1 to... ; Indicates the first A binary label for a Gaussian body, generated from the gradient norm-based judgment result in step S500; if the gradient norm is greater than a threshold... but Represents an important Gaussian body if its gradient norm is less than or equal to a threshold. but Represents a redundant Gaussian body; Indicates the first The opacity value of a Gaussian body, ranging from [value range missing]. .

[0126] The mechanism by which this loss function works is: when At that time, in the loss function The term is activated, and in order to minimize the loss, the optimization process forces... The opacity tends towards 1, thus maintaining the high opacity of the important Gaussian volume and ensuring its effective contribution to the rendering result; when At that time, in the loss function The term is activated, and in order to minimize the loss, the optimization process forces... It tends to 0, thereby actively compressing the opacity of redundant Gaussian bodies.

[0127] Through the above opacity regularization constraints, the redundant Gaussian body ( The opacity of the important Gaussian body is gradually compressed to a minimum during the training iterations, while the opacity of the important Gaussian body is reduced to a minimum. The opacity of the Gaussian body is maintained at a high level. This differential constraint mechanism based on gradient self-supervised labels enables dynamic control of the importance of the Gaussian body, creating conditions for active pruning in step S700, thereby effectively suppressing the disorderly expansion of the number of Gaussian bodies and reducing memory usage.

[0128] S700: Combine the fractional distillation sampling loss, image mean square error loss and opacity regularization loss to iteratively optimize the set of three-dimensional Gaussian volumes, remove Gaussian volumes with opacity below the pruning threshold, and output a three-dimensional Gaussian model.

[0129] Furthermore, iteratively optimizing the three-dimensional Gaussian volume set by combining the fractional distillation sampling loss and the opacity regularization loss includes:

[0130] Weighting coefficients are set for the fractional distillation sampling loss, the image mean square error loss, and the opacity regularization loss, respectively;

[0131] The total loss is obtained by multiplying each loss by its corresponding weight coefficient and then summing the results.

[0132] The position, scaling, rotation, opacity, and color parameters of the three-dimensional Gaussian volume set are updated by backpropagation based on the total loss.

[0133] Specifically, during the iterative optimization process of the appearance refinement stage, a total optimization objective function is constructed. The fractional distillation sampling loss and image mean square error loss in step S400 are combined with the opacity regularization loss in step S600 by weighting:

[0134] ;

[0135] in, The fractional distillation sampling loss calculated using the pre-trained two-dimensional image diffusion model in step S400 is used to guide the rendering results to align with the semantic content of the text description. The mean squared error loss between the rendered image and the reference image is used to constrain image quality; The opacity regularization loss constructed for step S600 is used to actively compress the opacity of redundant Gaussian volumes. , , These are the weighting coefficients for the corresponding losses, used to balance the importance of each loss in the overall optimization objective. In a preferred embodiment, the opacity regularization weights... Set it to 0.1.

[0136] Based on the total loss function The system uses the Adam optimizer to optimize the 3D Gaussian volume set. The parameters of all Gaussian bodies are updated via backpropagation. The initial learning rate of the Adam optimizer can be set to... to Within the scope, in the preferred embodiment, it is set as follows: Differentiated learning rate strategies can be employed for different parameters: higher learning rates are used for geometric parameters (position, scaling, rotation) to accelerate geometric convergence, while lower learning rates are used for appearance parameters (opacity, color) to achieve smooth adjustment. Specifically, the learning rate for appearance parameters can be set to 0.01 to 0.1 times that of geometric parameters. Learning rate scheduling strategies, such as cosine annealing and step decay, can be used to gradually reduce the learning rate as training progresses. The total loss is calculated using an automatic differentiation mechanism. The gradient relative to each Gaussian body parameter, including position. Scaling Rotation Opacity and color coefficient The gradient is calculated. The Adam optimizer adaptively adjusts and updates the parameters based on the calculated gradient and its momentum estimate.

[0137] In this iterative optimization process, the redundant Gaussian body undergoes opacity regularization loss. Under the mandatory constraints, its opacity value It is actively compressed and rapidly decays. The system periodically checks the opacity values ​​of all Gaussian bodies, and when the opacity of a certain Gaussian body... If the opacity of a Gaussian body falls below a preset pruning threshold, its contribution to the final rendering result is deemed negligible, and it is removed from the 3D Gaussian body set G. The pruning threshold can be set between 0.001 and 0.01, and in a preferred embodiment, it is set to 0.005. The removal operation can employ a periodic pruning strategy, performing a pruning operation every several iterations. In a preferred embodiment, the pruning frequency can be set to be performed every 100 to 1000 steps, preferably every 300 to 500 steps. An adaptive pruning strategy can also be used, dynamically adjusting the pruning frequency according to the training phase. The pruning frequency is reduced in the early stages of training to maintain sufficient geometric exploration, and increased in the later stages of training to accelerate convergence. In each pruning operation, all Gaussian bodies are traversed, and those with opacity below the pruning threshold are removed from the parameter list, and the total number of Gaussian bodies N is updated, thus achieving active pruning of redundant Gaussian bodies.

[0138] Through iterative execution of the aforementioned joint optimization and active pruning mechanisms, the system effectively controls the expansion of the number of Gaussian volumes while optimizing appearance quality, significantly reducing memory usage. When training reaches a preset number of iterations, iterative optimization stops, and the final optimized 3D Gaussian model is output. In a preferred embodiment, the preset number of iterations is set to 15,000 steps. The output 3D Gaussian model maintains semantic consistency with the text description and high-quality rendering effects, while also achieving efficient deployment on resource-constrained devices through effective memory management.

[0139] Example 2: This example uses the low-memory text generation 3D model system based on 3D Gaussian as described in this invention for practical application verification. The system is as follows: Figure 4 As shown, the system includes an initialization module, a feature fusion module, an alternating attention module, a geometry optimization module, a label generation module, a regularization module, and an iterative optimization module. Through multiple 3D model generation tasks described by real text, the significant advantages of this invention in reducing memory usage, improving generation quality, and accelerating convergence are verified.

[0140] The input text description is "a fluffy gray rabbit". The initialization module generates an initial point cloud containing 2048 points and initializes a corresponding set of Gaussian volumes. The alternating attention module adaptively divides the region into 4 local regions based on the number of Gaussian volumes. ), reducing computational complexity from Reduce to At the same time, through the k-nearest neighbor algorithm ( A semantic nearest neighbor graph is constructed to maintain consistency of details across regions. The geometry optimization module uses a pre-trained diffusion model to calculate the SDS loss, generating accurate rabbit geometry through 8000 iterations. After entering the appearance refinement stage, the label generation module sets a threshold. Approximately 30% of redundant Gaussian bodies were identified. The regularization module, based on binary cross-entropy loss, compressed the opacity of redundant Gaussian bodies to below 0.01, while maintaining the opacity of important Gaussian bodies above 0.95. Iterative optimization module settings... An adaptive pruning strategy was adopted: pruning was performed every 500 steps in the early training phase (first 5000 steps), and every 300 steps in the later training phase (after 5000 steps). After 15000 training steps, the number of Gaussian bodies decreased from 2048 to 1420, reducing memory usage by approximately 31%. The generated model exhibited soft hair texture, natural ear shape, and delicate gray tones, such as... Figure 5 As shown.

[0141] The input text description is "a delicious hamburger," and the initialization module generates 3072 Gaussian volumes. The alternating attention module divides the data into 6 local regions (…). This significantly reduces peak memory usage. The system accurately captures the multi-layered structure of a hamburger, using local attention to achieve detailed modeling of layers such as bread, lettuce, tomato, and patty, while global attention ensures the overall shape's consistency. The label generation module identifies approximately 35% redundant Gaussian bodies, mainly distributed within the bread and in interlayer occlusion areas. The regularization module compresses the opacity of redundant Gaussian bodies through differential constraints, while preserving key details such as surface texture and sesame seed particles. After active pruning, the final number of Gaussian bodies is reduced to 2180, a 29% reduction from the initial number. Figure 6 As shown, the generated hamburger model has realistic colors, clear hierarchical structure and delicate surface texture, and the memory usage is reduced by about 28% compared with the traditional method.

[0142] The input text description is "a blue tulip". The system initializes 1536 Gaussian volumes, and the alternating attention module divides them into 3 local regions. During the geometric optimization process, the curved surface morphology of the petals and the slender structure of the stem were accurately modeled. The label generation module is based on the gradient norm threshold. Approximately 28% of redundant Gaussian volumes were identified, and their opacity was actively compressed by the regularization module. After 12,000 training steps and periodic pruning, 1,105 effective Gaussian volumes were ultimately retained, reducing memory usage by approximately 32%. Figure 7 As shown, the generated tulip model exhibits natural petal curls, delicate color gradients, and realistic lighting effects.

[0143] The input text description is "an old car covered in vines." The scene contains two different geometric and textural features: a rigid car body and flexible vines. The initialization module generates 4096 Gaussian volumes, and the alternating attention module divides the scene into 8 local regions. In each region, local features such as vehicle body surfaces, window reflections, and vine entanglements are independently modeled. Global attention is aggregated through two rounds of k-nearest neighbor graphs. The system establishes a spatial coverage relationship between the vines and the vehicle body, ensuring the vines naturally adhere to the vehicle's surface. The label generation module identifies approximately 38% of redundant Gaussian bodies, primarily located inside the vehicle body, in areas obscured by vines, and in visually invisible parts. The regularization module then... The weight settings maximize redundancy suppression, and the iterative optimization module employs an adaptive pruning strategy: pruning is performed every 500 steps in the early training phase (first 5000 steps) and every 300 steps in the later training phase (after 5000 steps). After 15000 training steps, the number of Gaussian bodies decreased from 4096 to 2540, a reduction of approximately 38%, and memory usage decreased by approximately 35%. Figure 8 As shown, the generated model accurately presents the texture of the rusted car body, the mottled paint, and the emerald green color and natural growth form of the vine leaves, giving the overall scene a high degree of realism.

[0144] Comparative experiments show that the method of this invention achieves significant performance improvements in the four application scenarios compared to traditional global self-attention methods. Specifically, peak memory usage is reduced by an average of approximately 28%, enabling this method to run stably on mainstream consumer-grade GPUs (8GB-12GB of VRAM), while traditional methods typically require more than 16GB of VRAM; training time is shortened by 20% to 30%; and the peak number of Gaussian bodies during training is reduced by 30% to 40%. In terms of generation quality, using CLIP Score to evaluate text-image semantic consistency, the method of this invention improves by approximately 8% compared to the baseline method; using Chamfer Distance to evaluate geometric accuracy, it improves by approximately 10% to 15%; and using PSNR to evaluate rendering quality, it improves by approximately 3% to 5%, indicating that this invention maintains or improves generation quality while reducing resource consumption.

[0145] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A low-memory text-to-3D model generation method based on 3D Gaussian, characterized in that, include: S100: Receive text input, extract text semantic features through a pre-trained text encoder, generate an initial three-dimensional point cloud from the text input three-dimensional point cloud diffusion model, and initialize a three-dimensional Gaussian volume set based on the initial three-dimensional point cloud. S200: The position information and time step information of the three-dimensional Gaussian volume set are fused to generate a Gaussian volume feature sequence; S300: Alternately perform local attention calculation and global attention calculation on the Gaussian feature sequence, and output geometric prediction; S400: Render the geometric prediction as a two-dimensional image, calculate the fractional distillation sampling loss using a pre-trained two-dimensional image diffusion model, and update the geometric parameters of the three-dimensional Gaussian body set by combining the three-dimensional point cloud diffusion loss. S500: Calculate the loss gradient norm at each Gaussian body position in the set of three-dimensional Gaussian bodies, and generate binary labels for the corresponding Gaussian bodies based on the comparison between the loss gradient norm and a preset threshold. S600: Construct an opacity regularization loss based on the binary labels, constraining the opacity of the Gaussian body with a label of 1 to tend to 1, and constraining the opacity of the Gaussian body with a label of 0 to tend to 0; S700: Combine the fractional distillation sampling loss, image mean square error loss and opacity regularization loss to iteratively optimize the set of three-dimensional Gaussian volumes, remove Gaussian volumes with opacity below the pruning threshold, and output a three-dimensional Gaussian model. The steps of alternately performing local attention calculation and global attention calculation on the Gaussian feature sequence include: The Gaussian feature sequence is divided into multiple non-overlapping local regions based on the current total number of Gaussian bodies, and multi-head attention calculation is performed independently in each local region. A semantic nearest neighbor graph is constructed in the feature space for the Gaussian body feature sequence using the k-nearest neighbor algorithm, and feature aggregation is performed on the semantic nearest neighbor graph; The multi-head attention computation within the local region is alternately and sequentially performed in series with feature aggregation based on semantic nearest neighbor graphs; Feature aggregation of the semantic nearest neighbor graph includes: Linear projection is performed on the Gaussian feature sequence to generate query vector, key vector, and value vector; The k-nearest neighbor algorithm is used to construct a semantic nearest neighbor graph in the feature space for the value vector and to perform the first round of feature aggregation; Based on the features obtained from the first round of aggregation, the nearest neighbor relationships are recalculated for the second round of feature aggregation to obtain enhanced features. The enhanced features are then subjected to multi-head attention computation with the query vector and key vector.

2. The method for generating 3D models from low-memory text based on 3D Gaussian as described in claim 1, characterized in that, The fusion of the position information and time step information of the three-dimensional Gaussian volume set includes: The positional information of the three-dimensional Gaussian volume set is linearly projected; The time step information is embedded and encoded; The projected position information is fused with the embedded encoded time step information and then normalized to obtain the Gaussian body feature sequence.

3. The method for generating 3D models from low-memory text based on 3D Gaussian as described in claim 1, characterized in that, The step of dividing the Gaussian body feature sequence into multiple non-overlapping local regions based on the current total number of Gaussian bodies includes: Divide the total number of Gaussian volumes by the threshold for the number of Gaussian volumes in a single local region to obtain the number of local regions. The Gaussian body feature sequence is divided evenly according to the number of local regions.

4. The method for generating 3D models from low-memory text based on 3D Gaussian as described in claim 1, characterized in that, In the second round of feature aggregation, the number of neighbors when calculating nearest neighbor relationships is less than the number of neighbors in the first round of feature aggregation.

5. The method for generating 3D models from low-memory text based on 3D Gaussian as described in claim 1, characterized in that, The opacity regularization loss constructed based on the binary labels includes: For each Gaussian body in the set of three-dimensional Gaussian bodies, calculate the binary cross-entropy loss between its binary label and its opacity; The opacity regularization loss is obtained by averaging the binary cross-entropy loss over all Gaussian bodies.

6. The method for generating 3D models from low-memory text based on 3D Gaussian as described in claim 1, characterized in that, Iterative optimization of the 3D Gaussian volume set by combining the fractional distillation sampling loss and the opacity regularization loss includes: Weighting coefficients are set for the fractional distillation sampling loss, the image mean square error loss, and the opacity regularization loss, respectively; The total loss is obtained by multiplying each loss by its corresponding weight coefficient and then summing the results. The position, scaling, rotation, opacity, and color parameters of the three-dimensional Gaussian volume set are updated by backpropagation based on the total loss.

7. A low-memory text generation system for 3D models based on 3D Gaussian, characterized in that, The system is used to execute the method for generating 3D models from low-memory text based on 3D Gaussian as described in any one of claims 1-6, the system comprising: An initialization module is used to receive text input, extract text semantic features through a pre-trained text encoder, generate an initial three-dimensional point cloud from the text input three-dimensional point cloud diffusion model, and initialize a three-dimensional Gaussian volume set based on the initial three-dimensional point cloud. The feature fusion module is used to fuse the position information and time step information of the three-dimensional Gaussian volume set to generate a Gaussian volume feature sequence; An alternating attention module is used to alternately perform local attention calculations and global attention calculations on the Gaussian feature sequence and output geometric predictions. The geometry optimization module is used to render the geometric prediction as a two-dimensional image, calculate the fractional distillation sampling loss using a pre-trained two-dimensional image diffusion model, and update the geometric parameters of the three-dimensional Gaussian body set by combining the three-dimensional point cloud diffusion loss. The label generation module is used to calculate the loss gradient norm of each Gaussian body position in the three-dimensional Gaussian body set, and generate binary labels for the corresponding Gaussian bodies based on the comparison between the loss gradient norm and a preset threshold. The regularization module is used to construct an opacity regularization loss based on the binary labels, constraining the opacity of the Gaussian body with label 1 to tend to 1, and constraining the opacity of the Gaussian body with label 0 to tend to 0. The iterative optimization module is used to combine the fractional distillation sampling loss, the image mean square error loss, and the opacity regularization loss to iteratively optimize the three-dimensional Gaussian volume set, remove Gaussian volumes with opacity below the pruning threshold, and output a three-dimensional Gaussian model.

Images