A Text-Driven 3D Gaussian Scene Stylization Method

Abstract

The present invention discloses a three-dimensional Gaussian scene stylization method driven by this article, which belongs to the field of computer vision. The method is based on deep learning and is divided into two parts: scene reconstruction and stylization. In the reconstruction part, the RGB image sequence of the scene is collected and the reconstruction is completed using the three-dimensional Gaussian splashing technology; in the stylization part, a text prompt of the target style is given, and according to the style transfer strategy, the text-image multimodal model technology is used to gradually integrate the potential features of the style text prompt into the scene, and the Gaussian field representing the scene is iteratively optimized, and finally a three-dimensional scene Gaussian field that accurately represents the appearance and geometry of the scene and meets the requirements of the style text is obtained. The present invention only needs to input a simple text prompt to achieve the expected scene stylization effect, has strong versatility, and provides a feasible way to build customized scenes in the fields of virtual reality, games, etc.

Description

Technical Field

[0001] This invention relates to the fields of 3D reconstruction, image processing, and natural language processing, specifically a text-driven method for implementing 3D Gaussian scene stylization. Background Technology

[0002] In numerous fields such as virtual reality, augmented reality, game design, and film and television production, the stylization of 3D scenes plays a crucial role. By converting 3D scenes into specific styles, visual effects can be enhanced, providing users with diverse and customized options, while also offering artists and creators richer means of expression.

[0003] Traditional 3D scene modeling and stylization typically utilizes 3D modeling software and rendering tools such as Maya and Blender, requiring manual operation by experienced technicians—a tedious and time-consuming process. Existing methods for reconstructing 3D scenes using deep learning primarily include neural radiation fields and 3D Gaussian splashing, which learn the 3D representation of the scene from RGB image sequences. Among these, 3D Gaussian splashing offers significant advantages in training time and rendering speed. Existing scene stylization methods include those using images and text as style references. The image-based method requires providing existing images with rich color and texture information, learning the style features of the reference image through a deep neural network, and then transferring these features to the 3D scene using existing image style transfer techniques. The text-based method requires user input of natural language prompts. Leveraging the capabilities of multimodal deep learning models, it aligns more intuitive and abstract textual information with the feature information of the 3D scene in a joint space, performing the required stylization processing on the 3D scene model. The advantage of this method is that it eliminates the need for existing images, allowing direct processing of the 3D scene using natural language prompts, lowering the technical barrier for users and simplifying the creation process. At the same time, compared with using images as a priori reference, using text as a reference can achieve richer and more diverse effects.

[0004] Currently, there is no technology that uses text prompts to directly stylize scenes represented by 3D Gaussian fields. Therefore, if simple natural language input could be used as style prompts, while 3D Gaussian fields could be used as the scene representation, it would be of great significance in practical applications to quickly and conveniently meet the needs of scene-level customized style processing, thereby improving the efficiency and flexibility of scene processing. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies. Addressing the need for stylizing scenes using natural language as a stylization reference, this invention designs a deep learning-based method that reconstructs scenes using 3D Gaussian splashing technology and stylizes the scenes based on text prompts. Compared to existing methods that use images as stylization references, this invention, by inputting target style text prompts, is simpler, more direct, more diverse, and more user-friendly. Furthermore, the method proposed in this invention can achieve scene stylization results comparable in quality to existing image-based methods.

[0006] The technical solution of this invention is as follows: This invention first provides a text-driven 3D Gaussian scene stylization method, including the following steps:

[0007] (1) Scene reconstruction: RGB image sequences of the scene are acquired using an RGB image sensor and used as input to the scene reconstruction neural network. The scene reconstruction neural network uses three-dimensional Gaussian splashing technology to reconstruct the input RGB image sequence into a three-dimensional scene. The three-dimensional scene is represented by a Gaussian field composed of Gaussian ellipsoids with various parameters. Finally, the network outputs a three-dimensional Gaussian field that represents the scene.

[0008] (2) Scene reconstruction post-processing: Gaussian filtering strategy is used to trim the Gaussian ellipsoid in the Gaussian field. On this basis, the scene Gaussian field is further optimized to finally obtain the post-processed three-dimensional Gaussian field representing the original scene.

[0009] (3) Scene stylization: Input the text prompt of the target style, and use the text prompt and the three-dimensional Gaussian field representing the original scene obtained in step (2) as the input of the scene stylization neural network; through the scene stylization neural network, the semantic information of the scene and the style text are aligned, and the parameter values ​​of the Gaussian ellipsoid in the Gaussian field are gradually optimized, and finally a Gaussian field of a stylized three-dimensional scene is obtained.

[0010] As a preferred embodiment of the present invention, step (1) is specifically processed as follows:

[0011] 1.1) Input a sequence of RGB images of the scene and use motion structures to recover a sparse representation of the scene;

[0012] 1.2) Convert the sparse point cloud representation of the scene into the initial distribution state of a three-dimensional Gaussian field;

[0013] 1.3) Use differentiable Gaussian rasterization to render an image of a Gaussian field at a random viewpoint;

[0014] 1.4) Calculate the loss error based on the difference between the rendered image and the real image from the same viewpoint. The loss error includes L1 loss and structural similarity loss.

[0015] 1.5) Based on the loss error, backpropagation is used to adjust the parameter values ​​of the Gaussian ellipsoid, which include position, color, density, scale, and rotation direction;

[0016] 1.6) For each training round set at each interval, an automatic densification and pruning strategy is applied to the overall Gaussian field to dynamically adjust the Gaussian distribution;

[0017] 1.7) The Gaussian field dynamically adjusted in 1.6) is returned to 1.3) for rasterization operation. Steps 1.3) to 1.6) are repeated continuously until the target training rounds are reached and the loss function converges, and finally the Gaussian field representing the three-dimensional scene is obtained.

[0018] As a preferred embodiment of the present invention, step (2) sets a filtering threshold range for Gaussian ellipsoid parameter values ​​according to the Gaussian filtering strategy, trims Gaussian ellipsoids that exceed or fall below the filtering threshold range, and removes geometric noise from the scene.

[0019] Under the supervision of the loss function, the parameter values ​​of the filtered Gaussian ellipsoid are fine-tuned to fix the problems of inconsistent scene views and reduced rendering realism caused by the Gaussian filtering strategy, and the fine-tuned Gaussian field is output.

[0020] As a preferred embodiment of the present invention, in step (3), the scene stylization neural network includes a stylization module, an original content preservation module, and a Gaussian regularization module; the stylization module uses a text-image multimodal model technology based on contrastive learning to align semantic information between the image rendered by the Gaussian field and the stylized text in the joint encoding space, thereby controlling the degree of stylization of the scene; the original content preservation module uses a deep convolutional neural network to extract multi-dimensional features of the rendered image of the scene and the real image of the original scene during the stylization training process, and supervises the degree of information preservation of the original scene through feature error; the Gaussian regularization module constrains the change of the scale and density attributes of Gaussian before and after training through a loss function, controls the rapid change of Gaussian, and reduces the geometric noise generated in the scene after stylization.

[0021] Step (3) specifically includes:

[0022] 3.1) Input the target style text prompts and the reconstructed 3D Gaussian field;

[0023] 3.2) Randomly select a viewpoint and use differentiable Gaussian rasterization to obtain the rendered image of the current Gaussian field;

[0024] 3.3) Calculate the stylization loss error based on the rendered image and the text cues of the target style;

[0025] 3.4) The rendered image and the real image of the reconstructed original scene are fed into the original content preservation module to calculate the original content preservation loss error;

[0026] 3.5) Input the post-processed original scene 3D Gaussian field from 3.1) and the Gaussian field obtained from the current step into the Gaussian regularization module to calculate the Gaussian regularization loss error.

[0027] 3.6) Construct a total loss error using the three loss errors mentioned above. Based on the total loss error, backpropagate to adjust the parameter values ​​of the Gaussian ellipsoid. The parameter values ​​include position, color, density, scale, and rotation direction.

[0028] 3.7) The Gaussian field adjusted by backpropagation in 3.6) is returned to 3.2) for rasterization operation. Steps 3.2)-3.6) are repeated until the target number of training rounds is reached and the loss function converges, and finally the Gaussian field of the stylized 3D scene is obtained.

[0029] This invention utilizes text-driven prompts to reconstruct and stylize scenes. Compared to existing image-reference-based stylization methods mentioned in the background section, this method offers a simpler, more direct, diverse, and user-friendly approach to obtaining style references. The proposed method achieves high-quality stylization results comparable to image references, possesses strong versatility, and provides a new approach for constructing customized scenes. This method can be widely applied in fields such as virtual reality, augmented reality, game design, and film production. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating the present invention;

[0031] Figure 2 This is a schematic diagram of the overall network framework of the scenario reconstruction network structure of the present invention;

[0032] Figure 3 This is a schematic diagram of the overall network framework of the scene stylization network structure of the present invention;

[0033] Figure 4 This is a schematic diagram of the stylization module of the present invention;

[0034] Figure 5 This is a rendering of the scene reconstruction effect of the present invention;

[0035] Figure 6 This is a stylized rendering of the scene from the present invention. Detailed Implementation

[0036] The representative embodiments shown in the accompanying drawings will now be further refined. It should be understood that the following description is not intended to limit the embodiments to a single preferred embodiment. Rather, it is intended to cover alternatives, modifications, and equivalents that may be included within the substance and scope of the embodiments defined by the appended claims.

[0037] like Figure 1 As shown in the flowchart, the text-driven 3D Gaussian scene stylization method of the present invention includes the following steps:

[0038] (1) Scene reconstruction: RGB image sequences of the scene are acquired using an RGB image sensor and used as input to the scene reconstruction neural network. The scene reconstruction neural network uses three-dimensional Gaussian splashing technology to reconstruct the input RGB image sequence into a three-dimensional scene. The three-dimensional scene is represented by a Gaussian field composed of Gaussian ellipsoids with various parameters. Finally, the network outputs a three-dimensional Gaussian field that represents the scene.

[0039] (2) Scene reconstruction post-processing: Gaussian filtering strategy is used to trim the Gaussian ellipsoid in the Gaussian field. On this basis, the scene Gaussian field is further optimized to finally obtain the post-processed three-dimensional Gaussian field representing the original scene.

[0040] (3) Scene stylization: Input the text prompt of the target style, and use the text prompt and the three-dimensional Gaussian field representing the original scene obtained in step (2) as the input of the scene stylization neural network; through the scene stylization neural network, the semantic information of the scene and the style text are aligned, and the parameter values ​​of the Gaussian ellipsoid in the Gaussian field are gradually optimized, and finally a Gaussian field of a stylized three-dimensional scene is obtained.

[0041] A schematic diagram of the overall network framework (scene reconstruction neural network) of the scene reconstruction part of this invention is shown below. Figure 2As shown, the scene reconstruction neural network uses 3D Gaussian splashing technology to reconstruct the scene in 3D. It trains the parameters of the Gaussian ellipsoid within the Gaussian field to obtain the Gaussian field representing the original scene. The specific processing flow is as follows: Inputting the RGB image sequence of the scene, firstly, a sparse representation of the scene is recovered using motion structure analysis, and the sparse point cloud representation of the scene is converted into the initial distribution state of the 3D Gaussian field; then, differentiable Gaussian rasterization is used to render the image of the Gaussian field at a random viewpoint, and the difference between the rendered image and the real image is used to calculate the loss error, which includes L1 loss and summation. The Structural Similarity (SSIM) loss is used, with error weights typically set to 0.8 and 0.2. The position, color, density, scale, and rotation direction of the Gaussian ellipsoid are updated based on backpropagation of the error. The learning rate for these parameters is set empirically. At regular training epochs, an automatic densification and pruning strategy is applied to the overall Gaussian field to dynamically adjust the Gaussian distribution. This training process is repeated until the target number of reconstruction training epochs is reached and the loss function converges. The number of reconstruction training epochs is typically set to 30,000. The final output is a Gaussian field representing the original 3D scene.

[0042] In a specific embodiment of the present invention, the Gaussian field post-processing involves the following steps: In the Gaussian filtering module, for Gaussian ellipsoids that are too large in scale or too small in density, a filtering threshold range is set empirically (for example, the attribute values ​​corresponding to the top 1% of Gaussian scale and the bottom 1% of Gaussian density can be set as the filtering threshold range), and some Gaussian ellipsoids are cropped and removed; In the Gaussian optimization module, a viewpoint is randomly selected, and rasterization is used to obtain a rendered image of the filtered Gaussian field. The loss error function between the rendered image and the real image is calculated. The loss error includes L1 loss and structural similarity (SSIM) loss. The loss error weights are generally set to 0.8 and 0.2, respectively. The position, color, density, scale, rotation direction, and other parameter values ​​of the existing Gaussian ellipsoids are updated based on backpropagation of the error. The learning rate of the parameter values ​​is set empirically, but it is one to two orders of magnitude smaller than the parameter learning rate set during reconstruction. The post-reconstruction processing training rounds are generally set to 7000 times, and finally, a post-processed three-dimensional Gaussian field representing the original scene is obtained.

[0043] A schematic diagram of the overall network framework for the scene stylization part of this invention is shown below. Figure 3As shown, the stylization neural network includes a stylization module, a content preservation module, and a Gaussian regularization module. The stylization module first renders an image and styled text from a Gaussian field using randomly selected viewpoints. These are simultaneously fed into the image encoder and text encoder of a pre-trained text-image multimodal model to obtain the encoded features of the image and text in the same feature space. The relative cosine similarity of the two encoded features is measured for stylization trajectory control. Then, based on a contrastive learning strategy, positive and negative sample text data are constructed, and a random cropping strategy is used to obtain local images. The overall and local images are then compared using the positive and negative sample data to enhance the stylization level, ultimately resulting in a stylized image of the rendered image under the styled text cues.

[0044] In a specific embodiment of the present invention, the stylized neural network is trained to optimize the Gaussian field. The specific processing flow is as follows: Input a text prompt of the target style (for example, if the expected target style for converting the original scene is an abstract and concise line art style, then input the corresponding text prompt: Line). The process involves: first, randomly selecting a viewpoint and using differentiable Gaussian rasterization to obtain a rendered image of the current Gaussian field; second, feeding the rendered image and the target style text prompt into the stylization module to calculate the stylization loss error; then feeding the rendered image and the real image of the original scene after reconstruction into the original content preservation module to calculate the content preservation loss error; next, feeding the 3D Gaussian field into the Gaussian regularization module to calculate the Gaussian regularization loss error; the loss error weights for the three parts are generally set to 1.0, 0.01, and 0.02, respectively. Finally, based on the total loss error, backpropagation is used to adjust the position, color, density, scale, and rotation direction of the Gaussian ellipsoid. The learning rate for the parameters is set empirically, but is one to two orders of magnitude smaller than the learning rate set during reconstruction. The training operation described above is repeated until the target stylization training rounds are reached and the total loss function converges. The stylization training rounds are generally set to 1000 times, and the final output is the stylized 3D scene's Gaussian field.

[0045] Preferably, in a specific embodiment of the present invention, the stylization module first renders an image and stylized text by randomly selecting a viewpoint from a Gaussian field, and feeds them into a pre-trained CLIP text-image multimodal model. ViT is used as the image feature encoder, and BERT is used as the text feature encoder to obtain the encoded features of the image and text in the same feature space. The relative CLIP distance loss between the two encoded features is measured for stylization trajectory control. Then, based on a contrastive learning strategy, positive and negative sample text data are constructed. For each text, approximately 20 similar positive samples are manually labeled, and all other texts are considered as such. Negative samples of the text are used, and local images are obtained by random cropping using image enhancement strategies. For each local image, eight positive and negative samples are randomly selected to calculate the contrast learning loss error between the local image and the target style text. For the overall image, all positive and negative samples are used to calculate the contrast learning loss error between the local image and the target style text. The image stylization is enhanced by contrast learning using positive and negative sample data. The weights of the loss errors of the three parts are generally set to 0.5, 0.05, and 0.1. The weighted total stylization loss error is used as part of the overall loss error of the stylization network and is used for backpropagation to optimize the Gaussian field parameters.

[0046] To further illustrate the present invention, the following describes an embodiment implemented according to the complete method of the present invention, the implementation process of which is as follows:

[0047] Using LLFF as a known dataset as an example, this paper describes the idea and specific implementation steps of a text-driven 3D Gaussian scene stylization method. The RGB images in the example are from the known LLFF dataset, and the target style text LineArt is from user input.

[0048] Step 1: Using the known LLFF dataset partitioning, select the Trex scene. The training set contains 34 RGB images. Then, proceed with Step 2 using the RGB images provided in the training set.

[0049] Step Two: For the RGB ground truth images in the training set described in Step One, use motion structure reconstruction to extract the sparse point cloud of the Trex scene, and convert it into the initial position and color of the Gaussian ellipsoid in a Gaussian field. Input the initialized Gaussian field into the network structure of the reconstruction part. The overall network framework of the scene reconstruction part of this invention is as follows: Figure 2 As shown. The initialized Gaussian field is used to render an image through rasterization. The loss error is then calculated by comparing it with the real image. The calculation method is as follows:

[0050]

[0051] Among them, I render Represents the rendered image, I gt Represents a real image. Let λ1 and λ2 represent the L1 loss and structural similarity (SSIM) loss, respectively; λ1 (set to 0.8 in this example) and λ2 (set to 0.2 in this example) are the weighting coefficients of the loss error and the structural loss error, respectively, to obtain the reconstruction loss error.

[0052] The training of the neural network structure is based on the reconstruction loss error. The Gaussian parameters are iteratively updated using the gradient backpropagation method and accelerated using a GPU. Training stops when the total loss is reduced to within a set threshold or the number of network iterations meets the requirements, and the Gaussian field of the Trex scene is output.

[0053] Step 3: Input the Gaussian field obtained in Step 2 into the post-reconstruction processing section. After filtering out the Gaussian portion, the reconstruction loss error described above is applied again. Supervise the fine-tuning training of the Gaussian field. The final result of the Gaussian field after training, filtering, and optimization using the above method is as follows: Figure 5 As shown in the figure, the reconstructed image of the present invention is basically the same as the real RGB of the scene, and the reconstruction rendering effect of the present invention restores the information of the real scene very well.

[0054] Step 4: Input the Gaussian field obtained in Step 3 and the target style text input by the user into the network structure of the stylization part. The overall network structure of the scene stylization part of this invention is as follows: Figure 3 As shown. The total loss error of the stylization part is calculated as follows:

[0055]

[0056] in, Represents stylization loss error, This represents the loss of original content retention error. λ represents the Gaussian regularization loss error. sty (In this example, it is set to 1.0), λ con (In this example, it is set to 0.01) and λ reg (In this example, it is set to 0.02) is the weighting coefficient for the three errors.

[0057] To address stylization loss errors, a rendered image is obtained by rasterizing a Gaussian field. The rendered image and the target style text prompt are then fed into the stylization module. The stylization module framework of this invention is as follows: Figure 4 As shown. The stylization loss error is calculated as follows:

[0058]

[0059] Among them, I render T represents the rendered image. tgt Represents the target style text. Represents the relative CLIP distance loss. and λ represents the relative loss between the overall rendered image and the locally rendered image and the positive and negative samples, respectively. clip (In this example, it is set to 0.5), λ global (In this example, it is set to 0.1) and λ local (In this example, it is set to 0.05) is the weighting coefficient for the three errors.

[0060] Regarding the original content retention error, the rendered image and the original scene real image mentioned above are sent to the original content retention error module. The original content retention loss error is calculated as follows:

[0061]

[0062] Among them, I render Represents the rendered image, I gt Represents the real image, and ε(·) represents the VGG network for feature extraction. This represents L2 loss.

[0063] For the Gaussian regularization error, the aforementioned Gaussian field is fed into the Gaussian regularization module, and the Gaussian regularization loss error is calculated as follows:

[0064]

[0065] in, Represents Gaussian-scale variation loss. λ represents the loss due to Gaussian density variation. sca In this example, the values ​​are set to 0.05 and λ. opa In this example, the weighting factor is set to 0.05, which represents the weighting coefficients for the two errors.

[0066] Based on the total loss error of the stylized network, the Gaussian parameters are optimized using gradient backpropagation and accelerated using a GPU. Training stops when the total loss is reduced to within a set threshold or the number of network iterations meets the requirements, and the output is a stylized Gaussian field of the Trex scene after the input style text prompt line art.

[0067] The final stylization effect trained using the above method is as follows: Figure 6 As shown in the figure, the stylization effect of this invention extracts the abstract semantic information suggested in this paper and performs style transfer on the original scene. Comparing the stylized effect image with the original scene image, it can be seen that the method of this invention, while transferring the style, also preserves the appearance information and geometric structure of the original scene very well.

[0068] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes to the present invention within the spirit and scope of the claims fall within the protection scope of the present invention.

Claims

1. A text-driven 3D Gaussian scene stylization method, characterized in that, The method includes the following steps: (1) Scene reconstruction: RGB image sequences of the scene are acquired using an RGB image sensor and used as input to the scene reconstruction neural network. The scene reconstruction neural network uses three-dimensional Gaussian splashing technology to reconstruct the input RGB image sequence into a three-dimensional scene. The three-dimensional scene is represented by a Gaussian field composed of Gaussian ellipsoids with various parameters. Finally, the network outputs a three-dimensional Gaussian field that represents the scene. (2) Scene reconstruction post-processing: Gaussian filtering strategy is used to trim the Gaussian ellipsoid in the Gaussian field. On this basis, the scene Gaussian field is further optimized to finally obtain the post-processed three-dimensional Gaussian field representing the original scene. (3) Scene stylization: Input the text prompt of the target style, and use the text prompt and the three-dimensional Gaussian field representing the original scene obtained in step (2) as the input of the scene stylization neural network; through the scene stylization neural network, the semantic information of the scene and the style text are aligned, and the parameter values ​​of the Gaussian ellipsoid in the Gaussian field are gradually optimized, and finally a Gaussian field of a stylized three-dimensional scene is obtained. The scene stylization neural network comprises a stylization module, an original content preservation module, and a Gaussian regularization module. The stylization module utilizes a contrastive learning-based text-image multimodal model to align semantic information between the Gaussian field-rendered image and the stylized text in a joint encoding space, controlling the degree of scene stylization. The original content preservation module uses a deep convolutional neural network to extract multi-dimensional features from the rendered image of the scene and the original real image of the scene during stylization training, and supervises the degree of information preservation of the original scene through feature error. The Gaussian regularization module constrains the changes in the scale and density attributes of Gaussians before and after training through a loss function, controlling the abrupt changes in Gaussians and reducing geometric noise generated in the stylized scene.

2. The text-driven 3D Gaussian scene stylization method according to claim 1, characterized in that: The specific processing of step (1) is as follows: 1.1) Input a sequence of RGB images of the scene and use motion structures to recover a sparse representation of the scene; 1.2) Convert the sparse point cloud representation of the scene into the initial distribution state of a three-dimensional Gaussian field; 1.3) Use differentiable Gaussian rasterization to render an image of a Gaussian field at a random viewpoint; 1.4) Calculate the loss error based on the difference between the rendered image and the real image from the same viewpoint. The loss error includes L1 loss and structural similarity loss. 1.5) Based on the loss error, backpropagation is used to adjust the parameter values ​​of the Gaussian ellipsoid, which include position, color, density, scale, and rotation direction; 1.6) For each training round set at each interval, an automatic densification and pruning strategy is applied to the overall Gaussian field to dynamically adjust the Gaussian distribution; 1.7) The Gaussian field dynamically adjusted in 1.6) is returned to 1.3) for rasterization operation. Steps 1.3) to 1.6) are repeated continuously until the target training rounds are reached and the loss function converges, and finally the three-dimensional Gaussian field representing the scene is obtained.

3. The method for text-driven 3D Gaussian scene stylization according to claim 1, characterized in that: Step (2) is as follows: Based on the Gaussian filtering strategy, a filtering threshold range for Gaussian ellipsoid parameter values ​​is set, and Gaussian ellipsoids that exceed or fall below the filtering threshold range are trimmed to remove geometric noise from the scene. Under the supervision of the loss function, the parameter values ​​of the filtered Gaussian ellipsoid are fine-tuned to fix the problems of inconsistent scene views and reduced rendering realism caused by the Gaussian filtering strategy, and the fine-tuned Gaussian field is output.

4. The method for text-driven 3D Gaussian scene stylization according to claim 3, characterized in that: The post-processing of scene reconstruction in step (2) is performed by a Gaussian filtering module and a Gaussian optimization module; In the Gaussian filtering module, a filtering threshold range is set to clip and remove Gaussian ellipsoids that are too large in scale or too small in density. In the Gaussian optimization module, a random viewpoint is selected, and rasterization is used to obtain a rendered image of the filtered Gaussian field. The loss error function between the Gaussian field and the real image is calculated. The position, color, density, scale, and rotation direction of the existing Gaussian ellipsoid are updated according to the backpropagation of the error. The iteration is repeated until the target training round is reached and the loss function converges, and finally a post-processed 3D Gaussian field representing the original scene is obtained.

5. The method for text-driven 3D Gaussian scene stylization according to claim 1, characterized in that: Step (3) specifically includes: 3.1) Input the target style text prompts and the reconstructed 3D Gaussian field; 3.2) Randomly select a viewpoint and use differentiable Gaussian rasterization to obtain the rendered image of the current Gaussian field; 3.3) Calculate the stylization loss error based on the rendered image and the text cues of the target style; 3.4) The rendered image and the real image of the reconstructed original scene are fed into the original content preservation module to calculate the original content preservation loss error; 3.5) Input the post-processed original scene 3D Gaussian field from 3.1) and the Gaussian field obtained from the current step into the Gaussian regularization module to calculate the Gaussian regularization loss error. 3.6) Construct a total loss error using the three loss errors mentioned above. Based on the total loss error, backpropagate to adjust the parameter values ​​of the Gaussian ellipsoid. The parameter values ​​include position, color, density, scale, and rotation direction. 3.7) The Gaussian field adjusted by backpropagation in 3.6) is returned to 3.2) for rasterization operation. Steps 3.2)-3.6) are repeated until the target number of training rounds is reached and the loss function converges, and finally the Gaussian field of the stylized 3D scene is obtained.

6. The method for text-driven 3D Gaussian scene stylization according to claim 5, characterized in that: In step 3.6), the total loss error is: in, Represents stylization loss error, This represents the loss of original content retention error. λ represents the Gaussian regularization loss error. sty , λ con and λ reg These are the weighting coefficients for the three errors; The stylization loss error is: Among them, I render T represents the rendered image. tgt Represents the target style text. Represents the relative CLIP distance loss. λ represents the relative loss between the rendered image and the positive and negative samples. clip and λ' con These are the weighting coefficients for the two errors; The original content retention loss error is: Among them, I render Represents the rendered image, I gt Represents the real image, and ε(·) represents the VGG network for feature extraction. Represents L2 loss; The Gaussian regularization loss error is: in, Represents Gaussian-scale variation loss. λ represents the loss due to Gaussian density variation. sca and λ opa These are the weighting coefficients for the two errors.

7. The method for text-driven 3D Gaussian scene stylization according to claim 5, characterized in that: The stylization module first renders an image and style text by randomly selecting a viewpoint from a Gaussian field. These are then simultaneously fed into the image encoder and text encoder of a pre-trained text-image multimodal model to obtain the encoded features of the image and text in the same feature space. The relative cosine similarity of the two encoded features is measured for stylization trajectory control. Then, based on a contrastive learning strategy, positive and negative sample text data are constructed, and a random cropping strategy is used to obtain local images. The overall and local images are compared and learned using positive and negative sample data to enhance the stylization degree. Finally, a stylized image of the rendered image under the style text prompt is obtained.

Images