Three-dimensional Gaussian scene stylization method based on text driving

By employing deep learning-based 3D Gaussian splashing technology and a text-driven stylization method, this approach addresses the shortcomings of existing technologies in natural language stylization scenarios, achieving user-friendly, efficient, and diverse 3D scene stylization processing, and obtaining high-quality stylization effects.

CN119006760AActive Publication Date: 2024-11-22ZHEJIANG UNIV

Patent Information

Application Number
CN202411099210.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-22
Estimated Expiration
2044-08-12

AI Technical Summary

Technical Problem

Existing technologies lack methods for stylizing 3D Gaussian scenes using natural language as a stylization reference, resulting in an unfriendly user experience and low efficiency, and making it impossible to achieve fast and convenient customized style processing.

Method used

A deep learning-based approach is adopted to reconstruct the scene using 3D Gaussian splashing technology and to perform stylization processing using text prompts. The process includes scene reconstruction, post-processing, and stylization steps. A Gaussian filtering strategy and a multimodal deep learning model are used to align semantic information and optimize the Gaussian ellipsoid parameters.

Benefits of technology

It enables direct stylization of 3D scenes through natural language input, simplifying user operations, improving processing efficiency and flexibility, and achieving high-quality stylization effects comparable to image references.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006760A_ABST
    Figure CN119006760A_ABST
Patent Text Reader

Abstract

The invention discloses a three-dimensional Gaussian scene stylization method based on text driving, and belongs to the field of computer vision. The method is divided into a scene reconstruction part and a stylization part based on deep learning. In the reconstruction part, an RGB image sequence of a scene is collected, and reconstruction is completed by using a three-dimensional Gaussian splashing technology; in the stylization part, a text prompt of a target style is given, according to a style migration strategy, a text-image multi-modal model technology is utilized, potential features of the text prompt of the style are gradually fused into a scene, iterative optimization is performed on a Gaussian field representing the scene, and a target style is obtained. And finally obtaining a three-dimensional scene Gaussian field which accurately represents the scene appearance and geometry and meets the style text requirement. According to the method, the expected scene stylization effect can be achieved only by inputting simple text prompts, the universality is very high, and a feasible way is provided for constructing customized scenes in the fields of virtual reality, games and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of three-dimensional reconstruction, image processing and natural language processing, and in particular to a method for implementing three-dimensional Gaussian scene stylization based on text driving. Background Art

[0002] In many fields such as virtual reality, augmented reality, game design, and film and television production, 3D scene stylization plays an important role. By converting 3D scenes into specific styles, the visual effects can be enhanced, providing users with diverse and customized choices, while also providing artists and creators with richer means of expression.

[0003] Traditional 3D scene modeling and stylization usually use 3D modeling software and rendering tools such as Maya and Blender, and require manual operation by experienced technicians. The process is cumbersome and time-consuming. Existing methods for reconstructing 3D scenes using deep learning mainly include neural radiance fields and 3D Gaussian splashing, which learn the 3D representation of the scene from the RGB image sequence of the scene. Among them, 3D Gaussian splashing has more significant advantages in training time and rendering rate. Existing scene stylization methods include two methods of using images and text as style references. Among them, the method using images as style references requires providing existing images with rich color and texture information, learning the style features of the reference image through deep neural networks, using existing image style transfer technology, and then transferring these features to the 3D scene. The method using text as style reference requires users to input natural language prompts, and uses the capabilities of multimodal deep learning models to align more intuitive and abstract text information and feature information of the 3D scene in the joint space, and perform the required stylization processing on the 3D scene model. The advantage of this method is that there is no need to provide existing images, and natural language can be used directly to realize the processing of 3D scenes, which reduces the technical threshold for users and simplifies the creation process. At the same time, compared to using images as a priori references, using text as a reference can achieve richer and more diverse effects.

[0004] There is no technology in the prior art that uses text prompt input to directly stylize scenes represented by three-dimensional Gaussian field modeling. Therefore, if we can use simple natural language input as style prompts and use three-dimensional Gaussian as a scene representation form to quickly and conveniently meet the needs of scene-level customized style processing, improve the efficiency and flexibility of scene processing, it will be of great significance in practical applications. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art. Aiming at the need to use natural language as a stylized reference to stylize the scene, a deep learning-based method is designed to reconstruct the scene using a three-dimensional Gaussian splashing technique and stylize the scene according to text prompts. Compared with the existing method of using images as stylized references, the present invention is simpler, more direct, more diverse and more user-friendly by inputting target style text prompts. However, the method proposed by the present invention can obtain a scene stylization effect comparable to the quality of existing image-based methods.

[0006] The technical solution of the present invention is as follows: the present invention first provides a text-driven 3D Gaussian scene stylization method, comprising the following steps:

[0007] (1) Scene reconstruction: Use an RGB image sensor to collect RGB image sequences of the scene as input to the scene reconstruction neural network. The scene reconstruction neural network uses a three-dimensional Gaussian splashing technique to reconstruct the input RGB image sequence into a three-dimensional scene. The three-dimensional scene is represented by a Gaussian field composed of a Gaussian ellipsoid with various parameters. Finally, the network outputs a three-dimensional Gaussian field representing the scene.

[0008] (2) Post-processing of scene reconstruction: The Gaussian ellipsoid in the Gaussian field is trimmed using the Gaussian filtering strategy, and the scene Gaussian field is further optimized on this basis, and finally a three-dimensional Gaussian field representing the original scene is obtained after post-processing;

[0009] (3) Scene stylization: Input the text prompt of the target style, and use the text prompt and the three-dimensional Gaussian field representing the original scene obtained in step (2) as the input of the scene stylization neural network; through the scene stylization neural network, align the semantic information of the scene with the style text, gradually optimize the various parameter values ​​of the Gaussian ellipsoid in the Gaussian field, and finally obtain a Gaussian field of a stylized three-dimensional scene.

[0010] As a preferred embodiment of the present invention, the step (1) is specifically processed as follows:

[0011] 1.1) Input a sequence of RGB images of a scene and use motion structure to recover a sparse representation of the scene;

[0012] 1.2) Convert the sparse point cloud representation of the scene into the initial distribution state of the three-dimensional Gaussian field;

[0013] 1.3) Use differentiable Gaussian rasterization to render the image of the Gaussian field at a random viewing angle;

[0014] 1.4) Calculating the loss error based on the difference between the rendered image and the real image at the same viewing angle, wherein the loss error includes L1 loss and structural similarity loss;

[0015] 1.5) According to the loss error, back propagate to adjust the parameter values ​​of the Gaussian ellipsoid, which include position, color, density, scale and rotation direction;

[0016] 1.6) At each set training round, apply automatic densification and pruning strategies to the overall Gaussian field to dynamically adjust the Gaussian distribution;

[0017] 1.7) Return the Gaussian field dynamically adjusted in 1.6) to 1.3) for rasterization operation, and repeat steps 1.3) to 1.6) until the target training round is reached and the loss function converges, and finally a Gaussian field representing the three-dimensional scene is obtained.

[0018] As a preferred solution of the present invention, the step (2) sets a filtering threshold range for the Gaussian ellipsoid parameter value according to the Gaussian filtering strategy, trims the Gaussian ellipsoid that exceeds or falls below the filtering threshold range, and removes the geometric noise of the scene;

[0019] Under the supervision of the loss function, the parameter values ​​of the filtered Gaussian ellipsoid are fine-tuned to fix the multi-view inconsistency of the scene and the weakening of rendering realism caused by the Gaussian filtering strategy, and the fine-tuned Gaussian field is output.

[0020] As a preferred embodiment of the present invention, in step (3), the scene stylization neural network includes a stylization module, an original content preservation module and a Gaussian regularization module; the stylization module uses a text-image multimodal model technology based on contrastive learning to align semantic information between the image rendered by the Gaussian field and the style text in a joint coding space to control the stylization degree of the scene; the original content preservation module uses a deep convolutional neural network to extract multi-dimensional features of the rendered image of the scene and the real image of the original scene during the stylization training process, and supervises the information preservation degree of the original scene through feature errors; the Gaussian regularization module constrains the change range of the scale and density attributes of the Gaussian before and after training through a loss function, controls the rapid change of the Gaussian, and reduces the geometric noise generated in the stylized scene.

[0021] The step (3) specifically includes:

[0022] 3.1) Input the target style text prompt and the reconstructed 3D Gaussian field;

[0023] 3.2) Randomly select a viewing angle and use differentiable Gaussian rasterization to obtain the rendered image of the current Gaussian field;

[0024] 3.3) Calculate the stylization loss error based on the rendered image and the textual hint of the target style;

[0025] 3.4) Send the rendered image and the real image of the reconstructed original scene to the original content preservation module to calculate the original content preservation loss error;

[0026] 3.5) The post-processed original scene three-dimensional Gaussian field of 3.1) and the Gaussian field obtained by training in the current step are sent to the Gaussian regularization module to calculate the Gaussian regularization loss error;

[0027] 3.6) constructing a total loss error with the above three loss errors, and adjusting the parameter values ​​of the Gaussian ellipsoid by back propagation according to the total loss error, wherein the parameter values ​​include position, color, density, scale and rotation direction;

[0028] 3.7) The Gaussian field adjusted by backpropagation in 3.6) is returned to 3.2) for rasterization operation, and steps 3.2)-3.6) are repeated continuously until the target training round is reached and the loss function converges, and finally the Gaussian field of the stylized three-dimensional scene is obtained.

[0029] The present invention completes the reconstruction and stylization of the scene based on text-driven prompts. Compared with the existing stylization method based on image reference mentioned in the background technology, the method is simpler, more direct, more diverse and more user-friendly in obtaining style references. The method proposed in the present invention can obtain high-quality stylization effects comparable to image references, has strong versatility, and can provide a new way to build customized scenes. The method of the present invention can be widely used in the fields of virtual reality, augmented reality, game design, and film and television production. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the process of the present invention;

[0031] Figure 2 It is a schematic diagram of the overall network framework of the scene reconstruction network structure of the present invention;

[0032] Figure 3 It is a schematic diagram of the overall network framework of the scene stylized network structure of the present invention;

[0033] Figure 4 is a schematic diagram of a stylized module of the present invention;

[0034] Figure 5 It is the effect diagram of the scene reconstruction of the present invention;

[0035] Figure 6 It is a scene stylized effect diagram of the present invention. DETAILED DESCRIPTION

[0036] The representative embodiments shown in the accompanying drawings will now be further refined. It should be understood that the following description is not intended to limit the embodiments to a preferred embodiment. On the contrary, it is intended to cover alternatives, modifications and equivalents that may be included in the essence and scope of the embodiments defined by the appended claims.

[0037] like Figure 1 As shown in the flowchart, the text-driven 3D Gaussian scene stylization method of the present invention comprises the following steps:

[0038] (1) Scene reconstruction: Use an RGB image sensor to collect RGB image sequences of the scene as input to the scene reconstruction neural network. The scene reconstruction neural network uses a three-dimensional Gaussian splashing technique to reconstruct the input RGB image sequence into a three-dimensional scene. The three-dimensional scene is represented by a Gaussian field composed of a Gaussian ellipsoid with various parameters. Finally, the network outputs a three-dimensional Gaussian field representing the scene.

[0039] (2) Post-processing of scene reconstruction: The Gaussian ellipsoid in the Gaussian field is trimmed using the Gaussian filtering strategy, and the scene Gaussian field is further optimized on this basis, and finally a three-dimensional Gaussian field representing the original scene is obtained after post-processing;

[0040] (3) Scene stylization: Input the text prompt of the target style, and use the text prompt and the three-dimensional Gaussian field representing the original scene obtained in step (2) as the input of the scene stylization neural network; through the scene stylization neural network, align the semantic information of the scene with the style text, gradually optimize the various parameter values ​​of the Gaussian ellipsoid in the Gaussian field, and finally obtain a Gaussian field of a stylized three-dimensional scene.

[0041] The schematic diagram of the overall network framework (scene reconstruction neural network) of the scene reconstruction part of the present invention is as follows: Figure 2As shown in the figure, the scene reconstruction neural network uses the three-dimensional Gaussian splashing technology to reconstruct the scene in three dimensions, trains the various parameter values ​​of the Gaussian ellipsoid in the Gaussian field, and obtains the Gaussian field that represents the original scene. The specific processing flow is as follows: the RGB image sequence of the input scene is firstly used to restore the sparse representation of the scene using the motion structure, and the sparse point cloud representation of the scene is converted into the initial distribution state of the three-dimensional Gaussian field; then differentiable Gaussian rasterization is used to render the image of the Gaussian field at a random perspective, and the difference between the rendered image and the real image is used to calculate the loss error, which includes L1 loss and Structural similarity (SSIM) loss, the loss error weight is generally set to 0.8 and 0.2, and the parameter values ​​of the Gaussian ellipsoid such as position, color, density, scale, and rotation direction are updated according to the error back propagation. The learning rate of the parameter value is set according to experience. After each certain training round, the automatic densification and pruning strategy will be applied to the overall Gaussian field to dynamically adjust the Gaussian distribution; finally, the training operation described above is repeated until the target reconstruction training round is reached and the loss function converges. The reconstruction training round is generally set to 30,000 times, and the Gaussian field representing the three-dimensional original scene is finally output.

[0042] In a specific embodiment of the present invention, the Gaussian field post-processing, in the Gaussian filtering module, for Gaussian ellipsoids with too large scale or too small density, sets the filtering threshold range according to experience (for example, the attribute corresponding values ​​of Gaussians with a scale size of the first 1% and a density size of the last 1% can be set as the filtering threshold range), and cuts out part of the Gaussian; in the Gaussian optimization module, randomly selects a viewing angle, rasterizes to obtain a rendered image of the filtered Gaussian field, calculates the loss error function between the rendered image and the real image, the loss error includes L1 loss and structural similarity (SSIM) loss, and the loss error weight is generally set to 0.8 and 0.2. The existing Gaussian ellipsoid's position, color, density, scale, rotation direction and other parameter values ​​are updated according to the error back propagation. The learning rate of the parameter value is set according to experience, but is one to two orders of magnitude smaller than the setting of the parameter learning rate during reconstruction. The number of training rounds for post-reconstruction processing is generally set to 7000 times, and finally a three-dimensional Gaussian field representing the original scene after post-processing is obtained.

[0043] The overall network framework diagram of the scene stylization part of the present invention is as follows: Figure 3As shown, the stylized neural network includes a stylization module, an original content preservation module, and a Gaussian regularization module. The stylized module first randomly selects a perspective from the Gaussian field to render the image and style text, and simultaneously sends them to the image encoder and text encoder of the pre-trained text-image multimodal model to obtain the encoding features of the image and text in the same feature space, and measures the relative cosine similarity of the two encoding features for stylized trajectory control; then, based on the contrastive learning strategy, constructs positive and negative sample text data, and uses the image enhancement strategy to randomly crop to obtain local images, and uses the positive and negative sample data for contrastive learning of the overall and local images to enhance the degree of stylization, and finally obtains a stylized image of the rendered image under the prompt of the style text.

[0044] In a specific embodiment of the present invention, the stylized neural network optimizes and trains the Gaussian field. The specific processing flow is as follows: input the text prompt of the target style (for example, the target style of the original scene to be converted is an abstract and concise line art style, then input the corresponding text prompt: Line Art) and the reconstructed three-dimensional Gaussian field; first randomly select the perspective, use differentiable Gaussian rasterization to obtain the rendered image of the current Gaussian field; secondly, send the rendered image and the text prompt of the target style to the stylization module to calculate the stylization loss error; then send the rendered image and the real image of the original scene after reconstruction to the original content retention module to calculate the content retention loss error; then send the three-dimensional Gaussian field to the Gaussian regularization module to calculate the Gaussian regularization loss error; the loss error weights of the three parts are generally set to 1.0, 0.01 and 0.02, and finally, according to the total loss error, back propagation is used to adjust the parameter values ​​such as the position, color, density, scale, and rotation direction of the Gaussian ellipsoid. The learning rate of the parameter value is set according to experience, but it is one to two orders of magnitude smaller than the setting of the parameter learning rate during reconstruction; repeat the training operation described above until the target stylization training round is reached and the total loss function converges. The stylization training round is generally set to 1000 times, and finally the Gaussian field of the stylized three-dimensional scene is output.

[0045] Preferably, in a specific embodiment of the present invention, the stylization module first renders the image and style text by randomly selecting a perspective from the Gaussian field, sends them to the pre-trained CLIP text-image multimodal model, uses ViT as the image feature encoder, uses BERT as the text feature encoder, obtains the encoding features of the image and text in the same feature space, and measures the relative CLIP distance loss of the two encoding features for stylization trajectory control; then, based on the strategy of contrastive learning, constructs positive and negative sample text data, and for each text, manually annotates about 20 positive samples similar to it, and all other texts are regarded as the positive samples. For the local image, 8 positive and negative samples are randomly selected to calculate the contrast learning loss error between them and the target style text. For the whole image, all positive and negative samples are used to calculate the contrast learning loss error between them and the target style text. The positive and negative sample data are used to enhance the local and overall image stylization degree through contrast learning. The loss error weights of the three parts are generally set to 0.5, 0.05 and 0.1. The weighted total loss error of the stylization is used as part of the overall loss error of the stylization network for back-propagation optimization of Gaussian field parameters.

[0046] To further illustrate the present invention, the following is an embodiment of the present invention according to the complete method, and the implementation process is as follows:

[0047] Taking LLFF as a known data set as an example, the idea and specific implementation steps of the text-driven 3D Gaussian scene stylization method are described. The RGB image of the embodiment comes from the LLFF known data set, and the target style text LineArt of the embodiment comes from the user input.

[0048] Step 1: Using the division of the LLFF known data set, select the Trex scene, and the training set contains 34 RGB images. Implement step 2 for the RGB images provided by the training set.

[0049] Step 2: For the RGB true value image in the training set described in step 1, use motion structure recovery to extract the sparse point cloud of the Trex scene, and convert it into the initial position and color of the Gaussian ellipsoid in the Gaussian field. Input the initialized Gaussian field into the network structure of the reconstruction part. The overall network framework of the scene reconstruction part of the present invention is as follows: Figure 2 As shown. The initialized Gaussian field is used to render the image through rasterization, and the loss error is calculated by comparing it with the real image. The calculation method is:

[0050]

[0051] Among them, I render Represents the rendered image, I gt represents the real image, They represent L1 loss and structural similarity (SSIM) loss respectively; λ1 (set to 0.8 in this example) and λ2 (set to 0.2 in this example) are the weight coefficients of loss error and structural loss error, and the reconstruction loss error is obtained.

[0052] The training of the neural network structure is based on the reconstruction loss error, and the Gaussian parameters are iteratively updated according to the gradient back propagation method, and the GPU is used for acceleration until the total loss is reduced to within the set threshold or the number of network iterations meets the requirements. The training is stopped and the Gaussian field of the Trex scene is output.

[0053] Step 3: Input the Gaussian field obtained in step 2 into the reconstruction post-processing part. After filtering some Gaussians, the reconstruction loss error is also used as above. Supervised fine-tuning training of Gaussian field. The final result of the Gaussian field after training, filtering and optimizing using the above method is as follows Figure 5 As shown in the figure, it can be seen that the reconstructed image of the present invention is substantially the same as the real RGB of the scene, and the reconstructed rendering effect of the present invention well restores the information of the real scene.

[0054] Step 4: Input the Gaussian field obtained by training in step 3 and the target style text input by the user into the network structure of the stylized part. The overall network structure of the scene stylized part of the present invention is as follows: Figure 3 As shown. The total loss error of the stylized part is calculated as:

[0055]

[0056] in, represents the stylized loss error, Represents the original content preservation loss error, represents the Gaussian regularization loss error, λ sty (set to 1.0 in this example), λ con (set to 0.01 in this example) and λ reg (set to 0.02 in this example) is the weight coefficient of the three errors.

[0057] For the stylized loss error, the rendered image is obtained by rasterizing the Gaussian field, and the rendered image and the target style text prompt are sent to the stylized module. The stylized module framework of the present invention is as follows: Figure 4 As shown. The stylization loss error is calculated as:

[0058]

[0059] Among them, I render represents the rendered image, T tgt Represents the target style text, represents the relative CLIP distance loss, and Represents the relative loss of the overall rendered image and the local rendered image to the positive and negative samples, λ clip (set to 0.5 in this example), λ global (set to 0.1 in this example) and λ local (set to 0.05 in this example) is the weight coefficient of the three errors.

[0060] For the original content preservation error, the above-mentioned rendered image and the original scene real image are sent to the original content preservation error module, and the original content preservation loss error is calculated as follows:

[0061]

[0062] Among them, I render Represents the rendered image, I gt represents the real image, ε(·) represents the VGG network for extracting features, represents L2 loss.

[0063] For the Gaussian regularization error, the above Gaussian field is sent to the Gaussian regularization module, and the Gaussian regularization loss error is calculated as:

[0064]

[0065] in, represents the Gaussian scale change loss, represents the Gaussian density change loss, λ sca In this example, it is set to 0.05) and λ opa In this example, it is set to 0.05) is the weight coefficient of the two errors.

[0066] According to the total loss error of the stylized network, the Gaussian parameters are optimized using the gradient back propagation method and accelerated using the GPU until the total loss is reduced to the set threshold or the number of network iterations meets the requirements. The training is stopped and the Gaussian field of the Trex scene is stylized after the input of the style text prompt line art is output.

[0067] The final stylized effect of training using the above method is as follows Figure 6 As shown in the figure, the stylized effect of the present invention extracts the abstract semantic information suggested in this paper and transfers the style of the original scene. Comparing the stylized effect image with the original scene image, it can be seen that the method of the present invention not only transfers the style, but also well preserves the appearance information and geometric structure of the original scene.

[0068] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A text-driven 3D Gaussian scene stylization method, characterized in that: The method comprises the following steps: (1) Scene reconstruction: Use an RGB image sensor to collect RGB image sequences of the scene as input to the scene reconstruction neural network. The scene reconstruction neural network uses a three-dimensional Gaussian splashing technique to reconstruct the input RGB image sequence into a three-dimensional scene. The three-dimensional scene is represented by a Gaussian field composed of a Gaussian ellipsoid with various parameters. Finally, the network outputs a three-dimensional Gaussian field representing the scene. (2) Post-processing of scene reconstruction: The Gaussian ellipsoid in the Gaussian field is trimmed using the Gaussian filtering strategy, and the scene Gaussian field is further optimized on this basis, and finally a three-dimensional Gaussian field representing the original scene is obtained after post-processing; (3) Scene stylization: Input the text prompt of the target style, and use the text prompt and the three-dimensional Gaussian field representing the original scene obtained in step (2) as the input of the scene stylization neural network; through the scene stylization neural network, align the semantic information of the scene with the style text, gradually optimize the various parameter values ​​of the Gaussian ellipsoid in the Gaussian field, and finally obtain a Gaussian field of a stylized three-dimensional scene.

2. The text-driven 3D Gaussian scene stylization method according to claim 1, characterized in that: The step (1) is specifically processed as follows: 1.1) Input a sequence of RGB images of a scene and use motion structure to recover a sparse representation of the scene; 1.2) Convert the sparse point cloud representation of the scene into the initial distribution state of the three-dimensional Gaussian field; 1.3) Use differentiable Gaussian rasterization to render the image of the Gaussian field at a random viewing angle; 1.4) Calculating the loss error based on the difference between the rendered image and the real image at the same viewing angle, wherein the loss error includes L1 loss and structural similarity loss; 1.5) According to the loss error, back propagate to adjust the parameter values ​​of the Gaussian ellipsoid, which include position, color, density, scale and rotation direction; 1.6) At each set training round, apply automatic densification and pruning strategies to the overall Gaussian field to dynamically adjust the Gaussian distribution; 1.7) Return the Gaussian field dynamically adjusted in 1.6) to 1.3) for rasterization operation, and repeat steps 1.3) to 1.6) until the target training round is reached and the loss function converges, and finally a Gaussian field representing the three-dimensional scene is obtained.

3. The method for text-driven 3D Gaussian scene stylization according to claim 1, characterized in that: The step (2) is: According to the Gaussian filtering strategy, the filtering threshold range of the Gaussian ellipsoid parameter value is set, and the Gaussian ellipsoid exceeding or below the filtering threshold range is trimmed to remove the geometric noise of the scene; Under the supervision of the loss function, the parameter values ​​of the filtered Gaussian ellipsoid are fine-tuned to fix the multi-view inconsistency of the scene and the weakening of rendering realism caused by the Gaussian filtering strategy, and the fine-tuned Gaussian field is output.

4. The method for text-driven 3D Gaussian scene stylization according to claim 3, characterized in that: The scene reconstruction post-processing in step (2) is performed by a Gaussian filtering module and a Gaussian optimization module; In the Gaussian filtering module, set the filtering threshold range to cut out Gaussian ellipsoids that are too large or too small in density; In the Gaussian optimization module, the viewing angle is randomly selected, and the rasterization is performed to obtain the rendered image of the filtered Gaussian field. The loss error function between the rendered image and the real image is calculated, and the position, color, density, scale, and rotation direction of the existing Gaussian ellipsoid are updated according to the error back propagation. The iteration is repeated until the target training round is reached and the loss function converges, and finally the three-dimensional Gaussian field representing the original scene is obtained after post-processing.

5. The method for text-driven 3D Gaussian scene stylization according to claim 1, characterized in that: In the step (3), the scene stylization neural network includes a stylization module, an original content preservation module and a Gaussian regularization module; the stylization module uses a text-image multimodal model technology based on contrastive learning to align semantic information between an image rendered by a Gaussian field and a style text in a joint coding space to control the stylization degree of the scene; the original content preservation module uses a deep convolutional neural network to extract multi-dimensional features of the rendered image of the scene and the real image of the original scene during the stylization training process, and monitors the information preservation degree of the original scene through feature errors; the Gaussian regularization module constrains the change range of the scale and density attributes of the Gaussian before and after training through a loss function, controls the rapid change of the Gaussian, and reduces the geometric noise generated in the scene after stylization.

6. The method for text-driven 3D Gaussian scene stylization according to claim 5, characterized in that: The step (3) specifically includes: 3.1) Input the target style text prompt and the reconstructed 3D Gaussian field; 3.2) Randomly select a viewing angle and use differentiable Gaussian rasterization to obtain the rendered image of the current Gaussian field; 3.3) Calculate the stylization loss error based on the rendered image and the textual hint of the target style; 3.4) Send the rendered image and the real image of the reconstructed original scene to the original content preservation module to calculate the original content preservation loss error; 3.5) The post-processed original scene three-dimensional Gaussian field of 3.1) and the Gaussian field obtained by training in the current step are sent to the Gaussian regularization module to calculate the Gaussian regularization loss error; 3.6) constructing a total loss error with the above three loss errors, and adjusting the parameter values ​​of the Gaussian ellipsoid by back propagation according to the total loss error, wherein the parameter values ​​include position, color, density, scale and rotation direction; 3.7) The Gaussian field adjusted by backpropagation in 3.6) is returned to 3.2) for rasterization operation, and steps 3.2)-3.6) are repeated continuously until the target training round is reached and the loss function converges, and finally the Gaussian field of the stylized three-dimensional scene is obtained.

7. The method for text-driven 3D Gaussian scene stylization according to claim 6, characterized in that: In step 3.6), the total loss error is: in, represents the stylized loss error, Represents the original content preservation loss error, represents the Gaussian regularization loss error, λ sty , con and λ reg are the weight coefficients of the three errors; The stylization loss error is: Among them, I render represents the rendered image, T tgt Represents the target style text, represents the relative CLIP distance loss, represents the relative loss between the rendered image and the positive and negative samples, λ clip and λ con is the weight coefficient of the two errors; The original content preservation loss error is: Among them, I render Represents the rendered image, I gt represents the real image, ε(·) represents the VGG network for extracting features, represents L2 loss; The Gaussian regularization loss error is: in, represents the Gaussian scale change loss, represents the Gaussian density change loss, λ sca and λ opa is the weight coefficient of the two errors.

8. The method for text-driven 3D Gaussian scene stylization according to claim 6, characterized in that: The stylization module first renders the image and style text by randomly selecting a perspective from the Gaussian field, and simultaneously sends them to the image encoder and text encoder of the pre-trained text-image multimodal model to obtain the encoding features of the image and text in the same feature space, and measures the relative cosine similarity of the two encoding features for stylization trajectory control; then, based on the contrastive learning strategy, constructs positive and negative sample text data, and uses the image enhancement strategy to randomly crop to obtain local images, and uses the positive and negative sample data for contrastive learning of the overall and local images to enhance the degree of stylization, and finally obtains a stylized image of the rendered image under the prompt of the style text.

Citation Information

Patent Citations

  • Three-dimensional scene style migration method and device, equipment and storage medium

    CN116934936A

  • Three-dimensional scene style migration method, three-dimensional scene style migration system and computer equipment

    CN117274042A

  • Three-dimensional human body generation method based on text prompt

    CN118229860A

  • Rapid high-precision dense reconstruction method and system based on 3D Gaussian rasterization

    CN118314271A

Cited By

  • Four-dimensional Gaussian model generation method, system and equipment based on Gaussian sputtering

    CN119338966A

  • Semantic three-dimensional scene understanding method and device based on large language model, equipment and storage medium

    CN119941989A

  • Semantic three-dimensional scene understanding method and device based on large language model, equipment and storage medium

    CN119941989B

  • Dynamic Gaussian scene reconstruction method based on depth regularization

    CN119991974A

  • 3D Gaussian monomerization and semantization method based on multi-modal visual large model

    CN120164125A