Method for finely editing three-dimensional Gaussian scene through text

Abstract

The method for finely editing the three-dimensional Gaussian scene through the text is characterized by comprising the following steps that S1, description of an input scene is obtained, specifically, the description of the input scene is obtained from the input scene through a multi-modal model and a large language model; s2, extracting a text of the region of interest: extracting the text of the region of interest from a given edited text by using a large language model; s3, converting the text space into an image space: segmenting an initial image input into scene rendering according to the text of the region of interest by using a segmentation module to obtain an image of the region of interest; s4, raising the dimension of the image space: raising the dimension of the image of the region of interest to a three-dimensional region of interest through a dimension raising module; and S5, editing a three-dimensional Gaussian scene: finely editing the three-dimensional Gaussian of the input scene by using the obtained three-dimensional region of interest as a constraint. According to the method, the problems of low editing speed and insufficient detail quality are solved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a method for finely editing a three-dimensional Gaussian scene of text based on a large language model and a two-dimensional diffusion model. Background Art

[0002] Generative AI and 3D models are very important in many fields today and have a wide range of application scenarios, such as virtual reality, game development, and video production. The power of large language models such as ChatGPT is also obvious to all. Three-dimensional Gaussian splatter (3DGS) was proposed in 2023 and is the most revolutionary innovation in the field of 3D reconstruction in recent years. Compared with the implicit expression of the previous NeRF-based method, 3DGS draws on the advantages of explicit expression from the point cloud-based 3D reconstruction method. Compared with NeRF, which needs to query the neural network to calculate the scene in real time, 3DGS has faster rendering speed and shorter training time.

[0003] The introduction of the two-dimensional diffusion model makes it possible to edit two-dimensional images with text. Furthermore, some recent achievements have also achieved the use of text to edit three-dimensional scenes. However, the main problem with the existing technology is that the current methods based on the two-dimensional diffusion model often apply the editing to the entire image, while most users need to perform fine editing on specific areas of the three-dimensional scene. The existing methods are slow in obtaining three-dimensional scenes, and the quality of the results of text editing of three-dimensional scenes is low, and there will be problems with inconsistency in the background part.

[0004] The difficulty in solving the above problems and defects is: in order to realize editing of specific areas of the input scene with text, the editing position needs to be precisely constrained, multiple modules are required to operate the input scene and the editing text, and additional parameters are added to the three-dimensional Gaussian.

[0005] The significance of solving the above problems and defects is: it provides a practical idea of ​​using text to finely edit three-dimensional Gaussian scenes, and is a successful attempt to apply large language models and diffusion models to three-dimensional Gaussian splashing. Summary of the invention

[0006] The present invention provides a method for fine-tuning text editing of three-dimensional Gaussian scenes, which can use text descriptions to fine-tune three-dimensional scenes, improve the quality of the output model, and ensure the consistency of the background of the input and output three-dimensional scenes; the present invention combines multimodal models, large language models, and image segmentation technology to solve the problem of editing specific areas of three-dimensional scenes; the present invention provides a method for adding, deleting, and selecting edited areas, further enhancing the fine-tuning capabilities of three-dimensional scenes. Based on three-dimensional Gaussian splashing, the problems of slow editing speed and insufficient detail quality are solved by utilizing its explicit characteristics and the advantages of efficient and differentiable rendering.

[0007] The technical solution of the present invention is as follows: The method for finely editing a three-dimensional Gaussian scene with text of the present invention is characterized in that it comprises the following steps: S1. Obtaining a description of an input scene: using a multimodal model and a large language model to obtain a description of the input scene from the input scene; S2. Extracting text of an area of ​​interest: using a large language model to extract text of an area of ​​interest from a given edited text; S3. Converting text space into image space: using a segmentation module to segment an initial image rendered by the input scene according to the text of the area of ​​interest to obtain an image of the area of ​​interest; S4. Upgrading the image space: upgrading the image of the area of ​​interest to a three-dimensional area of ​​interest through a dimension upgrading module; S5. Editing a three-dimensional Gaussian scene: using the obtained three-dimensional area of ​​interest as a constraint to finely edit the three-dimensional Gaussian of the input scene.

[0008] Optionally, in the above method for finely editing a three-dimensional Gaussian scene with text, in step S2, a predefined template is used. To organize the description of the scene output in step S1 and the edit text entered by the user , thus obtaining the user text , the expression is: ; Then use the large language model According to the refined prompt words Distill the user text into the text of the area of ​​interest, denoted as , the expression is: .

[0009] Optionally, in the above method for finely editing a three-dimensional Gaussian scene using text, in step S3, using the input scene Rendering the initial image , using the segmentation module Based on the text of the region of interest The initial image Segmentation to obtain the image space of the region of interest , the expression is: .

[0010] Optionally, in the above method for finely editing a three-dimensional Gaussian scene with text, in step S4, an additional attribute is added to the three-dimensional Gaussian , this property indicates whether this 3D Gaussian is in the 3D region of interest Inside, Initialized to 0, indicating that the three-dimensional Gaussian is not inside, if is 1, it means that the three-dimensional Gaussian is located Inside; The set of ,in It is the input scene The number of three-dimensional Gaussians in ; the changed three-dimensional Gaussians are represented as ,in represents the center point of the three-dimensional Gaussian, represents the three-dimensional covariance matrix, Represents RGB color, Indicates opacity, In the original 3D Gaussian splatter, the rendering process is expressed as: use Rewrite it to get the rendered image of the area of ​​interest : ; A loss function is used to supervise the rendering of the image using the region of interest The process of training 3D Gaussian: in, and is a hyperparameter, parameter Will be updated by gradient descent; Get the trained three-dimensional Gaussian , provides an optional operation for the user, the user can add 3D Gaussian, delete 3D Gaussian and use 3D box To modify , the output is a three-dimensional region of interest , whose expression is: in, represents the three-dimensional Gaussian that the user wants to add, represents the 3D Gaussian that the user wants to remove from the region of interest, are the coordinates of the three-dimensional box, which is used to limit the area of ​​interest within the box; Through the dimensional module Get 3D region of interest It is expressed as: .

[0011] Optionally, in the above method for finely editing a three-dimensional Gaussian scene with text, in step S5, the image of the region of interest Upgrading to 3D region of interest After that, the input scene Randomly sample views to get rendered images ; Using a two-dimensional diffusion model Edit text as entered For rendered images Edit and export as edited image ; Then, calculate the edited image and render the image The loss between the two and the 3D region of interest Gradient back propagation is performed within the kernel; finally, optimization is performed based on these gradients to obtain the edited 3D scene. .

[0012] According to the technical solution of the present invention, the beneficial effects produced are: The method of the present invention aims at the problems of low quality and inconsistency of background parts caused by slow rendering speed and inability to perform fine editing of specific areas of the three-dimensional scene in existing text editing three-dimensional scene methods. A three-dimensional scene generation model based on a large language model, a two-dimensional diffusion model and a three-dimensional Gaussian splash is constructed, which effectively improves the speed and quality of text editing three-dimensional scenes and can edit specific areas of the three-dimensional scene. In the prior art, the three-dimensional scene after text editing often cannot maintain consistency in the background part, and it is difficult to obtain high-quality three-dimensional scenes through text editing in the prior art. The method of the present invention uses text precise positioning through a series of operations, and uses this as a constraint to finely edit the input scene. The present invention uses a three-dimensional Gaussian splash model with explicit characteristics to enhance the refinement of editing, improve the quality of the results, and increase the rendering speed.

[0013] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0015] Figure 1 It is a flow chart of the method for finely editing a three-dimensional Gaussian scene of text of the present invention; Figure 2 The scene description generation module in step S1 of the method of the present invention Flowchart of Figure 3 is a flowchart of extracting text in the region of interest in step S2 of the method of the present invention; Figure 4is a flow chart of converting text space into image space in step S3 of the method of the present invention; Figure 5 It is the comparison between the present invention and the existing method (Edited text: turning the benches red and the road into a river). DETAILED DESCRIPTION

[0016] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.

[0017] The method of the present invention for fine-tuning text editing of three-dimensional Gaussian scenes based on a large language model and a two-dimensional diffusion model benefits from the explicit characteristics of three-dimensional Gaussians, and designs a series of modules to realize editing of specific areas of three-dimensional scenes with text descriptions. Specifically, the present invention first obtains a description of the input scene and uses it to extract the area of ​​interest, then converts the edited text from the text control to the image space, and then performs dimensionality upgrade to control the editing process, thereby effectively solving the problem of low quality and inconsistency of background parts caused by the inability of the old method to perform fine editing of specific areas of the three-dimensional scene. The present invention uses a three-dimensional Gaussian splash model as a way to express the three-dimensional scene, solving the problem of slow rendering speed of the old method.

[0018] The principle of the present invention is: 1) Using a large language model and a two-dimensional diffusion model to edit specific areas of a three-dimensional scene with text; 2) Process the input scene through a multimodal model and a large language model to generate scene descriptions and text of the region of interest, thereby achieving preliminary positioning of the editing area; 3) Sample the input scene to obtain an initial image, and use the segmentation module to segment the initial image according to the text in the area of ​​interest to achieve the conversion from text space to image space; 4) Upscaling the image space and allowing users to manually edit it to obtain a three-dimensional region of interest; 5) Use the two-dimensional diffusion model to edit several randomly selected images, calculate the loss of the previous and next images, use it to backpropagate the gradient of the three-dimensional region of interest, and use three-dimensional Gaussian splashing to obtain the edited three-dimensional scene.

[0019] The method of the present invention provides a fast and high-quality idea for fine text editing of three-dimensional scenes. The multimodal model and the large language model are used to obtain the text description of the input scene, so as to locate the editing area according to the edited text, and the sampled initial image is segmented and dimensionally upgraded. Then, the randomly sampled image is edited using a two-dimensional diffusion model, and the loss of the previous and next images is calculated for the gradient backpropagation of the three-dimensional region of interest, and the edited three-dimensional scene is obtained using three-dimensional Gaussian splashing.

[0020] like Figure 1 As shown, the method for finely editing a three-dimensional Gaussian scene of text of the present invention comprises the following steps: S1. Obtain a description of the input scene: Use a multimodal model and a large language model to obtain a description of the input scene from the input scene.

[0021] In step S1, a module called scene description generation is used , according to the input scenario Get the description of the input scene .

[0022] S2. Extracting text of the region of interest: Use a large language model to extract the text of the region of interest from the given edited text.

[0023] In step S2, the description of the input scene obtained in step S1 is and the edit text entered Input to the large language model To determine the location to edit in the scene, the output of this step is the text of the region of interest, denoted as .

[0024] S3. Convert text space to image space: Use the segmentation module to segment the initial image rendered by the input scene according to the text in the region of interest to obtain an image of the region of interest.

[0025] In step S3, the input scene is used Rendering the initial image , using the segmentation module Based on the text of the region of interest The initial image Segmentation to obtain the image of the region of interest .

[0026] S4. Upgraded image space: The image of the region of interest is upgraded to a three-dimensional region of interest through the dimension upgrading module.

[0027] S5. Edit the 3D Gaussian scene: Use the obtained 3D region of interest as a constraint to finely edit the 3D Gaussian of the input scene.

[0028] In step S5, the image of the region of interest Upgrading to 3D region of interest After that, the input scene Randomly sample views to get rendered images Using a two-dimensional diffusion model Edit text as entered For rendered images Edit and export as edited image . Then, the edited image is calculated and render the image The loss between the two and the 3D region of interest Gradient back propagation is performed in the back propagation to ensure that only the region of interest can obtain the corresponding gradient in the back propagation. Finally, optimization is performed based on these gradients to obtain the edited 3D scene .

[0029] The method of the present invention uses point clouds as geometric guides for inpainting images with diffusion models, and smoothly connects the point clouds generated successively, so that the generated three-dimensional scene is consistent under multiple views. Using three-dimensional Gaussian splashing as the rendering method of the three-dimensional scene improves the rendering speed, thereby solving the problems of discontinuity, low rendering quality and slow rendering speed of the three-dimensional scene under multiple views in the existing method. The specific implementation steps are as follows:

[0030] S1. Get the description of the input scene: First, obtain the user's 3D Gaussian scene input, and the input scene is recorded as , the present invention adopts a module called scene description generation , according to the input scenario Get the description of the input scene , whose expression is: Specifically, the scene description generation module used in the present invention By multimodal model and large language models Composition, its design is as follows Figure 2 As shown. Using the differentiable Gaussian splashing technique, according to the input scene Generate a set of 2D image samples , and then input into the multimodal model Generate a set of corresponding text descriptions in :

[0031] in, is a prompt word, such as "what is in the image", used in multimodal models to obtain a precise description.

[0032] Then, this set of corresponding text descriptions is fed into a large language model In the example above, use a merge prompt word Will Merge into a detailed scene description, that is, the description of the input scene , the expression is: The output of step S1 is obtained .

[0033] S2. Extract the text of the region of interest: First, the present invention uses a predefined template To organize the description of the scene output in step S1 and the edit text entered by the user , an example showing the template format is: Scenario description: Edit text: This gets the user text , the expression is: Then use the large language model According to the refined prompt words Distill the user text into the text of the area of ​​interest, denoted as , the expression is: Figure 3 This is an example of step S2.

[0034] S3. Convert text space to image space: Step S3: Convert text space to image space. The flowchart is shown in Figure 4. First, use the input scene Rendering the initial image , using the segmentation module Based on the text of the region of interest The initial image Segmentation to obtain the image space of the region of interest , the expression is: S4. Up-dimensional image space: The present invention trains the image space output in step S3 Upgrade to three dimensions and get the three-dimensional region of interest .

[0035] Specifically, the present invention adds an additional property to the three-dimensional Gaussian , this property indicates whether this 3D Gaussian is in the 3D region of interest Inside, Initialized to 0, indicating that the three-dimensional Gaussian is not inside, if is 1, it means that the three-dimensional Gaussian is located Inside. The set of ,in It is the input scene The number of three-dimensional Gaussians in . The changed three-dimensional Gaussian can be expressed as ,in represents the center point of the three-dimensional Gaussian, represents the three-dimensional covariance matrix, Represents RGB color, Indicates opacity.

[0036] In the original 3D Gaussian splatter, the rendering process can be expressed as: The present invention uses Rewrite it to get the rendered image of the area of ​​interest : The present invention adopts a loss function to supervise the rendering of the image with the region of interest The process of training 3D Gaussian: in, and is a hyperparameter, parameter It will be updated via gradient descent.

[0037] So we get the trained three-dimensional Gaussian In order to avoid missing the region of interest, the present invention provides an optional operation for the user, the user can add a 3D Gaussian, delete a 3D Gaussian and use a 3D box To modify , the output is a three-dimensional region of interest , whose expression is: in, Indicates the three-dimensional Gaussian that the user wants to add; represents the 3D Gaussian that the user wants to remove from the region of interest; are the coordinates of the 3D box that confines the region of interest within the box.

[0038] Therefore, through the dimension-upgrading module Get 3D region of interest It can be expressed as: Step S4 is the core invention of the present invention.

[0039] S5. Edit 3D Gaussian scene S5: Image of the region of interest Upgrading to 3D region of interest After that, the input scene Randomly sample the view to get the rendered image .Will and the random noise level Insert into the two-dimensional diffusion model In, according to the edit text and the initial image Get the edited image .

[0040] In step S5, the present invention uses the same method as the three-dimensional Gaussian splashing. and Loss function.

[0041] Use the above loss function to calculate the edited image and render the image After that, in step S4, the three-dimensional region of interest is output Perform gradient backpropagation in: Finally, the Adam algorithm is used to optimize the three-dimensional Gaussian. After 30,000 iterations, the edited three-dimensional scene is obtained. .

[0042] Table 1: Comparison results of parameters involved in the scenes generated by the text editing 3D scene method of the present invention and the existing model Model Text-image directional similarity Image-image similarity Freche generation distance Average rendering time Instrust-nerf2nerf 0.22 0.85 103 51 minutes The present invention 0.28 0.95 51 20 minutes The present invention and the existing Instrust-nerf2nerf model are experimented in the scenario of Mip-NeRF360 dataset, and four parameters are used as indicators to compare the two to show the improvement of the performance of the present invention as shown in Table 1.

[0043] Text-image directional similarity is used to measure the similarity between images and text, especially their similarity in a specific direction. Specifically, it is measured by calculating the directional consistency of the image and text in the embedding space. Using the CLIP model, the text and image are mapped to a common embedding space, and then the cosine similarity between them is calculated to evaluate whether they are aligned in the same semantic direction.

[0044] Image-image similarity is used to measure the similarity between two images. It quantifies their similarity by calculating the distance or similarity (such as Euclidean distance or cosine similarity) between the feature embedding space of the two images. The image-image similarity parameter can be used to compare the difference between the original image and the generated image, thereby evaluating the effect of the generative model.

[0045] In such tasks, the higher the values ​​of text-image directional similarity and image-image similarity, the more accurate the method can be in editing specific areas of the scene in accordance with the edited text.

[0046] The Fréchet generation distance is a common metric for measuring the quality of generated images, especially in the evaluation of generative adversarial networks and other generative models. The quality of an image is measured by calculating the distance between the feature distributions extracted from the generated image and the real image in the Inception v3 model. Specifically, the Fréchet generation distance calculates the Fréchet distance (i.e., the difference in the mean and covariance of the Gaussian distribution) between the feature distributions of the generated image and the real image. Lower values ​​of the Fréchet generation distance generally mean that the generated images are more similar to the real images in terms of quality and diversity.

[0047] The average rendering time is the time it takes to render each scene in the statistical data set, and then the average is calculated. The lower the average rendering time, the faster the rendering speed.

[0048] From the results in Table 1, it can be seen that the text-image directional similarity and image-image similarity of the present invention are higher than those of the existing Instrust-nerf2nerf model, which means that the present invention can more accurately edit specific areas of the input scene instead of editing content outside the specific area, and the rendered scene is more in line with the description of the edited text, that is, the scene can be edited more accurately according to the requirements of the edited text. The Fleche generation distance of the present invention is lower than that of the existing Instrust-nerf2nerf model, which means that the scenes generated by the rendering of the present invention are more similar to real images in quality and diversity, that is, closer to reality. The average rendering time of the present invention has a strong advantage over the existing Instrust-nerf2nerf model, thanks to the explicit expression of three-dimensional Gaussian splashing and excellent rendering strategy.

[0049] Figure 5 It is a comparison between the present invention and the existing method, which can more intuitively demonstrate the improvement of the present invention. The editing text used here is "turn the bench red and turn the road into a river". The bicycle scene of the Mip-NeRF360 dataset is edited using the existing Instrust-nerf2nerf model and the present invention respectively. It can be seen that the existing Instrust-nerf2nerf model not only turns part of the bench red, but also turns the surrounding environment, including the road that should have turned into a river, into red, and the overall scene tone is reddish, and the background part has many discrepancies with the input scene. The present invention can accurately turn the bench red without affecting the color of the surrounding environment and the bicycle, and the editing text of "turn the road into a river" is also well implemented. The road did not turn red but successfully turned into a river. This reflects that the present invention has achieved strong robustness and can use text to perform fine editing of specific areas of the input scene.

[0050] The present invention utilizes a large language model, a two-dimensional diffusion model, and a three-dimensional Gaussian explicit representation to achieve a refined text editing three-dimensional Gaussian scene. The multimodal model and the large language model are used to locate the editing position, and the segmentation module is used to perform image segmentation, and then the image of the region of interest is upgraded in dimension, and the region to be edited is allowed to be manually modified to achieve refined three-dimensional scene editing. Specifically, after inputting the three-dimensional scene, the description of the input scene is obtained through the multimodal model and the large language model, and the text of the region of interest is extracted. This is used as a guide to convert it into image space and then upgrade the dimension. The sampled image is edited using a two-dimensional diffusion model, and the gradient back propagation is performed on the three-dimensional region of interest by calculating the loss of the previous and next images, thereby obtaining the edited three-dimensional scene. Using three-dimensional Gaussian splashing as the rendering method of the three-dimensional scene improves the rendering speed and quality. Many problems of the old method are solved.

[0051] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.

Claims

1. A method for finely editing a three-dimensional Gaussian scene with text, characterized in that: The following steps are involved: S1. Obtaining a description of an input scene: obtaining a description of an input scene from the input scene using a multimodal model and a large language model; S2. Extracting text in the region of interest: extracting text in the region of interest from a given edited text using the large language model; S3. Converting the text space into the image space: using a segmentation module, segmenting the initial image rendered by the input scene according to the text in the region of interest to obtain an image of the region of interest; S4. Upgrading the image space: Upgrading the image of the region of interest to a three-dimensional region of interest through a dimensionality-upgrading module; S5. Editing the three-dimensional Gaussian scene: using the obtained three-dimensional region of interest as a constraint, finely editing the three-dimensional Gaussian of the input scene.

2. The method for finely editing a three-dimensional Gaussian scene according to claim 1, characterized in that: In step S2, a predefined template is used To organize the description of the scene output in step S1 and the edit text entered by the user , thus obtaining the user text , the expression is: ； Then use the large language model According to the refined prompt words Distill the user text into the text of the area of ​​interest, denoted as , the expression is: 。 3. The method for finely editing a three-dimensional Gaussian scene according to claim 1, characterized in that: In step S3, the input scene is used Rendering the initial image , using the segmentation module Based on the text of the area of ​​interest The initial image Segmentation to obtain the image space of the region of interest , the expression is: 。 4. The method for finely editing a three-dimensional Gaussian scene according to claim 1, characterized in that: In step S4, an additional attribute is added to the 3D Gaussian , this property indicates whether this 3D Gaussian is in the 3D region of interest Inside, Initialized to 0, indicating that the three-dimensional Gaussian is not inside, if is 1, it means that the three-dimensional Gaussian is located Inside; The set of ,in It is the input scene The number of three-dimensional Gaussians in ; the changed three-dimensional Gaussians are represented as ,in represents the center point of the three-dimensional Gaussian, represents the three-dimensional covariance matrix, Represents RGB color, Indicates opacity, In the original 3D Gaussian splatter, the rendering process is expressed as: use Rewrite it to get the rendered image of the area of ​​interest : ； A loss function is used to supervise the rendering of the image using the region of interest The process of training 3D Gaussian: in, and is a hyperparameter, parameter Will be updated by gradient descent; Get the trained three-dimensional Gaussian , provides an optional operation for the user, the user can add 3D Gaussian, delete 3D Gaussian and use 3D box To modify , the output is a three-dimensional region of interest , whose expression is: in, represents the three-dimensional Gaussian that the user wants to add, represents the 3D Gaussian that the user wants to remove from the region of interest, are the coordinates of the three-dimensional box, which is used to limit the area of ​​interest within the box; Through the dimensional module Get 3D region of interest It is expressed as: 。 5. The method for finely editing a three-dimensional Gaussian scene according to claim 1, characterized in that: In step S5, the image of the region of interest Upgrading to 3D region of interest After that, the input scene Randomly sample views to get rendered images ; Using a two-dimensional diffusion model Edit text as entered For rendered images Edit and export as edited image ; Then, the edited image is calculated and render the image The loss between the two and the 3D region of interest Gradient back propagation is performed within the kernel; finally, optimization is performed based on these gradients to obtain the edited 3D scene. .

Images