Editing method for three-dimensional Gaussian

Abstract

The invention provides a three-dimensional Gaussian editing method, which comprises the following steps of: S1, performing source three-dimensional Gaussian rendering, depth estimation and inversion to obtain a source two-dimensional image and a depth map corresponding to different visual angles, and inverting the source two-dimensional image back to noise; s2, carrying out semantic segmentation on the source two-dimensional image to obtain a corresponding mask, and dividing the image into an edited region and an unedited region; s3, the frequency-space consistency is enhanced, and the consistency of noise is enhanced; s4, consistency alignment of the semantics and the reference view angle: inputting the noise after consistency enhancement, the editing text indication and the depth map into a pre-trained two-dimensional diffusion model for two-dimensional editing, performing consistency alignment on the current editing view angle, and combining the current editing view angle with an unedited area to obtain a two-dimensional editing result; and S5, three-dimensional Gaussian optimization: optimizing the source three-dimensional Gaussian according to the two-dimensional editing result to obtain the finally edited three-dimensional Gaussian. According to the method, the three-dimensional editing is optimized by using the two-dimensional editing result, so that the multi-view consistency of the three-dimensional editing result is ensured.

Description

Technical Field

[0001] The present invention relates to the field of editing technology based on three-dimensional Gaussian input, and in particular to a three-dimensional Gaussian editing method for improving the multi-view consistency of three-dimensional Gaussian editing. Background Art

[0002] With the emergence of 3D editing technology and the latest breakthroughs in neural rendering technology [6-7] , breaking the technical barriers between two-dimensional and three-dimensional modes, making it possible to transform the two-dimensional diffusion model into [9-11] The transfer of powerful generation capabilities to 3D generation models has become a trend [8] This trend provides more diverse result generation and editing, but at the same time makes the related network models more dependent on the editing results of the two-dimensional diffusion model to better edit and optimize the three-dimensional Gaussian.

[0003] The current mainstream paradigm is to render a set of images from different viewpoints from a 3D model, then perform 2D editing using a pre-trained 2D diffusion model and textual hints, and finally reconstruct these edited viewpoints back into the 3D model for further 3D model editing. [1-5] However, the large differences between different perspectives, as well as the diversity introduced by the textual cues themselves, may lead to inconsistencies between edited perspectives.

[0004] There are currently several ways to address this issue: (1) Optimize the 2D editing results by interacting with the rendered images from different perspectives during editing to improve the consistency between images, thereby improving the consistency of the 3D editing results. However, the interaction during image editing is not always effective, which may result in large differences between perspectives. (2) Fine-tune the 3D model by fine-tuning the 3D model using small batches of image editing results multiple times to improve multi-perspective consistency. However, if the perspective differences of the image editing results of this batch are too large, it may have a counter-effect in the fine-tuning, making it impossible to effectively resolve the inconsistency of the 3D editing. In general, how to effectively alleviate the inconsistency problem in 3D editing is challenging.

[0005] The difficulty of solving this problem lies in: how to ensure effective interaction between 2D editing perspectives and the consistency of 2D editing results; and how to effectively use 2D editing results to improve the consistency of 3D editing tasks.

[0006] The significance of solving this problem is: providing a 3D Gaussian editing method that can effectively solve the multi-view inconsistency problem in 3D Gaussian editing tasks; and providing a new solution to the multi-view inconsistency problem in 3D generation systems. Summary of the Invention

[0007] The present invention provides a three-dimensional Gaussian editing method, which uses two-dimensional editing results to optimize three-dimensional editing to ensure the multi-view consistency of the three-dimensional editing results.

[0008] The technical solutions of the present invention are as follows: The three-dimensional Gaussian editing method of the present invention includes the following steps: S1. Source three-dimensional Gaussian rendering, depth estimation and inversion: by rendering and depth estimation of different perspectives of the source three-dimensional Gaussian, source two-dimensional images and depth maps corresponding to different perspectives are obtained, and the source two-dimensional image is inverted back to noise; S2. Source two-dimensional image semantic segmentation: using a pre-trained segmentation model to semantically segment the foreground object and background of the source two-dimensional image, obtain corresponding masks and divide the image into an edited area and an unedited area; S3. Frequency-space consistency enhancement: by constructing a joint operation of the frequency domain and the spatial domain on the edited area, the consistency of the noise is enhanced; S4. Semantic and reference perspective consistency alignment: by inputting the noise after consistency enhancement, the editing text indication and the depth map obtained in step S1 into the pre-trained two-dimensional diffusion model for two-dimensional editing, during the editing process, the current editing perspective is aligned consistently by semantics and reference perspective, and combined with the unedited area retained in step S2 to obtain a two-dimensional editing result; S5. Three-dimensional Gaussian optimization: using the two-dimensional editing result obtained in step S4 to optimize the source three-dimensional Gaussian to obtain the final edited three-dimensional Gaussian.

[0009] Optionally, in the above three-dimensional Gaussian editing method, in step S1, the noise is obtained through DDIM inversion in the two-dimensional diffusion field, and the inversion process uses the source text indication corresponding to the source three-dimensional Gaussian.

[0010] Optionally, in the above-mentioned three-dimensional Gaussian editing method, in step S2, the pre-trained semantic segmentation model is the open-source LangSAM (a model that segments everything through natural language).

[0011] Optionally, in the above-mentioned three-dimensional Gaussian editing method, in step S3, the frequency domain is processed by adjusting the high-frequency and low-frequency ratio and then adding Gaussian noise; the spatial domain is processed by bilateral filtering and then adding Gaussian noise, and the result is obtained by weighting the results of the frequency domain and the spatial domain.

[0012] Optionally, in the above three-dimensional Gaussian editing method, in step S3, In the frequency domain, Gaussian low-pass filter and Gaussian high-pass filter are first used to convert the latent code of the perspective to be edited Convert from spatial domain to frequency domain to obtain low-frequency and high frequency :

[0013]

[0014]

[0015] Secondly, the relative weights of the high-frequency and low-frequency components are adjusted by the weight factor α to create a combined frequency component :

[0016] Finally, apply a mask of the part of the image to be edited , controlling Gaussian noise Add and detail preservation areas to obtain the final output of the frequency domain branch : ; In terms of spatial domain, first Apply a bilateral filter to calculate for each pixel The filtered output on :

[0017] in, is a normalization factor that ensures the sum of the weights is 1, It's a pixel The local neighborhood around is the standard deviation of the spatial Gaussian kernel, and then, by adding Gaussian noise and applying the mask To adjust ,get :

[0018] Next, Apply position-based attention mechanism to obtain the output of spatial domain branch :

[0019]

[0020] in, It's location The attention weight at It is the central location. is the maximum possible distance from the center; The fusion of frequency domain and spatial domain, the frequency domain branch outputs the result And the spatial domain branch output results Fusion is performed to obtain enhanced latent coding : .

[0021] Optionally, in the above-mentioned three-dimensional Gaussian editing method, in step S4, the consistency alignment of semantics and reference perspective is performed on the U-Net module and the ControlNet module in the two-dimensional diffusion model denoising process, and the pre-trained two-dimensional diffusion model is the open source released Stable Diffusion v1.5.

[0022] Optionally, in the above three-dimensional Gaussian editing method, in step S4, the self-attention output is first calculated , Attention output based on reference perspective and semantic-based attention output , then and Add together to get the cross attention output :

[0023]

[0024]

[0025]

[0026] in, 、 and is a linear transformation used to obtain query, key and value, 、 and The latent encoding of the editor's perspective is , latent encoding of reference perspective and text instructions The hidden layer of and After that, the attention output is generated by integrating self-attention and cross-attention :

[0027] in, Used to balance the contributions of self-attention and cross-attention.

[0028] Optionally, in the above three-dimensional Gaussian editing method, in step S5, the two-dimensional editing result obtained in S4 is used as a data set, and the source three-dimensional Gaussian is trained for a certain number of steps to obtain the edited three-dimensional Gaussian.

[0029] According to the technical solution of the present invention, the beneficial effects produced are: This invention provides a method for 3D Gaussian editing, designed to improve the multi-view consistency of 3D Gaussian editing and, consequently, the quality of the resulting 3D Gaussian editing results. This method employs frequency-spatial consistency enhancement, enabling images from different viewpoints to share the same diffusion style, ensuring style consistency across different viewpoints during 2D editing. Furthermore, during the 2D editing process, semantic-reference consistency alignment is implemented, utilizing semantic information and the reference viewpoint as a guide for aligning the editing perspective. This approach resolves inconsistencies in images from different viewpoints during editing, providing a novel solution for addressing multi-view inconsistencies in 3D generation systems.

[0030] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below with reference to the accompanying drawings and through specific embodiments: BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the specific implementation or the description of the prior art.

[0032] Figure 1 is a flow chart of the three-dimensional Gaussian editing method of the present invention; Figure 2 Schematic diagram of the two consistency enhancements and consistency alignment proposed by the method of the present invention. DETAILED DESCRIPTION

[0033] To make the purpose, technical methods and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and specific examples. These examples are merely illustrative and not intended to limit the present invention.

[0034] like Figure 1 As shown, the present invention provides a three-dimensional Gaussian editing method, comprising the following steps: S1. Source 3D Gaussian rendering, depth estimation, and inversion: By rendering the source 3D Gaussian at different viewpoints and performing depth estimation, we obtain the source 2D image and depth map corresponding to different viewpoints, and then invert the source 2D image back to noise.

[0035] In this step, the noise is obtained by inverting the implicit denoising diffusion model (DDIM) in the two-dimensional diffusion domain, and the inversion process uses the source text indication corresponding to the source three-dimensional Gaussian.

[0036] S2. Semantic Segmentation of the Source 2D Image: Use a pre-trained segmentation model to semantically segment the foreground and background of the source 2D image, obtaining corresponding masks and dividing the image into edited and unedited regions. The pre-trained semantic segmentation model is the open-source LangSAM language segmentation network.

[0037] S3. Frequency-space consistency enhancement: By constructing a joint operation in the frequency and space domains on the edited region, the consistency of the noise is enhanced.

[0038] In this step, the frequency domain is processed by adjusting the high-frequency to low-frequency ratio and then adding Gaussian noise; the spatial domain is processed by bilateral filtering and then adding Gaussian noise. The result is obtained by weighting the results of the frequency domain and spatial domain.

[0039] S4. Consistent alignment of semantics and reference perspectives: Two-dimensional editing is performed by inputting the noise after consistency enhancement, the editing text indication, and the depth map obtained in step S1 into the pre-trained two-dimensional diffusion model. During the editing process, the current editing perspective is consistently aligned through semantics and reference perspectives, and combined with the unedited area retained in step S2 to obtain the two-dimensional editing result.

[0040] In this step, semantic and reference perspective consistency alignment is performed in the U-Net module and the ControlNet module during the denoising process of the two-dimensional diffusion model. The pre-trained two-dimensional diffusion model used is the open-source Stable Diffusion v1.5.

[0041] S5. 3D Gaussian optimization: Use the 2D editing result obtained in step S4 to optimize the source 3D Gaussian to obtain the final edited 3D Gaussian.

[0042] In this step, the two-dimensional editing result obtained in S4 is used as a data set, and the source three-dimensional Gaussian is trained for a certain number of steps to obtain the edited three-dimensional Gaussian.

[0043] The present invention provides a method for three-dimensional Gaussian editing, which is used to improve the multi-perspective consistency of three-dimensional Gaussian editing, and thus improve the quality of three-dimensional Gaussian editing results. In order to achieve this goal, the present invention designs frequency-space consistency enhancement on the one hand, so that images of different perspectives share the same diffusion style to ensure the style consistency of images of different perspectives when performing two-dimensional editing. On the other hand, it designs semantic-reference consistency alignment in the two-dimensional editing process, and uses semantic information and reference perspective as alignment guidance for editing perspectives to solve the inconsistency problem of images of different perspectives when editing. The schematic diagram of the two consistency enhancements and consistency alignment proposed by the present invention is shown as follows: Figure 2 As shown. Among them: Frequency-Spatial Consistency Enhancement. The Frequency-Spatial Consistency Enhancement module enforces multi-view consistency and improves the quality of 3D Gaussian editing. It performs joint operations in the frequency and spatial domains to enhance the latent code, reduce artifacts, and ensure that different views share a consistent diffusion style.

[0044] In the frequency domain, Gaussian low-pass filter and Gaussian high-pass filter are first used to transform the latent code of the perspective to be edited Convert from spatial domain to frequency domain to obtain low-frequency and high frequency :

[0045]

[0046]

[0047] Secondly, the relative weights of the high-frequency and low-frequency components are adjusted by the weight factor α to create a combined frequency component :

[0048] Finally, apply a mask of the part of the image to be edited , controlling Gaussian noise Add and detail preservation areas to obtain the final output of the frequency domain branch :

[0049] In terms of spatial domain. First, Apply a bilateral filter to calculate for each pixel The filtered output on :

[0050] in, is a normalization factor that ensures the sum of the weights is 1, It's a pixel The local neighborhood around is the standard deviation of the spatial Gaussian kernel. Then, by adding Gaussian noise and applying the mask To adjust ,get :

[0051] Next, Apply position-based attention mechanism to obtain the output of spatial domain branch :

[0052]

[0053] in, It's location The attention weight at It is the central location. is the maximum possible distance from the center. The spatial domain helps to achieve a smoother and more natural appearance in the details of the edited image, significantly improving multi-view consistency and editing quality.

[0054] The fusion of frequency domain and spatial domain. Output the result of frequency domain branch And the spatial domain branch output results Fusion is performed to obtain enhanced latent coding :

[0055] Semantic-reference consistency alignment. By leveraging the reference view and semantic information as alignment guidance for the edit view, consistent feature propagation across different views is ensured. A custom attention mechanism is employed that combines self-attention, reference view-based attention, and semantic-based attention to address multi-view inconsistency.

[0056] First calculate the self-attention output , Attention output based on reference perspective and semantic-based attention output , then and Add together to get the cross attention output :

[0057]

[0058]

[0059]

[0060] in, 、 and is a linear transformation used to obtain query, key and value, 、 and The latent encoding of the editor's perspective is , latent encoding of reference perspective and text instructions The hidden layer of . and After that, the attention output is generated by integrating self-attention and cross-attention :

[0061] in, Used to balance the contributions of self-attention and cross-attention.

[0062] Reference-based attention helps the model maintain consistency between generated content and reference images and textual cues. Self-attention captures dependencies between features within the current hidden state. Semantic-based attention effectively guides the attention mechanism to focus on more semantically relevant features, thereby enhancing the semantic consistency of the generated results. Integrating these components significantly improves editing quality and reduces inconsistencies between different editing perspectives.

[0063] The present invention was tested on editing tasks in multiple complex 3D Gaussian scenes. To verify the effectiveness of the present invention, various scenes from multiple existing datasets were collected for evaluation. Specifically, four 360-degree scenes and two frontal scenes were collected. For each scene, the present invention was evaluated based on multiple text instructions and compared using two evaluation metrics: CLIP text-image orientation similarity and user research. The results are shown in Tables 1 and 2. It can be seen that the present invention can better adapt to 3D Gaussian editing tasks and provide more consistent editing results.

[0064] Table 1 CLIP text-image directional similarity Scene IN2N(GS) ViCA-NeRF GaussianEditor GaussCtrl The present invention Bear 0.0765 0.1104 0.1280 0.1370 0.1692 Dinosaur 0.0590 0.0723 0.0623 0.1609 0.1902 Garden 0.1663 0.2903 0.1509 0.3089 0.2885 Stone horse 0.1390 0.1926 0.1677 0.1581 0.2502 Ark 0.2032 0.1809 0.1887 0.1904 0.2008 Face 0.1115 0.1357 0.1503 0.1498 0.1803 Table 2 User research

[0065] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the scope of protection of the present claims. Other embodiments and combinations of implementations of the present invention are also within the scope of protection of the present invention.

[0066] References 1.Haque, Ayaan, Matthew Tancik, Alexei A. Efros, Aleksander Holynski, and Angjoo Kanazawa. "Instruct-nerf2nerf: Editing 3d scenes withinstructions." In Proceedings of the IEEE / CVF International Conference on Computer Vision , pp. 19740-19750. 2023. 2.Wang, Junjie, Jiemin Fang, Xiaopeng Zhang, Lingxi Xie, and Qi Tian."Gaussianeditor: Editing 3d gaussians delicately with text instructions."In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition , pp. 20902-20911. 2024. 3.Wu, Jing, Jia-Wang Bian, Xinghui Li, Guangrun Wang, Ian Reid,Philip Torr, and Victor Adrian Prisacariu. "Gaussctrl: Multi-view consistenttext-driven 3d gaussian splatting editing." In European Conference on Computer Vision , pp. 55-71. Cham: Springer Nature Switzerland, 2024. 4.Dong, Jiahua, and Yu-Xiong Wang. "Vica-nerf: View-consistency-aware3d editing of neural radiance fields." Advances in Neural Information Processing Systems 36 (2023): 61466-61477. 5.Wang, Yuxuan, Xuanyu Yi, Zike Wu, Na Zhao, Long Chen, and HanwangZhang. "View-consistent 3d editing with gaussian splatting." In European Conference on Computer Vision , pp. 404-420. Cham: Springer NatureSwitzerland, 2024. 6.Mildenhall, Ben, Pratul P. Srinivasan, Matthew Tancik, Jonathan T.Barron, Ravi Ramamoorthi, and Ren Ng. "Nerf: Representing scenes as neuralradiance fields for view synthesis." Communications of the ACM 65, no. 1(2021): 99-106. 7.Kerbl, Bernhard, Georgios Kopanas, Thomas Leimkühler, and GeorgeDrettakis. "3D Gaussian Splatting for Real-Time Radiance Field Rendering." ACM Transactions on Graphics 42, no. 4 (2023): 1-14. 8.Dihlmann, Jan-Niklas, Andreas Engelhardt, and Hendrik Lensch. "Signerf: Scene integrated generation for neural radiance fields."In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition , pp. 6679-6688. 2024. 9.Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusionprobabilistic models." Advances in neural information processing systems 33(2020): 6840-6851. 10.Nichol, Alexander Quinn, and Prafulla Dhariwal. "Improveddenoising diffusion probabilistic models." In International conference on machine learning , pp. 8162-8171. PMLR, 2021. 11.Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoisingdiffusion implicit models." arXiv preprint arXiv:2010.02502 (2020).

Claims

1. A three-dimensional Gaussian editing method, characterized in that: The following steps are involved: S1. Source 3D Gaussian rendering, depth estimation and inversion: By rendering the source 3D Gaussian at different perspectives and estimating the depth, the source 2D image and depth map corresponding to the different perspectives are obtained, and the source 2D image is inverted back to noise; S2. Semantic segmentation of the source 2D image: Using a pre-trained segmentation model, semantically segment the foreground object and background of the source 2D image, obtain corresponding masks, and divide the image into edited and unedited areas; S3. Frequency-space consistency enhancement: by constructing a joint operation of the frequency domain and the spatial domain on the editing area, the consistency of the noise is enhanced; S4. Semantic and reference perspective alignment: 2D editing is performed by inputting the noise after consistency enhancement, the edit text indication, and the depth map obtained in step S1 into a pre-trained 2D diffusion model. During the editing process, the current editing perspective is aligned using semantics and the reference perspective, and the result is combined with the unedited area retained in step S2 to obtain a 2D edited result. S5. Three-dimensional Gaussian optimization: Use the two-dimensional editing result obtained in step S4 to optimize the source three-dimensional Gaussian to obtain the final edited three-dimensional Gaussian.

2. The three-dimensional Gaussian editing method according to claim 1, characterized in that: In step S1, the noise is obtained by inverting an implicit denoising diffusion model in a two-dimensional diffusion domain, and the inversion process uses a source text indication corresponding to a source three-dimensional Gaussian.

3. The three-dimensional Gaussian editing method according to claim 1, wherein: In step S2, the pre-trained semantic segmentation model is the open-source language segmentation network LangSAM.

4. The three-dimensional Gaussian editing method according to claim 1, wherein: In step S3, the frequency domain is processed by adjusting the high-frequency to low-frequency ratio and then adding Gaussian noise; the spatial domain is processed by bilateral filtering and then adding Gaussian noise. The result is obtained by weighting the results of the frequency domain and the spatial domain.

5. The three-dimensional Gaussian editing method according to claim 1, characterized in that: In step S3, In the frequency domain, Gaussian low-pass filter and Gaussian high-pass filter are first used to transform the potential code z of the perspective to be edited into T Convert from spatial domain to frequency domain to obtain low-frequency and high frequency f T = Fourier transform (z T ) Secondly, the relative weights of the high-frequency and low-frequency components are adjusted by the weight factor α to create a combined frequency component Finally, the mask M of the part to be edited in the image is applied to control the Gaussian noise η addition and detail preservation area, thus obtaining the final output of the frequency domain branch In terms of spatial domain, first we need to T Apply a bilateral filter to compute the filtered output at each pixel x Among them, W p is a normalization factor that ensures that the weights sum to 1, Ω is the local neighborhood around pixel x, and σ s is the standard deviation of the spatial Gaussian kernel, and then, is adjusted by adding Gaussian noise and applying the mask M get Next, Apply position-based attention mechanism to obtain the output of spatial domain branch Among them, A(i,j) is the attention weight at position (i,j), (i0,j0) is the center position, and D max is the maximum possible distance from the center; The fusion of frequency domain and spatial domain, the frequency domain branch outputs the result And the spatial domain branch output results Fusion is performed to obtain enhanced latent coding 6. The three-dimensional Gaussian editing method according to claim 1, characterized in that: In step S4, the semantic and reference perspective consistency alignment is performed in the U-Net module and the ControlNet module in the two-dimensional diffusion model denoising process, and the pre-trained two-dimensional diffusion model is the open source released Stable Diffusion v1.

5.

7. The three-dimensional Gaussian editing method according to claim 1, characterized in that: In step S4, First calculate the self-attention output Attn self , Attention output Attn based on reference perspective crossref and semantic-based attention output Attn crossp , then Attn crossref and Attn crossp Add together to get the cross attention output Attn cross : Attn cross =Attn crossref +Attn crcssp Among them, W q (·), W k (·) and W v (·) is the linear transformation used to obtain query, key and value, H edit 、H ref and H p The latent encoding of the editor's perspective is Latent encoding of reference perspective and text indicating p e The hidden layer; after calculating Attn self and Attn cross Finally, the attention output H is generated by integrating self-attention and cross-attention attn : H attn =γ·Attn self +(1-γ)·Attn cross Among them, γ∈[0,1] is used to balance the contribution of self-attention and cross-attention.

8. The three-dimensional Gaussian editing method according to claim 1, characterized in that: In step S5, the two-dimensional editing result obtained in step S4 is used as a data set, and a certain number of training steps are performed on the source three-dimensional Gaussian to obtain the edited three-dimensional Gaussian.

Images