A text-driven three-dimensional gaussian scene editing method without training

Abstract

The application discloses a text-driven three-dimensional Gaussian scene editing method without training, solves the problem that in the prior art, the artifacts such as floaters, ghosting, texture breakage and perspective flicker are introduced in three-dimensional reconstruction, and the authenticity and stability of the editing result are seriously affected, and realizes the editing of a three-dimensional scene which can be rendered in real time, is stable across perspectives and meets text editing instructions; the method comprises the following steps: acquiring multi-perspective images, camera parameters and text editing instructions; generating a candidate set for the first and second perspectives by using a two-dimensional diffusion model; obtaining an initial editing result through cross-perspective consistency screening; generating a geometrically consistent editing image pair by diffusion inversion and polar line constraint denoising; inputting an original image and an editing image into a fusion network, injecting editing semantics into a geometric channel, and obtaining an edited three-dimensional Gaussian set; rendering a new perspective, and outputting an edited scene which meets the instructions and is consistent.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a text-driven 3D Gaussian scene editing method that requires no training. Background Technology

[0002] 3D content generation and editing have significant applications in virtual reality, film and television production, digital twins, robot simulation, and games. Regarding the acquisition, representation, and editable rendering of 3D scenes, industry and academia have proposed various 3D representation methods, such as triangular meshes, voxels, point clouds, and implicit radiation fields. Traditional mesh representations typically require a relatively complete geometry and texture unfolding process. Editing often relies on manual modeling, texture repainting, or complex texture consistency constraints, resulting in high engineering costs and difficulty in directly responding to natural language commands.

[0003] In recent years, implicit representations, represented by Neural Radiance Fields (NeRF), have been able to reconstruct scenes with high quality by optimizing network parameters. However, their training and rendering costs are high, and when performing text-driven editing, each scene often requires long-term iterative optimization. The overall process is sensitive to hyperparameters such as initialization, learning rate, and regularization weights, making it difficult to stably implement in interactive applications.

[0004] 3D Gaussian Splatting (3DGS), as an explicit scene representation, models scenes using a large number of Gaussian primitives with attributes such as position, scale, orientation, transparency, and color. It boasts both high rendering quality and real-time performance and has gained widespread attention in multi-view reconstruction and new viewpoint compositing. However, the editability of 3DGS still faces challenges: on the one hand, Gaussian parameters are strongly coupled with geometry and appearance, and simple modifications to color or transparency can easily cause geometric drift or detail destruction; on the other hand, text instructions typically describe high-level language, such as changes in materials, styles, and attributes. How to stably inject these semantic changes into the 3DGS representation and maintain consistency across viewpoints is a problem that urgently needs to be solved.

[0005] Existing technologies for text-driven 3D generation and editing can be broadly categorized as follows. The first category is scene-by-scene optimization methods, such as introducing text-based guided losses into NeRF or 3DGS, including but not limited to score distillation sampling (SDS) type score distillation, contrastive language-image pre-training (CLIP) guidance, and diffused back gradients followed by iterative updates. While these methods are general, they typically require numerous iterations and high computational power, and are prone to local optima, structural collapse, over-smoothing, or texture noise. Furthermore, when the editing scope is large, they may disrupt the geometric topology and multi-view consistency of the original scene. The second category is learning-based or fine-tuning methods, which train specialized 3D editing networks, adapters, or low-rank adaptation (LoRA) to achieve the mapping from text to 3D appearance. These methods often rely on large-scale paired or synthetic data and have high training costs. Their generalization ability is limited in real-world scenes, complex occlusion, or out-of-distribution data, often requiring additional fine-tuning for the target scene, thus failing to meet the demand for training-free rapid editing. The third category involves multi-view 2D editing followed by reconstruction. This method first decomposes the scene into multi-view images, then performs image editing on each view using a pre-trained 2D diffusion model, and finally inputs the edited multi-view images into a 3D reconstruction pipeline to obtain the 3D result. This approach does not require direct backpropagation in 3D space, making engineering implementation relatively simple, but it has significant drawbacks: 2D diffusion editing is random, and independent editing per view can easily lead to cross-view semantic inconsistencies and structural mismatches; when the view baseline is large or occlusion exists, the generated content from each view is difficult to align geometrically, introducing artifacts such as floating objects, ghosting, texture breaks, and view flickering in 3D reconstruction, severely affecting the realism and stability of the editing results.

[0006] Therefore, there is an urgent need for a text-driven 3D Gaussian scene editing scheme that allows for free training. Without requiring scene-by-scene training or long-term optimization, it can explicitly introduce geometric constraints during the 2D generation stage to obtain consistent editing results across different viewpoints. Furthermore, it can reliably propagate consistent semantic changes to the 3D Gaussian representation, thereby obtaining an editable 3D scene that can be rendered in real time, is stable across different viewpoints, and conforms to text instructions. Summary of the Invention

[0007] This invention provides a text-driven 3D Gaussian scene editing method that does not require training. This method solves the problem in existing technologies where artifacts such as floating objects, ghosting, texture breaks, and viewpoint flicker are introduced in 3D reconstruction, which seriously affect the realism and stability of the editing results. It enables real-time rendering, cross-viewpoint stability, and text-instruction-compliant 3D scene editing.

[0008] This invention provides a text-driven 3D Gaussian scene editing method that requires no training, the method comprising:

[0009] Obtain a set of multi-view input images of the scene to be edited, the camera pose parameters corresponding to the viewpoints, and the text editing instructions input by the user;

[0010] Under the conditions of the text editing instructions, based on the first and second perspectives in the multi-view input image set, a first candidate set and a second candidate set are obtained using a pre-trained two-dimensional diffusion editing model;

[0011] Cross-perspective consistency screening and semantic alignment are performed on the first candidate set and the second candidate set. The comprehensive score of each candidate edit pair is calculated, and the candidate edit pair with the highest comprehensive score is selected as the initial edit result.

[0012] Based on the initial editing results and the corresponding camera pose parameters, the corresponding initial noise vector is obtained through diffusion inversion, and denoising is performed by combining cross-view epipolar constraints to generate edited image pairs with geometric consistency; wherein, the corresponding camera pose parameters include: the camera pose parameters corresponding to the first viewpoint and the camera pose parameters corresponding to the second viewpoint.

[0013] The multi-view input image set and the edited image pair are injected into a cross-view fusion network, and the editing semantics are injected from the editing features into the geometric anchoring path through attention reconfiguration to obtain the edited 3D Gaussian set;

[0014] Based on the three-dimensional Gaussian set, a new perspective rendering or output of three-dimensional results is performed to obtain an editable three-dimensional scene that satisfies the text editing instructions and is consistent across perspectives.

[0015] One or more technical solutions provided in this invention have at least the following technical effects or advantages:

[0016] This invention acquires a multi-view input image set of the scene to be edited, the camera pose parameters corresponding to the viewpoints, and the user-inputted text editing commands. By acquiring multi-view images and camera parameters, it provides a complete scene geometry and appearance benchmark for subsequent editing, enabling the editing process to be carried out under a known spatial structure and avoiding geometric ambiguity caused by missing viewpoints. Simultaneously, the user inputs commands in natural language, lowering the interaction threshold and improving the intuitiveness and convenience of editing. Under the condition of text editing commands, based on the first and second viewpoints in the multi-view input image set, a pre-trained two-dimensional diffusion editing model is used to obtain a first candidate set and a second candidate set. Utilizing the powerful semantic generation capability of the pre-trained two-dimensional diffusion model, multiple candidate editing results conforming to the text commands can be generated without additional training for the current scene, significantly reducing computational overhead. By generating multiple candidates, diversity is provided for subsequent selection, improving the robustness of the final editing effect. Cross-perspective consistency screening and semantic alignment are performed on the first and second candidate sets. The comprehensive score of each candidate editing pair is calculated, and the candidate editing pair with the highest comprehensive score is selected as the initial editing result. Cross-perspective consistency screening is introduced in the two-dimensional generation stage, which effectively suppresses the randomness and inconsistency of independent editing from one perspective and avoids subsequent reconstruction artifacts caused by semantic conflicts between perspectives. By comprehensively scoring the original structure preservation, text semantic conformity and cross-perspective consistency, the high quality of the selected initial editing result in terms of geometry and semantics is ensured. Based on the initial editing results and corresponding camera pose parameters, the initial noise vector is obtained through diffusion inversion, and denoising is performed by combining cross-view epipolar constraints to generate geometrically consistent edited image pairs. The corresponding camera pose parameters include the camera pose parameters corresponding to the first viewpoint and the camera pose parameters corresponding to the second viewpoint. Obtaining consistent initial noise through diffusion inversion makes the subsequent generation process repeatable. Cross-view epipolar constraint attention is introduced in the denoising process, and the geometric correspondence is explicitly embedded into the self-attention mechanism of the diffusion model, which significantly enhances the geometric consistency of the two-view editing results and effectively eliminates artifacts such as viewpoint flicker, ghosting, and floating objects, providing high-quality, cross-viewpoint aligned edited images for 3D reconstruction. A cross-view fusion network is constructed by injecting a multi-view input image set and an edited image pair into the input. Through attention reconfiguration, the editing semantics are injected from the editing features into the geometric anchoring path, resulting in an edited 3D Gaussian set. The cross-view injection fusion network decouples the editing semantics from the original geometric structure through an attention mechanism in the geometric anchoring path. This ensures that the edited 3D Gaussian representation can faithfully reflect the semantic changes required by the text instruction while maximizing the preservation of the geometric details and structural stability of the original scene. This feedforward generation method avoids scene-by-scene iterative optimization and enables rapid editing with flexible training.Based on a 3D Gaussian set, new perspective rendering or output of 3D results is performed to obtain editable 3D scenes that satisfy text editing instructions and are consistent across different perspectives; a 3D Gaussian splash representation that can be rendered in real time is directly output, supporting instant rendering of any new perspective to meet the needs of interactive applications; finally, the edited scene maintains geometric and appearance consistency under multiple perspectives, ensuring visual realism and stability. Attached Figure Description

[0017] Figure 1 A flowchart illustrating the steps of a text-driven 3D Gaussian scene editing method without training, as provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0019] This invention provides a text-driven 3D Gaussian scene editing method that requires no training; see [link to relevant documentation]. Figure 1 The method includes the following steps S101 to S106.

[0020] S101, obtain the multi-view input image set of the scene to be edited, the camera pose parameters corresponding to the viewpoints, and the text editing instructions input by the user; here, the multi-view input image set is represented as: Text editing commands are represented as: .

[0021] For example, for each perspective Acquire image Camera intrinsic parameter matrix External reference This allows us to obtain a set of multi-view input images of the scene to be edited. Secondly, obtain text editing instructions. And determine the number of diffusion steps. First candidate set With the second candidate set number of candidates A set of hyperparameters such as editing intensity.

[0022] S102, in text editing instructions Under the condition of multi-view input image set Using the first and second perspectives in the model, a pre-trained two-dimensional diffusion editing model is used to obtain the first candidate set and the second candidate set.

[0023] Here, the first candidate set is denoted as The second candidate set is represented as The set of multi-view input images is represented as: Text editing commands are represented as: .

[0024] Specifically, in step S102, based on the multi-view input image set Using the first and second perspectives in the model, a pre-trained two-dimensional diffusion editing model is used to obtain the first candidate set. With the second candidate set The steps include S1021 to S1023.

[0025] S1021, Text editing instructions Edit as text embed ;

[0026] S1022, Random sampling is performed on the first and second viewpoints respectively to obtain multiple sets of noise. and multiple groups of noise As the initial input to the pre-trained two-dimensional diffusion editing model;

[0027] S1023 utilizes a pre-trained two-dimensional diffusion editing model to provide text editing instructions. Perform image-to-image diffusion editing to obtain multiple candidate edit images corresponding to each key viewpoint. This leads to the first candidate set. With the second candidate set .

[0028] For example, the image corresponding to the first-person perspective Image corresponding to the second perspective Text editing instructions Encoded as text embedding Text embedding Represented as:

[0029] (1.1);

[0030] in, For text encoders.

[0031] First-person and second-person perspectives Random sampling Group noise And perform image-to-image diffusion editing to obtain candidates:

[0032] (1.2);

[0033] in, For diffusion editing operations; Indicates perspective The original image; Indicates text embedding; express Group noise; This represents the set of parameters that guide the diffusion editing process; Indicates the number of candidates.

[0034] This leads to the first candidate set. With the second candidate set .

[0035] S103, perform cross-perspective consistency screening and semantic alignment on the first and second candidate sets, calculate the comprehensive score of each candidate edit pair, and select the candidate edit pair with the highest comprehensive score as the initial editing result; here, the first candidate set is represented as... The second candidate set is represented as Each candidate editor is represented as The overall score is expressed as follows: Candidate editors are represented as .

[0036] Specifically, in step S103, the first candidate set is... With the second candidate set Perform cross-perspective consistency screening and semantic alignment, and calculate the edit pairs for each candidate. Overall rating The steps include S1031 to S1037.

[0037] S1031, the first candidate set With the second candidate set The corresponding element combination forms multiple candidate edit pairs ;

[0038] S1032, respectively, for each candidate editor The first image block feature map and the second image block feature map are extracted from each candidate image in the dataset.

[0039] S1033, calculate the first average structural similarity between the first candidate image and the input image corresponding to the first view in the multi-view input image set, and the second average structural similarity between the second candidate image and the input image corresponding to the second view in the multi-view input image set.

[0040] S1034, Calculate the first cosine similarity between the first candidate image and the text editing instruction, and the second cosine similarity between the second candidate image and the text editing instruction;

[0041] S1035, based on the camera pose parameters corresponding to the first and second viewpoints, calculates the fundamental matrix between the first and second viewpoints. ;

[0042] S1036, for each first image block in the feature map of the first image block, according to the fundamental matrix... Calculate the corresponding epipolar line of the first image patch in the second viewpoint;

[0043] S1037, Based on a preset bandwidth threshold, in the feature map of the second image block, determine the image block whose distance from the corresponding epipolar line is less than the bandwidth threshold and record it as the second image block, and form the second image block into a matching candidate set. ;

[0044] S1038, Calculate the first image block in the feature map of the first image block and the matching candidate set. The feature similarity between the second image patches is used to select the maximum similarity as the cross-view consistency score of the first image patch. ;

[0045] S1039, Cross-view consistency score for all first image patches Calculate the average to obtain the average cross-perspective consistency score;

[0046] S10310, the average structural similarity of the first perspective, the average structural similarity of the second perspective, the cosine similarity of the first perspective, the cosine similarity of the second perspective, and the average cross-perspective consistency score are weighted and summed to obtain the candidate editor's... Overall rating Here, the overall score is... Represented as:

[0047] (1.3);

[0048] in, This represents the average structural similarity weight from the first-person perspective. It is a constant; This indicates the calculation of average structural similarity; Represents the first candidate set The Middle 1 candidate image; This represents the input image corresponding to the first viewpoint in a set of multi-view input images; Indicates the first average structural similarity; The structural similarity weights represent the second-person perspective. It is a constant; Represents the second candidate set The Middle 1 candidate image; This represents the input image corresponding to the second perspective in a set of multi-view input images; Indicates the second average structural similarity; The cosine similarity weight represents the first-person perspective. It is a constant; This indicates the calculation of cosine similarity; Indicates the first cosine similarity; The cosine similarity weights represent the second-person perspectives. It is a constant; Indicates the second cosine similarity; This indicates text editing instructions processed by a text encoder; Indicates the cross-perspective consistency score weight. It is a constant; This indicates the cross-perspective consistency score.

[0049] here, = =0.1; = =0.25; =0.3; , ; , ;in, Indicates an image encoder; This indicates a text encoder.

[0050] For example, candidate edit images from first and second perspectives Extracting image patch feature maps yields a first image patch feature map and a second image patch feature map. The first and second image patch feature maps are represented as follows:

[0051] (1.4);

[0052] in, For the feature extraction network, diffused U-Net visual backbone features are used here; Represents the first candidate set The Middle 1 candidate image; Represents the second candidate set The Middle 1 candidate image; Represents the feature map of the first image block; This represents the feature map of the second image block.

[0053] Perform the following steps on the feature maps of the first and second image blocks: Normalization:

[0054] (1.5);

[0055] in, This represents the feature map of the first image block or the feature map of the second image block; This represents the normalized feature map of the first or second image patch, i.e. This is the transformation function.

[0056] Next, based on the camera pose parameters corresponding to the first and second viewpoints, the fundamental matrix between the first and second viewpoints is calculated. .

[0057] First-person perspective The Middle Homogeneous coordinates of the center pixel of each patch From a second perspective Corresponding polar lines for:

[0058] (1.6);

[0059] in, This represents the first component of the polar line; This represents the second component of the polar line; This represents the third component of the polar line; This indicates the transpose; Indicates the first The center pixel of each patch is in the second perspective. The corresponding polar line.

[0060] Second perspective The Middle The center pixel homogeneous coordinates of each patch Define its distance to the polar line:

[0061] (1.7);

[0062] in, Indicates second perspective The Middle Homogeneous coordinates of the center pixel of each patch; First-person perspective The Middle The homogeneous coordinates of the center pixel of each patch in the second viewpoint The corresponding polar lines; This represents the square of the first component value of the polar line; This represents the square of the second component value of the polar line.

[0063] Preset bandwidth threshold in the polar neighborhood Internally, construct a matching candidate set:

[0064] (1.8);

[0065] Cross-view consistency for each patch is defined using the "maximum cosine similarity within the epipolar neighborhood":

[0066] (1.9);

[0067] in, Indicates perspective Transformation function; Indicates perspective Transformation function;

[0068] For the sampled patch set The average cross-perspective consistency score is obtained by averaging:

[0069] (1.10);

[0070] The overall score is expressed as formula (1.3), thus obtaining the initial editing result. .

[0071] S104, based on the initial editing results and corresponding camera pose parameters, the corresponding initial noise vector is obtained through diffusion inversion, and denoising is performed by combining cross-view epipolar constraints to generate geometrically consistent edited image pairs; wherein, the corresponding camera pose parameters include: the camera pose parameters corresponding to the first viewpoint and the camera pose parameters corresponding to the second viewpoint; here, the geometrically consistent edited image pairs are represented as: .

[0072] Specifically, in step S104, based on the noise scheduling of the forward diffusion process in the pre-trained two-dimensional diffusion editing model, deterministic mapping is performed on the two images in the initial editing result, transforming each image from the image space to the noise space to obtain the initial noise vector corresponding to each image.

[0073] Specifically, in step S104, denoising is performed by combining cross-view epipolar constraints to obtain edited image pairs with geometric consistency. The steps include S1041 to S1043.

[0074] S1041, based on the camera pose parameters corresponding to the first and second viewpoints, calculate the fundamental matrix between the first and second viewpoints. ;

[0075] S1042, in the denoising generation process of the pre-trained diffusion model, a self-attention computation layer based on the fundamental matrix is ​​introduced into the cross-view self-attention computation layer. The constructed epipolar attention constraint mask yields cross-view epipolar attention;

[0076] Specifically, in step S104, during the denoising generation process of the pre-trained diffusion model, a matrix-based approach is introduced into the cross-view self-attention computation layer. The constructed epipolar attention constraint mask yields cross-view epipolar attention, including:

[0077] (1) Obtain the pixel coordinates of the query token in the image corresponding to the first view and the pixel coordinates of the key token in the image corresponding to the second view;

[0078] (2) Based on the fundamental matrix By using the pixel coordinates of the query token, the corresponding epipolar line in the image corresponding to the second viewpoint can be calculated;

[0079] (3) Calculate the distance from the pixel coordinates of the key token to the corresponding epipolar line, and construct the epipolar attention constraint mask based on the comparison result between the distance and the preset bandwidth threshold.

[0080] (4) Introduce the epipolar attention constraint mask into the self-attention computation layer of the pre-trained diffusion model to obtain cross-view epipolar attention.

[0081] S1043, the initial noise vector is input into a diffusion model including cross-view epipolar attention, and denoising generation is performed iteratively across multiple denoising time steps, outputting edited image pairs with geometric consistency. .

[0082] Here, the initial noise vector is input into a diffusion model that includes cross-view epipolar attention, and denoising generation is performed iteratively across multiple denoising time steps, outputting edited image pairs with geometric consistency. ,include:

[0083] (1) The initial noise vector is used as the starting point for denoising, and the diffusion model including cross-view epipolar attention is input.

[0084] (2) In multiple preset denoising time steps, denoising operations including cross-view epipolar attention are executed sequentially to gradually update the noise vector and obtain the updated noise vector.

[0085] (3) After completing all denoising time steps, the updated noise vector is converted into the image space through the sampler, and the edited image pair with geometric consistency is output. .

[0086] For example, the initial edited image ( ),Right now Perform diffusion inversion or noise coding to obtain the initial noise vector. .

[0087] Taking the forward diffusion noise addition process as an example:

[0088] (1.11)

[0089] in, For initial image editing The latent variables are represented as follows: This is the noise dispatch coefficient; Indicates a Gaussian distribution; This represents random noise sampled from the Gaussian distribution.

[0090] The initialization noise for regeneration is obtained through the inversion process. ,here, The initialization noise corresponding to the first-person perspective, Regarding the initialization noise corresponding to the second perspective, this invention does not limit the inversion method; for example, DDIM inversion or equivalent methods can be used.

[0091] Furthermore, the essential and fundamental matrices are calculated based on the intrinsic and extrinsic parameters of the key viewpoint camera. Let the relative pose be:

[0092] (1.12);

[0093] but:

[0094] (1.13);

[0095] (1.14);

[0096] in, Representing vectors The antisymmetric matrix; Indicates perspective rotation matrix; Indicates perspective rotation matrix; Indicates perspective The translation vector; Indicates perspective The translation vector; Represents the transformed vector; Indicates perspective The intrinsic parameter matrix; Indicates perspective The intrinsic parameter matrix;

[0097] In the diffusion denoising process, a cross-view self-attention computation layer is injected to form CVEA. For the first viewpoint... The corresponding pixel coordinates of the token in the image. Second perspective The pixel coordinates of the key token in the corresponding image ,definition:

[0098] (1.15);

[0099] in, First-person perspective The corresponding pixel coordinates of the token in the image; Indicates perspective Query the image coordinates of the token After transformation matrix Then, mapped to Image coordinates corresponding to the viewpoint; Indicates second perspective The pixel coordinates of the key token in the corresponding image; This represents the square of the first component value of the polar line; This represents the square of the second component value of the polar line; Indicates the first and the The distance metric between query tokens;

[0100] Constructing an epipolar attention constraint mask:

[0101] (1.16);

[0102] Standard attention is:

[0103] (1.17);

[0104] in, Represents the query matrix; Represents the key matrix; Represents a value matrix; This indicates the dimensions of the query and the key.

[0105] Introducing a mask to obtain cross-view epipolar attention:

[0106] (1.18);

[0107] in, Indicates perspective The query matrix; Indicates perspective The key matrix; Indicates perspective The value matrix; Indicates a line attention constraint mask;

[0108] By repeatedly injecting CVEA at multiple diffusion time steps and multiple attention layers, and using shared time step scheduling for regeneration, edited image pairs with strong geometric consistency are output. .

[0109] S105, the multi-view input image set and the edited image pair are injected into the cross-view fusion network. Through attention reconfiguration, the editing semantics are injected from the editing features into the geometric anchoring pathway, resulting in the edited 3D Gaussian set. Here, the multi-view input image set is represented as: Editing image pairs is represented as: The 3D Gaussian set is represented as: .

[0110] Specifically, in step S105, the multi-view input image set is... Editing Image Pair The input is injected into a cross-view fusion network, and the editing semantics are injected from the editing features into the geometric anchoring pathway through attention reconfiguration, resulting in an editable 3D Gaussian set. The steps include S1051 to S1053.

[0111] S1051, for multi-view input image set Multi-scale features are extracted to obtain the original geometric features, and the edited image pairs are then processed. Extract multi-scale features to obtain editing semantic features;

[0112] S1052, the original geometric features are used as queries, and the edited semantic features are used as keys and values. The input is given to the attention reconfiguration module in the cross-view injection fusion network to enable the edited semantic injection geometric anchoring path and obtain the fused features.

[0113] S1053, input the fused features into the 3D Gaussian regression head, perform regression processing, and obtain the edited 3D Gaussian set; where each Gaussian element in the 3D Gaussian set includes a center parameter, a scale parameter, a rotation parameter, an opacity parameter, and a color parameter.

[0114] For example, for a multi-view input image set Multi-scale features are extracted to obtain the original geometric features, and the edited image pairs are then processed. Extracting multi-scale features yields edited semantic features and original geometric features: Edit semantic features: ;in, Indicates scale level. This is a feature extraction network.

[0115] Furthermore, semantic injection is performed, with the original geometric feature being the Query, i.e. Edit the semantic features as Key and Value, that is and :

[0116] (1.19);

[0117] (1.20);

[0118] in, The weight matrix representing the query; This represents the weight matrix of the Key; The weight matrix representing the Value; The dimension of the vector representing the query and key.

[0119] Furthermore, CVIF-Net completes cross-view information fusion under the geometric anchoring path and outputs the edited 3D Gaussian set predicted by the 3D Gaussian parameter network, which can be represented as:

[0120] (1.21);

[0121] in, Centered on Gauss, For scale parameters, For rotation parameters, For opacity, For color parameters.

[0122] S106, based on a 3D Gaussian set, renders or outputs a 3D result from a new perspective, obtaining an editable 3D scene that satisfies text editing instructions and is consistent across different perspectives. Here, the 3D Gaussian set is represented as: Text editing commands are represented as: .

[0123] Specifically, in step S106, based on the three-dimensional Gaussian set... To render from a new perspective or output 3D results that satisfy text editing instructions. Furthermore, editing a 3D scene with consistent perspective across different viewpoints includes the following steps S1061 to S1063.

[0124] S1061, Obtain the target camera pose parameters;

[0125] S1062, 3D Gaussian set Input the 3D Gaussian renderer, perform rendering operations under the target camera pose parameters, and generate a rendered image from a new perspective;

[0126] S1063, 3D Gaussian set The rendered image from the new perspective is used as output to obtain text editing instructions. Furthermore, it allows for consistent editing of 3D scenes across different perspectives.

[0127] For example, based on To render or output 3D results from a new perspective. Input a 3D Gaussian renderer to generate a new perspective rendering result under the target camera pose, and output the edited 3DGS representation and / or its rendered image sequence, thereby obtaining a result that satisfies the text editing instructions. Furthermore, it allows for consistent editing of 3D scenes across different perspectives.

[0128] In a simulation provided by this invention, the average performance (Avg-acc) of multiple experiments was calculated. The evaluation results are shown in Table 1. In the following table, CLIP Similarity represents the similarity measure between the image and text calculated based on the CLIP model; PSI represents the structural similarity score; Avg.Editing time represents the average time spent in the editing process; GaussianEditor represents a 3D Gaussian editing method; DGE represents a 3D Gaussian editing method; GaussCtrl represents a 3D Gaussian editing method; IP2P+MVSplat represents direct use of 2D editing and 3D reconstruction; TF3DEdit(IP2P+MVSplat) represents the method of this invention, in which the 3D reconstruction part uses the MVSplat technology; TF3DEdit stands for "Training-Free Text-Driven 3D Editing". This scheme adopts the cross-view consistency screening, epipolar constraint denoising generation, and cross-view injection fusion network proposed in this invention; TF3DEdit(IP2P +DepthSplat) represents the method of this invention, in which the 3D reconstruction part uses the DepthSplat technology.

[0129] Table 1 Comparison of results between the method of the present invention and existing methods

[0130]

[0131] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this invention can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0132] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.

Claims

1. A text-driven 3D Gaussian scene editing method without training, characterized in that, include: Obtain a set of multi-view input images of the scene to be edited, the camera pose parameters corresponding to the viewpoints, and the text editing instructions input by the user; Under the conditions of the text editing instructions, based on the first and second perspectives in the multi-view input image set, a first candidate set and a second candidate set are obtained using a pre-trained two-dimensional diffusion editing model; wherein, obtaining the first candidate set and the second candidate set based on the first and second perspectives in the multi-view input image set using a pre-trained two-dimensional diffusion editing model includes: editing the text editing instructions into text embeddings; randomly sampling the first and second perspectives respectively to obtain multiple sets of noise, and using the multiple sets of noise as the initial input of the pre-trained two-dimensional diffusion editing model; using the pre-trained two-dimensional diffusion editing model to perform image-to-image diffusion editing with the text editing instructions to obtain multiple candidate edited images corresponding to each key perspective, thereby obtaining the first candidate set and the second candidate set; The corresponding elements of the first candidate set and the second candidate set are combined into multiple candidate edit pairs. Cross-perspective consistency screening and semantic alignment are performed on the first candidate set and the second candidate set. The comprehensive score of each candidate edit pair is calculated, and the candidate edit pair with the highest comprehensive score is selected as the initial editing result. Based on the initial editing results and the corresponding camera pose parameters, the corresponding initial noise vector is obtained through diffusion inversion, and denoising is performed by combining cross-view epipolar constraints to generate edited image pairs with geometric consistency; wherein, the corresponding camera pose parameters include: the camera pose parameters corresponding to the first viewpoint and the camera pose parameters corresponding to the second viewpoint. The multi-view input image set and the edited image pair are injected into a cross-view fusion network, and the editing semantics are injected from the editing features into the geometric anchoring path through attention reconfiguration to obtain the edited 3D Gaussian set; Based on the three-dimensional Gaussian set, a new perspective rendering or output of three-dimensional results is performed to obtain an editable three-dimensional scene that satisfies the text editing instructions and is consistent across perspectives.

2. The text-driven 3D Gaussian scene editing method without training according to claim 1, characterized in that, The process of performing cross-perspective consistency screening and semantic alignment on the first and second candidate sets, and calculating the comprehensive score for each candidate edit pair, includes: The first image block feature map and the second image block feature map are extracted from each candidate image in each candidate edit pair; Calculate the first average structural similarity between the first candidate image and the input image corresponding to the first viewpoint in the multi-view input image set, and the second average structural similarity between the second candidate image and the input image corresponding to the second viewpoint in the multi-view input image set; Calculate the first cosine similarity between the first candidate image and the text editing instruction, and the second cosine similarity between the second candidate image and the text editing instruction; Based on the camera pose parameters corresponding to the first viewpoint and the second viewpoint, the fundamental matrix between the first viewpoint and the second viewpoint is calculated; For each first image block in the feature map of the first image block, the corresponding epipolar line of the first image block in the second view is calculated according to the fundamental matrix; Based on a preset bandwidth threshold, image blocks whose distance from the corresponding epipolar line is less than the bandwidth threshold are identified in the feature map of the second image block and recorded as the second image block, and the second image blocks are used to form a matching candidate set; Calculate the feature similarity between the first image block in the feature map of the first image block and the second image block in the matching candidate set, and select the maximum similarity as the cross-view consistency score of the first image block; The average cross-view consistency score is obtained by averaging the cross-view consistency scores of all the first image patches. The average structural similarity of the first perspective, the average structural similarity of the second perspective, the cosine similarity of the first perspective, the cosine similarity of the second perspective, and the average cross-perspective consistency score are weighted and summed to obtain the comprehensive score of the candidate edit pair.

3. The text-driven 3D Gaussian scene editing method without training according to claim 1, characterized in that, The comprehensive score is expressed as follows: ； in, This represents the average structural similarity weight from the first-person perspective. It is a constant; This indicates the calculation of average structural similarity; Represents the first candidate set The Middle 1 candidate image; This represents the input image corresponding to the first viewpoint in a set of multi-view input images; Indicates the first average structural similarity; The structural similarity weights represent the second-perspective structural similarity. It is a constant; Represents the second candidate set The Middle 1 candidate image; This refers to the input image corresponding to the second perspective in the multi-view input image set; Indicates the second average structural similarity; The cosine similarity weight represents the first-person perspective. It is a constant; This indicates the calculation of cosine similarity; Indicates the first cosine similarity; The cosine similarity weights represent the second-person perspectives. It is a constant; Indicates the second cosine similarity; This indicates text editing instructions processed by a text encoder; Indicates the cross-perspective consistency score weight. It is a constant; This indicates the cross-perspective consistency score.

4. The text-driven 3D Gaussian scene editing method without training according to claim 1, characterized in that, The step of obtaining the corresponding initial noise vector through diffusion inversion based on the initial editing result and the corresponding camera pose parameters includes: Based on the noise scheduling of the forward diffusion process in the pre-trained two-dimensional diffusion editing model, deterministic mapping is performed on the two images in the initial editing result, transforming each image from the image space to the noise space to obtain the initial noise vector corresponding to each image.

5. The text-driven 3D Gaussian scene editing method without training according to claim 1, characterized in that, The denoising generation process, which combines cross-view epipolar constraints, yields edited image pairs with geometric consistency, including: Based on the camera pose parameters corresponding to the first viewpoint and the second viewpoint, the fundamental matrix between the first viewpoint and the second viewpoint is calculated; In the denoising generation process of the pre-trained diffusion model, an epipolar attention constraint mask based on the aforementioned fundamental matrix is ​​introduced into the cross-view self-attention computation layer to obtain cross-view epipolar attention. The initial noise vector is input into a diffusion model that includes the cross-view epipolar attention, and denoising generation is performed iteratively in multiple denoising time steps to output the edited image pair with geometric consistency.

6. The text-driven 3D Gaussian scene editing method without training according to claim 5, characterized in that, In the denoising generation process of the pre-trained diffusion model, an epipolar attention constraint mask based on the fundamental matrix is ​​introduced into the cross-view self-attention computation layer to obtain cross-view epipolar attention, including: Obtain the pixel coordinates of the query token in the image corresponding to the first viewpoint, and the pixel coordinates of the key token in the image corresponding to the second viewpoint; Based on the fundamental matrix Using the pixel coordinates of the query token, the corresponding epipolar line in the image corresponding to the second viewpoint is calculated; Calculate the distance from the pixel coordinates of the key token to the corresponding epipolar line, and construct an epipolar attention constraint mask based on the comparison result of the distance and a preset bandwidth threshold. The epipolar attention constraint mask is introduced into the self-attention computation layer of the pre-trained diffusion model to obtain the cross-view epipolar attention.

7. The text-driven 3D Gaussian scene editing method without training according to claim 5, characterized in that, The step of inputting the initial noise vector into a diffusion model including the cross-view epipolar attention, iteratively performing denoising generation in multiple denoising time steps, and outputting edited image pairs with geometric consistency includes: The initial noise vector is used as the starting point for denoising, and the diffusion model including the cross-view epipolar attention is input. In multiple preset denoising time steps, the denoising operation including the cross-view epipolar attention is executed sequentially to gradually update the noise vector and obtain the updated noise vector. After all denoising time steps are completed, the updated noise vector is converted into the image space by the sampler, and the geometrically consistent edited image pair is output.

8. The text-driven 3D Gaussian scene editing method without training according to claim 1, characterized in that, The process of injecting the multi-view input image set and the edited image pair into the input cross-view fusion network, and injecting the editing semantics from the editing features into the geometric anchoring pathway through attention reconfiguration, to obtain the edited 3D Gaussian set, includes: Multi-scale features are extracted from the multi-view input image set to obtain the original geometric features, and multi-scale features are extracted from the edited image pair to obtain the edited semantic features; The original geometric features are used as queries, and the edited semantic features are used as keys and values. These are then input into the attention reconfiguration module in the cross-view injection fusion network to enable the edited semantic injection geometric anchoring path and obtain the fused features. The fused features are input into a 3D Gaussian regression head for regression processing to obtain an edited 3D Gaussian set; wherein each Gaussian element in the 3D Gaussian set includes a center parameter, a scale parameter, a rotation parameter, an opacity parameter, and a color parameter.

9. The text-driven 3D Gaussian scene editing method without training according to claim 1, characterized in that, The process of rendering or outputting 3D results from a new perspective based on the 3D Gaussian set to obtain an editable 3D scene that satisfies the text editing instructions and is consistent across different perspectives includes: Obtain the target camera pose parameters; The three-dimensional Gaussian set is input into the three-dimensional Gaussian renderer, and a rendering operation is performed under the target camera pose parameters to generate a rendered image from a new perspective. The three-dimensional Gaussian set and the rendered image from the new perspective are used as outputs to obtain an editable three-dimensional scene that satisfies the text editing instructions and is consistent across different perspectives.

Images