Point-based three-dimensional Gaussian scene dragging and editing method and device and readable medium

Abstract

The invention discloses a point-based three-dimensional Gaussian scene dragging and editing method and device and a readable medium, and relates to the field of three-dimensional Gaussian process.The method comprises the steps that a Gaussian center coordinate is input into a three-dimensional Gaussian scene dragging and editing model, and the three-dimensional Gaussian scene dragging and editing model sequentially passes through a multi-resolution three-plane encoder and an area specific position decoder; obtaining an offset of a Gaussian center coordinate, constructing a deformation loss function and a deformation Gaussian scene, and performing rendering by adopting a second renderer to obtain an RGB rendering image; initializing the Gaussian scene to obtain an initial Gaussian scene and rendering the initial Gaussian scene to obtain an initial RGB rendering image; and carrying out fractional distillation guide by adopting the drag-based potential diffusion model to construct a distillation loss function, constructing a total loss function according to the deformation loss function and the distillation loss function, and training the three-dimensional Gaussian scene drag editing model based on the total loss function to obtain a trained three-dimensional Gaussian scene drag editing model. According to the method, the problem of editing the reconstructed three-dimensional Gaussian scene is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional Gaussian processing, and in particular to a point-based three-dimensional Gaussian scene dragging editing method and device and readable medium. BACKGROUND

[0002] Representation and processing of three-dimensional scenes have become increasingly important in multiple fields such as virtual reality (VR) and augmented reality (AR). Traditional three-dimensional representation methods, such as meshes, voxels, and point clouds, have facilitated many technological advancements but often face challenges in scalability, efficiency, and expressiveness. The development of neural scene representations, such as Neural Radiance Fields (NeRF), has demonstrated remarkable capabilities in synthesizing photorealistic new views. However, NeRF-related methods rely on a heavy sampling process and are computationally intensive, making them less suitable for interactive editing tasks. Recently, three-dimensional Gaussian Splatting (3DGS) has attracted widespread attention due to its ability to represent volumetric data using sparse Gaussian primitives. By replacing the dense neural network with lightweight and interpretable Gaussian primitives, 3DGS enables real-time rendering and fast updates, making it a promising three-dimensional representation method for three-dimensional editing. Therefore, expressing three-dimensional scenes through three-dimensional Gaussian primitives for three-dimensional reconstruction and generation has become a research hotspot. Compared to NeRF, 3DGS has obvious advantages in reconstruction quality and speed. However, how to edit the reconstructed three-dimensional scene has become a problem.

[0003] To address these limitations of geometric editing, some methods provide precise control and intuitive image editing capabilities by using pairs of control points (including control points and target points). However, applying two-dimensional drag-based generation models to guide 3DGS optimization for drag-based three-dimensional editing presents a new challenge: target regions often exhibit sparse distribution of three-dimensional Gaussian points, making it challenging to effectively edit three-dimensional Gaussian fields. As a result, the model tends to align the textures of nearby three-dimensional Gaussians around the target region rather than accurately generating the desired geometric structure. This issue significantly affects the accuracy and realism of the editing results.

[0004] Recent three-dimensional Gaussian scene editing work mainly focuses on text instruction editing, which edits the existing three-dimensional Gaussian scene through the given text prompt words. These methods mainly focus on texture and style color editing, and have obvious limitations in geometry structure editing. Some image editing methods edit images through control points. Therefore, we will extend the point-based editing method from two-dimensional images to three-dimensional Gaussian scenes to achieve more friendly geometry editing effects. The patents with application publication numbers CN119417972A and CN119006760A can realize the editing of three-dimensional Gaussian scenes, but both are limited to editing the texture style and other colors, limiting the changes of scene geometry, and are difficult to adapt to the needs of three-dimensional geometry scene editing. The geometry structure in three-dimensional space is complex and difficult to edit. SUMMARY

[0005] The purpose of the present application is to propose a point-based three-dimensional Gaussian scene drag editing method, device and readable medium for the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a point-based three-dimensional Gaussian scene drag editing method, comprising the following steps:

[0007] A three-dimensional Gaussian scene drag editing model, a first renderer and a drag-based latent diffusion model are constructed. The three-dimensional Gaussian scene drag editing model includes a multi-resolution tri-plane encoder, a region-specific position decoder and a second renderer connected in turn;

[0008] The center coordinates of one of the Gaussians in the Gaussian scene in the training data are input into the three-dimensional Gaussian scene drag editing model, and the center coordinates of the Gaussian are obtained by sequentially passing through the multi-resolution tri-plane encoder and the region-specific position decoder. The deformation loss function is constructed based on the center coordinates of the Gaussian in the third subset obtained by soft local editing. The center coordinates of the Gaussian are added to the corresponding center coordinates of the Gaussian to obtain the corresponding deformed Gaussian and construct the deformed Gaussian scene. A camera pose c' is randomly sampled from the reconstruction view angle, and the second renderer is used to render the deformed Gaussian scene to obtain an RGB rendering image. The Gaussian scene in the training data is initialized to obtain the corresponding initial Gaussian scene, and the first renderer is used to render the initial Gaussian scene under the camera pose c' to obtain an initial RGB rendering image. The editing region mask and the three-dimensional editing control point in the training data are processed respectively to obtain a two-dimensional mask image and a two-dimensional control point. The score distillation guide is constructed using the drag-based latent diffusion model based on the initial RGB rendering image, the RGB rendering image, the two-dimensional mask image and the two-dimensional control point to construct a distillation loss function. The total loss function is constructed based on the deformation loss function and the distillation loss function, and the three-dimensional Gaussian scene drag editing model is trained based on the total loss function to obtain the trained three-dimensional Gaussian scene drag editing model.

[0009] The Gaussian scene to be edited is obtained and input into the trained three-dimensional Gaussian scene drag-editing model to obtain a corresponding rendering image.

[0010] As a preference, the i-th Gaussian in the Gaussian scene is represented as g i ={p i ,f i ,o i ,s i ,r i ,m i}, wherein p i represents the center coordinates of the i-th Gaussian g i , f i represents the spherical harmonic color of the i-th Gaussian g i , o i represents the opacity of the i-th Gaussian g i , s i represents the scaling of the i-th Gaussian g i , r i represents the rotation of the i-th Gaussian g i , and m i represents the mask of the i-th Gaussian g i ; m i =1 indicates that the i-th Gaussian g i is expected to be edited, and m i =0 indicates that the i-th Gaussian g i is not expected to be edited; the Gaussian scene is represented as

[0011] The Gaussian scene g is divided into three groups of subsets, namely a first subset g , a second subset g , and a third subset g , wherein the first subset g is a set of Gaussians with m i =1; the set of Gaussians with m i =0 is further divided into a second subset g and a third subset g , i.e. Different learning rates are set for different subsets to perform soft local editing;

[0012] The multi-resolution three-plane encoder is three learnable orthogonal planes with different resolutions For the center coordinates p=(x, y, z) of the Gaussian scene, first, normalization is performed to obtain normalized coordinates Then, bilinear interpolation is performed on the different resolution orthogonal planes, as shown in the following formula:

[0013]

[0014] wherein π c denotes the projection of the point p' to the orthogonal plane ψ s denotes the bilinear interpolation to the resolution s, c denotes the index of the orthogonal plane, denotes the orthogonal plane and the resolution s;

[0015] The hybrid scale feature is calculated for different orthogonal planes and resolutions of the feature, as shown in the following formula:

[0016]

[0017] wherein Π denotes the hadamard product, C = {xy, xz, yz} denotes the set of indices of the orthogonal plane, Θ denotes the first multi-layer perception, concat denotes the tensor concatenation operation, f con denotes the hybrid scale feature;

[0018] The hybrid scale feature is decoded using the region-specific position decoder to obtain the offset of the center coordinates of the Gaussian scene, as shown in the following formula:

[0019]

[0020] wherein and denote the second multi-layer perception and the third multi-layer perception, respectively, sg denotes the gradient stop operation, and Δp denotes the offset of the center coordinates of the Gaussian scene.

[0021] One of the deformed Gaussians in the deformed Gaussian scene is denoted as g' i = {Δp i +p i , f i , o i , s i , r i , m i}.

[0022] As a preferred, the region-specific position decoder comprises the second multi-layer perception and the third multi-layer perception, and the first multi-layer perception, the second multi-layer perception and the third multi-layer perception comprise a first fully connected layer, a ReLU activation function layer and a second fully connected layer connected in sequence.

[0023] As a preferred, the editing region mask and the three-dimensional editing control point in the training data are processed respectively to obtain a two-dimensional mask graph and a two-dimensional control point, specifically comprising:

[0024] The edit region mask in the training data is rendered by using a first renderer to obtain a rendered edit region mask image; an inflation operation is performed on the rendered edit region mask image to obtain a two-dimensional mask image, as shown in the following formula:

[0025] I dilate = dilate(I, kernel, iteration);

[0026] Wherein, I represents the rendered edit region mask image, kernel represents the convolution kernel of the inflation operation, iteration represents the iteration number of the inflation operation, I dilate represents the two-dimensional mask image;

[0027] The three-dimensional edit control points in the training data are projected to the camera pose c' to obtain two-dimensional control points, as shown in the following formula:

[0028] P 2d = M proj ·P 3d ;

[0029] Wherein, P 3d represents the three-dimensional edit control point, P 2d represents the two-dimensional control point, M proj is a perspective projection matrix, n represents the near plane, f' represents the far plane, (l, b, n) and (r, t', f') are the boundary coordinates of the camera view cone.

[0030] As preferred, the score distillation guide is constructed by using a drag-based latent diffusion model according to the initial RGB rendering image, the RGB rendering image, the two-dimensional mask image and the two-dimensional control points, and the total loss function is constructed according to the deformation loss function and the distillation loss function, which specifically includes:

[0031] The two-dimensional control points are respectively input into the pre-trained point encoder and the appearance encoder together with the initial RGB rendering image and the two-dimensional mask image to obtain point embedding features and appearance embedding features;

[0032] The drag-based latent diffusion model includes a VAE decoder in the original latent diffusion model, a UNet network in the original latent diffusion model and a repair model with the same structure as the UNet network in the original latent diffusion model; the RGB rendering image is encoded by the VAE decoder to obtain a latent variable z0;

[0033] In the noise adding process, random noise ∈ ~ N(0, 1) is sampled from the standard Gaussian distribution, and the latent variable z0 is added with noise to obtain a noisy latent variable Z t at time step t, as shown in the following formula:

[0034]

[0035] wherein, denotes a first weight function related to time step t, ~ denotes subject to, N denotes a Gaussian distribution;

[0036] In the denoising process, the UNet network in the inpainting model and the original latent diffusion model is used to repair the noisy latent variable z t Noise prediction is performed to obtain predicted noise As shown in the following formula:

[0037]

[0038] wherein, ∈ src is the source distribution estimation noise, ∈ tgt is the target distribution estimation noise;

[0039] The SDS loss function is constructed, as shown in the following formula:

[0040]

[0041] wherein, denotes the SDS loss function, denotes expectation, denotes L2 norm, w(t) is a second weight function related to time step t; denotes the estimated value of the noisy latent variable at time step t,

[0042] The generated conditional embedding spliced by the point embedding feature and the appearance embedding is taken as the condition of the inpainting model, and the noisy latent variable Z t at time step t is input into the inpainting model to predict the conditional noise; under the unconditional condition, the noisy latent variable Z t at time step t is input into the inpainting model to predict the unconditional noise; the conditional noise and the unconditional noise are guided using the classifier-free guidance to obtain the target distribution estimation noise, as shown in the following formula:

[0043]

[0044] wherein, e is the generated conditional embedding, s' is the ratio of the current iteration round number to the total round number, ω(s') is the classifier-free guidance weight, and is represented as: ω(s')=(ω max -1)·(1-s') 2 +1, ω max denotes the maximum weight; denotes the unconditional, θ' denotes the parameters of the inpainting model, denotes the unconditional noise, ∈ θ′ (Z t ,t,e) denotes the conditional noise;

[0045] The condition is a learnable condition embedding of the UNet network in the original latent diffusion model, and the noisy latent variable Z t is input into the UNet network in the original latent diffusion model to obtain a source distribution estimation noise, as shown in the following formula:

[0046]

[0047] wherein δ represents the weight of LoRA, is a learnable condition embedding, initialized as all zeros; the UNet network in the original latent diffusion model is trained in the manner of LoRA to determine φ and The LoRA loss function used in the training process is:

[0048]

[0049] wherein, represents the LoRA loss function;

[0050] The latent variable is decoded from the latent space to the pixel space by the VAE decoder in the original latent diffusion model, and a pixel space loss function is constructed in the pixel space, as shown in the following formula:

[0051]

[0052] wherein, represents the pixel space loss function; X t represents the pixel space variable corresponding to the noisy latent variable at time step t, represents the pixel space variable corresponding to the estimation value of the noisy latent variable at time step t, and respectively represents: is a function corresponding to the VAE decoder in the original latent diffusion model;

[0053] The distillation loss function is constructed according to the SDS loss function, the LoRA loss function and the pixel space loss function as shown in the following formula:

[0054]

[0055] wherein λ SDS , λ img and λ lora respectively represent the weights corresponding to the SDS loss function, the LoRA loss function and the pixel space loss function;

[0056] The deformation loss function is represented as:

[0057]

[0058] wherein, represents a deformation loss function;

[0059] The total loss function is represented as:

[0060]

[0061] wherein, represents a total loss function, λ RR and λ Drag-SDS respectively represent the weights corresponding to the deformation loss function and the distillation loss function.

[0062] As a preference, the training process of the three-dimensional Gaussian scene drag editing model is a two-stage training process, in the first stage of the training process, the parameters of the first renderer and the original parameters of the UNet network in the patch model and the original latent diffusion model are frozen; in the second stage of the training process, the original parameters of the UNet network in the patch model and the original latent diffusion model are frozen.

[0063] In a second aspect, the present application provides a point-based three-dimensional Gaussian scene drag editing device, comprising:

[0064] A model construction module is configured to construct a three-dimensional Gaussian scene drag editing model, a first renderer and a drag-based latent diffusion model, the three-dimensional Gaussian scene drag editing model comprising a multi-resolution three-plane encoder, a region-specific position decoder and a second renderer connected in sequence.

[0065] The fractional distillation module is configured to input the center coordinate of one of the Gaussians in the Gaussian scene in the training data into the three-dimensional Gaussian scene drag editing model, sequentially pass through the multi-resolution three-plane encoder and the region-specific position decoder to obtain the offset of the center coordinate of the Gaussian, and construct a deformation loss function based on the offset of the center coordinate of the Gaussian in the third subset obtained based on the soft local editing; add the offset of the center coordinate of the Gaussian to the center coordinate of the corresponding Gaussian to obtain a corresponding deformed Gaussian and construct a deformed Gaussian scene; randomly sample a camera pose c' from the reconstruction perspective and use the second renderer to render the deformed Gaussian scene to obtain an RGB rendering image; initialize the Gaussian scene in the training data to obtain a corresponding initial Gaussian scene, and use the first renderer to render the initial Gaussian scene under the camera pose c' to obtain an initial RGB rendering image; process the editing region mask and the three-dimensional editing control point in the training data respectively to obtain a two-dimensional mask image and a two-dimensional control point; use the drag-based latent diffusion model to construct a distillation loss function based on fractional distillation guided by the initial RGB rendering image, the RGB rendering image, the two-dimensional mask image, and the two-dimensional control point; construct a total loss function based on the deformation loss function and the distillation loss function, and train the three-dimensional Gaussian scene drag editing model based on the total loss function to obtain a trained three-dimensional Gaussian scene drag editing model.

[0066] The rendering module is configured to obtain a Gaussian scene to be edited and input the Gaussian scene into the trained three-dimensional Gaussian scene drag editing model to obtain a corresponding rendering image.

[0067] In a third aspect, the present application provides an electronic device, including one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the implementation manners of the first aspect.

[0068] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method described in any of the implementation manners of the first aspect.

[0069] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the method described in any of the implementation manners of the first aspect.

[0070] Compared with the prior art, the present application has the following beneficial effects:

[0071] (1) The point-based three-dimensional Gaussian scene dragging editing method provided by the application first obtains the deformation characteristics of the three-dimensional Gaussian center coordinates through a multi-resolution three-plane encoder, and decodes the offset of the three-dimensional Gaussian center position through a region-specific position decoder to realize the deformation of the Gaussian scene. Next, the Gaussians are divided into three groups through the nearest neighbor strategy, and different learning rates are set to prevent the geometric tearing problem of the Gaussian scene, and smooth local editing and Gaussian center regularization are applied to realize smooth editing. Finally, the potential diffusion model based on dragging is distilled through Drag-SDS to realize the three-dimensional Gaussian scene dragging editing.

[0072] (2) The point-based three-dimensional Gaussian scene dragging editing method provided by the application realizes smooth and consistent geometric editing effect through smooth geometric editing process, and encodes and decodes the Gaussian geometric deformation characteristics through the multi-resolution three-plane encoder and the region-specific position decoder. Compared with directly optimizing the center coordinates of the Gaussian, the implicit encoding-decoding structure can better fit the Gaussian deformation. The region-specific position encoder decodes the Gaussian deformation hybrid scale characteristics in different ways according to whether the Gaussian primitive belongs to the editing region or the non-editing region, so as to realize the smooth and consistent geometric editing effect.

[0073] (3) The point-based three-dimensional Gaussian scene dragging editing method provided by the application uses fractional distillation to guide the distillation of priori from the potential diffusion model through the Drag-SDS loss to optimize the second renderer, the multi-resolution three-plane encoder and the region-specific position decoder; two UNet networks are used to predict the target distribution estimation noise and the source distribution estimation noise respectively, so that the editing results of the editing region and the non-editing region are more consistent. In order to improve the quality of the editing result, in the whole optimization process, the UNet network in the original potential diffusion model is loaded with LoRA model for fine tuning, and a zero-initialized learnable conditional embedding is used. And use a two-stage training strategy, the first stage optimizes the second renderer, the multi-resolution three-plane encoder and the region-specific position decoder and freezes the first renderer, encourages Gaussians to fit the target distribution through position deformation, the second stage reduces the learning rate of the multi-resolution three-plane encoder, to stabilize the scene geometric structure and start optimizing the first renderer, aiming to provide a user-friendly three-dimensional Gaussian scene editing method, further strengthen the interactivity of the three-dimensional Gaussian scene, and better edit the scene geometric structure. BRIEF DESCRIPTION OF DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0075] Figure 1 A flowchart of a point-based three-dimensional Gaussian scene drag editing method according to an embodiment of the present application;

[0076] Figure 2 A framework diagram of a point-based three-dimensional Gaussian scene drag editing method according to an embodiment of the present application;

[0077] Figure 3 A flowchart of a point-based three-dimensional Gaussian scene drag editing method according to an embodiment of the present application;

[0078] Figure 4 A visualization diagram of a position change of a center coordinate of a Gaussian according to a point-based three-dimensional Gaussian scene drag editing method according to an embodiment of the present application;

[0079] Figure 5 A schematic diagram of a point-based three-dimensional Gaussian scene drag editing device according to an embodiment of the present application;

[0080] Figure 6 A hardware structure schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0081] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0082] Figure 1 A point-based three-dimensional Gaussian scene drag editing method according to an embodiment of the present application is shown, including the following steps:

[0083] S1, a three-dimensional Gaussian scene drag editing model, a first renderer and a drag-based potential diffusion model are constructed, the three-dimensional Gaussian scene drag editing model including a multi-resolution three-plane encoder, a region-specific position decoder and a second renderer connected in sequence.

[0084] Specifically, refer to Figure 2Embodiments of the present application propose a point-based three-dimensional Gaussian scene drag editing model architecture, which is composed of two processes, namely a smooth geometry editing process and a fractional distillation guide process. The smooth geometry editing process utilizes a multi-resolution three-plane encoder and a region-specific position decoder to achieve smooth and consistent geometry editing results. The fractional distillation guide process distills prior knowledge from the drag-based latent diffusion model to optimize the second renderer, the multi-resolution three-plane encoder and the region-specific position decoder. The Drag-SDS loss utilizes the UNet network in the patch model and the original latent diffusion model to predict the target distribution estimation noise and the source distribution estimation noise for high-quality three-dimensional drag editing scene optimization.

[0085] S2, input the center coordinates of one of the Gaussians in the Gaussian scene in the training data into the three-dimensional Gaussian scene drag editing model, sequentially pass through the multi-resolution three-plane encoder and the region-specific position decoder to obtain the offset of the center coordinates of the Gaussian, and construct a deformation loss function based on the offset of the center coordinates of the Gaussian in the third subset obtained based on soft local editing; add the offset of the center coordinates of the Gaussian to the center coordinates of the corresponding Gaussian to obtain the corresponding deformed Gaussian and construct a deformed Gaussian scene; randomly sample a camera pose c' from the reconstruction view and render the deformed Gaussian scene using a second renderer to obtain an RGB rendering image; initialize the Gaussian scene in the training data to obtain a corresponding initial Gaussian scene, and render the initial Gaussian scene using a first renderer under the camera pose c' to obtain an initial RGB rendering image; process the editing region mask and the three-dimensional editing control points in the training data respectively to obtain a two-dimensional mask image and a two-dimensional control point; construct a distillation loss function based on fractional distillation guide using the drag-based latent diffusion model according to the initial RGB rendering image, the RGB rendering image, the two-dimensional mask image and the two-dimensional control point; construct a total loss function according to the deformation loss function and the distillation loss function; train the three-dimensional Gaussian scene drag editing model based on the total loss function to obtain a trained three-dimensional Gaussian scene drag editing model.

[0086] In specific embodiments, the i-th Gaussian in the Gaussian scene is represented as g i ={p i ,f i ,o i ,s i ,r i ,m i}, where p i represents the center coordinates of the i-th Gaussian g i , f i represents the spherical harmonic color of the i-th Gaussian g i , o i represents the opacity of the i-th Gaussian g i , si denotes the scaling of the i-th Gaussian g i i denotes the rotation of the i-th Gaussian g i i denotes the mask of the i-th Gaussian g i i m i = 1 indicates that the i-th Gaussian g i is expected to be edited, m i = 0 indicates that the i-th Gaussian g i is not expected to be edited; the Gaussian scene is denoted as

[0087] The Gaussian scene G is divided into three groups of subsets, denoted as the first subset G the second subset G and the third subset G where the first subset G is the set of Gaussians with m i = 1; the set of Gaussians with m c = 0 is further divided into the second subset G and the third subset G i.e. Different learning rates are set for different subsets for soft local editing;

[0088] The multi-resolution three-plane encoder is three learnable orthogonal planes with different resolutions For the center coordinate p = (x, y, z) of the Gaussian scene, first, the normalization operation is performed to obtain the normalized coordinate Then, it is projected onto the orthogonal plane with different resolutions for bilinear interpolation, as shown in the following formula:

[0089]

[0090] where π s denotes the projection of the point p' onto the orthogonal plane ψ c denotes the bilinear interpolation with the resolution s, c denotes the index of the orthogonal plane, denotes the characteristics of the orthogonal plane and the resolution s;

[0091] The hybrid scale feature is calculated according to the characteristics of different orthogonal planes and resolutions, as shown in the following formula:

[0092]

[0093] ​​wherein, Π denotes a hadamard product, C = {xy, xz, yz} denotes a set of indices of orthogonal planes, Θ denotes a first multi-layer perceptron, concat denotes a tensor concatenation operation, f con denotes a mixed scale feature;

[0094] The mixed scale feature is decoded using a region-specific position decoder to obtain an offset of a center coordinate of a Gaussian scene, as shown in the following formula:

[0095]

[0096] wherein, and denote a second multi-layer perceptron and a third multi-layer perceptron respectively, sg denotes a gradient stop operation, and Δp denotes the offset of the center coordinate of the Gaussian scene;

[0097] One of the deformed Gaussians in the deformed Gaussian scene is denoted as g′ i = {Δp i +p i , f i , o i , s i , r i , m i}.

[0098] In specific embodiments, the region-specific position decoder comprises the second multi-layer perceptron and the third multi-layer perceptron, and the first multi-layer perceptron, the second multi-layer perceptron and the third multi-layer perceptron comprise a first fully connected layer, a ReLU activation function layer and a second fully connected layer connected in sequence.

[0099] Specifically, a Gaussian scene obtained by a three-dimensional Gaussian splatting method, an editing region mask and three-dimensional editing control points input by a user are obtained to construct training data. The offset of the Gaussian center coordinate is predicted by a smooth geometry editing process to obtain an RGB rendering image. The steps of the smooth geometry editing process specifically include:

[0100] The Gaussian scene is divided into three groups of subsets Then, different learning rates are set to achieve soft local editing. In one embodiment, the learning rate ratio of is 1:0.1:0.0001.

[0101] The center coordinate p i of the i-th Gaussian g i in the Gaussian scene is taken as an input of a multi-resolution three-plane encoder and a bilinear interpolation is performed to obtain a mixed scale feature. The center coordinate p iis a three-dimensional coordinate, first normalized to obtain normalized coordinates, and then projected onto different resolution orthogonal planes for bilinear interpolation. Then, the features of different orthogonal planes under the same resolution are multiplied element by element to obtain multiplied features, and the multiplied features under different resolutions are spliced, and a lightweight first multilayer perception machine is used to fuse the spliced features to obtain mixed scale features. Then, the mixed scale features are decoded using a region-specific position decoder composed of two identical lightweight multilayer perception machines, and the center coordinates p i of the i-th Gaussian g i are predicted in different ways according to different subsets of the Gaussian scene. i The offset Δp of the i-th Gaussian g i The center coordinates p i of the i-th Gaussian g i The offset Δp of the i-th Gaussian g i is added to the center coordinates p dilate of the original i-th Gaussian g dilate to obtain the i-th morphed Gaussian. All morphed Gaussians constitute a morphed Gaussian scene, and a morphing loss function is used for the Gaussian scene to constrain the third subset located in the non-editing region to morph. Randomly sampling a camera pose c' from the reconstructed view, and rendering the morphed Gaussian scene using the second renderer under the camera pose c' can obtain an RGB rendering image.

[0102] In specific embodiments, the editing region mask and the three-dimensional editing control points in the training data are processed respectively to obtain a two-dimensional mask graph and a two-dimensional control point, specifically including:

[0103] The editing region mask in the training data is rendered using the first renderer to obtain a rendered editing region mask graph; an inflation operation is performed on the rendered editing region mask graph to obtain a two-dimensional mask graph, as shown in the following formula:

[0104] I dilate =dilate(I,kernel,iteration);

[0105] Wherein, I represents the rendered editing region mask graph, kernel represents the convolution kernel of the inflation operation, iteration represents the iteration number of the inflation operation, and I dilate represents the two-dimensional mask graph.

[0106] The three-dimensional editing control points in the training data are projected under the camera pose c' to obtain a two-dimensional control point, as shown in the following formula:

[0107] P 2d =M proj ·P 3d ;

[0108] where P 3d represents a three-dimensional editing control point, P 2d represents a two-dimensional control point, M proj is a perspective projection matrix, n represents a near plane, f' represents a far plane, and (l, b, n) and (r, t', f') are boundary coordinates of a camera view frustum.

[0109] In specific embodiments, a score distillation guided construction distillation loss function is constructed by using a drag-based latent diffusion model according to the initial RGB renderings, the RGB renderings, the two-dimensional maskings, and the two-dimensional control points, and a total loss function is constructed according to the deformation loss function and the distillation loss function, and specifically includes:

[0110] The two-dimensional control points are respectively passed through a pre-trained point encoder and an appearance encoder with the initial RGB renderings and the two-dimensional maskings to obtain point embedding features and appearance embedding features;

[0111] The drag-based latent diffusion model includes a VAE decoder in the original latent diffusion model, a UNet network in the original latent diffusion model, and a patch model with the same structure as the UNet network in the original latent diffusion model; the RGB renderings are encoded by the VAE decoder to obtain latent variables z0;

[0112] In the noise adding process, random noise ∈ ~ N(0, 1) is sampled from a standard Gaussian distribution, and the latent variables z0are added with noise to obtain the noisy latent variables z t , at time step t, as shown in the following formula:

[0113]

[0114] wherein, represents a first weight function related to time step t, ~ represents a Gaussian distribution, and N represents a Gaussian distribution.

[0115] In the denoising process, the noisy latent variables z t are denoised by the patch model and the UNet network in the original latent diffusion model to obtain the predicted noise , as shown in the following formula:

[0116]

[0117] wherein, ∈ src is a source distribution estimation noise, and ∈ tgt is a target distribution estimation noise.

[0118] The SDS loss function is constructed, as shown in the following formula:

[0119]

[0120] wherein, denotes the SDS loss function, denotes expectation, denotes L2 norm, w(t) is a second weight function related to time step t; denotes the estimated value of the noisy latent variable at time step t,

[0121] the generated conditional embedding spliced by the point embedding feature and the appearance embedding as the condition of the repair model, and the noisy latent variable Z t at time step t is input into the repair model to predict the conditional noise; without condition, the noisy latent variable Z t at time step t is input into the repair model to predict the unconditional noise; the conditional noise and the unconditional noise are guided using a classifier-free guidance to obtain the target distribution estimation noise, as shown in the following formula:

[0122]

[0123] wherein, e is the generated conditional embedding, s' is the ratio of the current iteration round number to the total round number, ω(s') is the classifier-free guidance weight, and is represented as: ω(s')=(ω max -1)·(1-s') 2 +1, ω max denotes the maximum weight; denotes the unconditional, θ' denotes the parameters of the repair model, denotes the unconditional noise, ∈ θ′ (Z t ,t,e) denotes the conditional noise;

[0124] the learnable conditional embedding as the condition of the UNet network in the original latent diffusion model, the noisy latent variable Z t at time step t is input into the UNet network in the original latent diffusion model to predict the source distribution estimation noise, as shown in the following formula:

[0125]

[0126] wherein, δ denotes the weight of LoRA, is the learnable conditional embedding, initialized as all zeros; the UNet network in the original latent diffusion model is trained in the manner of LoRA to determine δ and The LoRA loss function used in the training process is:

[0127]

[0128] wherein, denotes the LoRA loss function;

[0129] The latent variable is decoded from the latent space to the pixel space by the VAE decoder in the original latent diffusion model, and a pixel space loss function is constructed in the pixel space, as shown in the following formula:

[0130]

[0131] wherein, represents the pixel space loss function; X t represents the pixel space variable corresponding to the noisy latent variable at the time step t, represents the pixel space variable corresponding to the estimated value of the noisy latent variable at the time step t, and respectively represents: is a function corresponding to the VAE decoder in the original latent diffusion model;

[0132] The distillation loss function is constructed according to the SDS loss function, the LoRA loss function and the pixel space loss function as shown in the following formula:

[0133]

[0134] wherein, λ SDS , λ img and λ lora respectively represent the weights corresponding to the SDS loss function, the LoRA loss function and the pixel space loss function;

[0135] The deformation loss function is represented as:

[0136]

[0137] wherein, represents the deformation loss function;

[0138] The total loss function is represented as:

[0139]

[0140] wherein, represents the total loss function, λ RR and λ Drag-SDS respectively represent the weights corresponding to the deformation loss function and the distillation loss function.

[0141] In specific embodiments, the training process of the three-dimensional Gaussian scene dragging editing model is a two-stage training process, in the first stage of the training process, the parameters of the first renderer and the original parameters of the UNet network in the patch model and the original latent diffusion model are frozen; in the second stage of the training process, the original parameters of the UNet network in the patch model and the original latent diffusion model are frozen.

[0142] Specifically, the prior knowledge is distilled from the drag-based latent diffusion model through a fractional distillation guiding process, and the second renderer, the multi-resolution three-plane positional encoder, and the region-specific positional decoder are optimized using the fractional distillation guiding process for two-stage training, and the specific process includes:

[0143] First, the Gaussian scene is initialized to obtain an initial Gaussian scene, and the initial RGB rendering image is obtained by rendering the initial Gaussian scene using the first renderer at the camera pose c' sampled in step S1. The parameters of the first renderer are frozen during the training process in the first stage. The editing region mask is rendered using the first renderer and an expansion operation is performed to obtain a two-dimensional mask image; the input three-dimensional editing control points are projected under the same camera pose c' to obtain two-dimensional control points; the two-dimensional control points, the initial RGB rendering image, and the two-dimensional mask image are respectively input into the pre-trained point encoder and appearance encoder to obtain point embedding features and appearance embedding features.

[0144] In the drag-based latent diffusion model, the RGB rendering image x0 rendered in step S1 is encoded into latent variable Z0 through the VAE decoder of the original latent diffusion model, then random noise ∈ ~ N(0, 1) is sampled from the standard Gaussian distribution, and the noisy latent variable Zt is obtained by adding noise to the latent variable Z0. t The point embedding features and the appearance embedding features are used as the condition of the UNet network through the cross-attention mechanism, and the noisy latent variable z t is noise-predicted through the UNet network of the latent diffusion model to obtain the predicted noise Finally, the second renderer, the multi-resolution three-plane encoder, and the region-specific positional decoder are optimized through the Drag-SDS.

[0145] SDS is a method of distilling prior knowledge from a pre-trained latent diffusion model to optimize scene parameters θ, and specifically SDS uses the following loss:

[0146]

[0147] wherein, denotes the gradient of the SDS loss function, w(t) is the second weight function related to the time step t, θ is the scene parameter, c' is the camera pose, denotes the rendering function corresponding to the second renderer, represents rendering the scene parameter θ under the camera pose c', and ε is the function corresponding to the VAE encoder of the original latent diffusion model. According to the denoising formula of the latent diffusion model, the SDS loss function can be further rewritten as:

[0148] Reference Figure 3Embodiments of the present application construct a patching model which has the same structure as the UNet network in the original stable diffusion model, both of which contain cross-attention layers, while the generated conditional embedding obtained by splicing the point embedding feature and the appearance embedding feature is used as the key and value of the cross-attention layer in the patching model, and the output feature of the self-attention layer is used as the query to perform cross-attention operation, so that the generated conditional embedding and the unconditional are used as the condition of the patching model to guide the prediction of the target distribution estimation noise. Unlike the standard SDS which models the source noise as a standard Gaussian noise ∈ src Embodiments of the present application use the UNet network in the original stable diffusion model to predict the source distribution estimation noise ∈ src , while achieving regionally consistent editing results. The UNet network in the original stable diffusion model uses the LoRA method to fine-tune its parameters, and at the same time uses a learnable conditional embedding as a condition to perform the cross-attention mechanism of the UNet network in the original stable diffusion model, and predict the source distribution estimation noise ∈ src . The UNet network in the original stable diffusion model is trained by the LoRA loss function, and during the training process, the original parameters of the UNet network in the original stable diffusion model are frozen, and only the weights of LoRA are adjusted.

[0149] In order to generate realistic editing results, embodiments of the present application also use a pixel space loss function The latent variable Z located in the latent space is decoded to the pixel space by the VAE decoder of the original stable diffusion model to obtain the pixel space variable X, and the pixel space loss function is used in the pixel space. Finally, the distillation loss function is constructed by combining the SDS loss function, the LoRA loss function and the pixel space loss function. The distillation loss function is further weighted with the above deformation loss function to obtain the total loss function.

[0150] During the entire training process, embodiments of the present application adopt a two-stage training strategy. In the first stage, the parameters of the first renderer are frozen, and the second renderer, the multi-resolution three-plane encoder and the region-specific decoder are trained to promote the Gaussian to fit the optimization target through geometric deformation; in the second stage, the learning rate of the multi-resolution plane encoder and the region-specific position decoder is reduced to stabilize the geometric structure of the entire Gaussian scene, and the second renderer is started to be optimized. In the above two stages, the original parameters of the patching model and the UNet network in the original stable diffusion model are in a frozen state, and after the above two-stage training, the trained three-dimensional Gaussian scene dragging editing model is obtained. In Figure 4 the effect of two-stage training is visualized in Figure 4It is clearly shown that the center coordinates of Gauss have been optimized to the target position at the end of the first stage, and the center position of Gauss has almost no change from the first stage to the end of the second stage, indicating that the scene geometry has been stabilized.

[0151] S3, obtaining the Gaussian scene to be edited and inputting into the trained three-dimensional Gaussian scene drag editing model to obtain the corresponding rendering graph.

[0152] Specifically, the trained three-dimensional Gaussian scene drag editing model is deployed, and the Gaussian scene to be edited is input into the trained three-dimensional Gaussian scene drag editing model. First, a multi-resolution three-plane encoder and a region-specific decoder are used to perform a smooth geometry editing process to obtain a corresponding deformation Gaussian. A second renderer is used to render a deformation Gaussian scene composed of all deformation Gaussians g'i to obtain a corresponding rendering graph.

[0153] Further referring to Figure 5 , as an implementation of the method shown in the above figures, the present application provides an embodiment of a point-based three-dimensional Gaussian scene drag editing device. The device embodiment corresponds to the method embodiment shown in Figure 1 , and the device can be applied to various electronic devices.

[0154] The present application provides a point-based three-dimensional Gaussian scene drag editing device, which includes:

[0155] A model construction module 1 is configured to construct a three-dimensional Gaussian scene drag editing model, a first renderer and a drag-based latent diffusion model. The three-dimensional Gaussian scene drag editing model includes a multi-resolution three-plane encoder, a region-specific position decoder and a second renderer connected in sequence.

[0156] The fractional distillation module 2 is configured to input the center coordinates of one of the Gaussians in the Gaussian scene in the training data into the three-dimensional Gaussian scene drag editing model, sequentially pass through the multi-resolution three-plane encoder and the region-specific position decoder, obtain the offset of the center coordinates of the Gaussian, construct a deformation loss function based on the offset of the center coordinates of the Gaussian in the third subset obtained based on the soft local editing; add the offset of the center coordinates of the Gaussian to the center coordinates of the corresponding Gaussian to obtain the corresponding deformed Gaussian and construct a deformed Gaussian scene, randomly sample a camera pose c' from the reconstruction perspective and use the second renderer to render the deformed Gaussian scene to obtain an RGB rendering image; initialize the Gaussian scene in the training data to obtain a corresponding initial Gaussian scene, use the first renderer to render the initial Gaussian scene under the camera pose c' to obtain an initial RGB rendering image; process the editing region mask and the three-dimensional editing control points in the training data respectively to obtain a two-dimensional mask image and a two-dimensional control point; construct a distillation loss function using the drag-based latent diffusion model based on the initial RGB rendering image, the RGB rendering image, the two-dimensional mask image and the two-dimensional control point, construct a total loss function based on the deformation loss function and the distillation loss function, train the three-dimensional Gaussian scene drag editing model based on the total loss function to obtain a trained three-dimensional Gaussian scene drag editing model.

[0157] The rendering module 3 is configured to obtain a Gaussian scene to be edited and input into the trained three-dimensional Gaussian scene drag editing model to obtain a corresponding rendering image.

[0158] Figure 6 The hardware structure schematic diagram of the electronic device provided by the embodiment of the present application is shown in the figure. Figure 6 As shown, the electronic device of the embodiment includes a processor 601 and a memory 602; the memory 602 is used to store computer execution instructions; the processor 601 is used to execute the computer execution instructions stored in the memory to realize each step executed by the electronic device in the above-mentioned embodiment. For details, please refer to the related description in the foregoing method embodiment.

[0159] Optionally, the memory 602 can be independent or integrated with the processor 601.

[0160] When the memory 602 is independently set, the electronic device further includes a bus 603 for connecting the memory 602 and the processor 601.

[0161] The embodiment of the present application further provides a computer storage medium, and the computer storage medium stores computer execution instructions; when the processor 601 executes the computer execution instructions, the method as above is realized.

[0162] The embodiment of the present application further provides a computer program product comprising a computer program, which, when executed by the processor 601, implements the method as above.

[0163] In the embodiments of the present application, it should be understood that the disclosed device and method can be implemented in other manners. For example, the described device embodiment is merely schematic. For example, the division of the modules is merely a logical function division. There can be another division manner for the actual implementation. For example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or modules, and can be electrical, mechanical or in other forms.

[0164] The modules illustrated as separated components can or can not be physical separated, and the components illustrated as modules can or can not be physical units. That is, they can be located in one position or distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to implement the embodiments.

[0165] In addition, the various function modules in the various embodiments of the present application can be integrated in a processing unit, or each module can be a physical unit, or two or more modules can be integrated in one unit. The units formed by the above modules can be implemented in the form of hardware, or in the form of hardware and software function units.

[0166] The integrated modules implemented in the form of software function modules can be stored in a computer readable storage medium. The above software function modules stored in a storage medium include a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or the processor 601 to perform some steps of the various embodiment methods of the present application.

[0167] It should be understood that the processor 601 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor, or the processor 601 can also be any conventional processor, etc. The steps of the method disclosed in the present application can be directly embodied as the execution of the processor 601 in hardware, or the combination of hardware and software modules in the processor 601.

[0168] The memory 602 can include a high-speed RAM memory, and can also include a non-volatile storage NVM, for example at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic or optical disk, etc.

[0169] The bus 603 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 603 can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, the bus 603 in the drawings of the present application does not limit to only one bus 603 or one type of bus 603.

[0170] The storage medium described above can be realized by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0171] An exemplary storage medium is coupled to the processor 601, so that the processor 601 can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be an integral part of the processor 601. The processor 601 and the storage medium can be located in an Application Specific Integrated Circuit (ASIC). Of course, the processor 601 and the storage medium can also exist as discrete components in an electronic device or a host device.

[0172] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction-related hardware. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the foregoing storage medium includes various storage media that can store program codes, such as ROM, RAM, magnetic disk or optical disk.

[0173] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A point-based 3D Gaussian scene dragging and editing method, characterized in that, Includes the following steps: A 3D Gaussian scene drag-and-edit model, a first renderer, and a drag-based potential diffusion model are constructed. The 3D Gaussian scene drag-and-edit model includes a multi-resolution three-plane encoder, a region-specific location decoder, and a second renderer connected in sequence. The center coordinates of one of the Gaussian sigmas in the Gaussian scene from the training data are input into the 3D Gaussian scene drag-and-edit model. The model passes through a multi-resolution three-plane encoder and a region-specific location decoder to obtain the offset of the Gaussian's center coordinates. A deformation loss function is constructed based on the offset of the Gaussian's center coordinates in the third subset obtained through soft local editing. The offset of the Gaussian's center coordinates is added to the center coordinates of its corresponding Gaussian sigma to obtain the corresponding deformed Gaussian sigma, and a deformed Gaussian scene is constructed. The camera pose c' is randomly sampled from the reconstructed viewpoint, and the deformed Gaussian scene is rendered using the second renderer to obtain an RGB rendering image. The Gaussian scene in the training data is initialized to obtain the corresponding initial Gaussian scene. The initial Gaussian scene is rendered using the first renderer under the camera pose c' to obtain an initial RGB rendering image. The edit region mask and 3D edit control points in the training data are processed separately to obtain a 2D mask image and 2D control points. Based on the initial RGB rendering image, RGB rendering image, 2D mask image, and 2D control points, a fractional distillation-guided construction of a distillation loss function is performed using a drag-based latent diffusion model. A total loss function is constructed based on the deformation loss function and the distillation loss function. The 3D Gaussian scene drag-editing model is trained based on the total loss function to obtain a trained 3D Gaussian scene drag-editing model. The Gaussian scene to be edited is obtained and input into the trained 3D Gaussian scene drag-and-drop editing model to obtain the corresponding rendering image.

2. The point-based 3D Gaussian scene dragging and editing method according to claim 1, characterized in that, The i-th Gaussian in the Gaussian scene is denoted as g. i ={p i ,f i ,o i ,s i ,r i ,m i }, where p i Represents the i-th Gaussian g i The center coordinates, f i Represents the i-th Gaussian g i The color of the ball, o i Represents the i-th Gaussian g i Opacity, s i Represents the i-th Gaussian g i Scaling, r i Represents the i-th Gaussian g i rotation; m i Represents the i-th Gaussian g i The mask, m i =1 indicates that the expectation for the i-th Gaussian g is 1. i Edit, m i =0 indicates that it is not expected that the i-th Gaussian g... i Editing is performed; the Gaussian scene is represented as The Gaussian scene Divided into three subsets, namely the first subset Second Subset and the third subset Among them, the first subset For m i The set consisting of Gaussians of 1; m i The set of Gaussians with a value of 0 is further divided into a second subset with no intersection. and the third subset Right now Soft local editing is performed by setting different learning rates for different subsets; The multi-resolution three-plane encoder consists of three learnable orthogonal planes with different resolutions. For the center coordinates p = (x, y, z) of the Gaussian scene, a normalization operation is first performed to obtain the normalized coordinates. Then, the projection is performed onto orthogonal planes of different resolutions for bilinear interpolation, as shown in the following equation: Where, π c This means that point p ′ Projected onto an orthogonal plane ψ s This indicates bilinear interpolation over resolution s, where c represents the index of the orthogonal plane. Represents an orthogonal plane and the characteristics of resolution s; Mixed-scale features are calculated for features at different orthogonal planes and resolutions, as shown in the following equation: Where Π represents the Hadamard product, C = {xy, xz, yz} represents the set of indices of the orthogonal plane, Θ represents the first multilayer perceptron, concat represents the tensor concatenation operation, and f con Indicates mixed-scale features; The region-specific location decoder is used to decode the mixed-scale features to obtain the offset of the center coordinates of the Gaussian scene, as shown in the following formula: in, and These represent the second and third multilayer perceptrons, respectively; sg represents the gradient stopping operation; and Δp represents the offset of the center coordinates of the Gaussian scene. One of the deformation Gaussians in the deformation Gaussian scenario is denoted as g′. i ={Δp i +p i ,f i ,o i ,s i ,r i ,m i } 3. The point-based 3D Gaussian scene dragging and editing method according to claim 2, characterized in that, The region-specific location decoder includes a second multilayer perceptron and a third multilayer perceptron. The first multilayer perceptron, the second multilayer perceptron, and the third multilayer perceptron include a first fully connected layer, a ReLU activation function layer, and a second fully connected layer connected in sequence.

4. The point-based 3D Gaussian scene dragging and editing method according to claim 1, characterized in that, The edit region mask and 3D edit control points in the training data are processed separately to obtain a 2D mask image and 2D control points, specifically including: The edit region mask in the training data is rendered using the first renderer to obtain a rendered edit region mask image; an inflation operation is performed on the rendered edit region mask image to obtain a two-dimensional mask image, as shown in the following formula: I dilate =dilate(I,kernel,iteration); Where I represents the mask image of the rendered editing region, kernel represents the convolution kernel of the dilation operation, and iteration represents the number of iterations of the dilation operation. dilate Represents a two-dimensional mask image; Projecting the 3D editing control points from the training data onto the camera pose c' yields the 2D control points, as shown in the following equation: P 2d =M proj ·P 3d ; Among them, P 3d P represents the 3D editing control point. 2d M represents a two-dimensional control point. proj For perspective projection matrix, n represents the near plane, f′ represents the far plane, and (l,b,n) and (r,t′,f′) are the boundary coordinates of the camera's view frustum.

5. The point-based 3D Gaussian scene dragging and editing method according to claim 2, characterized in that, Based on the initial RGB rendering image, RGB rendering image, 2D mask image, and 2D control points, a drag-based latent diffusion model is used to guide the construction of a distillation loss function through fractional distillation. The total loss function is then constructed based on the deformation loss function and the distillation loss function, specifically including: The two-dimensional control points are fed into the initial RGB rendering image and the two-dimensional mask image respectively through a pre-trained point encoder and appearance encoder to obtain point embedding features and appearance embedding features. The drag-based latent diffusion model includes the VAE decoder in the original latent diffusion model, the UNet network in the original latent diffusion model, and a patched model with the same structure as the UNet network in the original latent diffusion model; the latent variable z0 is obtained by encoding the RGB rendering image through the VAE decoder. During the noise addition process, random noise ∈ ~N(0,1) is sampled from a standard Gaussian distribution and added to the latent variable z0 to obtain the noisy latent variable Z at time step t. t As shown in the following formula: in, Let represent the first weighting function associated with time step t, ~ indicates that it follows a Gaussian distribution; During the denoising process, the noisy latent variable z is processed using the UNet network in both the repair model and the original latent diffusion model. t Perform noise prediction to obtain predicted noise. As shown in the following formula: Where, ∈ src To estimate the noise for the source distribution, ∈ tgt Estimate the noise for the target distribution; The SDS loss function is constructed as shown in the following equation: in, Represents the SDS loss function. Expressing expectations, Let w(t) denote the L2 norm, and w(t) be the second weighting function related to time step t. This represents the estimated value of the noisy latent variable at time step t. The generated conditional embedding, formed by concatenating the point embedding features and the appearance embedding, is used as the condition for the repair model, and the noisy latent variable Z at time step t is used. t The input is fed into the repair model to predict conditional noise; under no conditions, the noisy latent variable Z at time step t is... t The input is fed into the repair model to predict unconditional noise; the conditional and unconditional noises are then guided by a classifier-free method to obtain the target distribution estimate noise, as shown in the following equation: Where e is the generating conditional embedding, s′ is the ratio of the current iteration round to the total number of rounds, and ω(s′) is the classifier-free guiding weight, expressed as: ω(s′)=(ω max -1)ω(1-s′) 2 +1, ω max This represents the maximum weight. θ′ represents unconditional, and θ′ represents the parameters of the repair model. Represents unconditional noise, ∈ θ′ (Z t ,t,e) represents conditional noise; Using learnable conditional embeddings as conditions for the UNet network in the original latent diffusion model, the noisy latent variable Z at time step t is... t The noise from the source distribution is estimated by inputting the data into the UNet network of the original potential diffusion model, as shown in the following equation: Where φ represents the weight of LoRA, For learnable conditional embeddings, initialization is set to all zeros; the UNet network in the original latent diffusion model is trained using LoRA to determine φ and The LoRA loss function used during training is: in, Represents the LoRA loss function; The latent variables are decoded from the latent space to the pixel space using the VAE decoder in the original latent diffusion model, and a pixel space loss function is constructed in the pixel space, as shown in the following equation: in, X represents the pixel space loss function; t This represents the pixel spatial variable corresponding to the noisy latent variable at time step t. The pixel spatial variables corresponding to the estimated values ​​of the noisy latent variables at time step t are expressed as follows: This refers to the function corresponding to the VAE decoder in the original potential diffusion model; A distillation loss function is constructed based on the SDS loss function, LoRA loss function, and pixel space loss function. As shown in the following formula: Where, λ SDS , λ img and λ lora These represent the weights corresponding to the SDS loss function, LoRA loss function, and pixel space loss function, respectively. The deformation loss function is expressed as: in, Represents the deformation loss function; The total loss function is expressed as: in, Let λ represent the total loss function. RR and λ Drag-SDS These represent the weights corresponding to the deformation loss function and the distillation loss function, respectively.

6. The point-based 3D Gaussian scene dragging and editing method according to claim 1, characterized in that, The training process of the 3D Gaussian scene drag-and-drop editing model is a two-stage training process. In the first stage of training, the parameters of the first renderer and the original parameters of the UNet network in the patch model and the original latent diffusion model are frozen. In the second stage of training, the original parameters of the UNet network in the patch model and the original latent diffusion model are frozen.

7. A point-based 3D Gaussian scene dragging and editing device, characterized in that, include: The model building module is configured to build a 3D Gaussian scene drag-and-edit model, a first renderer, and a drag-based potential diffusion model. The 3D Gaussian scene drag-and-edit model includes a multi-resolution three-plane encoder, a region-specific location decoder, and a second renderer connected in sequence. The fractional distillation module is configured to input the center coordinates of one of the Gaussian spheres in the Gaussian scene from the training data into the 3D Gaussian scene drag-and-edit model. The model passes through a multi-resolution three-plane encoder and a region-specific location decoder sequentially to obtain the offset of the Gaussian sphere's center coordinates. A deformation loss function is constructed based on the offset of the Gaussian sphere's center coordinates in the third subset obtained by soft local editing. The offset of the Gaussian sphere's center coordinates is added to the corresponding center coordinates of the Gaussian sphere to obtain the corresponding deformed Gaussian sphere, and a deformed Gaussian scene is constructed. The camera pose c' is randomly sampled from the reconstructed viewpoint, and the deformed Gaussian scene is rendered using the second renderer to obtain an RGB rendering image. The Gaussian scene in the training data is initialized to obtain a corresponding initial Gaussian scene. The initial Gaussian scene is rendered using the first renderer under the camera pose c' to obtain an initial RGB rendering image. The edit region mask and 3D edit control points in the training data are processed separately to obtain a 2D mask image and 2D control points. Based on the initial RGB rendering image, RGB rendering image, 2D mask image, and 2D control points, a fractional distillation-guided construction of a distillation loss function is performed using a drag-based latent diffusion model. A total loss function is constructed based on the deformation loss function and the distillation loss function. The 3D Gaussian scene drag-editing model is trained based on the total loss function to obtain a trained 3D Gaussian scene drag-editing model. The rendering module is configured to acquire the Gaussian scene to be edited and input it into the trained 3D Gaussian scene drag-and-drop editing model to obtain the corresponding rendering image.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images