Three-dimensional scene editing method and device based on consistent distillation multi-view and storage medium

Abstract

The invention relates to the technical field of computer vision, in particular to a three-dimensional scene editing method and device based on multiple view angles of consistent distillation and a storage medium. The method comprises the following steps: acquiring a multi-view image set of a three-dimensional scene; adjusting parameters of a preset three-dimensional NVS diffusion model according to the multi-view image set; according to the adjusted three-dimensional NVS diffusion model, a preset two-dimensional editor is trained, and training is used for indicating the prior knowledge of the adjusted three-dimensional NVS diffusion model to be transmitted to the two-dimensional editor through knowledge distillation; and editing the multi-view image set through the trained two-dimensional editor. According to the embodiment of the invention, a brand new knowledge distillation framework is provided, the priori knowledge in the three-dimensional NVS diffusion model can be transmitted to the two-dimensional editor, so that the editor can learn the relationship among different visual angles in the three-dimensional scene, multi-visual-angle consistency is kept during editing, and the high-quality three-dimensional scene is generated.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision, and particularly relates to a three-dimensional scene editing method and device based on consistency distillation multi-view and a storage medium. BACKGROUND

[0002] At present, in the field of generative artificial intelligence, the generation technology based on diffusion model (DM) has made remarkable breakthroughs in the field of two-dimensional image generation and editing. With the help of the step-by-step denoising process of Markov chain, this technology shows unique advantages in image synthesis, style transfer and other tasks. However, in the field of three-dimensional content creation, the traditional creation process highly depends on professional three-dimensional software, and the professional skills and experience of creators are required extremely high, which greatly limits the popularity of three-dimensional content creation. Therefore, introducing generative artificial intelligence technology into the field of three-dimensional creation has extremely important significance for promoting the development of virtual reality, film and television production and other industries.

[0003] At present, the three-dimensional scene editing technology based on diffusion model mainly develops along three technical routes: the first is the score distillation sampling (SDS) framework represented by DreamFusion, which uses the gradient signal generated by the two-dimensional diffusion model to drive the update of the neural radiance field (NeRF) parameter. On this basis, Instruct-NeRF2NeRF, Gaussian editor (English: GaussianEditor) and other schemes further improve the editing effect by introducing iterative dataset update (IDU) strategy and other improvements. The second is the multi-view consistency editing architecture, such as direct Gaussian editing (DGE), which regards the multi-view sequence as video frames and realizes joint editing through feature propagation by attention mechanism; GaussCtrl controls the editing process through depth information propagation. Although these schemes have made certain progress in the feasibility of three-dimensional editing, they have not effectively solved the essential contradiction between two-dimensional diffusion priori knowledge and three-dimensional geometric constraints. This contradiction limits the performance of three-dimensional scene editing in multi-view consistency and high-quality generation, and new technical solutions are needed to break through the existing bottlenecks. Figure One Figure One SUMMARY

[0004] Therefore, the present disclosure proposes a three-dimensional scene editing method and device based on consistency distillation multi-view and a storage medium.

[0005] ​​In an aspect of the present disclosure, a method for editing a three-dimensional scene based on consistent multi-view distillation is provided, which comprises:

[0006] obtaining a multi-view image set of the three-dimensional scene, the multi-view image set being a two-dimensional image set of the three-dimensional scene under multiple views;

[0007] adjusting parameters of a preset three-dimensional Novel View Synthesis (NVS) diffusion model based on the multi-view image set, the three-dimensional NVS diffusion model being used to generate a new view image of the three-dimensional scene based on two-dimensional images of the three-dimensional scene under a single view or multiple views;

[0008] training a preset two-dimensional editor based on the adjusted three-dimensional NVS diffusion model, the training being used to indicate that prior knowledge of the adjusted three-dimensional NVS diffusion model is transferred to the two-dimensional editor through knowledge distillation, the prior knowledge being used to indicate consistent characteristics of the three-dimensional scene under different views;

[0009] editing the multi-view image set through the trained two-dimensional editor.

[0010] In a possible implementation, the adjusting parameters of the preset three-dimensional Novel View Synthesis (NVS) diffusion model based on the multi-view image set comprises:

[0011] determining a reference image from the multi-view image set;

[0012] adding first Gaussian noise to each two-dimensional image in the multi-view image set to obtain a first multi-view image set;

[0013] generating predicted noise through the preset three-dimensional NVS diffusion model based on the first multi-view image set and the reference image;

[0014] adjusting parameters of the three-dimensional NVS diffusion model by minimizing the difference between the predicted noise and the first Gaussian noise.

[0015] In another possible implementation, the three-dimensional NVS diffusion model comprises a convolution layer with fixed parameters, a spatial attention layer with fixed parameters, and a temporal attention layer with parameters to be adjusted, and the adjusting parameters of the three-dimensional NVS diffusion model by minimizing the difference between the predicted noise and the first Gaussian noise comprises:

[0016] calculating a temporal loss based on the difference between the predicted noise and the first Gaussian noise;

[0017] Parameters of a low-rank adaptation (LoRA) module of the temporal attention layer are adjusted by minimizing the temporal loss.

[0018] In another possible implementation, the training of the preset two-dimensional editor according to the adjusted three-dimensional NVS diffusion model comprises:

[0019] According to the input editing instruction, the set of multi-view images is edited by the preset two-dimensional editor to obtain a second set of multi-view images;

[0020] Second Gaussian noise is added to each two-dimensional image in the second set of multi-view images to obtain a third set of multi-view images;

[0021] According to the third set of multi-view images, the editing distribution and the target consistency distribution of the two-dimensional editor are estimated by the adjusted three-dimensional NVS diffusion model, the editing distribution is the actual probability distribution of the set of multi-view images after editing by the two-dimensional editor, and the target consistency distribution is the probability distribution of the set of multi-view images after editing by the two-dimensional editor to maintain three-dimensional consistency between views;

[0022] According to the difference between the editing distribution and the target consistency distribution, a distillation loss is calculated;

[0023] The two-dimensional editor is trained by minimizing the distillation loss.

[0024] In another possible implementation, the estimation of the editing distribution and the target consistency distribution of the two-dimensional editor by the adjusted three-dimensional NVS diffusion model comprises:

[0025] A parameter trainable parameterized model is copied from the adjusted three-dimensional NVS diffusion model;

[0026] The editing distribution of the two-dimensional editor is estimated by the copied parameter trainable parameterized model, and the target consistency distribution of the two-dimensional editor is estimated by the adjusted three-dimensional NVS diffusion model with parameter freezing.

[0027] In another possible implementation, the method further comprises:

[0028] During the training process, a second reference image is obtained by editing the reference image according to the two-dimensional editor;

[0029] According to the difference between the second reference image and the reference image in each round of training, a regularization loss is calculated;

[0030] The training of the two-dimensional editor by minimizing the distillation loss includes:

[0031] The training of the two-dimensional editor by minimizing the distillation loss and the regularization loss.

[0032] In another possible implementation, the editing of the multi-view image set by the trained two-dimensional editor includes:

[0033] According to the input editing instruction, the multi-view image set is edited by the trained two-dimensional editor to obtain a fourth multi-view image set;

[0034] The fourth multi-view image set is integrated into the three-dimensional scene to obtain an edited three-dimensional scene representation.

[0035] According to another aspect of the present disclosure, a three-dimensional scene editing device based on consistent distillation multi-view is provided, and the device includes:

[0036] An acquisition module is configured to acquire a multi-view image set of a three-dimensional scene, the multi-view image set being a two-dimensional image set of the three-dimensional scene under multiple views;

[0037] An adjustment module is configured to adjust parameters of a preset three-dimensional NVS diffusion model according to the multi-view image set, the three-dimensional NVS diffusion model being used to generate a new view image of the three-dimensional scene based on a two-dimensional image of the three-dimensional scene under a single view or multiple views;

[0038] A training module is configured to train a preset two-dimensional editor according to the adjusted three-dimensional NVS diffusion model, the training being used to indicate that prior knowledge of the adjusted three-dimensional NVS diffusion model is transferred to the two-dimensional editor through knowledge distillation, the prior knowledge being used to indicate consistent characteristics of the three-dimensional scene under different views;

[0039] An editing module is configured to edit the multi-view image set by the trained two-dimensional editor.

[0040] In a possible implementation, the adjustment module is further configured to:

[0041] determine a reference image from the multi-view image set;

[0042] add first Gaussian noise to each two-dimensional image in the multi-view image set to obtain a first multi-view image set;

[0043] generate predicted noise through the preset three-dimensional NVS diffusion model according to the first multi-view image set and the reference image.

[0044] adjust parameters of the three-dimensional NVS diffusion model by minimizing the difference between the predicted noise and the first Gaussian noise.

[0045] In another possible implementation, the three-dimensional NVS diffusion model includes a convolution layer with fixed parameters, a spatial attention layer with fixed parameters, and a temporal attention layer with parameters to be adjusted, and the adjustment module is further configured to:

[0046] calculate a temporal loss according to the difference between the predicted noise and the first Gaussian noise;

[0047] adjust parameters of a LoRA module of the temporal attention layer by minimizing the temporal loss.

[0048] In another possible implementation, the training module is further configured to:

[0049] edit the multi-view image set by using a preset two-dimensional editor according to an input editing instruction to obtain a second multi-view image set;

[0050] add second Gaussian noise to each two-dimensional image in the second multi-view image set to obtain a third multi-view image set;

[0051] estimate an editing distribution and a target consistency distribution of the two-dimensional editor according to the third multi-view image set by using the adjusted three-dimensional NVS diffusion model, the editing distribution being an actual probability distribution of the multi-view image set after being edited by using the two-dimensional editor, and the target consistency distribution being a probability distribution of the multi-view image set after being edited by using the two-dimensional editor while maintaining three-dimensional consistency among views;

[0052] calculate a distillation loss according to the difference between the editing distribution and the target consistency distribution;

[0053] train the two-dimensional editor by minimizing the distillation loss.

[0054] In another possible implementation, the training module is further configured to:

[0055] copy the adjusted three-dimensional NVS diffusion model to obtain a parameterizable model with trainable parameters;

[0056] estimate the editing distribution of the two-dimensional editor by using the copied parameterizable model with trainable parameters, and estimate the target consistency distribution of the two-dimensional editor by using the adjusted three-dimensional NVS diffusion model with frozen parameters.

[0057] In another possible implementation, the training module is further configured to:

[0058] In the training process, a second reference image is obtained by editing the reference image according to the two-dimensional editor;

[0059] A regularization loss is calculated according to a difference between the second reference image and the reference image in each round of training.

[0060] The two-dimensional editor is trained by minimizing the distillation loss and the regularization loss.

[0061] In another possible implementation, the editing module is further configured to:

[0062] According to the input editing instruction, the multi-view image set is edited by the trained two-dimensional editor to obtain a fourth multi-view image set.

[0063] The fourth multi-view image set is integrated into the three-dimensional scene to obtain an edited three-dimensional scene representation.

[0064] According to another aspect of the present disclosure, there is provided a three-dimensional scene editing device based on consistent distillation multi-view, comprising a memory, a processor and a computer program stored in the memory, the processor executes the computer program to implement the steps of the above method.

[0065] According to another aspect of the present disclosure, there is provided a non-volatile computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the above method.

[0066] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, or a non-volatile computer readable storage medium carrying a computer program, the computer program being executed by a processor to implement the steps of the above method.

[0067] The embodiment of the present disclosure provides a three-dimensional scene editing method based on consistent distillation multi-view, which comprises the following steps: acquiring a multi-view image set of a three-dimensional scene, the multi-view image set being a two-dimensional image set of the three-dimensional scene under multiple views; adjusting parameters of a preset three-dimensional NVS diffusion model according to the multi-view image set, the three-dimensional NVS diffusion model being used for generating a new view image of the three-dimensional scene based on a two-dimensional image of the three-dimensional scene under a single view or multiple views; training a preset two-dimensional editor according to the adjusted three-dimensional NVS diffusion model, the training being used for indicating that prior knowledge of the adjusted three-dimensional NVS diffusion model is transmitted to the two-dimensional editor through knowledge distillation, and the prior knowledge being used for indicating consistent characteristics of the three-dimensional scene under different views; and editing the multi-view image set through the trained two-dimensional editor. A new knowledge distillation framework is proposed, which can efficiently transmit the prior knowledge in the three-dimensional NVS diffusion model to the two-dimensional editor, so that the two-dimensional editor can learn the relationship between different views in the three-dimensional scene, thereby maintaining the consistency of multi-views during editing.

[0068] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0069] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0070] Figure 1 A structural schematic diagram of a computing device provided by one exemplary embodiment of the present disclosure is shown.

[0071] Figure 2 A flowchart of a three-dimensional scene editing method based on consistent distillation multi-view provided by one exemplary embodiment of the present disclosure is shown.

[0072] Figure 3 A principle schematic diagram of a three-dimensional scene editing method based on consistent distillation multi-view provided by one exemplary embodiment of the present disclosure is shown.

[0073] Figure 4 A flowchart of a training process of a two-dimensional editor provided by one exemplary embodiment of the present disclosure is shown.

[0074] Figure 5 A principle schematic diagram of a three-dimensional scene editing method based on consistent distillation multi-view provided by another exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0075] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in different drawings represent the same or similar elements / function. Although various aspects of embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically noted.

[0076] As used herein, the terms "comprise", "comprising", "have", "having", "include", "including", "contain", "containing", "provide", "providing", or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components or functions but do not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof.

[0077] When an element is referred to as being "connected", "coupled", "responsive", or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element, or intervening elements can be present.

[0078] Although the terms first, second, third, etc. can be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus, a first element / operation in some embodiments could be termed a second element / operation in other embodiments without departing from the teachings of the present disclosure embodiment concepts.

[0079] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0080] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known functions, procedures, components, and circuits will not be described in detail herein. It will be appreciated that the present disclosure can be practiced with the elements, procedures and circuits being replaced by other

[0081] Current three-dimensional scene editing techniques based on diffusion models have some fundamental flaws. First, the single-view iterative update paradigm (e.g., the IDU framework) lacks a cross-view consistency constraint mechanism, which can easily cause signal conflicts when optimizing the three-dimensional representation, resulting in slow convergence speed, blurred textures, and other issues. Second, some research focuses on achieving multi-view consistency editing by updating the three-dimensional representation through implicit depth information or attention feature propagation schemes (e.g., DGE, GaussCtrl, etc.). However, due to the weak constraints of these methods, fine-grained inconsistencies still occur. In addition, when dealing with complex scenes with large changes in view content, these methods tend to accumulate alignment errors in geometric discontinuous areas, leading to semantic mismatch in 360-degree scene editing due to the lack of a view relationship coordination mechanism. These flaws are essentially due to the lack of implicit constraints on the two-dimensional editing space and the coordination mechanism between multiple views.

[0082] Current artificial intelligence technology has made many breakthroughs in content creation, especially in three-dimensional scene editing. The present disclosure aims to implement three-dimensional scene editing through artificial intelligence technology, further to edit the original three-dimensional scene content under the driving of text instructions to obtain a high-quality and consistent three-dimensional scene that meets the text semantics.

[0083] First, some terms related to the present disclosure are introduced.

[0084] 1. Gaussian Splatting (GS): A three-dimensional scene representation and rendering technique based on Gaussian distribution, which realizes high-fidelity dynamic scene modeling through discrete Gaussian function point cloud, supports real-time rendering, and is suitable for virtual reality (VR) / augmented reality (AR) and real-time three-dimensional reconstruction scenes.

[0085] 2. Reward Feedback Learning (ReFL): A method mechanism to improve the ability of text-to-image diffusion models.

[0086] 3. NVS: An image generation technology based on multi-view geometry or deep learning, which generates any new view of a three-dimensional scene by inputting two-dimensional images from limited views and reconstructs the three-dimensional scene.

[0087] 4. Classifier-Free Guidance (CFG): An optimization strategy for generative models, which replaces explicit classifiers with implicit conditional embedding to dynamically adjust the matching degree of generated content and target semantics, improving image generation quality and diversity, and is used for text-to-image generation, controllable content creation, and multi-modal data synthesis.

[0088] 5. Contrastive Language-Image Pretraining (CLIP): A multimodal model based on contrastive learning, which aligns visual and semantic features through joint training of massive text-image pairs, enabling cross-modal retrieval and zero-shot reasoning, and supporting image generation guidance, open-domain content understanding, and human-computer interaction intent analysis.

[0089] 6. Iterative Dataset Update (IDU): A framework for three-dimensional scene editing by iterating between editing rendering perspectives and updating three-dimensional representations.

[0090] 7. LoRA: An efficient fine-tuning technique for optimizing pre-trained model parameters through low-rank matrix decomposition, training only low-rank incremental matrices to reduce computational resource consumption.

[0091] Next, the application scenarios related to the present disclosure are introduced. Please refer to Figure 1 which shows the structural schematic diagram of the computing device provided by one exemplary embodiment of the present disclosure.

[0092] The computing device can be a terminal or a server. The terminal includes a mobile terminal or a fixed terminal, such as a mobile phone, a tablet computer, a laptop computer, and a desktop computer, etc. The server can be a server, or a server cluster composed of several servers, or a cloud computing service center.

[0093] The computing device includes a processor 10, a memory 20, and a communication interface 30. Those skilled in the art can understand that Figure 1 the structure shown in the above figure does not constitute a limitation on the computing device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements. Among them:

[0094] The processor 10 is the control center of the computing device, which connects all parts of the computing device through various interfaces and lines, executes the software programs and / or modules stored in the memory 20 and calls the data stored in the memory 20, executes various functions of the computing device and processes data, thereby overall controls the computing device. The processor 10 can be implemented by a central processing unit (CPU) or a graphics processing unit (GPU).

[0095] The memory 20 can be used to store software programs and modules. The processor 10 performs various functional applications and data processing by running the software programs and modules stored in the memory 20. The memory 20 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system 21, an acquisition module 22, an adjustment module 23, a training module 24, an editing module 25, and an application program 26 required by at least one function, etc.; and the data storage area can store data created according to the use of the computing device, etc. The memory 20 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk. Accordingly, the memory 20 can also include a memory controller to provide the processor 10 with access to the memory 20.

[0096] The processor 10 performs the following functions by running the acquisition module 22: acquiring a multi-view image set of a three-dimensional scene, the multi-view image set being a set of two-dimensional images of the three-dimensional scene under multiple views; the processor 10 performs the following functions by running the adjustment module 23: adjusting parameters of a preset three-dimensional NVS diffusion model according to the multi-view image set, the three-dimensional NVS diffusion model being used to generate a new view image of the three-dimensional scene based on a two-dimensional image of the three-dimensional scene under a single view or multiple views; the processor 10 performs the following functions by running the training module 24: training a preset two-dimensional editor according to the adjusted three-dimensional NVS diffusion model, the training being used to indicate that prior knowledge of the adjusted three-dimensional NVS diffusion model is transferred to the two-dimensional editor through knowledge distillation, the prior knowledge being used to indicate consistency characteristics of the three-dimensional scene under different views; and the processor 10 performs the following functions by running the editing module 25: editing the multi-view image set through the trained two-dimensional editor.

[0097] In the following, the method provided by the embodiments of the present disclosure is introduced by using several exemplary embodiments.

[0098] Please refer to Figure 2FIG. 1 shows a flowchart of a method for editing a three-dimensional scene based on consistent multi-view distillation according to an example embodiment of the present disclosure. The method can be used for Figure 1 The method can be implemented by a computing device as shown in FIG. 2. The method includes the following steps.

[0099] At step 201, a multi-view image set of a three-dimensional scene is obtained. The multi-view image set is a set of two-dimensional images of the three-dimensional scene from multiple perspectives.

[0100] A three-dimensional scene refers to a virtual or real space composed of elements such as objects, environment, and lighting in three-dimensional space. It can be a complex virtual environment (such as a city in a game), a building model, or a three-dimensional reconstruction of a real-world scene.

[0101] A multi-view image set refers to a set of two-dimensional images of the same three-dimensional scene generated from different angles.

[0102] The computing device obtains the multi-view image set of the three-dimensional scene, including but not limited to the following possible implementation ways: in one possible implementation way, the images of the three-dimensional scene are taken from multiple angles by a camera to form the multi-view image set. In another possible implementation way, the pre-taken multi-view image set is loaded from a storage device (such as a hard disk, cloud storage). In another possible implementation way, the multi-view image set of the three-dimensional scene is rendered from a three-dimensional GS model, which is a model used to describe the geometric structure and appearance characteristics of the three-dimensional scene.

[0103] At step 202, the parameters of a pre-set three-dimensional NVS diffusion model are adjusted based on the multi-view image set. The three-dimensional NVS diffusion model is used to generate a new perspective image of a three-dimensional scene based on two-dimensional images of the three-dimensional scene from a single perspective or multiple perspectives.

[0104] The three-dimensional NVS diffusion model is a deep learning model based on a diffusion model, which is used to generate a new perspective image of a three-dimensional scene. The diffusion model generates target images by gradually removing noise, and can generate high-quality and realistic images. The pre-set three-dimensional NVS diffusion model can be trained based on two-dimensional image samples of a three-dimensional scene from a single perspective or multiple perspectives, and the corresponding new perspective image labels, based on the training method in the prior art.

[0105] A novel view image refers to a two-dimensional image of a three-dimensional scene generated from a different angle than any view in the multi-view image set. It is the target output of the three-dimensional NVS diffusion model, used to supplement or expand the perspective information of the scene. Suppose there is a three-dimensional scene containing a building and the surrounding environment. The multi-view image set is an image taken from four directions (east, south, west, north) of the building. If an image is needed from the northeast direction of the building, then the image from the northeast direction is the novel view image.

[0106] The computing device can adjust the parameters of the three-dimensional NVS diffusion model using a preset algorithm to improve the performance of the model on the multi-view image set. It should be noted that the details of adjusting the parameters of the preset three-dimensional NVS diffusion model can be referred to the related description in the following embodiments, which will not be introduced here.

[0107] In step 203, the preset two-dimensional editor is trained according to the adjusted three-dimensional NVS diffusion model, and the training is used to indicate that the prior knowledge of the adjusted three-dimensional NVS diffusion model is transferred to the two-dimensional editor through knowledge distillation, and the prior knowledge is used to indicate the consistency characteristics of the three-dimensional scene under different perspectives.

[0108] The two-dimensional editor refers to a pre-trained editing model based on a two-dimensional diffusion framework (such as Instuct-pix2pix), which can edit two-dimensional images based on language instructions. In the embodiments of the present disclosure, a LoRA module is added to the two-dimensional editor, and the LoRA module is trained to edit the multi-view image set.

[0109] The prior knowledge refers to the rules and characteristics of the data distribution learned by the model during the parameter adjustment process of the three-dimensional NVS diffusion model. These knowledge are essentially a summary of the consistency characteristics or rules exhibited by the three-dimensional scene under different perspectives. They are refined through the analysis of the geometric structure, appearance features, and physical properties of the three-dimensional scene, and can provide important guidance for the two-dimensional editor, enabling it to better understand and generate novel view images that conform to the laws of the real world.

[0110] In the three-dimensional scene, the prior knowledge can cover multiple aspects of information, including one or more of geometric consistency prior knowledge, appearance consistency prior knowledge, and physical consistency prior knowledge. The geometric consistency prior knowledge is used to indicate the characteristics of the shape, structure, and spatial relationship of objects in the three-dimensional scene remaining consistent under different perspectives. The appearance consistency prior knowledge is used to indicate the characteristics of the appearance attributes of objects in the three-dimensional scene, such as color, texture, lighting, and reflection, remaining consistent under different perspectives. The physical consistency prior knowledge is used to indicate the characteristics of the physical properties of objects in the three-dimensional scene, such as material, motion rules, etc., remaining consistent under different perspectives.

[0111] In the process of transferring the prior knowledge of the three-dimensional NVS diffusion model to the two-dimensional editor, these prior knowledge plays a key guiding role. The prior knowledge can be used as a constraint condition to ensure that the output results obtained by the two-dimensional editor in editing the multi-view images meet the three-dimensional consistency between views. For example, through the geometric consistency prior, the two-dimensional editor can avoid changes in the shape of objects in multiple views after editing.

[0112] Knowledge distillation is a model compression technique that transfers the knowledge of a complex model (teacher model) to a simple model (student model). In this process, the teacher model scores the output of the student model and updates the student model to obtain similar knowledge and performance. In the embodiments of the present disclosure, the three-dimensional NVS diffusion model is used as the teacher model, and the two-dimensional editor is used as the student model. It should be noted that the details of transferring the prior knowledge of the adjusted three-dimensional NVS diffusion model to the two-dimensional editor through knowledge distillation can be referred to the related description in the following embodiments, which will not be introduced here.

[0113] Step 204, editing the multi-view image set by the trained two-dimensional editor.

[0114] The computing device uses the trained two-dimensional editor to edit the multi-view image set. The two-dimensional editor edits the multi-view image set based on the prior knowledge obtained from the three-dimensional NVS diffusion model through knowledge distillation. These editing operations can include changes to the properties of the images, texture adjustments, style transformations, tone modifications, etc.

[0115] After editing, the computing device uses the edited multi-view image set to reconstruct or update the three-dimensional scene. Through this step, the computing device can effectively reflect the modification results of the two-dimensional editor to the three-dimensional scene, realize the reconstruction or update of the three-dimensional scene, and thus generate a three-dimensional scene that is more consistent with user needs and real-world laws.

[0116] In summary, the embodiments of the present disclosure propose a brand-new multi-view three-dimensional scene editing framework based on consistency distillation. The framework distills the prior knowledge of the three-dimensional NVS diffusion model into a two-dimensional instruction-guided editor, thereby achieving multi-view consistent editing results. Subsequently, the edited consistent multi-view images are integrated into the three-dimensional scene, and finally a high-quality three-dimensional scene that meets the editing instructions is generated. In some embodiments, the principle diagram of the multi-view three-dimensional scene editing method based on consistency distillation provided by an example embodiment of the present disclosure is as follows: Figure 3The three-dimensional scene is rendered from the three-dimensional GS model at the selected camera perspective, and then processed through three core stages. Stage one: source scene cross-view motion extraction stage. Fine-tune the three-dimensional NVS diffusion model through the multi-view image set to extract consistent information along the camera trajectory. Stage two: consistency distillation stage of the editing model. According to the adjusted three-dimensional NVS diffusion model, train the preset two-dimensional editor through the distillation loss and regularization loss. Stage three: three-dimensional reconstruction and mask optimization stage. Edit the multi-view image set through the trained two-dimensional editor, and use the edited multi-view image set to reconstruct the three-dimensional scene. Through this innovative framework, the three-dimensional scene can be efficiently edited under the guidance of text instructions while ensuring consistency between multiple views, thereby providing strong support for high-quality three-dimensional content creation. The following is a further elaboration of the three stages.

[0117] Stage one: source scene cross-view motion extraction stage.

[0118] In this stage, first, a reference image is determined from the multi-view image set of the three-dimensional scene. Then, first Gaussian noise is added to each two-dimensional image in the multi-view image set to generate a first multi-view image set. Next, a pre-set three-dimensional NVS diffusion model is used to generate predicted noise by combining the first multi-view image set and the reference image. By minimizing the difference between the predicted noise and the first Gaussian noise, the parameters of the three-dimensional NVS diffusion model are adjusted.

[0119] The reference image is a specific image selected from the multi-view image set to guide the model's training or generation process. It is usually used as a benchmark to help the model better understand and generate images of other perspectives. In the view synthesis task, the reference image can be a user-specified view image, and the model generates images of other views based on it.

[0120] The first Gaussian noise is random noise in each two-dimensional image in the multi-view image set. The first Gaussian noise is used for parameter adjustment of the three-dimensional NVS diffusion model to help the model learn how to recover the original image from the noise. Optionally, the first Gaussian noise is sampled from a standard normal distribution.

[0121] The first multi-view image set is the image set obtained by adding the first Gaussian noise to each two-dimensional image in the multi-view image set. These images are used to adjust the parameters of the three-dimensional NVS diffusion model to help the model learn how to handle noisy data.

[0122] The predicted noise is an estimate of the noise contained in the current input image by the three-dimensional NVS diffusion model. It is used to guide the learning process of the model by minimizing the difference between the predicted noise and the true noise, optimizing the model parameters. The predicted noise is an intermediate result in the training phase, while the final output of the model is the new view image, which is generated through the denoising process.

[0123] In some embodiments, the three-dimensional NVS diffusion model includes a parameter-fixed convolutional layer, a parameter-fixed spatial attention layer, and a parameter-adjustable temporal attention layer. Further, the model parameters can be adjusted by the following steps: first, calculate the temporal loss according to the difference between the predicted noise and the first Gaussian noise; then, by minimizing the temporal loss, fine-tune the parameters of the LoRA module of the temporal attention layer.

[0124] The temporal attention layer is a kind of neural network layer used to process information between multiple frames. In the three-dimensional NVS diffusion model, it helps the model capture the relationship between different view images.

[0125] The LoRA module is used to adjust the model parameters by introducing a low-rank matrix, rather than directly modifying the original weight matrix. That is, a low-rank matrix is added to the temporal attention layer of the three-dimensional NVS diffusion model to be adjusted, and the other parameters of the three-dimensional NVS diffusion model are frozen, and only the low-rank matrix parameters of the LoRA module are adjusted.

[0126] The temporal loss is a loss function that measures the difference between the predicted noise and the true noise (i.e., the first Gaussian noise), with a particular focus on the difference in the time dimension. Optionally, it can be defined by calculating the squared Euclidean distance between the predicted noise and the true noise.

[0127] The three-dimensional NVS diffusion model is trained based on a three-dimensional diffusion architecture. After training is completed, the model can receive one or more two-dimensional view images as input, and can output the corresponding new view image of the three-dimensional scene with the camera parameters of the new view as additional condition information in a display or implicit form. The additional condition information can be one of the inputs of the three-dimensional NVS diffusion model, which means that in the generation task, in addition to the input image itself, additional information is provided to the model. These information is used to guide the model to generate output that meets specific requirements, and the additional condition information can include the position, direction, focal length of the camera, etc., which helps the model to generate new view images more accurately.

[0128] Although recent diffusion-based NVS methods have shown great potential, their geometric consistency and texture fidelity are still insufficient under sparse camera trajectories. To maintain the geometric and texture consistency of the source scene's multi-view under fixed camera trajectories, a LoRA module is injected into the temporal attention layer of the 3D-Unet, a convolutional neural network-based architecture designed specifically for processing three-dimensional image data, in the three-dimensional NVS diffusion model, and the parameters of the LoRA module are fine-tuned to enhance the capabilities of the three-dimensional NVS diffusion model. The reasons for fine-tuning the parameters of the LoRA module to enhance the capabilities of the three-dimensional NVS diffusion model are as follows: 1. Parameter efficiency: The LoRA module adjusts model parameters by introducing a low-rank structure, which is more parameter efficient than directly adjusting the original weight matrix. This means that better performance can be achieved with fewer parameters. 2. Flexibility: The LoRA module only introduces a small number of trainable parameters, and its low-rank matrix can capture the specific features that the model needs when processing new tasks. By fine-tuning the parameters of these low-rank matrices, the model can better adapt to the needs of new tasks.

[0129] In some embodiments, a multi-view image set of a three-dimensional scene under a given fixed camera trajectory is provided A reference image is selected from the multi-view image set as a guide. The multi-view image set is encoded into a latent space representation, resulting in true latent variables z = {z0, z1, …, z N-1 According to different three-dimensional NVS diffusion models, additional conditional information c is obtained in combination with the camera trajectory.

[0130] A standard diffusion optimization objective is used as the training target of the LoRA module, defined as follows:

[0131]

[0132] The definitions of the parameters in the above formula are as follows:

[0133] This is a multi-view image set of a three-dimensional scene, where each represents a two-dimensional image from a certain perspective, and N is the total number of perspectives.

[0134] This is a reference image selected from the multi-view image set, used as a guide image to help the model understand and generate new perspective images.

[0135] z = {z0, z1, …, z N-1}: This is the latent space representation of the multi-view image set after encoding, where each z i is the latent variable corresponding to

[0136] ​c: This is the additional conditioning information obtained from the three-dimensional NVS diffusion model and the camera trajectory.

[0137] This is the loss function that measures the difference between the predicted noise of the model and the true noise (i.e., the first Gaussian noise). The loss function is the target of optimization in the training process, by minimizing this loss, the model can learn how to better generate new view images.

[0138] This is the representation of the expected value, where t is the time step sampled from a uniform distribution , ∈ is the noise sampled from a standard normal distribution . The expected value represents the average of the loss for all possible time steps and noise.

[0139] This is the predicted noise of the model given the latent variable z t , time step t, conditioning information c, and reference view . The difference between this predicted noise and the true noise ∈ is measured by the loss function .

[0140] This is the square of the Euclidean distance used to calculate the difference between the predicted noise and the true noise.

[0141] Phase two: consistency distillation phase of the editing model.

[0142] In this phase, it aims to improve the performance of the two-dimensional editor in the multi-view editing task through consistency distillation, ensuring that it can maintain consistency across views during the editing process. Further, first, according to the input editing instruction, the preset two-dimensional editor is used to edit the multi-view image set, thereby obtaining a second multi-view image set. Then, add a second Gaussian noise to each two-dimensional image in the second multi-view image set to generate a third multi-view image set. Based on the third multi-view image set, the editing distribution and target consistency distribution of the two-dimensional editor are estimated by the adjusted three-dimensional NVS diffusion model. By calculating the difference between the editing distribution and the target consistency distribution, the distillation loss is obtained, and the two-dimensional editor is trained by minimizing the loss.

[0143] The editing instruction is an instruction provided by the user to guide image editing, which can be a text description, used to specify the content and target of editing. For example, the user inputs the editing instruction as "replace the sky in the image with blue".

[0144] The two-dimensional editor can be a diffusion architecture-based model used to edit two-dimensional images. It can modify images according to the input editing instruction.

[0145] The second multi-view image set is a multi-view image set edited by the two-dimensional editor. Each image in the edited multi-view image set is modified according to the editing instruction.

[0146] The second Gaussian noise is random noise added to each two-dimensional image in the second multi-view image set.

[0147] The third multi-view image set is obtained by adding the second Gaussian noise to the second multi-view image set. The third multi-view image set with added noise is used to train the two-dimensional editor.

[0148] The editing distribution describes the actual probability distribution of the results obtained after the multi-view image set is edited by the two-dimensional editor. The editing distribution can be estimated by a fully parameterized trainable three-dimensional NVS diffusion model.

[0149] The target consistency distribution reflects the probability distribution of maintaining three-dimensional consistency between multi-view images. The target consistency distribution can be estimated by a fully parameterized frozen three-dimensional NVS diffusion model.

[0150] The distillation loss is a loss function that measures the difference between the editing distribution and the target consistency distribution. It can be defined by calculating the Kullback-Leibler (KL) divergence between the two distributions.

[0151] In some embodiments, a parameterized model is obtained by copying the adjusted three-dimensional NVS diffusion model. The trainable parameterized model updates the parameters by calculating the standard diffusion loss on the third multi-view image. The editing distribution of the two-dimensional editor is estimated by the copied parameterized model, and the target consistency distribution of the two-dimensional editor is estimated by the parameter frozen adjusted three-dimensional NVS diffusion model. In addition, during the training process, the second reference image is obtained by editing the reference image by the two-dimensional editor; the regularization loss is calculated according to the difference between the second reference image and the reference image in each round of training. By minimizing the distillation loss and the regularization loss, the training process of the two-dimensional editor is further optimized.

[0152] Parameterized model refers to a deep model obtained by copying a frozen parameter model. These parameters can be trained or adjusted to enable the model to better fit the data or complete a specific task. In the embodiments of the present disclosure, the parameterized model is obtained by copying the three-dimensional NVS diffusion model and making its parameters trainable. It is used to estimate the editing distribution of the two-dimensional editor, helping the two-dimensional editor learn how to maintain consistency under multi-view.

[0153] The regularization loss is a loss function that measures the difference between the edited reference image and the reference image. It can be defined by calculating the pixel-level difference or feature-level difference between the images.

[0154] Although the two-dimensional editing model performs well in single-view editing tasks, it is often difficult to maintain consistency across views when adapting to multiple views. To solve this problem, a distillation framework is constructed: the fine-tuned three-dimensional NVS diffusion model is used as the teacher model, and a two-dimensional editor based on the diffusion architecture is used as the student model. During training, the student model is initialized by loading the pre-trained weights, and only the LoRA module is strategically inserted into the self-attention layer to maintain the integrity of the editing instruction representation in the original cross-attention layer. In this way, the performance of the student model can be improved without significantly increasing the computational cost.

[0155] In the process of implementing consistency distillation, the Reward Feedback Learning (ReFL) method is used to sample the two-dimensional editor. Further, the Euler Discretization with Momentum (EDM) noise scheduler is used, which can obtain satisfactory results in fewer denoising steps. The denoising process generates a set of multi-view images I s from the edited image set I e Based on I e , the loss is calculated and the gradient is returned to optimize the two-dimensional editor.

[0156] However, the cost of directly calculating the gradient on the multi-step denoising result is too high. Therefore, the following method can be used: randomly select a step t (such as t∈[15,20]) in the later stage of the denoising process, only calculate the gradient for the last denoising step, and do not calculate the gradient for the remaining steps. This method not only reduces the computational cost, but also improves the training efficiency.

[0157] Then, consistency distillation is achieved by minimizing the KL divergence between the target consistency distribution p cons and the editing distribution p θ of the two-dimensional editor E edit The formula is as follows:

[0158]

[0159] The formula for calculating the gradient of the parameters of the two-dimensional editor E θ is as follows:

[0160]

[0161] where, and This is the score function for the corresponding distribution.

[0162] The parameters in the above formula are defined as follows:

[0163] p cons : is the target consistency distribution, representing the probability distribution of ideal, consistent multi-view editing results.

[0164] p edit : is the editing distribution of the 2D editor, representing the probability distribution of multi-view editing results generated by the 2D editor.

[0165] D KL KL divergence is used to measure the divergence between two probability distributions, i.e., p. cons and p edit The difference between them. In this case, it measures the difference between the editing distribution of the 2D editor and the target consistency distribution.

[0166] This indicates that the distribution follows a standard normal distribution. The latent variable z sampled in the middle and the image set I generated by the 2D editor e Perform the expected value calculation. Expectation value operator; From the standard normal distribution Latent variables z and I sampled in the middle e =E θ (z,I s ): 2D editor E θ Based on the latent variable z and the multi-view image set I of the 3D scene s The generated edited image set I e .

[0167] logp cons (I e )-logp edit (I e ) represents the difference between the log probabilities of two probability distributions.

[0168] This represents the gradient with respect to the two-dimensional editor parameter θ; D represents KL Regarding the gradient of the 2D editor parameter θ, by calculating this gradient, we can determine how to adjust θ to minimize D. KL This means making the collection of multi-view images generated by the 2D editor closer to the target consistent distribution.

[0169] s cons (I e ): Target Consistency Distribution p cons Regarding image set I ethe score function of the target consistency distribution, i.e. the log-probability of the target consistency distribution with respect to the image set I e .

[0170] s edit (I e ): the score function of the edit distribution p edit with respect to the image set I e , i.e. the log-probability of the edit distribution with respect to the image set I e .

[0171] In some embodiments, Gaussian perturbations can be introduced when computing the gradient to avoid zero-probability regions of the data distribution. Therefore, a diffusion denoiser is employed to estimate the score function of these probability distributions through Gaussian diffusion. Further, an adjusted three-dimensional NVS diffusion model is used to model the consistency distribution under noise perturbation, and the pre-trained model is copied to obtain a parameterizable model φ that is parameterically trainable, for approximating the distribution of the actual edit multi-view under noise perturbation.

[0172] The score function is defined as and where I e (t) obeys the diffusion forward process:

[0173] I e (t) = a t E θ (z, I s ) + s t e

[0174] The definitions of the various parameters in the above formula are as follows:

[0175] θ: parameterizable model. A parameterizable model that is parameterically trainable is obtained by copying the parameters of the adjusted three-dimensional NVS diffusion model, for approximating the distribution of the multi-view under noise perturbation output by the two-dimensional editor in the distillation process.

[0176] The score function of the target consistency distribution at time step t, denoted as s e (t) , represents the gradient of the distribution with respect to the image set I

[0177] s edit (I e (t) , t): the score function of the edit distribution at time step t, denoted as s e (t) , represents the gradient of the distribution with respect to the image set I

[0178] I e (t): The set of images subject to diffusion forward process at time step t.

[0179] α t and σ t : The preset noise schedule coefficient at time step t.

[0180] Noise sampled from standard normal distribution to simulate randomness in diffusion process.

[0181] In some embodiments, the initial time step is applied to the reference image to obtain the set of output images Consistent with the source scene cross-view motion extraction stage, according to different three-dimensional NVS diffusion models, the corresponding additional condition information is obtained from the camera trajectory and denoted as c e . In the training process, the set of images generated by the two-dimensional editor with noise is I e (t) , which is encoded as z e (t) . The score function s edit (I e (t) , t) of the editing distribution is parameterized by the network , which is dynamically trained on the two-dimensional editor output by the standard diffusion target:

[0182]

[0183] In the above formula, each parameter is defined as follows:

[0184] min φ : Denotes optimization of the parameter φ to minimize the objective function.

[0185] Denotes the expected calculation of the time step t and the noise ∩ sampled from the uniform distribution and the standard normal distribution .

[0186] ∈ φ : This is the objective function parameterized by the network, which is used to predict the noise ∩ at time step t.

[0187] z e (t) : This is the latent space representation of the set of images I e (t) encoded at time step t.

[0188] c e: This is additional conditioning information obtained from the 3D NVS diffusion model and the camera trajectory.

[0189] This is the set of images generated after editing the reference image by the 2D editor.

[0190] represents the standard diffusion loss, which is the squared Euclidean distance between the predicted noise and the true noise, which is the objective to be minimized.

[0191] In addition, the adjusted 3D NVS diffusion model is used to approximate the score function s of the target consistency distribution cons (I e (t) , t). The gradient of the update of θ is: φ with respect to the 2D editor parameters θ.

[0192]

[0193] The definitions of the above parameters are as follows:

[0194] represents the gradient of the distillation loss with respect to the 2D editor parameters θ.

[0195] represents the expectation calculation over the noise z sampled from the standard normal distribution , and the time step t and the noise ε sampled from the uniform distribution.

[0196] ω(t): is a weight function with respect to the time step t, used to adjust the influence of different time steps in gradient calculation.

[0197] represents the output of the 3D NVS diffusion model at time step t, given the latent variable z e (t) , the condition c e and the reference image .

[0198] represents the output of the trainable parameterized model φ at time step t, given the latent variable z e (t) , the condition c e and the reference image .

[0199] represents the gradient of the output E θ (z, I s ) with respect to θ with respect to the 2D editor parameters θ.

[0200] where t ~ U(0.02T, 0.98T) denotes that the time step t is sampled from the uniform distribution U(0.02T, 0.98T), where T is the total number of time steps. The CFG mechanism can be used when calculating the score function of the target consistency distribution, which can effectively enhance the consistency and coherence of the generated results, so as to better approximate the ideal consistency state. While the mechanism can be ignored when calculating the score function of the editing distribution, which can avoid introducing additional complexity, making the training process more stable, and at the same time can more directly reflect the actual performance of the two-dimensional editor, which is convenient for optimization.

[0201] In order to encourage the original editor to generate more multi-perspective consistent editing results, a distillation loss is used. However, since the teacher model is fitted based on the source scene, its optimization direction may not always ensure that the output result is both consistent and semantically faithful to the editing intention (even if conditioned on In addition, in order to balance between maintaining editing semantics and scene consistency, the output result may become ambiguous, or even weaken the original strong editing ability of the editor.

[0202] To solve this problem, a simple and effective regularization loss is introduced when updating the parameters θ. This loss function can ensure that the editing ability of the editor will not be damaged during the distillation process, and will not produce unexpected ambiguity effect.

[0203] Further, first, when storing the initial time step, based on the editing reference view The output obtained As the training proceeds, when the parameters are updated from θ to θ', the corresponding output becomes The regularization loss is defined as:

[0204]

[0205] The final training target of the two-dimensional editor parameters θ is:

[0206]

[0207] The above parameters are defined as follows:

[0208] The distillation loss represents the distillation loss, which is used to encourage the two-dimensional editor to generate more multi-perspective consistent editing results, and to optimize the output consistency of the two-dimensional editor by comparison with the teacher model.

[0209] The regularization loss represents the regularization loss, which is used to ensure that the editing ability of the two-dimensional editor will not be damaged during the distillation process, while avoiding unintended ambiguity or semantic deviation caused by excessive optimization of consistency.

[0210] a: is a preset hyperparameter used to balance the weight between distillation loss and regularization loss to achieve a trade-off between the two.

[0211] The final training target of the two-dimensional editor parameter θ is the weighted sum of distillation loss and regularization loss. By minimizing this comprehensive loss function, the balance performance of the two-dimensional editor between multi-view consistency and editing ability can be effectively improved.

[0212] In some embodiments, the flowchart of the training process of the two-dimensional editor is as shown in Figure 4 The entire training process is an iterative optimization process, aiming to adjust the parameters of the two-dimensional editor to generate high-quality editing results according to the input reference image. The training process involves data preparation, model initialization, parameter update, etc. The training process includes but is not limited to the following steps:

[0213] Step 401, prepare the dataset: prepare a multi-view image set of a three-dimensional scene Select the reference image as the guide view. Prepare the model: two-dimensional editor with LoRA module, denoted as E θ .

[0214] The LoRA module is a parameter-efficient fine-tuning method that adds a low-rank matrix to the pre-trained model to achieve fine-tuning.

[0215] Step 402, initialize the LoRA parameters of the two-dimensional editor, initialize the loss function: distillation loss and regularization loss, and edit the reference image

[0216] Distillation loss is used for knowledge distillation, which can be used to guide the model to learn the features of the pre-trained model. The purpose of the regularization loss is to prevent the destruction of the editing ability of the two-dimensional editor.

[0217] Step 403, set the time step T, the time step range of fine-tuning is [T1, T2].

[0218] Set the time step T of the noise scheduler, which is used to control the noise addition and removal process of the three-dimensional NVS diffusion model. Define the time step range of the fine-tuning process, which is used to adjust the sampling process of the three-dimensional NVS diffusion model during the training process.

[0219] Step 404, start the iterative training.

[0220] The training process is an iterative process, and the parameters of the editor are updated in the i-th iteration to optimize the editing result, and the initial value of i is 1.

[0221] Step 405, randomly select t in the range of [T, T2].

[0222] Step 406, j = T, …, t + 1, loop to perform diffusion model sampling: Determine whether j is greater than t.

[0223] From j = T, step down to t + 1, loop to perform sampling of the three-dimensional NVS diffusion model: Determine whether j is greater than t.

[0224] where z j is the latent code of the diffusion model at time step j, is the editor model of the current round. If j is greater than t, the gradient is not preserved, and step 406 is continued.

[0225] Step 407, if j is less than or equal to t, perform sampling: Preserve the gradient.

[0226] Step 408, predict the original latent code, and decode it into the editing result

[0227] From z t-1 predict the original latent code according to the noise scheduler, and decode it into the editing result

[0228] Step 409, according to the editing result calculate the distillation loss and regularization loss.

[0229] Step 410, update the two-dimensional editor parameters

[0230] According to the calculated distillation loss and regularization loss, update the two-dimensional editor parameters

[0231] Step 411, determine whether the maximum number of training rounds is reached.

[0232] Determine whether i reaches the maximum number of training rounds, which can be a preset value. If the maximum number of training rounds is reached, the training is ended. If the maximum number of training rounds is not reached, return to step 404, and i + 1 is continued for the next iteration.

[0233] ​​The core of this training process is to utilize the three-dimensional NVS diffusion model and the LoRA module, iteratively optimizing the parameters of the two-dimensional editor, so that it can generate high-quality editing results according to the input reference image. During the training process, random selection of time steps, gradient preservation and other methods are used to ensure that the model can learn effective features, and the model parameters are adjusted through the optimization of the loss function.

[0234] Stage three: three-dimensional reconstruction and mask optimization stage.

[0235] In this stage, based on the trained two-dimensional editor, further reconstruction and optimization of the three-dimensional scene are realized. The process is as follows: according to the input editing instruction, the trained two-dimensional editor is used to edit the multi-view image set to obtain a fourth multi-view image set; the fourth multi-view image set is integrated into the three-dimensional scene to obtain the edited three-dimensional scene representation. The trained editor can generate multi-view images that maintain strong consistency and are faithful to the editing semantics, which are used to reconstruct or update the three-dimensional GS model of the three-dimensional scene

[0236] The fourth multi-view image set is the multi-view image set edited by the trained two-dimensional editor. It contains images modified according to the editing instruction. In the embodiments of the present disclosure, the fourth multi-view image set is the output of the trained two-dimensional editor. These images will be integrated into the three-dimensional scene to generate the final editing result.

[0237] The edited three-dimensional scene representation is the final representation form of the edited three-dimensional scene. It can be a three-dimensional model, a rendered image, or a complete description of the three-dimensional scene containing edited images. In the embodiments of the present disclosure, the edited three-dimensional scene representation is the final output result. It reflects the modification of the three-dimensional scene by the editing instruction and maintains multi-view consistency.

[0238] In the process of reconstructing the three-dimensional GS model, the loss between the rendered image and the corresponding edited view is calculated Loss and learning perceptual image patch similarity (LPIPS) loss. Through the optimization of these loss functions, the robust reconstruction process can further alleviate the slight inconsistency, so as to ensure that the final edited scene representation is consistent and accurate in vision and semantics.

[0239] In addition, the trained editor can generate multi-view consistent output synchronously, without the need for complex iterative optimization in the three-dimensional GS model optimization stage as in previous work. This improvement significantly simplifies the reconstruction process and improves efficiency.

[0240] Due to the use of explicit three-dimensional GS representation, the GS mask is generated by using the semantic segmentation model, so that the reconstruction process of the final three-dimensional GS model is only applied to the mask area, and accurate editing under specific instructions is realized. This method not only improves the efficiency of editing, but also ensures the accuracy and pertinence of editing.

[0241] Through the above process, efficient and accurate editing of the three-dimensional scene is realized on the basis of maintaining the consistency of the editing semantics, which significantly improves the overall performance of three-dimensional reconstruction and optimization.

[0242] Please refer to Figure 5 , which shows the principle diagram of the three-dimensional scene editing method based on consistency distillation multi-view provided by another exemplary embodiment of the present disclosure. The method includes but is not limited to the processing of the following three core stages:

[0243] Stage one: source scene cross-view motion extraction stage.

[0244] 1. Source scene three-dimensional GS: render a multi-view image set of the three-dimensional scene from the original three-dimensional GS model, i.e. multiple two-dimensional view images of the three-dimensional scene.

[0245] 2. Gaussian noise: add Gaussian noise to each two-dimensional image in the multi-view image set to simulate the noise in the real scene.

[0246] 3. Three-dimensional NVS diffusion model: use the three-dimensional NVS diffusion model to process the image set with added noise to generate a new view image set.

[0247] 4. Time series loss: during training, the three-dimensional NVS diffusion model is optimized by time series loss to better generate a new view image set.

[0248] Among them, the frozen convolution layer: represents the convolution layer that is not updated during training. The frozen spatial attention layer: represents the spatial attention layer that is not updated during training. The time series attention layer with trainable LoRA: represents the time series attention layer that can be updated during training.

[0249] Stage two: consistency distillation stage of the editing model.

[0250] 1. Source scene multi-view: obtain a multi-view image set of the three-dimensional scene.

[0251] 2. Two-dimensional editor: according to the user's input editing instruction (for example, changing a certain person into a clown), use the two-dimensional editor to edit the multi-view image set to obtain an edited image set.

[0252] 3. Regularization loss: optimize the two-dimensional editor through the regularization loss to make it better maintain the editing ability.

[0253] 4. Distillation loss: use the distillation loss to further optimize the two-dimensional editor to output multi-view consistent editing results. Calculate the noisy target consistency distribution through the three-dimensional NVS diffusion model with all parameters frozen, and calculate the noisy editing distribution through the three-dimensional NVS diffusion model with all parameters trainable. According to the difference between the noisy target consistency distribution and the noisy editing distribution, the distillation loss is obtained, and the two-dimensional editor is trained by minimizing the loss.

[0254] 5. Standard diffusion loss: train the three-dimensional NVS diffusion model with all parameters trainable by minimizing the standard diffusion loss.

[0255] Stage three: three-dimensional reconstruction and mask optimization stage.

[0256] 1. Trained two-dimensional editor: use the trained two-dimensional editor to process the multi-view image set of the three-dimensional scene.

[0257] 2. Edited consistent multi-view: generate edited consistent multi-view images.

[0258] 3. Reconstruction: integrate the edited multi-view images into the three-dimensional scene to generate the edited three-dimensional GS model, that is, the high-quality three-dimensional result that meets the editing instructions.

[0259] It should be noted that the related details of each step of the three stages can be referred to the related description in the above embodiments, which will not be repeated here.

[0260] Therefore, the scheme provided by the embodiments of the present disclosure has the following beneficial effects:

[0261] On the one hand, by taking the three-dimensional NVS diffusion model as the knowledge source, the view consistency knowledge is transferred to the two-dimensional editor through the distillation process, so that the two-dimensional editor can accurately maintain the view consistency with the three-dimensional scene when editing images. Traditional two-dimensional editors usually have difficulty in directly understanding the view relationship in the three-dimensional scene, and consistency distillation can make up for this deficiency to ensure that the editing results maintain high consistency in multi-view.

[0262] On the other hand, by selecting appropriate camera views, rendering the three-dimensional scene from multiple angles to generate multi-view images, multi-view rendering can comprehensively capture the omnidirectional information of the three-dimensional scene; then fine-tune the NVS diffusion model to extract consistency along the camera trajectory, ensure that images under different views maintain consistency in object position, color, shape, etc., and make the model better adapt to the specific view characteristics of the current scene, thereby significantly improving the quality and consistency of the generated images.

[0263] On the other hand, the 2D editor is trained using distillation loss and regularization loss. Distillation loss is used to measure the gap between the image generated by the 2D editor and the consistency of the 3D scene. By optimizing this loss function, the image generated by the 2D editor can conform to the perspective relationship of the 3D scene. Regularization loss is used to prevent the 2D editor from being compromised and to ensure that it can faithfully reproduce the editing semantics while outputting consistent results.

[0264] In summary, the solution provided in this disclosure successfully transfers the strong consistency prior of the 3D NVS diffusion model to a 2D editor through a consistency distillation framework, achieving efficient and refined multi-view consistent editing. This solution not only solves many problems in existing technologies but also demonstrates superior performance and stability in various complex scenarios. Through multi-view rendering and fine-tuning, as well as optimized training of the 2D editor, this solution achieves an ideal balance between semantic fidelity and geometric consistency, providing strong support for high-quality 3D content creation.

[0265] In one illustrative embodiment, a 3D NVS diffusion model based on ViewCrafter is employed. To reduce computational overhead, a low-resolution variant with an output resolution of 320×512 is selected. After fine-tuning on the source scene, the model provides strong consistent prior knowledge. In the cross-view motion extraction stage of the source scene, the 25-frame segment configuration of ViewCrafter is retained, and the view coverage is expanded to 49 directions. The first view is set as a common reference frame, while the remaining views are organized into two staggered segments. The LoRA rank of the ViewCrafter temporal attention layer is set to 64, and 200 iterations of training are performed on the two preprocessed scene segments, with a fixed learning rate of 5×10⁻⁶. -4 Given a set of source images, i.e., a multi-view input of the source scene. Select reference image As a guide, firstly, the dense stereo matching module DUSt3R is used to... Reconstruct the point cloud, and then generate a set of rendered images along the target camera trajectory. The source image set and the rendered image set are encoded into latent space representations, which generate true latent variables z = {z0, z1, ..., z} respectively. N-1} and conditional latent variables These conditional latent variables Compared with the ground true latent variable z after noise disturbance t (in and σ t Channel splicing is performed for noise scheduling coefficients, and cross-attention injection is used.

[0266] The training target of the timing LoRA is defined as:

[0267]

[0268] When the ViewCrafter is used as the NVS model, the gradient of updating the parameter θ in the two-dimensional editor consistency distillation process is correspondingly:

[0269]

[0270] wherein, is obtained by reconstructing the point cloud from using the dense stereo matching module DUSt3R, then generating a rendered atlas along the same camera trajectory of the source scene, and encoding into the latent space.

[0271] The editor adopts the InstructPix2Pix model, and the LoRA rank configuration is 32. In the distillation process, the source view is edited by batch, and the edited view and the associated condition signal are then input into the three-dimensional NVS diffusion model for consistency distillation. The multi-view consistency can be achieved in about 100 training iterations.

[0272] In addition, the non-classifier guided generation mechanism is adopted, and the default guided scale is set to s T = 7.5 and s I = 1.5. The parameters can be dynamically adjusted according to the specific scene characteristics and the text prompt. The learning rate strategy of 4x10 -4 is adopted in the parameter updating process.

[0273] In summary, the embodiment of the disclosure proposes an innovative consistency distillation framework, which can efficiently transfer the powerful three-dimensional consistency prior in the three-dimensional NVS diffusion model to the two-dimensional editor, thereby realizing fine and multi-view consistent editing effect. By introducing a regularization strategy, the framework performs well in balancing the editing semantic fidelity and geometric consistency, and can effectively prevent distortion while maintaining the original ability of the editor. These high-consistency editing results can be directly integrated into the three-dimensional GS expression without complex iterative updating process. Compared with related technologies, the embodiment of the disclosure solves the problems of slow convergence and texture blur in the traditional IDU framework. Compared with other consistency editing schemes based on attention features or depth information propagation (such as DGE, GaussCtrl, etc.), the embodiment of the disclosure explicitly constrains the editor output space, and uses the guided view to ensure editing fidelity, while ensuring the equality of other views, and ensures consistency through joint optimization. This not only solves the fine-grained inconsistency in related technologies, but also exhibits stable performance in various scenes.

[0274] The scheme provided by the embodiments of the present disclosure is stable in various scene data sets and can achieve consistent and stable three-dimensional editing. Especially in a 360-degree scene with large changes in viewing angle, the scheme of the related technology often produces blur artifacts or even fails due to inconsistent error accumulation between viewing angles, while the present scheme shows strong and stable editing capability. In the 3 scenes of the Tanks-and-Temples data set, using 10 different editing instructions for experiments, the CLIP text-image direction similarity index and the CLIP direction consistency index of the scheme provided by the embodiments of the present disclosure are improved by 19% and 3% respectively compared with the existing scheme, fully proving its significant advantages in editing quality and consistency.

[0275] The following is a device embodiment of the embodiments of the present disclosure. For parts not elaborated in the device embodiment, refer to the technical details disclosed in the above method embodiments.

[0276] The embodiments of the present disclosure provide a three-dimensional scene editing device based on consistent distillation multi-view. The device can be implemented by software, hardware, and a combination of the two to realize all or part of the computing device. The device comprises an acquisition module, an adjustment module, a training module, and an editing module.

[0277] The acquisition module is configured to acquire a multi-view image set of a three-dimensional scene, the multi-view image set being a two-dimensional image set of the three-dimensional scene under multiple viewing angles;

[0278] The adjustment module is configured to adjust parameters of a preset three-dimensional NVS diffusion model according to the multi-view image set, the three-dimensional NVS diffusion model being used to generate a new viewing angle image of the three-dimensional scene based on a two-dimensional image of the three-dimensional scene under a single viewing angle or multiple viewing angles;

[0279] The training module is configured to train a preset two-dimensional editor according to the adjusted three-dimensional NVS diffusion model, the training being used to indicate that prior knowledge of the adjusted three-dimensional NVS diffusion model is transferred to the two-dimensional editor through knowledge distillation, the prior knowledge being used to indicate consistency characteristics of the three-dimensional scene under different viewing angles;

[0280] The editing module is configured to edit the multi-view image set through the trained two-dimensional editor.

[0281] In a possible implementation, the adjustment module is further configured to:

[0282] determine a reference image from the multi-view image set;

[0283] add first Gaussian noise to each two-dimensional image in the multi-view image set to obtain a first multi-view image set;

[0284] According to the first multi-view image set and the reference image, a prediction noise is generated through a preset three-dimensional NVS diffusion model;

[0285] Parameters of the three-dimensional NVS diffusion model are adjusted by minimizing the difference between the prediction noise and the first Gaussian noise.

[0286] In another possible implementation, the three-dimensional NVS diffusion model includes a convolution layer with fixed parameters, a spatial attention layer with fixed parameters, and a temporal attention layer with parameters to be adjusted, and the adjusting module is further configured to:

[0287] According to the difference between the prediction noise and the first Gaussian noise, a temporal loss is calculated;

[0288] Parameters of a LoRA module of the temporal attention layer are adjusted by minimizing the temporal loss.

[0289] In another possible implementation, the training module is further configured to:

[0290] According to the input editing instruction, a second multi-view image set is obtained by editing the multi-view image set through a preset two-dimensional editor;

[0291] Each two-dimensional image in the second multi-view image set is added with a second Gaussian noise to obtain a third multi-view image set;

[0292] According to the third multi-view image set, an editing distribution of the two-dimensional editor and a target consistency distribution are estimated through the adjusted three-dimensional NVS diffusion model, the editing distribution is an actual probability distribution of the multi-view image set after being edited by the two-dimensional editor, and the target consistency distribution is a probability distribution of the multi-view image set after being edited by the two-dimensional editor while maintaining three-dimensional consistency among the views;

[0293] According to the difference between the editing distribution and the target consistency distribution, a distillation loss is calculated;

[0294] The two-dimensional editor is trained by minimizing the distillation loss.

[0295] In another possible implementation, the training module is further configured to:

[0296] A parameter trainable parameterized model is obtained by copying the adjusted three-dimensional NVS diffusion model;

[0297] The editing distribution of the two-dimensional editor is estimated through the copied parameter trainable parameterized model, and the target consistency distribution of the two-dimensional editor is estimated through the adjusted three-dimensional NVS diffusion model with parameters frozen.

[0298] In another possible implementation, the training module is further configured to:

[0299] In the training process, the second reference image is obtained according to editing the reference image by using the two-dimensional editor;

[0300] The regularization loss is calculated according to the difference between the second reference image and the reference image in each round of training;

[0301] The two-dimensional editor is trained by minimizing the distillation loss and the regularization loss.

[0302] In another possible implementation, the editing module is further configured to:

[0303] According to the input editing instruction, the multi-view image set is edited by using the trained two-dimensional editor to obtain a fourth multi-view image set;

[0304] The fourth multi-view image set is integrated into the three-dimensional scene again to obtain an edited three-dimensional scene representation.

[0305] It should be noted that the apparatus provided in the above embodiments is only used as an example to divide the above various functional modules to achieve its functions, and in actual applications, the above functions can be completed by different functional modules according to actual needs, that is, the content structure of the device is divided into different functional modules to complete all or part of the above described functions.

[0306] As to the apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the method, and will not be described in detail here.

[0307] The disclosure embodiments also provide a three-dimensional scene editing apparatus based on consistent distillation multi-view, which includes a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above method.

[0308] The disclosure embodiments also provide a non-volatile computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the above method.

[0309] The disclosure embodiments also provide a computer program product, which includes a computer program or a non-volatile computer readable storage medium carrying the computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0310] Computer readable storage media can be any media that can be read by a machine. Such media can include, but is not limited to, optical discs, magnetic discs, magnetic tapes, electronic memories, and / or any combination thereof. Computer readable storage media can be non-transitory, in that it can be a tangible medium. In some embodiments, computer readable storage media can be non-transitory, in that it can not be a signal per se. In other embodiments, computer readable storage media can be a transitory medium, in that it can be a signal. In some embodiments, computer readable storage media can be non-transitory, in that it can not be a signal per se, but can be a tangible medium. In other embodiments, computer readable storage media can be a transitory medium, in that it can be a signal. In some embodiments, computer readable storage media can be non-transitory, in that it can not be a signal per se, but can be a tangible medium. In other embodiments, computer readable storage media can be a transitory medium, in that it can be a signal.

[0311] The computer programs (or computer readable program instructions) described herein can be downloaded from a computer readable storage medium to respective computing / processing devices or to external computers or external storage devices via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0312] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0313] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0314] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage device. When the computer readable program instructions are loaded into the computer and other programmable data processing apparatus, a series of operational steps are implemented that provide processes such that the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0315] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0316] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0317] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not restrictive. Many modifications and variations of the described embodiments are possible and are within the scope of the disclosure. The selection of terms is intended to best describe the principles of the embodiments, practical application, or technical improvements in the art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for editing a three-dimensional scene based on consistent multi-view distillation, characterized in that, The method comprises: acquiring a multi-view image set of a three-dimensional scene, the multi-view image set being a two-dimensional image set of the three-dimensional scene under multiple views; adjusting parameters of a preset three-dimensional new view synthesis (NVS) diffusion model according to the multi-view image set, the three-dimensional NVS diffusion model being used to generate a new view image of the three-dimensional scene based on two-dimensional images of the three-dimensional scene under a single view or multiple views; training a preset two-dimensional editor according to the adjusted three-dimensional NVS diffusion model, the training being used to indicate that prior knowledge of the adjusted three-dimensional NVS diffusion model is transferred to the two-dimensional editor through knowledge distillation, the prior knowledge being used to indicate consistency characteristics of the three-dimensional scene under different views; editing the multi-view image set through the trained two-dimensional editor.

2. The method of claim 1, wherein, The adjusting of the parameters of the preset three-dimensional new view synthesis (NVS) diffusion model according to the multi-view image set comprises: determining a reference image from the multi-view image set; adding first Gaussian noise to each two-dimensional image in the multi-view image set to obtain a first multi-view image set; generating predicted noise through the preset three-dimensional NVS diffusion model according to the first multi-view image set and the reference image; adjusting the parameters of the three-dimensional NVS diffusion model by minimizing the difference between the predicted noise and the first Gaussian noise.

3. The method of claim 2, wherein, The three-dimensional NVS diffusion model comprises a convolution layer with fixed parameters, a spatial attention layer with fixed parameters, and a time attention layer with parameters to be adjusted, and the adjusting of the parameters of the three-dimensional NVS diffusion model by minimizing the difference between the predicted noise and the first Gaussian noise comprises: calculating a time loss according to the difference between the predicted noise and the first Gaussian noise; adjusting the parameters of a low-rank adaptive LoRA (Long and Short Range Attention) module of the time attention layer by minimizing the time loss.

4. The method of claim 1, wherein, The training of the preset two-dimensional editor according to the adjusted three-dimensional NVS diffusion model comprises: editing the multi-view image set through the preset two-dimensional editor according to an input editing instruction to obtain a second multi-view image set; adding second Gaussian noise to each two-dimensional image in the second multi-view image set to obtain a third multi-view image set; estimating an editing distribution and a target consistency distribution of the two-dimensional editor according to the third multi-view image set through the adjusted three-dimensional NVS diffusion model, the editing distribution being an actual probability distribution of the multi-view image set after editing by the two-dimensional editor, and the target consistency distribution being a probability distribution of the multi-view image set after editing by the two-dimensional editor while maintaining three-dimensional consistency between views; calculating a distillation loss according to the difference between the editing distribution and the target consistency distribution; training the two-dimensional editor by minimizing the distillation loss.

5. The method of claim 4, wherein, The estimating, by the adjusted three-dimensional NVS diffusion model, of the edit distribution and the target consistency distribution of the two-dimensional editor comprises: copying the adjusted three-dimensional NVS diffusion model to obtain a parameter trainable parameterized model; estimating, by the copied parameter trainable parameterized model, the edit distribution of the two-dimensional editor, and estimating, by the parameter frozen adjusted three-dimensional NVS diffusion model, the target consistency distribution of the two-dimensional editor.

6. The method of claim 4, wherein, The method further comprises: during the training process, editing the reference image according to the two-dimensional editor to obtain a second reference image; calculating a regularization loss according to the difference between the second reference image and the reference image in each round of training; The training, by the two-dimensional editor, through the minimization of the distillation loss comprises: training, by the two-dimensional editor, through the minimization of the distillation loss and the regularization loss.

7. The method according to any one of claims 1 to 6, characterized in that, The editing, by the trained two-dimensional editor, of the multi-view image set comprises: editing, by the trained two-dimensional editor, the multi-view image set according to the input edit instruction to obtain a fourth multi-view image set; reintegrating the fourth multi-view image set into the three-dimensional scene to obtain an edited three-dimensional scene representation.

8. An apparatus for editing a multi-view three-dimensional scene based on consistent distillation, comprising: a consistency distillation unit configured to distill consistency from a multi-view three-dimensional scene; and a scene editing unit configured to edit the multi-view three-dimensional scene based on the consistency. The device comprises: an acquisition module configured to acquire a multi-view image set of a three-dimensional scene, the multi-view image set being a two-dimensional image set of the three-dimensional scene under multiple perspectives; an adjustment module configured to adjust, according to the multi-view image set, parameters of a preset three-dimensional NVS diffusion model, the three-dimensional NVS diffusion model being used to generate a new perspective image of the three-dimensional scene based on a two-dimensional image of the three-dimensional scene under a single perspective or multiple perspectives; a training module configured to train, according to the adjusted three-dimensional NVS diffusion model, a preset two-dimensional editor, the training being used to indicate that prior knowledge of the adjusted three-dimensional NVS diffusion model is transferred into the two-dimensional editor through knowledge distillation, the prior knowledge being used to indicate consistency characteristics of the three-dimensional scene under different perspectives; an editing module configured to edit, by the trained two-dimensional editor, the multi-view image set.

9. An apparatus for editing a multi-view three-dimensional scene based on consistent distillation, comprising a memory, a processor and a computer program stored in the memory, wherein the computer program comprises the following steps of: The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

10. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images