Text-driven three-dimensional content controllable editing method and device

Abstract

The embodiments of the present invention relate to the technical field of 3D scene editing, and provide a text-driven controllable editing method and device for three-dimensional content. The method comprises: in response to a text editing instruction, obtaining all original images of the original scene under different shooting angles, wherein all original images contain a target to be edited; constructing a Gaussian radiation field of the initial three-dimensional scene based on all original images; screening multiple key images based on a preset key view selection strategy, and performing two-dimensional diffusion editing on the target to be edited in the multiple key images according to the text editing instruction; using depth perception fusion to map the editing effects of the target to be edited in all original images according to the editing effects of multiple key views, and then optimizing a new Gaussian radiation field to generate and render a target three-dimensional scene that meets the text editing instruction. In this way, three-dimensional content that meets the editing target can be quickly generated, which is applicable to complex scenes and makes the editing results flexible and controllable.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D scene editing, and in particular to a text-driven three-dimensional content controllable editing method and device. Background Art

[0002] As demand for 3D content grows, while pipelines for representing real-world scenes as 3D models are relatively mature, editing these 3D assets remains challenging. Given multiple views of an original scene and user-specified textual instructions, text-driven 3D editing methods can reconstruct a new 3D scene that matches the textual description. Users can freely express their personalized editing needs, providing a new approach for conveniently building 3D content assets that meet user-specific needs.

[0003] Most traditional content editing methods are performed on 2D images. Directly applying such methods to 3D editing will lead to multi-view inconsistencies and artifacts. Current 3D editing methods face multiple difficulties such as multi-view inconsistency, poor editing quality, low controllability of editing effects, and long training and rendering times. The iterative view editing method based on neural radiation fields requires iteratively editing the entire training set and each view edit is independent, which leads to the final generated results often having multi-view inconsistencies and artifacts. In addition, this indirect 3D editing strategy achieved through iterative editing and optimization is very time-consuming and unstable, especially for complex and large-scale scenes. The distillation-based editing method cannot achieve fine editing control, and requires a large amount of video memory, which has high requirements for hardware configuration. Summary of the Invention

[0004] The present invention provides a text-driven controllable editing method and device for three-dimensional content, which is used to solve the defects of the existing technology such as multi-view inconsistency, poor editing quality, low controllability of editing effects, and long training and rendering time, and realizes the rapid generation of three-dimensional content that meets the editing goals, supports user interaction in complex scenes, and has flexible and controllable editing effects.

[0005] The present invention provides a text-driven three-dimensional content controllable editing method, comprising:

[0006] In response to the text editing instruction, obtaining all original images of the original scene at different shooting angles, wherein all the original images contain the object to be edited;

[0007] Constructing a Gaussian radiation field of an initial three-dimensional scene based on all the original images;

[0008] Screening out a plurality of key images from all the original images based on a preset key view selection strategy, and performing two-dimensional diffusion editing on the objects to be edited in the plurality of key images respectively according to the text editing instruction;

[0009] Using depth-aware fusion to map the editing effects of the multiple key images after two-dimensional diffusion editing to the objects to be edited in all the original images;

[0010] A new Gaussian radiation field is optimized based on all the original images after the editing effect mapping, and a target three-dimensional scene that meets the text editing instruction is generated and rendered.

[0011] In one possible implementation, the method further includes:

[0012] Randomly selecting an initial key image from all the original images;

[0013] Performing two-dimensional editing on the object to be edited in the initial key image according to the text editing instruction;

[0014] Mapping the initial key view after 2D editing to each original image pixel by pixel through depth-aware fusion;

[0015] An image in which the modified pixel ratio of the original image after pixel mapping meets a preset condition is used as the next key image to obtain multiple key images.

[0016] In one possible implementation, the method further includes:

[0017] The text editing instruction and the multiple key images are sequentially input into a diffusion editing model, and two-dimensional diffusion editing is performed on the objects to be edited in the multiple key images through the diffusion editing model.

[0018] In one possible implementation, the method further includes:

[0019] Rendering depth maps corresponding to the viewing angles of the multiple key views according to the Gaussian radiation field of the initial three-dimensional scene;

[0020] Depth-aware fusion is used to map the editing effects of the multiple key images after two-dimensional diffusion editing and the depth map to the objects to be edited in all the original images.

[0021] In one possible implementation, the method further includes:

[0022] Performing latent feature alignment and refinement processing on all the original images after editing effect mapping through cross attention and multi-channel normalization operations;

[0023] generating a mask for distinguishing the object to be edited from the background according to the text editing instruction;

[0024] After the editing effect mapping is performed on the mask, local editing control is performed on the objects to be edited in the entire original image.

[0025] In one possible implementation, the method further includes:

[0026] Performing multiple rounds of optimization training on the Gaussian radiation field of the initial three-dimensional scene according to all the original images after editing effect mapping;

[0027] The corresponding constraints are applied to the Gaussian radiation fields generated under different optimization training rounds to obtain a target three-dimensional scene that conforms to the text editing instructions.

[0028] The present invention also provides a text-driven 3D content controllable editing device, comprising the following modules:

[0029] an acquisition module, configured to acquire, in response to a text editing instruction, all original images of the original scene at different shooting angles, wherein the original images contain the object to be edited;

[0030] A construction module, configured to construct a Gaussian radiation field of an initial three-dimensional scene based on all the original images;

[0031] an editing module, configured to select a plurality of key images from the entire original image based on a preset key view selection strategy, and perform two-dimensional diffusion editing on the objects to be edited in the plurality of key images according to the text editing instruction;

[0032] The editing module is further configured to map the editing effects of the multiple key images after two-dimensional diffusion editing to the objects to be edited in all the original images using depth-aware fusion;

[0033] The optimization rendering module is used to optimize a new Gaussian radiation field based on all the original images after the editing effect mapping, and generate and render a target three-dimensional scene that meets the text editing instructions.

[0034] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the text-driven three-dimensional content controllable editing method as described above is implemented.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements any of the above-described text-driven controllable editing methods for three-dimensional content.

[0036] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-described text-driven three-dimensional content controllable editing methods.

[0037] The present invention provides a text-driven, controllable 3D content editing method and device. In response to a text editing instruction, the method obtains all original images of an original scene from different shooting angles, wherein the original images contain the target to be edited. The method then constructs a Gaussian radiation field of the initial 3D scene based on the original images. Multiple key images are selected from the original images based on a preset key view selection strategy, and two-dimensional diffusion editing is performed on the target to be edited in each of the key images according to the text editing instruction. Depth-aware fusion is used to map the editing effects of the multiple key images after the two-dimensional diffusion editing to the target to be edited in the original images. A new Gaussian radiation field is optimized based on the original images after the editing effect mapping, and a target 3D scene that conforms to the text editing instruction is generated and rendered. Compared to the existing techniques, which suffer from inconsistent multi-views, poor editing quality, low controllability of editing effects, and long training and rendering times, the present method can rapidly generate high-quality 3D content that conforms to the editing target. The method is applicable to complex scenes, supports user interaction, and enables flexible and controllable editing results. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 This is one of the flow charts of the text-driven three-dimensional content controllable editing method provided by the present invention.

[0040] Figure 2 This is the second flow chart of the text-driven three-dimensional content controllable editing method provided by the present invention.

[0041] Figure 3 It is a structural schematic diagram of the text-driven three-dimensional content controllable editing device provided by the present invention.

[0042] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0044] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below with reference to the accompanying drawings. The embodiments do not limit the embodiments of the present invention.

[0045] Figure 1 This is one of the flow charts of the text-driven three-dimensional content controllable editing method provided by the present invention, such as Figure 1 As shown, the method specifically includes:

[0046] S11 . In response to a text editing instruction, obtain all original images of the original scene at different shooting angles.

[0047] In this embodiment of the present invention, all original images of an original scene captured from different perspectives are first acquired. These original images contain the target to be edited, and the original images may be RGB images. Simultaneously, text editing instructions are acquired for the original images. These text editing instructions specify the target to be edited and the editing effect information for the original images, such as "change this bear sculpture into a panda" or "change this person's hair red."

[0048] S12. Constructing a Gaussian radiation field of an initial three-dimensional scene based on all the original images.

[0049] The three-dimensional scene is reconstructed under Gaussian representation using all the RGB images of the original scene at different shooting angles, and the initial three-dimensional scene under Gaussian radiation field is constructed.

[0050] Gaussian (GS) radiance field is a 3D scene representation method with good performance in the fields of 3D reconstruction and viewpoint generation. This method uses Gaussians to explicitly represent the scene, and each Gaussian has the following properties: position, covariance matrix, opacity, and spherical harmonic coefficients used to represent its color. The traditional Neural Radiance Field (NeRF) framework relies on multi-layer perceptrons to encode 3D scene information and dense sampling, which results in complex calculations in the training and rendering processes. In addition, NeRF's implicit scene representation limits direct modification of the scene, making the editing task more complex and difficult. Therefore, Gaussian, as a new form of scene construction, is used as the underlying scene framework of this method, and its explicit representation helps to achieve fast and accurate editing.

[0051] S13 , selecting a plurality of key images from all the original images based on a preset key view selection strategy, and performing two-dimensional diffusion editing on the objects to be edited in the plurality of key images according to the text editing instruction.

[0052] Key images are selected from the input original scene's multiple views using a key view selection strategy. The key view selection strategy aims to ensure an appropriate overlap between key views to ensure coverage of the edited area and consistency of the edited effect. This strategy can be described as follows: first, an initial key image is randomly selected. Then, using depth-aware fusion technology, the edited key image is mapped pixel by pixel to the rest of the original scene, achieving diffusion of the edited effect.

[0053] Furthermore, according to the ratio of pixels modified in each original image after mapping to select the next key image. A value of 0 indicates that the image has a larger overlap with the current key image. The value can not only maintain the consistency of the editing effect in the original image, but also effectively expand the editing range of the scene. Specifically, the key view selection strategy can be expressed by the following formula:

[0054] (1)

[0055] in, Indicates the expected ratio value, which can be determined through experience gained from experiments. The largest image will be selected as the next key image, and then the key image selection process will be iterated according to the above process until the ratio of all images is Exceeded the threshold .

[0056] Furthermore, a mask for distinguishing the target to be edited from the background is generated according to the text editing instruction; the target to be edited and the background in multiple key images are segmented according to the mask to obtain multiple key images after segmentation; the text editing instruction and the multiple key images after segmentation are input into the diffusion editing model, and the target to be edited in the multiple key images is two-dimensionally diffused and edited through the diffusion editing model.

[0057] Two-dimensional diffusion editing is achieved by inputting the image and text instructions to be edited into an image-conditional diffusion model. This model utilizes the diffusion process and conditional information (such as the input image and text) to generate or modify images. Its operating principle combines the noise modeling and conditional generation capabilities of the diffusion process, enabling it to generate images that meet the required conditions through an iterative denoising process. Optional image-conditional diffusion models include GLIDE, InstructPix2Pix (IP2P), and DALL E 2. The IP2P model is preferred in this embodiment of the present invention.

[0058] S14. Using depth-aware fusion, based on the editing effects of the multiple key images after two-dimensional diffusion editing, perform editing effect mapping on the objects to be edited in all the original images.

[0059] According to the editing effects of multiple key images after two-dimensional diffusion editing, depth maps corresponding to the multiple key images under corresponding viewing angles are rendered in the initial three-dimensional scene; depth perception fusion is used to map the editing effects of the to-be-edited targets in all original images according to the depth maps.

[0060] Specifically, firstly, the depth map of the key image corresponding to the viewing angle is rendered according to the Gaussian radiation field of the original scene, and then the depth map of each key image frame is Do the following: First, use the key image Internal reference and external reference , in depth data Under the constraint of , the corresponding point of each pixel of the key image in the world coordinate system is obtained :

[0061] (2)

[0062] in, Represents the mapping function from the 2D image coordinate system to the 3D world coordinate system. Then use the training image The internal and external parameters of , map the points in space to each training image, so as to achieve consistent transmission and diffusion of editing effects:

[0063] (3)

[0064] in, Represents the key image Mapping to image Pixel color. A depth map is an image representation used to describe the distance from each pixel in a scene to the camera or observer. It is a two-dimensional image composed of the depth value (i.e., distance) corresponding to each pixel in the image. In addition, in order to retain only relatively accurate mapping pairs and reduce the impact of depth errors and object occlusions, the reprojection error is calculated to filter out results with significant errors. In order to prevent repeated and invalid coverage of pixels in the mapping process of different key images, only unmodified pixels can be covered by new pixels. All original images are used as the training set for Gaussian rendering. It is the data set required for subsequent training of the edited Gaussian scenes. Generally, the image perspective of the training set is consistent with that of the given original image.

[0065] S15. Optimizing a new Gaussian radiation field based on all the original images after the editing effect mapping, generating and rendering a target three-dimensional scene that complies with the text editing instruction.

[0066] The initial three-dimensional scene is subjected to multiple rounds of Gaussian rendering based on all the original images after the editing effect mapping; the corresponding constraints are applied to the Gaussians generated in different rounds to optimize the rendering results, and the target three-dimensional scene that meets the text editing instructions is obtained.

[0067] The present invention provides a text-driven, controllable 3D content editing method. In response to a text editing instruction, all original images of an original scene captured from different viewing angles are obtained, wherein the original images contain a target to be edited. A Gaussian radiation field of an initial 3D scene is constructed based on the original images. Multiple key images are selected from the original images based on a preset key view selection strategy, and two-dimensional diffusion editing is performed on the target to be edited in each of the key images according to the text editing instruction. Depth-aware fusion is used to map the editing effects of the multiple key images after the two-dimensional diffusion editing to the target to be edited in the original images. New Gaussian radiation fields are optimized based on the original images after the editing effect mapping, and a target 3D scene that conforms to the text editing instruction is generated and rendered. Compared to existing techniques that suffer from inconsistent multi-views, poor editing quality, low controllability of editing effects, and long training and rendering times, this method can rapidly generate high-quality 3D content that conforms to the editing target. The method is applicable to complex scenes, supports user interaction, and enables flexible and controllable editing results.

[0068] Figure 2 This is the second flow chart of the text-driven three-dimensional content controllable editing method provided by the present invention, such as Figure 2 As shown, the method specifically includes:

[0069] S21. Randomly select an initial key image from all the original images.

[0070] In this embodiment of the present invention, all original images of an original scene captured from different perspectives are first acquired. These original images contain the target to be edited, and the original images may be RGB images. Simultaneously, text editing instructions are acquired for the original images. These text editing instructions specify the target to be edited and the editing effect information for the original images, such as "change this bear sculpture into a panda" or "change this person's hair red."

[0071] Furthermore, all the RGB images of the original scene obtained at different shooting angles are used to perform three-dimensional scene reconstruction under Gaussian representation to construct an initial three-dimensional scene under Gaussian radiation field.

[0072] Gaussian (GS) radiance field is a 3D scene representation method with good performance in the fields of 3D reconstruction and viewpoint generation. This method uses Gaussians to explicitly represent the scene, and each Gaussian has the following properties: position, covariance matrix, opacity, and spherical harmonic coefficients used to represent its color. The traditional Neural Radiance Field (NeRF) framework relies on multi-layer perceptrons to encode 3D scene information and dense sampling, which results in complex calculations in the training and rendering processes. In addition, NeRF's implicit scene representation limits direct modification of the scene, making the editing task more complex and difficult. Therefore, Gaussian, as a new form of scene construction, is used as the underlying scene framework of this method, and its explicit representation helps to achieve fast and accurate editing.

[0073] Furthermore, multiple key images are screened from all original images based on a preset key view selection strategy, and two-dimensional diffusion editing is performed on the objects to be edited in the multiple key images according to the text editing instruction.

[0074] Specifically, an initial key image is randomly selected from all original images. Key images are selected from the input original scene's multiple views using a key view selection strategy. The key view selection strategy aims to ensure an appropriate overlap between key views to ensure consistent coverage of the edited area and the effect of the edit. This strategy can be described as follows: first, an initial key image is randomly selected. Then, using depth-aware fusion technology, the edited key image is mapped pixel by pixel to the rest of the original images, achieving diffusion of the edit effect.

[0075] S22: Perform two-dimensional editing on the object to be edited in the initial key image according to the text editing instruction.

[0076] Furthermore, the target to be edited in the initial key image is two-dimensionally edited according to the text editing instructions. The text editing instructions can be a set of instructions, usually given in text form, that instruct image editing software or algorithms on how to modify the image. These instructions include, but are not limited to, operations such as moving, scaling, rotating, cropping, and color adjustment.

[0077] In the initial key image, there are one or more specific objects (targets to be edited) that need to be edited. These objects may be a person, object, background element, etc. in the image.

[0078] 2D editing refers to editing an image on a two-dimensional plane, without altering the three-dimensional space. 2D editing typically involves geometric transformations (such as translation, rotation, and scaling), color adjustments, and the application of filters.

[0079] S23, mapping the initial key view after 2D editing to each original image pixel by pixel through depth-aware fusion.

[0080] S24 , taking the image whose modified pixel ratio in the original image after pixel mapping meets the preset conditions as the next key image, to obtain multiple key images.

[0081] Furthermore, according to the ratio of pixels modified in each original image after mapping to select the next key image. A value of 0 indicates that the image has a larger overlap with the current key image. The value can not only maintain the consistency of the editing effect in the original image, but also effectively expand the editing range of the scene. Specifically, the key view selection strategy can be expressed by the following formula:

[0082] (1)

[0083] in, Indicates the expected ratio value, which can be determined through experience gained from experiments. The largest image will be selected as the next key image, and then the key image selection process will be iterated according to the above process until the ratio of all images is Exceeded the threshold .

[0084] S25 , inputting the text editing instruction and the multiple key images into a diffusion editing model in sequence, and performing two-dimensional diffusion editing on the objects to be edited in the multiple key images through the diffusion editing model.

[0085] Two-dimensional diffusion editing is achieved by inputting the image and text instructions to be edited into an image-conditional diffusion model. This model utilizes the diffusion process and conditional information (such as the input image and text) to generate or modify images. Its operating principle combines the noise modeling and conditional generation capabilities of the diffusion process, enabling it to generate images that meet the required conditions through an iterative denoising process. Optional image-conditional diffusion models include GLIDE, InstructPix2Pix (IP2P), and DALL E 2. The IP2P model is preferred in this embodiment of the present invention.

[0086] S26 , rendering depth maps corresponding to the multiple key views at corresponding viewing angles according to the Gaussian radiation field of the initial three-dimensional scene.

[0087] S27 , using depth-aware fusion to perform editing effect mapping on the objects to be edited in all the original images according to the editing effects of the multiple key images after two-dimensional diffusion editing and the depth map.

[0088] Depth-aware fusion is used to map the editing effects of the to-be-edited targets in all original images according to the editing effects of multiple key images after two-dimensional diffusion editing.

[0089] Specifically, firstly, the depth map of the key image corresponding to the viewing angle is rendered according to the Gaussian radiation field of the original scene, and then the depth map of each key image frame is Do the following: First, use the key image Internal reference and external reference , in depth data Under the constraint of , the corresponding point of each pixel of the key image in the world coordinate system is obtained :

[0090] (2)

[0091] in, Represents the mapping function from the 2D image coordinate system to the 3D world coordinate system. Then use the training image The internal and external parameters of , map the points in space to each training image, so as to achieve consistent transmission and diffusion of editing effects:

[0092] (3)

[0093] in, Represents the key image Mapping to image Pixel color. A depth map is an image representation used to describe the distance from each pixel in a scene to the camera or observer. It is a two-dimensional image composed of the depth value (i.e., distance) corresponding to each pixel in the image. In addition, in order to retain only relatively accurate mapping pairs and reduce the impact of depth errors and object occlusions, the reprojection error is calculated to filter out results with significant errors. In order to prevent repeated and invalid coverage of pixels in the mapping process of different key images, only unmodified pixels can be covered by new pixels. All original images are used as the training set for Gaussian rendering. It is the data set required for subsequent training of the edited Gaussian scenes. Generally, the image perspective of the training set is consistent with that of the given original image.

[0094] S28. Performing potential feature alignment and refinement processing on all the original images after editing effect mapping through cross attention and multi-channel normalization operations.

[0095] In the embodiment of the present invention, the potential features of the training views may be aligned and refined through cross-attention and multi-channel normalization operations.

[0096] Feature alignment and refinement are primarily achieved by sequentially feeding the entire original image after editing effects are mapped into two serially connected refinement editing modules. The first refinement editing module uses the entire original image of the original scene as image conditions, while the second refinement editing module uses the entire original image after editing effects are mapped as image conditions. Feature alignment and refinement further improve the consistency of the editing effects, remove artifacts, and enhance the overall quality of the editing.

[0097] The refinement editing module improves upon the IP2P model by introducing additional key view conditions and computing cross-attention with latent features to align the appearance of the image to be refined with the key image, effectively resolving inconsistencies that arise during independent editing. Furthermore, during the diffusion denoising process, the use of multi-channel independent updates and average normalization makes the editing results more stable and reliable, improving the overall quality of the edited image.

[0098] The refinement editing module is an encoder-decoder structure. After adding varying degrees of noise to the input image to be refined, it is fed into the VAE encoder to generate latent features. The text editing instructions are encoded into text conditions by the CLIP encoder and then fed into the U-Net network to generate images under the guidance of the instructions. The U-Net network's primary function is denoising, transforming the noisy image into the target image. The core module in this process is the alignment submodule, which uses attention operations to link the text conditions, image conditions, and latent features of the input image.

[0099] Cross attention refers to the cross attention between the additional key view conditions and the input view latent features. Unlike the IP2P model which only calculates the self-attention of the input image, the additional cross attention can align the appearance of the input image with the key image, further enhancing the consistency. Specifically, the two latent features and Attention can be defined as:

[0100] (4)

[0101] in, is the scaling factor; 、 and Denotes the linear projection operation, which is used to obtain the query, key, and value. Given the input image and key image, the alignment submodule can be described by the following formula:

[0102] (5)

[0103] in, is the total number of key images; is a weighting factor. Considering the self-attention of the input image helps improve the quality and uniqueness of the editing results, while the cross-attention between the input image and the key image improves the editing consistency across different images.

[0104] Multi-channel normalization is designed to mitigate the instability of the diffusion model during distribution prediction, which can lead to discrepancies between output images and affect overall quality. To mitigate this, the refinement editing module adds a variety of noise to the input image at an early stage. The resulting multiple latent features undergo independent diffusion denoising, then their average is calculated and fed into the VAE decoder for decoding. The features in the latent space are converted to pixel space by the decoder. The independent multi-channel updates and average normalization make the editing results more stable and reliable, improving the quality of the refinement results.

[0105] S29: Generate a mask for distinguishing the object to be edited from the background according to the text editing instruction.

[0106] Based on the text editing instructions, a mask is generated to distinguish the editing subject (the target to be edited) from the background, enabling local editing control. The mask is an image that uses 0 or 1 to represent the segmentation between the editing subject and the background. It is generated by a text-based segmentation model.

[0107] Localized editing control allows users to separate the subject from the background using the aforementioned mask, giving them the ability to perform localized editing, making editing more flexible and controllable. Furthermore, this operation can prevent unnecessary editing of the background, further improving overall editing quality.

[0108] S210 , performing local editing control on the objects to be edited in all the original images after the editing effect mapping is performed on the mask.

[0109] The mask and the text-based segmentation model are used to segment the target and background of the original image after the editing effect is mapped. The generated mask retains only the target and background of the original image, and the encoding invalidates the background edit, achieving flexible control over the editing results.

[0110] The aforementioned text-based segmentation model aims to segment images at the pixel level based on a given natural language description (text). Unlike traditional image segmentation, text-based segmentation allows users to describe objects or regions of interest using natural language, and the model generates corresponding segmentation masks based on this text description. Optional text-based segmentation models include CLIPSeg, Grounded SAM, and Lang-SAM. The Lang-SAM model is preferred in this embodiment of the present invention.

[0111] S211 , performing multiple rounds of optimization training on the Gaussian radiation field of the initial three-dimensional scene according to all the original images after editing effect mapping.

[0112] S212 , applying corresponding constraints to the Gaussian radiation fields generated in different optimization training rounds to obtain a target three-dimensional scene that conforms to the text editing instruction.

[0113] Multiple rounds of optimization training are performed on the Gaussian radiation field of the initial 3D scene, based on all original images after editing effects are mapped. Different generations of Gaussians are assigned different constraints to flexibly control the degrees of freedom of their updates. Older Gaussians are subject to stronger constraints to prevent arbitrary adjustments to the scene's infrastructure. Newer Gaussians are given higher degrees of freedom to enable targeted editing of the scene.

[0114] The properties of any Gaussian in the initial 3D scene are recorded as a reference. During training, the mean square error (MSE) loss between the reference state and the current state is used as a constraint loss component to control the generation of Gaussians. Furthermore, the L1 loss and LPIPS loss between the rendered image and the training view, which form the editing loss component, also participate in scene editing optimization. Each loss component is adjusted with a corresponding weight, and together they form the training loss function.

[0115] Specifically, different constraints are imposed on the Gaussians generated in different rounds to limit the disorder of the basic structure, optimize the edited radiation field, and finally quickly generate and render three-dimensional content that meets the text editing goals.

[0116] The disordered infrastructure stems from the limitations of Gaussian models when faced with the stochastic nature of generative guidance. This is because data updates in Gaussian models are directly tied to the loss function, lacking the implicit regulation of neural networks. For editing tasks, this can lead to disordered results and artifacts, hindering convergence in training.

[0117] Gaussians generated in different rounds can be distinguished by a new attribute "generation". The above attribute value Indicates that the Gaussian is in the The generation attribute value of the Gaussian is generated during the densification process. That is, as the Gaussian is cloned and split during training, the value of the "generation" attribute of the newly generated Gaussian will be larger. For the initial Gaussian, the attribute value is assigned to 0.

[0118] Imposing different constraints means that different generations of Gaussians will be given different constraints to flexibly control the update freedom of different Gaussians. The smaller the value of Gaussian, the stronger the constraints will be to prevent the basic structure of the scene from being adjusted arbitrarily. Gaussians with larger values ​​are given more degrees of freedom, allowing the scene to be edited according to the target. Specifically, the properties of any Gaussian in the initial scene will be recorded as a reference. During training, the MSE loss between the reference state and the current state is used to achieve generational control of the Gaussian:

[0119] (6)

[0120] Among them, n represents the total number of Gaussians in the scene; p represents each original attribute of Gaussian, including the center position x, rotation matrix R, scaling matrix S, opacity and color c; Indicates reference value; Represents the generational control loss weight applied to the i-th Gaussian, which is related to its g value.

[0121] The edited radiation field is optimized under the constraints of a training loss function, ultimately generating a 3D Gaussian scene model that conforms to the text editing instructions. The training loss function consists of two components: a constraint loss component that implements the aforementioned Gaussian generation control, and an edit loss component composed of the L1 loss and LPIPS loss between the rendered image and the training view, used for scene reconstruction. Each loss component is weighted accordingly:

[0122] (7)

[0123] in, is the loss weight of different attributes, and this value can be increased or decreased for different editing task attributes. For example, for an editing task that only changes color, the color weight Can be turned down to increase color change flexibility.

[0124] Fast rendering is the rendering of images from any perspective achieved through the splatting fast rasterization algorithm under an efficient parallel computing framework. The above-mentioned parallel computing framework refers to fully utilizing the parallel processing capabilities of the GPU to calculate the value of each pixel in parallel, thereby improving computing efficiency. The above-mentioned splatting fast rasterization algorithm is a rendering technology used to project three-dimensional data onto a two-dimensional screen. Its basic idea is to map each Gaussian ellipsoid onto a two-dimensional imaging plane and form a Gaussian ellipse. Each Gaussian ellipse affects an area around its center according to the Gaussian distribution, and then the distribution affecting the pixel is superimposed by weights to finally calculate the color value of the output pixel.

[0125] The present invention provides a text-driven, controllable 3D content editing method. In response to a text editing instruction, all original images of an original scene captured from different viewing angles are obtained, wherein the original images contain a target to be edited. A Gaussian radiation field of an initial 3D scene is constructed based on the original images. Multiple key images are selected from the original images based on a preset key view selection strategy, and two-dimensional diffusion editing is performed on the target to be edited in each of the key images according to the text editing instruction. Depth-aware fusion is used to map the editing effects of the multiple key images after the two-dimensional diffusion editing to the target to be edited in the original images. New Gaussian radiation fields are optimized based on the original images after the editing effect mapping, and a target 3D scene that conforms to the text editing instruction is generated and rendered. Compared to existing techniques that suffer from inconsistent multi-views, poor editing quality, low controllability of editing effects, and long training and rendering times, this method can rapidly generate high-quality 3D content that conforms to the editing target. The method is applicable to complex scenes, supports user interaction, and enables flexible and controllable editing results.

[0126] The text-driven three-dimensional content controllable editing method provided by the present invention is described in detail below with reference to specific embodiments:

[0127] Thirty multi-view images of a portrait scene taken from various viewpoints are selected as the original scene input. Given a text editing instruction, "Turn this person's hair red." Using multiple views of the original scene as input, the Gaussian radiation field of the original scene is constructed. Training is performed for 30,000 iterations, taking approximately 5 minutes.

[0128] Then, some views are selected as key views through the key view selection strategy, and the expected overlap ratio , when the scale of all views Exceeded the threshold The key view selection ends at , and eight key views are selected. The text editing instructions and the selected key views are sequentially fed into the InstructPix2Pix image-conditional diffusion model for image editing. The image guidance scale and text guidance scale in the IP2P model are set to 1.5 and 7.5, respectively.

[0129] After editing the eight key views selected by the key view selection strategy, the edits are then propagated to the 30 training views using a depth fusion perception operation. The perspective of the training views matches the perspective of the original scene input multi-view. Each training view has a resolution of 994×738, and the depth map rendered from the original Gaussian model also has a resolution of 994×738.

[0130] Optionally, the 30 training views are aligned and refined sequentially using cross-attention and multi-channel normalization. The alignment submodule is responsible for calculating the self-attention of the input view and the cross-attention between the input view and the key views, where the total number of key views is Set the self-attention weight factor to 8 To mitigate the instability of the diffusion model during distribution prediction, the refinement editing module adds three different noises to the input image at an early stage of image input. The three generated latent features undergo independent diffusion denoising, and then their average is calculated and input into the VAE decoder for decoding. The features in the latent space are converted to pixel space by the decoder to obtain the refined training view.

[0131] Furthermore, given the input text instruction "Turn this person's hair red," the text-based segmentation model Lang-SAM can identify the target subject of the edit as "this person's hair," and the remaining content as background. After Lang-SAM generates a mask to distinguish the edit subject from the background, this mask can be added to the corresponding training set to isolate the target subject for subsequent optimization training. This prevents invalid edits in non-target areas in the final result, further improving overall quality and controllability.

[0132] Furthermore, different constraints are applied to the Gaussians generated in different rounds to limit the disorder of the underlying structure and optimize the edited radiation field. Each loss component of the training loss function has a corresponding weight for adjustment. The L1 loss between the rendered image and the training view can be set to a weight of 1.5, the LPIPS loss between the rendered image and the training view can be set to a weight of 1, and the color attribute weight in the constraint loss component is set to 5, the rotation attribute weight is set to 50, the scale attribute weight is set to 50, the opacity attribute weight is set to 50, and the position attribute weight is set to 50. Training the edited radiation field requires approximately 7500 iterations, which takes about 10 minutes on an RTX3090 configuration.

[0133] After training is complete, the intrinsic and extrinsic parameter matrices for the desired viewpoint are input, and through splatting, rapid rasterization can be used to quickly render and generate high-quality color images of the edited scene at any viewpoint. Compared to other methods, the 3D content generated by this method not only better conforms to the user's given text editing instructions, but also provides more controllable and flexible editing effects. Furthermore, the rendering frame rate can reach 35fps at a view resolution of 994×738, and the total training time is 15 minutes, surpassing the training and rendering speeds of most existing 3D editing methods.

[0134] The text-driven 3D content controllable editing device provided by the present invention is described below. The text-driven 3D content controllable editing device described below and the text-driven 3D content controllable editing method described above can refer to each other.

[0135] Figure 3 Schematic diagram of the structure of the text-driven 3D content controllable editing device provided by the present invention, specifically comprising:

[0136] The acquisition module 301 is configured to acquire all original images of the original scene under different shooting angles in response to the text editing instruction, wherein the original images contain the object to be edited. Detailed descriptions are provided in the corresponding descriptions of the above method embodiments, which will not be repeated here.

[0137] The construction module 302 is configured to construct the Gaussian radiation field of the initial three-dimensional scene based on all the original images. Detailed descriptions can be found in the corresponding descriptions of the above method embodiments, which will not be repeated here.

[0138] Editing module 303 is configured to select multiple key images from the original images based on a preset key view selection strategy, and perform two-dimensional diffusion editing on the objects to be edited in the key images according to the text editing instructions. For detailed descriptions, please refer to the corresponding descriptions of the above method embodiments and will not be repeated here.

[0139] The editing module 303 is further configured to map the editing effects of the multiple key images after two-dimensional diffusion editing to the objects to be edited in all the original images using depth-aware fusion. Detailed descriptions are provided in the corresponding descriptions of the above method embodiments, which will not be repeated here.

[0140] The optimization rendering module 304 is configured to optimize a new Gaussian radiation field based on the entire original image after the editing effect mapping, and generate and render a target three-dimensional scene that conforms to the text editing instruction. Detailed descriptions are provided in the corresponding descriptions of the above method embodiments, which will not be repeated here.

[0141] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communications bus 840. The processor 810 may call logic instructions in the memory 830 to execute a text-driven controllable editing method for three-dimensional content, the method comprising: in response to a text editing instruction, obtaining all original images of an original scene from different shooting angles, wherein the original images contain an object to be edited; constructing a Gaussian radiation field of an initial three-dimensional scene based on the original images; selecting multiple key images from the original images based on a preset key view selection strategy, and performing two-dimensional diffusion editing on the objects to be edited in the key images in accordance with the text editing instruction; performing editing effect mapping on the objects to be edited in the original images using depth-aware fusion based on the editing effects of the multiple key images after the two-dimensional diffusion editing; optimizing a new Gaussian radiation field based on the original images after the editing effect mapping, and generating and rendering a target three-dimensional scene that complies with the text editing instruction.

[0142] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0143] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the text-driven three-dimensional content controllable editing method provided by the above methods, the method including: in response to a text editing instruction, obtaining all original images of the original scene under different shooting angles, wherein all the original images contain the target to be edited; constructing a Gaussian radiation field of the initial three-dimensional scene based on all the original images; screening out multiple key images from all the original images based on a preset key view selection strategy, and performing two-dimensional diffusion editing on the targets to be edited in the multiple key images according to the text editing instruction; using depth perception fusion to map the editing effects of the multiple key images after the two-dimensional diffusion editing to the targets to be edited in all the original images; optimizing a new Gaussian radiation field based on all the original images after the editing effect mapping, and generating and rendering a target three-dimensional scene that meets the text editing instruction.

[0144] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the text-driven three-dimensional content controllable editing method provided by the above-mentioned methods, the method comprising: in response to a text editing instruction, obtaining all original images of the original scene under different shooting angles, wherein all the original images contain the target to be edited; constructing a Gaussian radiation field of the initial three-dimensional scene based on all the original images; screening out multiple key images from all the original images based on a preset key view selection strategy, and performing two-dimensional diffusion editing on the targets to be edited in the multiple key images according to the text editing instruction; using depth perception fusion to map the editing effects of the multiple key images after the two-dimensional diffusion editing to the targets to be edited in all the original images; optimizing a new Gaussian radiation field based on all the original images after the editing effect mapping, and generating and rendering a target three-dimensional scene that meets the text editing instruction.

[0145] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0146] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A text-driven 3D content controllable editing method, characterized in that: include: In response to the text editing instruction, obtaining all original images of the original scene at different shooting angles, wherein all the original images contain the object to be edited; Constructing a Gaussian radiation field of an initial three-dimensional scene based on all the original images; Screening out a plurality of key images from all the original images based on a preset key view selection strategy, and performing two-dimensional diffusion editing on the objects to be edited in the plurality of key images respectively according to the text editing instruction; The step of selecting a plurality of key images from all the original images based on a preset key view selection strategy includes: Randomly selecting an initial key image from all the original images; Performing two-dimensional editing on the object to be edited in the initial key image according to the text editing instruction; Mapping the initial key view after 2D editing to each original image pixel by pixel through depth-aware fusion; An image in which the modified pixel ratio of the original image after pixel mapping meets a preset condition is used as the next key image to obtain multiple key images; Using depth-aware fusion to map the editing effects of the multiple key images after two-dimensional diffusion editing to the objects to be edited in all the original images; A new Gaussian radiation field is optimized based on all the original images after the editing effect mapping, and a target three-dimensional scene that meets the text editing instruction is generated and rendered.

2. The method according to claim 1, characterized in that The performing two-dimensional diffusion editing on the objects to be edited in the multiple key images according to the text editing instruction includes: The text editing instruction and the multiple key images are sequentially input into a diffusion editing model, and two-dimensional diffusion editing is performed on the objects to be edited in the multiple key images through the diffusion editing model.

3. The method according to claim 2, characterized in that The method of using depth-aware fusion to map the editing effects of the multiple key images after two-dimensional diffusion editing to the objects to be edited in all the original images includes: Rendering depth maps corresponding to the viewing angles of the multiple key views according to the Gaussian radiation field of the initial three-dimensional scene; Depth-aware fusion is used to map the editing effects of the multiple key images after two-dimensional diffusion editing and the depth map to the objects to be edited in all the original images.

4. The method according to claim 3, characterized in that The method further comprises: Performing latent feature alignment and refinement processing on all the original images after editing effect mapping through cross attention and multi-channel normalization operations; generating a mask for distinguishing the object to be edited from the background according to the text editing instruction; After the editing effect mapping is performed on the mask, local editing control is performed on the objects to be edited in the entire original image.

5. The method according to claim 4, characterized in that The step of optimizing a new Gaussian radiation field based on all original images after the editing effect mapping, and generating and rendering a target three-dimensional scene that complies with the text editing instruction, includes: Performing multiple rounds of optimization training on the Gaussian radiation field of the initial three-dimensional scene according to all the original images after editing effect mapping; The corresponding constraints are applied to the Gaussian radiation fields generated under different optimization training rounds to obtain a target three-dimensional scene that conforms to the text editing instructions.

6. A text-driven 3D content controllable editing device, characterized in that: include: an acquisition module, configured to acquire, in response to a text editing instruction, all original images of the original scene at different shooting angles, wherein the original images contain the object to be edited; A construction module, configured to construct a Gaussian radiation field of an initial three-dimensional scene based on all the original images; An editing module is configured to select a plurality of key images from all the original images based on a preset key view selection strategy, and perform two-dimensional diffusion editing on the objects to be edited in the plurality of key images according to the text editing instruction. The selecting of the plurality of key images from all the original images based on the preset key view selection strategy comprises: randomly selecting an initial key image from all the original images; performing two-dimensional editing on the objects to be edited in the initial key image according to the text editing instruction; mapping the initial key view after the two-dimensional editing to each original image pixel by pixel through depth-aware fusion; and selecting an image in the original image after pixel mapping whose modified pixel ratio meets a preset condition as the next key image, thereby obtaining a plurality of key images. The editing module is further configured to use depth-aware fusion to map the editing effects of the multiple key images after two-dimensional diffusion editing to the objects to be edited in all the original images; The optimization rendering module is used to optimize a new Gaussian radiation field based on all the original images after the editing effect mapping, and generate and render a target three-dimensional scene that meets the text editing instructions.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the text-driven three-dimensional content controllable editing method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text-driven three-dimensional content controllable editing method according to any one of claims 1 to 5 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the text-driven three-dimensional content controllable editing method according to any one of claims 1 to 5 is implemented.

Images