Three-dimensional content controllable editing method and device based on text driving

Abstract

The embodiment of the invention relates to the technical field of 3D scene editing, and provides a three-dimensional content controllable editing method and device based on text driving, and the method comprises the steps: responding to a text editing instruction, and obtaining all original images of an original scene under different shooting visual angles, and all original images comprise a to-be-edited target; constructing a Gaussian radiation field of the initial three-dimensional scene based on all the original images; screening out a plurality of key images based on a preset key view selection strategy, and performing two-dimensional diffusion editing on a to-be-edited target in the plurality of key images according to the text editing instruction; and performing editing effect mapping on the to-be-edited targets in all the original images according to the editing effects of the plurality of key views by adopting depth perception fusion, then optimizing a new Gaussian radiation field, and generating and rendering a target three-dimensional scene conforming to a text editing instruction. Therefore, the three-dimensional content conforming to the editing target can be quickly generated, the method is suitable for complex scenes, and the editing result is flexible and controllable.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D scene editing, and in particular to a text-driven three-dimensional content controllable editing method and device. Background Art

[0002] As people's demand for 3D content grows, although the pipeline for representing real scenes as 3D models is relatively mature, how to edit these 3D assets remains challenging. Given multiple views of the original scene and user-specified text instructions, the text-driven 3D editing method can reconstruct a new 3D scene that conforms to the effect described by the text, and users can express their personalized editing needs at will, providing a new idea for conveniently building 3D content assets that meet users' personalized needs.

[0003] Most traditional content editing methods are performed on 2D images. Directly applying such methods to 3D editing will lead to multi-view inconsistencies and artifacts. Current 3D editing methods face multiple difficulties such as multi-view inconsistency, poor editing quality, low controllability of editing effects, and long training and rendering time. The view iterative editing method based on neural radiation fields requires iterative editing of the entire training set and each view edit is independent, which leads to the final generated results often having multi-view inconsistencies and artifacts. In addition, this indirect 3D editing strategy achieved through iterative editing and optimization is very time-consuming and unstable, especially for complex large-scale scenes. The distillation-based editing method cannot achieve fine editing control, and requires a large amount of video memory, which has high requirements for hardware configuration. Summary of the invention

[0004] The present invention provides a text-driven controllable editing method and device for three-dimensional content, which are used to solve the defects of the prior art such as inconsistency of multiple views, poor editing quality, low controllability of editing effects, and long training and rendering time, and to achieve rapid generation of three-dimensional content that meets the editing goals and supports user interaction in complex scenes and has flexible and controllable editing effects.

[0005] The present invention provides a text-driven three-dimensional content controllable editing method, comprising: In response to the text editing instruction, all original images of the original scene under different shooting angles are acquired, wherein all the original images contain the object to be edited; Constructing a Gaussian radiation field of an initial three-dimensional scene based on all the original images; Screening out a plurality of key images from all the original images based on a preset key view selection strategy, and performing two-dimensional diffusion editing on the objects to be edited in the plurality of key images respectively according to the text editing instruction; Using depth perception fusion to map the editing effects of the multiple key images after two-dimensional diffusion editing to the objects to be edited in all the original images; A new Gaussian radiation field is optimized based on all the original images after the editing effect mapping, and a target three-dimensional scene that complies with the text editing instruction is generated and rendered.

[0006] In a possible implementation, the method further includes: Randomly selecting an initial key image from all the original images; Performing two-dimensional editing on the object to be edited in the initial key image according to the text editing instruction; The initial key view after 2D editing is mapped pixel by pixel to each original image through depth-aware fusion; An image whose modified pixel ratio in the original image after pixel mapping meets a preset condition is used as the next key image to obtain multiple key images.

[0007] In a possible implementation, the method further includes: The text editing instruction and the multiple key images are sequentially input into a diffusion editing model, and two-dimensional diffusion editing is performed on the objects to be edited in the multiple key images through the diffusion editing model.

[0008] In a possible implementation, the method further includes: Rendering depth maps corresponding to the multiple key views at corresponding viewing angles according to the Gaussian radiation field of the initial three-dimensional scene; Depth-aware fusion is used to map the editing effects of the multiple key images after two-dimensional diffusion editing and the depth map to the objects to be edited in all the original images.

[0009] In a possible implementation, the method further includes: Performing latent feature alignment and refinement processing on all the original images after editing effect mapping through cross attention and multi-channel normalization operations; Generating a mask for distinguishing the object to be edited from the background according to the text editing instruction; After the editing effect is mapped according to the mask, local editing control is performed on the objects to be edited in the entire original image.

[0010] In a possible implementation, the method further includes: Performing multiple rounds of optimization training on the Gaussian radiation field of the initial three-dimensional scene according to all the original images after editing effect mapping; The corresponding constraints are applied to the Gaussian radiation fields generated under different optimization training rounds to obtain the target three-dimensional scene that conforms to the text editing instructions.

[0011] The present invention also provides a text-driven 3D content controllable editing device, comprising the following modules: An acquisition module, configured to acquire all original images of the original scene at different shooting angles in response to the text editing instruction, wherein the all original images contain the object to be edited; A construction module, used for constructing a Gaussian radiation field of an initial three-dimensional scene based on all the original images; An editing module, configured to select a plurality of key images from all the original images based on a preset key view selection strategy, and perform two-dimensional diffusion editing on the objects to be edited in the plurality of key images respectively according to the text editing instruction; The editing module is further used to map the editing effects of the objects to be edited in all the original images by using depth perception fusion according to the editing effects of the multiple key images after two-dimensional diffusion editing; The optimization rendering module is used to optimize a new Gaussian radiation field according to all the original images after the editing effect mapping, and generate and render a target three-dimensional scene that meets the text editing instructions.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the text-driven three-dimensional content controllable editing method described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the text-driven three-dimensional content controllable editing method as described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned text-driven three-dimensional content controllable editing methods.

[0015] The text-driven controllable editing method and device of three-dimensional content provided by the present invention obtains all original images of the original scene under different shooting angles in response to text editing instructions, wherein the original images contain the target to be edited; constructs the Gaussian radiation field of the initial three-dimensional scene based on the original images; selects multiple key images from the original images based on a preset key view selection strategy, and performs two-dimensional diffusion editing on the target to be edited in the multiple key images according to the text editing instructions; uses depth perception fusion to map the editing effect of the target to be edited in the original images according to the editing effect of the multiple key images after two-dimensional diffusion editing; optimizes the new Gaussian radiation field according to the original images after the editing effect mapping, and generates and renders the target three-dimensional scene that meets the text editing instructions. Compared with the defects of inconsistent multiple views, poor editing quality, low controllability of editing effects, and long training and rendering time in the prior art, the present solution can quickly generate high-quality three-dimensional content that meets the editing target, is suitable for complex scenes, supports user interaction, and makes the editing results flexible and controllable. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is one of the flow charts of the text-driven three-dimensional content controllable editing method provided by the present invention.

[0018] Figure 2 This is the second flow chart of the text-driven three-dimensional content controllable editing method provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the text-driven three-dimensional content controllable editing device provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0023] Figure 1 is one of the flow charts of the text-driven three-dimensional content controllable editing method provided by the present invention, such as Figure 1 As shown, the method specifically includes: S11 . In response to a text editing instruction, all original images of the original scene at different shooting angles are acquired.

[0024] In the embodiment of the present invention, firstly, all original images of the original scene at different shooting angles are obtained, wherein all original images contain the target to be edited, and the original images may be RGB images. At the same time, text editing instructions for the original images are obtained, and the text editing instructions are used to specify the target to be edited and the editing effect information of editing the original images, such as "turn this sculpture bear into a panda", "turn this person's hair into red", etc.

[0025] S12. Constructing a Gaussian radiation field of an initial three-dimensional scene based on all the original images.

[0026] The three-dimensional scene under Gaussian representation is reconstructed using all the RGB images of the original scene at different shooting angles obtained above, and the initial three-dimensional scene under Gaussian radiation field is constructed.

[0027] Gaussian (GS) radiance field is a 3D scene representation method with good performance in the field of 3D reconstruction and viewpoint generation. This method uses Gaussians to explicitly represent the scene, each of which has the following properties: position, covariance matrix, opacity, and spherical harmonic coefficients used to represent its color. The traditional Neural Radiance Field (NeRF) framework relies on the encoding and dense sampling of 3D scene information by multi-layer perceptrons, which causes the training and rendering processes to be affected by complex calculations. In addition, the implicit scene representation of NeRF limits the direct modification of the scene, making the editing task more complex and difficult. Therefore, Gaussians, as a new form of scene construction, are used as the underlying scene framework of this method, and its explicit representation helps to achieve fast and accurate editing.

[0028] S13: Filter out a plurality of key images from all the original images based on a preset key view selection strategy, and perform two-dimensional diffusion editing on the objects to be edited in the plurality of key images respectively according to the text editing instruction.

[0029] The key image is a part of the image selected by the key view selection strategy from the input original scene multi-view. The key view selection strategy aims to make the key views have a suitable overlap ratio to ensure the coverage of the editing area and the consistency of the editing effect. The implementation of this strategy can be described as follows: first randomly select an initial key image, and then map the edited key image pixel by pixel to other images in all original images through depth perception fusion technology to achieve the diffusion of the editing effect.

[0030] Furthermore, according to the ratio of pixels modified in each original image after mapping to select the next key image. A value of 0 indicates that the image has a larger overlap with the current key image. The value can not only maintain the consistency of the editing effect in the original image, but also effectively expand the editing range of the scene. Specifically, the key view selection strategy can be expressed by the following formula: (1) in, Indicates the expected ratio value, which can be determined through experience gained from experiments. The largest image will be selected as the next key image, and then the key image selection process will be iterated according to the above process until the ratio of all images is Exceeded the threshold .

[0031] Furthermore, a mask for distinguishing the target to be edited and the background is generated according to the text editing instruction; the target to be edited and the background in multiple key images are segmented according to the mask to obtain multiple key images after segmentation; the text editing instruction and the multiple key images after segmentation are input into a diffusion editing model, and two-dimensional diffusion editing is performed on the target to be edited in the multiple key images through the diffusion editing model.

[0032] Two-dimensional diffusion editing is achieved by inputting the image and text instructions to be edited into the image-conditional diffusion model. The image-conditional diffusion model uses the diffusion process and conditional information (such as input image and text) to generate or modify images. Its working principle combines the noise modeling and conditional generation capabilities of the diffusion process, and can generate images that meet the conditional requirements through an iterative denoising process. Optional image-conditional diffusion models include GLIDE, InstructPix2Pix (IP2P), DALL·E 2, etc. The preferred embodiment of the present invention is the IP2P model.

[0033] S14, using depth perception fusion to map the editing effects of the multiple key images after two-dimensional diffusion editing to the objects to be edited in all the original images.

[0034] According to the editing effects of multiple key images after two-dimensional diffusion editing, depth maps corresponding to multiple key images under corresponding viewing angles are rendered in the initial three-dimensional scene; depth perception fusion is used to map the editing effects of the to-be-edited targets in all original images according to the depth maps.

[0035] Specifically, firstly, the depth map of the key image corresponding to the viewing angle is rendered according to the Gaussian radiation field of the original scene, and then the depth map of each key image frame is Do the following: First, use the key image Internal reference and external reference , in depth data Under the constraint of , we get the point corresponding to each pixel of the key image in the world coordinate system. : (2) in, Represents the mapping function from the 2D image coordinate system to the 3D world coordinate system. Then use the training image The internal and external parameters of , map the points in space to each training image, so as to achieve consistent transmission and diffusion of editing effects: (3) in, Represents the key image Mapping to image The pixel color of the scene. A depth map is a two-dimensional image composed of the depth value (i.e., distance) corresponding to each pixel in the image. In addition, in order to retain only relatively accurate mapping pairs and reduce the impact of depth errors and object occlusion, the results with significant errors are filtered out by calculating the reprojection error. In order to prevent repeated and invalid coverage of pixels in the mapping process of different key images, only unmodified pixels can be covered by new pixels. All original images are used as the training set for Gaussian rendering. They are the data sets required for subsequent training of edited Gaussian scenes. Generally, the image perspective of the training set is consistent with the perspective of the given original image.

[0036] S15, optimizing a new Gaussian radiation field according to all the original images after the editing effect mapping, generating and rendering a target three-dimensional scene that complies with the text editing instruction.

[0037] The initial three-dimensional scene is subjected to multiple rounds of Gaussian rendering according to all the original images after the editing effect mapping; the corresponding constraints are applied to the Gaussian optimization rendering results generated in different rounds to obtain the target three-dimensional scene that meets the text editing instructions.

[0038] The text-driven controllable 3D content editing method provided by the present invention obtains all original images of the original scene under different shooting angles in response to the text editing instruction, wherein the all original images contain the target to be edited; constructs the Gaussian radiation field of the initial 3D scene based on the all original images; selects multiple key images from the all original images based on the preset key view selection strategy, and performs two-dimensional diffusion editing on the target to be edited in the multiple key images respectively according to the text editing instruction; uses depth perception fusion to map the editing effect of the target to be edited in the all original images according to the editing effect of the multiple key images after the two-dimensional diffusion editing; optimizes the new Gaussian radiation field according to the all original images after the editing effect mapping, and generates and renders the target 3D scene that meets the text editing instruction. Compared with the defects of inconsistent multiple views, poor editing quality, low controllability of editing effect, and long training and rendering time in the prior art, the present method can quickly generate high-quality 3D content that meets the editing target, is suitable for complex scenes, supports user interaction, and makes the editing results flexible and controllable.

[0039] Figure 2 FIG. 2 is a flow chart of the text-driven three-dimensional content controllable editing method provided by the present invention. Figure 2 As shown, the method specifically includes: S21. Randomly select an initial key image from all the original images.

[0040] In the embodiment of the present invention, firstly, all original images of the original scene at different shooting angles are obtained, wherein all original images contain the target to be edited, and the original images may be RGB images. At the same time, text editing instructions for the original images are obtained, and the text editing instructions are used to specify the target to be edited and the editing effect information of editing the original images, such as "turn this sculpture bear into a panda", "turn this person's hair into red", etc.

[0041] Furthermore, all the RGB images of the original scene at different shooting angles obtained above are used to reconstruct the three-dimensional scene under Gaussian representation to construct an initial three-dimensional scene under Gaussian radiation field.

[0042] Gaussian (GS) radiance field is a 3D scene representation method with good performance in the field of 3D reconstruction and viewpoint generation. This method uses Gaussians to explicitly represent the scene, each of which has the following properties: position, covariance matrix, opacity, and spherical harmonic coefficients used to represent its color. The traditional Neural Radiance Field (NeRF) framework relies on the encoding and dense sampling of 3D scene information by multi-layer perceptrons, which causes the training and rendering processes to be affected by complex calculations. In addition, the implicit scene representation of NeRF limits the direct modification of the scene, making the editing task more complex and difficult. Therefore, Gaussians, as a new form of scene construction, are used as the underlying scene framework of this method, and its explicit representation helps to achieve fast and accurate editing.

[0043] Furthermore, multiple key images are screened from all original images based on a preset key view selection strategy, and two-dimensional diffusion editing is performed on the objects to be edited in the multiple key images according to the text editing instruction.

[0044] Specifically, first randomly select an initial key image from all the original images. The key image is a partial image selected by the key view selection strategy from the input original scene multi-view. The key view selection strategy aims to ensure that the key views have a suitable overlap ratio to ensure the coverage of the editing area and the consistency of the editing effect. The implementation of this strategy can be described as follows: first randomly select an initial key image, and then map the edited key image pixel by pixel to other images in all the original images through the depth perception fusion technology to achieve the diffusion of the editing effect.

[0045] S22: Perform two-dimensional editing on the object to be edited in the initial key image according to the text editing instruction.

[0046] Further, the target to be edited in the initial key image is two-dimensionally edited according to the text editing instructions. The text editing instructions can be a set of instructions, usually given in text form, for instructing image editing software or algorithms on how to modify the image. These instructions include but are not limited to operations such as moving, scaling, rotating, cropping, and color adjustment.

[0047] There are one or more specific objects (to-be-edited targets) in the initial key image that need to be edited. These objects may be certain people, objects, background elements, etc. in the image.

[0048] Two-dimensional editing refers to editing an image on a two-dimensional plane, without involving changes in three-dimensional space. Two-dimensional editing usually includes geometric transformations of images (such as translation, rotation, scaling), color adjustments, filter applications, etc.

[0049] S23, mapping the initial key view after two-dimensional editing to each original image pixel by pixel through depth-aware fusion.

[0050] S24, taking an image in which the modified pixel ratio in the original image after pixel mapping meets a preset condition as the next key image, and obtaining a plurality of key images.

[0051] Furthermore, according to the ratio of pixels modified in each original image after mapping to select the next key image. A value of 0 indicates that the image has a larger overlap with the current key image. The value can not only maintain the consistency of the editing effect in the original image, but also effectively expand the editing range of the scene. Specifically, the key view selection strategy can be expressed by the following formula: (1) in, Indicates the expected ratio value, which can be determined through experience gained from experiments. The largest image will be selected as the next key image, and then the key image selection process will be iterated according to the above process until the ratio of all images is Exceeded the threshold .

[0052] S25 , inputting the text editing instruction and the multiple key images into a diffusion editing model in sequence, and performing two-dimensional diffusion editing on the objects to be edited in the multiple key images through the diffusion editing model.

[0053] Two-dimensional diffusion editing is achieved by inputting the image and text instructions to be edited into the image-conditional diffusion model. The image-conditional diffusion model uses the diffusion process and conditional information (such as input image and text) to generate or modify images. Its working principle combines the noise modeling and conditional generation capabilities of the diffusion process, and can generate images that meet the conditional requirements through an iterative denoising process. Optional image-conditional diffusion models include GLIDE, InstructPix2Pix (IP2P), DALL·E 2, etc. The preferred embodiment of the present invention is the IP2P model.

[0054] S26. Rendering depth maps corresponding to the multiple key views at corresponding viewing angles according to the Gaussian radiation field of the initial three-dimensional scene.

[0055] S27, using depth-aware fusion to map the editing effects of the multiple key images after two-dimensional diffusion editing and the depth map to the objects to be edited in all the original images.

[0056] Depth-aware fusion is used to map the editing effects of the objects to be edited in all original images according to the editing effects of multiple key images after two-dimensional diffusion editing.

[0057] Specifically, firstly, the depth map of the key image corresponding to the viewing angle is rendered according to the Gaussian radiation field of the original scene, and then the depth map of each key image frame is Do the following: First, use the key image Internal reference and external reference , in depth data Under the constraint of , we get the point corresponding to each pixel of the key image in the world coordinate system. : (2) in, Represents the mapping function from the 2D image coordinate system to the 3D world coordinate system. Then use the training image The internal and external parameters of , map the points in space to each training image, so as to achieve consistent transmission and diffusion of editing effects: (3) in, Represents the key image Mapping to image The pixel color of the scene. A depth map is a two-dimensional image composed of the depth value (i.e., distance) corresponding to each pixel in the image. In addition, in order to retain only relatively accurate mapping pairs and reduce the impact of depth errors and object occlusion, the results with significant errors are filtered out by calculating the reprojection error. In order to prevent repeated and invalid coverage of pixels in the mapping process of different key images, only unmodified pixels can be covered by new pixels. All original images are used as the training set for Gaussian rendering. They are the data sets required for subsequent training of edited Gaussian scenes. Generally, the image perspective of the training set is consistent with the perspective of the given original image.

[0058] S28. Performing potential feature alignment and refinement processing on all the original images after editing effect mapping through cross attention and multi-channel normalization operations.

[0059] In the embodiment of the present invention, the potential features of the training views may be aligned and refined through cross-attention and multi-channel normalization operations.

[0060] Feature alignment and refinement are mainly achieved by inputting all the original images after the editing effect is mapped into two serially connected refinement editing modules in sequence. The first refinement editing module uses all the original images of the original scene as image conditions, and the second refinement editing module uses all the original images after the editing effect is mapped as image conditions. Feature alignment and refinement can further improve the consistency of the editing effect, remove artifacts, and improve the overall quality of the editing.

[0061] The above-mentioned refinement editing module is an improvement on the IP2P model. It introduces additional key view conditions and calculates cross attention with latent features to achieve alignment between the appearance of the image to be refined and the key image, effectively solving the inconsistency problem that occurs during independent editing. In addition, in the diffusion denoising process, the use of multi-channel independent updates and average normalization operations makes the editing results more stable and reliable, improving the overall quality of the edited image.

[0062] The refinement editing module is an encoder-decoder structure as a whole. After adding different degrees of noise to the input image to be refined, it is sent to the VAE encoder to generate latent features. The text editing instructions are encoded into text conditions by the CLIP encoder, and then sent to the U-Net network to generate images under the guidance of the instructions. The main function of the U-Net network is denoising, converting the noisy image into the target image. In this process, the core module is the alignment submodule, which uses attention operations to link the text conditions, image conditions, and latent features of the input image.

[0063] Cross-attention refers to the cross-attention between the additional key view conditions and the input view latent features. Unlike the IP2P model, which only calculates the self-attention of the input image, the additional cross-attention can align the input image with the key image appearance and further enhance the consistency. Specifically, the two latent features and Attention can be defined as: (4) in, is the scaling factor; , and represents the linear projection operation, which is used to obtain the query, key and value. Given the input image and key image, the alignment submodule can be described by the following formula: (5) in, is the total number of key images; is a weight factor. Considering the self-attention of the input image helps improve the quality and uniqueness of the editing results, while the cross-attention between the input image and the key image improves the editing consistency across different images.

[0064] The multi-channel normalization operation is to alleviate the instability of the diffusion model in the distribution prediction process, which may cause differences between the output images and affect the overall quality. To mitigate this effect, the refinement editing module adds multiple different noises to the input image at the early stage of image input. The multiple latent features generated are subjected to independent diffusion denoising, and then their average is calculated and input into the VAE decoder for decoding. The features in the latent space are converted to the pixel space by the decoder. The multi-channel independent update and average normalization operation make the editing results more stable and reliable, and improve the quality of the refinement results.

[0065] S29: Generate a mask for distinguishing the object to be edited from the background according to the text editing instruction.

[0066] According to the text editing instructions, a mask is generated to distinguish the editing subject (the target to be edited) from the background to achieve local editing control. The mask is an image that uses numbers 0 or 1 to represent the segmentation of the editing subject and the background. It is generated by a text-based segmentation model.

[0067] Local editing control means that with the support of the above mask, the editing target subject and background can be separated, thus giving users the ability to perform local editing control, making editing more flexible and controllable. In addition, this operation can prevent the background from being edited unnecessarily, further improving the overall editing quality.

[0068] S210, performing local editing control on the objects to be edited in all the original images after the editing effect mapping is performed on the mask.

[0069] According to the mask and the text-based segmentation model, the target to be edited and the background in the entire original image after the editing effect mapping are segmented. The mask generated above is used to retain only the target to be edited and the background of the original image, and the coding performs invalid editing on the background, thus realizing flexible control of the editing results.

[0070] The above-mentioned text-based segmentation model aims to perform pixel-level segmentation of an image based on a given natural language description (text). Different from traditional image segmentation, text-based segmentation allows users to use natural language to describe objects or regions of interest, and the model generates corresponding segmentation masks based on the text description. Optional text-based segmentation models include CLIPSeg, Grounded SAM, Lang-SAM, etc., and the Lang-SAM model is preferred in the embodiment of the present invention.

[0071] S211 . Perform multiple rounds of optimization training on the Gaussian radiation field of the initial three-dimensional scene according to all the original images after editing effect mapping.

[0072] S212, applying corresponding constrained optimized Gaussian radiation fields to the Gaussians generated in different optimization training rounds to obtain a target three-dimensional scene that conforms to the text editing instruction.

[0073] The Gaussian radiation field of the initial 3D scene is optimized for multiple rounds according to all the original images after the editing effect mapping. Different generations of Gaussians will be given different constraints to flexibly control the update freedom of different Gaussians. Stronger constraints are imposed on the earlier generated Gaussians to prevent the basic structure of the scene from being arbitrarily adjusted. Newer generated Gaussians are given higher degrees of freedom so that the scene can be edited according to the target.

[0074] The properties of any Gaussian in the initial 3D scene will be recorded as a reference. During the training process, the MSE loss between the reference state and the current state is used as the constraint loss component to achieve the generational control of the Gaussian. In addition, the L1 loss and LPIPS loss between the rendered image and the training view constitute the editing loss component and also participate in the editing optimization of the scene. Each loss component has a corresponding weight for adjustment, and together they constitute the training loss function.

[0075] Specifically, different constraints are imposed on the Gaussians generated in different rounds to limit the disorder of the underlying structure and optimize the edited radiation field, and finally three-dimensional content that meets the text editing goal is quickly generated and rendered.

[0076] The disorder of the infrastructure is due to the fact that the Gaussian model will expose some limitations when facing the randomness of the generation guide, because the data update of the Gaussian model is directly related to the loss function and lacks the implicit regulation of the neural network. For editing tasks, this characteristic will lead to disorder and artifacts in the editing results, hindering the convergence of the training process.

[0077] Gaussians generated in different rounds can be distinguished by a new attribute "generation". Indicates that the Gaussian is in The generation attribute value of the Gaussian is generated during the round of densification. That is, as the Gaussian is cloned and split during the training process, the value of the "generation" attribute of the newly generated Gaussian will be larger. For the initial Gaussian, the attribute value is assigned to 0.

[0078] Imposing different constraints means that Gaussians of different generations will be given different constraints to flexibly control the update freedom of different Gaussians. The smaller the value of the Gaussian, the stronger the constraints will be to prevent the basic structure of the scene from being adjusted arbitrarily. Gaussians with larger values ​​will be given more degrees of freedom, allowing the scene to be edited according to the goal. Specifically, the properties of any Gaussian in the initial scene will be recorded as a reference. During training, the MSE loss between the reference state and the current state is used to achieve generational control of the Gaussian: (6) Where n represents the total number of Gaussians in the scene; p represents each primitive attribute of the Gaussian, including the center position x, the rotation matrix R, the scaling matrix S, the opacity and color c; Indicates reference value; Represents the generational control loss weight applied to the i-th Gaussian, which is related to its g value.

[0079] The optimization of the edited radiation field is performed under the constraints of the training loss function, and finally a 3D Gaussian scene model that conforms to the text editing instructions is generated. The above training loss function contains two components, where the constraint loss component implements the above Gaussian generation control, and the L1 loss and LPIPS loss between the rendered image and the training view constitute the editing loss component for scene reconstruction. Each loss component has a corresponding weight for adjustment: (7) in, is the loss weight of different attributes. This value can be increased or decreased for different editing task attributes. For example, for an editing task that only changes color, the color weight Can be turned down to increase color change flexibility.

[0080] Fast rendering is to achieve the rendering of images at any viewing angle through the splatting fast rasterization algorithm under an efficient parallel computing framework. The above parallel computing framework refers to making full use of the parallel processing capabilities of the GPU to calculate the value of each pixel in parallel, thereby improving computing efficiency. The above splatting fast rasterization algorithm is a rendering technology for projecting three-dimensional data onto a two-dimensional screen. Its basic idea is to map each Gaussian ellipsoid on a two-dimensional imaging plane to form a Gaussian ellipse. Each Gaussian ellipse affects an area around its center according to the Gaussian distribution, and then the distribution affecting the pixel is superimposed through the weight, and finally the color value of the output pixel is calculated.

[0081] The text-driven controllable 3D content editing method provided by the present invention obtains all original images of the original scene under different shooting angles in response to the text editing instruction, wherein the all original images contain the target to be edited; constructs the Gaussian radiation field of the initial 3D scene based on the all original images; selects multiple key images from the all original images based on the preset key view selection strategy, and performs two-dimensional diffusion editing on the target to be edited in the multiple key images respectively according to the text editing instruction; uses depth perception fusion to map the editing effect of the target to be edited in the all original images according to the editing effect of the multiple key images after the two-dimensional diffusion editing; optimizes the new Gaussian radiation field according to the all original images after the editing effect mapping, and generates and renders the target 3D scene that meets the text editing instruction. Compared with the defects of inconsistent multiple views, poor editing quality, low controllability of editing effect, and long training and rendering time in the prior art, the present method can quickly generate high-quality 3D content that meets the editing target, is suitable for complex scenes, supports user interaction, and makes the editing results flexible and controllable.

[0082] The text-driven three-dimensional content controllable editing method provided by the present invention is described in detail below in conjunction with specific embodiments: 30 multi-view images of portrait scenes taken from multiple perspectives are selected as the input original scene images; the given text editing instruction is "turn this person's hair red". The multiple views of the original scene are used as input to construct the Gaussian radiation field of the original scene. The number of training iterations is 30,000 times, which takes about 5 minutes.

[0083] Then, the key view selection strategy is used to select some views as key views, and the expected overlap ratio , when the scale of all views Exceeded the threshold The selection of key views ends when 8 key views are selected. The text editing instructions and the selected key views are sequentially sent to the InstructPix2Pix image-conditional diffusion model for image editing. The image guidance scale and text guidance scale in the IP2P model are set to 1.5 and 7.5 respectively.

[0084] Then, after the 8 key views selected by the above key view selection strategy are edited, the editing effect is spread to the 30 training views in turn through the deep fusion perception operation. The perspective of the training view is consistent with the perspective of the input original scene multi-view. The resolution of each training view is 994×738, and the resolution of the depth map rendered from the original Gaussian model is also 994×738.

[0085] Optionally, the 30 training views are aligned and refined sequentially through cross-attention and multi-channel normalization operations. The alignment submodule is responsible for calculating the self-attention of the input view and the cross-attention between the input view and the key views, where the total number of key views is Set the self-attention weight factor to 8 To mitigate the instability of the diffusion model during distribution prediction, the refinement editing module adds three different noises to the input image at the early stage of image input. The three generated latent features are subjected to independent diffusion denoising, and then their average is calculated and input into the VAE decoder for decoding. The features in the latent space are converted to the pixel space by the decoder to obtain the refined training view.

[0086] Furthermore, according to the input text instruction "turn this person's hair red", the text-based segmentation model Lang-SAM can identify the target subject of the edit as "this person's hair" and the rest of the content as the background. After Lang-SAM generates a mask for distinguishing the edited subject from the background, the mask can be added to the corresponding training set to separate the target subject for subsequent optimization training, thereby avoiding invalid editing of non-target areas in the final result and further improving the overall quality and controllability.

[0087] Furthermore, different constraints are imposed on the Gaussians generated in different rounds to limit the disorder of the basic structure and optimize the edited radiation field. Each loss component of the training loss function has a corresponding weight for adjustment, where the L1 loss weight between the rendered image and the training view can be set to 1.5, the LPIPS loss weight between the rendered image and the training view can be set to 1, the color attribute weight in the constraint loss component is 5, the rotation attribute weight is 50, the scaling attribute weight is 50, the opacity attribute weight is 50, and the position attribute weight is 50. The training of the edited radiation field requires approximately 7,500 iterations, which takes approximately 10 minutes on an RTX3090.

[0088] After the training is completed, the intrinsic and extrinsic parameter matrices of the required viewing angle are input, and high-quality color images of the edited scene at any viewing angle can be quickly rendered through splatting fast rasterization. Compared with other methods, the three-dimensional content generated by this method is not only more in line with the text editing instructions given by the user, but also the editing effect is more controllable and flexible. In addition, the rendering frame rate can reach 35fps at a view resolution of 994×738, and the total training time is 15min. The training and rendering speed exceeds that of most existing 3D editing methods.

[0089] The text-driven three-dimensional content controllable editing device provided by the present invention is described below. The text-driven three-dimensional content controllable editing device described below and the text-driven three-dimensional content controllable editing method described above can be referred to each other.

[0090] Figure 3 : is a structural schematic diagram of a text-driven three-dimensional content controllable editing device provided by the present invention, which specifically includes: The acquisition module 301 is used to respond to the text editing instruction and acquire all original images of the original scene under different shooting angles, wherein all the original images contain the object to be edited. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0091] The construction module 302 is used to construct the Gaussian radiation field of the initial three-dimensional scene based on all the original images. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0092] The editing module 303 is used to select multiple key images from all the original images based on a preset key view selection strategy, and perform two-dimensional diffusion editing on the objects to be edited in the multiple key images according to the text editing instruction. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0093] The editing module 303 is further used to map the editing effects of the multiple key images after two-dimensional diffusion editing using depth perception fusion to the objects to be edited in all the original images. Detailed descriptions refer to the corresponding descriptions of the above method embodiments, which will not be repeated here.

[0094] The optimization rendering module 304 is used to optimize the new Gaussian radiation field according to the entire original image after the editing effect mapping, generate and render the target three-dimensional scene that meets the text editing instruction. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0095] Figure 4An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a text-driven three-dimensional content controllable editing method, the method comprising: in response to a text editing instruction, obtaining all original images of the original scene under different shooting angles, wherein the all original images contain the target to be edited; constructing a Gaussian radiation field of an initial three-dimensional scene based on the all original images; selecting a plurality of key images from the all original images based on a preset key view selection strategy, and performing two-dimensional diffusion editing on the targets to be edited in the plurality of key images respectively according to the text editing instruction; using depth perception fusion to perform editing effect mapping on the targets to be edited in the all original images according to the editing effects of the plurality of key images after the two-dimensional diffusion editing; optimizing a new Gaussian radiation field according to the all original images after the editing effect mapping, and generating and rendering a target three-dimensional scene that meets the text editing instruction.

[0096] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0097] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the text-driven controllable three-dimensional content editing method provided by the above-mentioned methods, the method including: in response to a text editing instruction, obtaining all original images of the original scene under different shooting angles, wherein all the original images contain targets to be edited; constructing a Gaussian radiation field of an initial three-dimensional scene based on all the original images; screening out multiple key images from all the original images based on a preset key view selection strategy, and performing two-dimensional diffusion editing on the targets to be edited in the multiple key images according to the text editing instruction; using depth perception fusion to map the editing effects of the multiple key images after the two-dimensional diffusion editing to the targets to be edited in all the original images; optimizing a new Gaussian radiation field based on all the original images after the editing effect mapping, and generating and rendering a target three-dimensional scene that meets the text editing instruction.

[0098] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the text-driven controllable three-dimensional content editing method provided by the above-mentioned methods, the method comprising: in response to a text editing instruction, obtaining all original images of the original scene under different shooting angles, wherein the all original images contain targets to be edited; constructing a Gaussian radiation field of an initial three-dimensional scene based on the all original images; screening out a plurality of key images from the all original images based on a preset key view selection strategy, and performing two-dimensional diffusion editing on the targets to be edited in the plurality of key images according to the text editing instruction; using depth perception fusion to perform editing effect mapping on the targets to be edited in the all original images according to the editing effects of the plurality of key images after the two-dimensional diffusion editing; optimizing a new Gaussian radiation field based on the all original images after the editing effect mapping, and generating and rendering a target three-dimensional scene that meets the text editing instruction.

[0099] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text-driven 3D content controllable editing method, characterized in that: include: In response to the text editing instruction, all original images of the original scene under different shooting angles are acquired, wherein all the original images contain the object to be edited; Constructing a Gaussian radiation field of an initial three-dimensional scene based on all the original images; Screening out a plurality of key images from all the original images based on a preset key view selection strategy, and performing two-dimensional diffusion editing on the objects to be edited in the plurality of key images respectively according to the text editing instruction; Using depth perception fusion to map the editing effects of the multiple key images after two-dimensional diffusion editing to the objects to be edited in all the original images; A new Gaussian radiation field is optimized based on all the original images after the editing effect mapping, and a target three-dimensional scene that complies with the text editing instruction is generated and rendered.

2. The method according to claim 1, characterized in that The step of selecting a plurality of key images from all the original images based on a preset key view selection strategy includes: Randomly selecting an initial key image from all the original images; Performing two-dimensional editing on the object to be edited in the initial key image according to the text editing instruction; The initial key view after 2D editing is mapped pixel by pixel to each original image through depth-aware fusion; An image whose modified pixel ratio in the original image after pixel mapping meets a preset condition is used as the next key image to obtain multiple key images.

3. The method according to claim 2, characterized in that The performing two-dimensional diffusion editing on the objects to be edited in the multiple key images respectively according to the text editing instruction includes: The text editing instruction and the multiple key images are sequentially input into a diffusion editing model, and two-dimensional diffusion editing is performed on the objects to be edited in the multiple key images through the diffusion editing model.

4. The method according to claim 3, characterized in that: The method of using depth perception fusion to map the editing effects of the multiple key images after two-dimensional diffusion editing to the objects to be edited in all the original images includes: Rendering depth maps corresponding to the multiple key views at corresponding viewing angles according to the Gaussian radiation field of the initial three-dimensional scene; Depth-aware fusion is used to map the editing effects of the multiple key images after two-dimensional diffusion editing and the depth map to the objects to be edited in all the original images.

5. The method according to claim 4, characterized in that The method further comprises: Performing latent feature alignment and refinement processing on all the original images after editing effect mapping through cross attention and multi-channel normalization operations; Generating a mask for distinguishing the object to be edited from the background according to the text editing instruction; After the editing effect is mapped according to the mask, local editing control is performed on the objects to be edited in the entire original image.

6. The method according to claim 5, characterized in that The step of optimizing a new Gaussian radiation field based on all the original images after the editing effect mapping, and generating and rendering a target three-dimensional scene that complies with the text editing instruction, includes: Performing multiple rounds of optimization training on the Gaussian radiation field of the initial three-dimensional scene according to all the original images after editing effect mapping; The corresponding constraints are applied to the Gaussian radiation fields generated under different optimization training rounds to obtain the target three-dimensional scene that conforms to the text editing instructions.

7. A text-driven 3D content controllable editing device, characterized in that: include: An acquisition module, configured to acquire all original images of the original scene at different shooting angles in response to the text editing instruction, wherein the all original images contain the object to be edited; A construction module, used for constructing a Gaussian radiation field of an initial three-dimensional scene based on all the original images; An editing module, configured to select a plurality of key images from all the original images based on a preset key view selection strategy, and perform two-dimensional diffusion editing on the objects to be edited in the plurality of key images respectively according to the text editing instruction; The editing module is further used to map the editing effects of the objects to be edited in all the original images by using depth perception fusion according to the editing effects of the multiple key images after two-dimensional diffusion editing; The optimization rendering module is used to optimize a new Gaussian radiation field according to all the original images after the editing effect mapping, and generate and render a target three-dimensional scene that meets the text editing instructions.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the text-driven three-dimensional content controllable editing method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text-driven three-dimensional content controllable editing method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the text-driven three-dimensional content controllable editing method according to any one of claims 1 to 6 is implemented.

Images