3D scene editing method

Abstract

The invention discloses a 3D scene editing method. The method comprises the following steps: S1, solving camera pose information according to COLMAP; s2, editing an input picture according to a text instruction input by a user to obtain a data set; s3, inputting a picture and an editing instruction of a user by adopting a conditional SD model, and outputting the edited picture; and S4, updating the neural radiation field through iteration. The method has the advantages that the designated area can be edited according to text input through the local image editing model, the non-edited area is kept consistent with the original image, local editing and global editing can be supported in a self-adaptive mode, other areas are not affected during local editing, and the image rendered by the updated neural radiation field has no artificial trace.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D scene editing, and in particular to a 3D scene editing method. Background Art

[0002] With the advancement of technology, 3D scene reconstruction has become very simple and efficient. You only need to take some photos of the scene and then use mature 3D reconstruction algorithms, such as NeRF, Gaussian splatting, etc., to obtain the implicit expression of the 3D scene. However, editing 3D scenes is not easy and requires professionals. The current common editing method is to input instructions and then edit automatically, but this method either only supports global editing or only supports local editing, and local editing may affect non-editing areas, and its artificial traces are obvious. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a 3D scene editing method.

[0004] The object of the present invention is achieved by the following technical solution: A 3D scene editing method comprises the following steps: S1: Solve the camera pose information according to COLMAP; S2: Edit the input image according to the text instructions input by the user to obtain the data set; S3: Use the conditional SD model to input the image and the user's editing instructions, and output the edited image; S4: Update the neural radiation field through iteration.

[0005] Preferably, in step S2, for a globally modified picture, the SD model is directly used to generate the pictures before and after editing; for a locally modified picture, the local picture editing model is used to generate the edited image.

[0006] Preferably, the local image editing model consists of a bounding box generation module and a text graph model. The bounding box generation module is used to generate a bounding box of the area to be edited according to the text description, and the text graph model is used to generate a new edited image according to the user's text input and image input.

[0007] Preferably, the training steps of the local image editing model are as follows: A1: Send the input image to the OV-DINO model, obtain the attention map and features, and generate K bounding boxes through the bounding box generation module; A2: Feed the generated bounding box and text hint into the pre-trained text-based graph model to generate the edited image.

[0008] Preferably, in step A1, the bounding box generation module adopts an end-to-end training method to calculate the similarity between the edited output image and the text prompt, and the loss function is as follows: ; ; in, To calculate the cosine distance between the generated image and the text prompt, To control the orientation in CLIP space, is the text description inferred from the input graph. A text prompt for user input.

[0009] Preferably, only the images with the highest scores are output through the scoring mechanism. ; in, represents the cosine similarity between the text prompt and the edited image, Represents the cosine similarity of the image before and after editing, where only the non-edited area is involved in the calculation.

[0010] Preferably, in step S3, the conditional SD model optimizes the target by supervised learning to minimize the distance between the target noise and the predicted noise in the latent space. ; in, is the initial noise, t represents the time step, is the hidden expression of the input image, For user input commands, is the predicted noise, is the actual noise.

[0011] Preferably, in step S4, a weighted regularization term is introduced when updating the neural radiation field.

[0012] The present invention has the following advantages: the present invention can edit the designated area according to the text input through the local picture editing model, and keep the non-editing area and the original Figure 1 This makes it possible to adaptively support local and global editing, and local editing will not affect other areas. The image rendered by the updated neural radiation field has no artificial traces. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A structural diagram of the 3D scene editing method process. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0015] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0017] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0018] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the product of the invention is usually placed when in use, or the positions or positional relationships commonly understood by those skilled in the art, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0019] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0020] In this embodiment, if Figure 1 As shown, a 3D scene editing method is characterized by: S1: Solve the camera pose information according to COLMAP; S2: Edit the input image according to the text instructions input by the user to obtain the data set; S3: Use the conditional SD model to input the image and the user's editing instructions, and output the edited image; S4: Update the neural radiation field through iteration. The local image editing model can edit the specified area according to the text input, and keep the non-edited area the same as the original. Figure 1 Therefore, it can adaptively support local editing and global editing, and local editing will not affect other areas, and the image rendered by the updated neural radiation field has no artificial traces. In this embodiment, step S1 adopts the existing method for solving, which is not improved here and will not be described in detail.

[0021] Further, in step S2, for globally modified pictures, the SD model is directly used to generate pictures before and after editing; for locally modified pictures, the local picture editing model is used to generate the edited image. Specifically, editing the input picture according to the text instruction input by the user is regarded as a supervised learning task, so a large number of training samples need to be prepared. The samples are divided into two parts: pictures and instructions. For example, the text description "a photo of a cat" is converted into "a photo of a black cat" by using the instruction "turn the cat black". The data set consists of a triple, namely the original description, the user instruction, and the new description modified according to the instruction. For each instruction sample, it is necessary to prepare corresponding picture samples, the picture before editing and the picture after editing. The picture before editing is generated by the SD model according to the text description before editing, and the picture after editing must be consistent with the text description after editing. Since the pictures generated by SD are very random, directly generating the picture pairs before and after editing cannot ensure the consistency of the pictures. Especially when only local modification is required, the two pictures obtained may be too different and cannot meet our needs. For this reason, the pictures are divided into two subcategories, one for those that need global modification, such as stylization; and the other for those that only need local modification.

[0022] In this embodiment, the local image editing model is composed of a bounding box generation module and a text image model. The bounding box generation module is used to generate a bounding box of the area to be edited according to the text description, and the text image model is used to generate a new edited image according to the user's text input and image input. Further, the training steps of the local image editing model are as follows: A1: Send the input image to the OV-DINO model, obtain the attention map and features, and generate K bounding boxes through the bounding box generation module; A2: The generated bounding box and text prompt are fed into the pre-trained text-based graph model to generate the edited image. Furthermore, in step A1, the bounding box generation module uses an end-to-end training method to calculate the similarity between the edited output image and the text prompt. The loss function is as follows: ; ; in, To calculate the cosine distance between the generated image and the text prompt, To control the orientation in CLIP space, is the text description inferred from the input graph. is the text prompt entered by the user. Specifically, since K bounding boxes are generated, there are K possibilities for the edited image. For this reason, a scoring mechanism is introduced, through which only the image with the highest score is output. ; in, represents the cosine similarity between the text prompt and the edited image, Represents the cosine similarity of the image before and after editing, where only the non-edited area is involved in the calculation.

[0023] In this embodiment, in step S3, the conditional SD model optimizes the target by supervised learning to minimize the distance between the target noise and the predicted noise in the latent space. ; in, is the initial noise, t represents the time step, is the hidden expression of the input image, For user input commands, is the predicted noise, is the actual noise.

[0024] Furthermore, in step S4, a weighted regularization term is introduced when updating the neural radiation field. Specifically, the main function of introducing the weighted regularization term when updating the neural radiation field is to suppress noise peaks and eliminate artificial traces.

[0025] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A 3D scene editing method, characterized in that: The following steps are involved: S1: Solve the camera pose information according to COLMAP; S2: Edit the input image according to the text instructions input by the user to obtain the data set; S3: Use the conditional SD model to input the image and the user's editing instructions, and output the edited image; S4: Update the neural radiation field through iteration.

2. The 3D scene editing method according to claim 1, characterized in that: In step S2, for a globally modified picture, the SD model is directly used to generate the pictures before and after editing; for a locally modified picture, the local picture editing model is used to generate the edited image.

3. The 3D scene editing method according to claim 2, characterized in that: The local image editing model consists of a bounding box generation module and a text image model. The bounding box generation module is used to generate a bounding box of the area to be edited according to the text description, and the text image model is used to generate a new edited image according to the user's text input and image input.

4. The 3D scene editing method according to claim 3, characterized in that: The training steps of the local image editing model are as follows: A1: Send the input image to the OV-DINO model, obtain the attention map and features, and generate K bounding boxes through the bounding box generation module; A2: Feed the generated bounding box and text hint into the pre-trained text-based graph model to generate the edited image.

5. The 3D scene editing method according to claim 4, characterized in that: In step A1, the bounding box generation module uses an end-to-end training method to calculate the similarity between the edited output image and the text prompt, and the loss function is as follows: ； ； in, To calculate the cosine distance between the generated image and the text prompt, To control the orientation in CLIP space, is the text description inferred from the input graph. A text prompt for user input.

6. The 3D scene editing method according to claim 5, characterized in that: The scoring mechanism only outputs the image with the highest score. ； in, represents the cosine similarity between the text prompt and the edited image, Represents the cosine similarity of the image before and after editing, where only the non-edited area is involved in the calculation.

7. The 3D scene editing method according to claim 6, characterized in that: In step S3, the conditional SD model optimizes the target by supervised learning to minimize the distance between the target noise and the predicted noise in the latent space. ； in, is the initial noise, t represents the time step, is the hidden expression of the input image, For user input commands, is the predicted noise, is the actual noise.

8. The 3D scene editing method according to claim 7, characterized in that: In step S4, a weighted regularization term is introduced when updating the neural radiation field.

Images