A method for editing a three-dimensional scene using a semantic graph

Abstract

The application discloses a method for three-dimensional scene editing using a semantic map. The method can allow a user to complete editing of an implicit three-dimensional scene by editing a two-dimensional semantic map, and render a two-dimensional photo after editing at any new view angle. The method mainly includes four steps of generalization training, fine-tuning, optimization and rendering. During generalization training, the method uses data of multiple scenes for training, improves the learning ability of the network for scene geometry, and further accelerates the training convergence speed of a new scene. During fine-tuning, the method takes an RGB picture and a semantic map of a new scene as supervision, trains an MLP in the network, and improves the expression ability of the network for the new scene. During optimization, the method takes a semantic map edited by the user as supervision, trains the MLP in the network, and completes editing of the three-dimensional scene. During rendering, the method can render a two-dimensional RGB photo after editing of the scene according to any camera pose input. The method can be used for scene editing in AR / VR devices.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction and editing, and in particular relates to a method for editing 3D scenes using semantic graphs. Background Technology

[0002] With the emergence of the metaverse trend, achieving high-quality 3D reconstruction of real-world scenes plays a crucial role in the digitization of the real world. In 2020, Neural Radiation Field (NeRF) technology pioneered the introduction of neural networks into the field of 3D reconstruction, enabling high-quality reconstruction of real-world scenes using only a few RGB images, allowing users to obtain high-resolution rendering results from new perspectives. In 2021, MVSNeRF technology integrated MVSNet, a network used for depth prediction, into the NeRF framework, significantly improving the network's training speed. Also in 2021, semanticNeRF technology enabled NeRF to render not only RGB images from new perspectives but also semantic images. In recent years, many other technologies have explored how to improve the rendering quality of NeRF, how to increase its training speed, and how to expand its application scenarios, but currently, few technologies have attempted to edit the implicit 3D scene in NeRF by editing the semantic graph.

[0003] Editing 3D scenes has significant commercial application value. Adding scene editing functionality to AR applications can greatly enhance the user experience. Currently, the main technologies for editing 3D scenes include modelers constructing 3D object models, and mobile apps using toolkits like ARcore to place these 3D objects into the scene; or modelers manually constructing 3D scenes and then manually modifying the information of 3D objects using 3D editing software. However, technologies for editing implicit 3D scenes constructed using NeRF technology are still relatively few. Recent technologies include object-NeRF, which constructs an MLP for each object and edits the scene by manipulating the corresponding MLP for each object; and IDE-3D technology, which uses GANs for face editing. However, there is currently no technology that can perform generalized editing of implicit 3D scenes both indoors and outdoors by editing semantic maps, and achieve 2D photo rendering with viewpoint consistency from any perspective. Summary of the Invention

[0004] The purpose of this invention is to provide a method for editing 3D scenes using semantic graphs, specifically for 3D implicit scenes constructed using NeRF technology.

[0005] The objective of this invention is achieved through the following technical solution: a method for editing 3D scenes using semantic graphs, characterized by comprising the following steps:

[0006] (1) In the generalization training phase, a 3D scene semantic editing network consisting of 2D convolutional modules, 3D convolutional modules, and an MLP is constructed. The input consists of RGB images of the scene, semantic maps, and the camera poses corresponding to each image. The 2D convolutional modules, 3D convolutional modules, and the MLP are trained to improve the network's ability to learn scene geometric information, thereby accelerating the training speed of subsequent steps, including:

[0007] (1.1) Obtain the 2D feature map of the input RGB image through a 2D convolution module;

[0008] (1.2) Use homography transformation to convert 2D feature maps from different perspectives to the reference perspective, and then obtain the loss cube by calculating the variance between 2D feature maps from different perspectives.

[0009] (1.3) The neural coding cube is obtained by extracting the features of the loss cube through the 3D convolution module;

[0010] (1.4) Based on the camera pose corresponding to a certain image, a ray is formed, and the neural coding cube is uniformly sampled along the ray to obtain the feature value f of each sampling point. k and three-dimensional coordinates x k For the three-dimensional coordinates x of the sampling point k , Ray direction d k Perform position encoding;

[0011] (1.5) The feature values ​​f of the sampling points k The position encoding result of the three-dimensional coordinates of the sampling point γ(x) k ), the position encoding result of the ray direction γ(d) k The input image and its RGB values ​​are used as inputs to a multilayer perceptron (MLP), where γ(·) is the position encoding function, and the output volume density value σ is... k Color value r k and semantic value sk;

[0012] (1.6) Based on the volume density value σ output by the MLP k Color value r k and semantic value s k The color value c of a 2D pixel corresponding to a ray is obtained using volume rendering methods. t and semantic value s t :transparency Regarding the color value c i Compared to the true color value The L2 norm is used to calculate the color loss value: Regarding the semantic value s i With true semantic value The semantic loss value is obtained by calculating the cross-entropy loss function. in The semantics of the point obtained from the volume rendering are the probabilities of category l. The probability that the true semantic graph belongs to category l at this point is represented by L, where L is the total number of semantic categories; the total loss value is obtained by weighted summing of the color loss value and the semantic loss value: L total =L c +λL s λ is the semantic loss weight;

[0013] (1.7) Update the parameters of the 2D convolutional module, 3D convolutional module and MLP using the total loss value through gradient backpropagation;

[0014] (1.8) Use the multi-scene data in the dataset to repeat the above steps to train the network, improve the network's ability to learn scene geometric information, and thus improve the scene generalization ability;

[0015] (2) In the fine-tuning stage, input the RGB image, semantic map and corresponding camera pose of the scene to be edited, repeat steps (1.1) to (1.6) to obtain the total loss value, but do not update the parameters in the 2D convolutional module and 3D convolutional module during gradient backpropagation, only update the parameters in the MLP to improve the expressive ability of the 3D scene semantic editing network for the scene to be edited.

[0016] (3) Optimization phase, specifically: the user edits the semantic graph; the feature values ​​f of the sampling points are... k The position encoding result of the three-dimensional coordinates of the sampling point γ(x) k ), the position encoding result of the ray direction γ(d) k The MLP takes the RGB values ​​of the input image as input and the volume density value σ as output. k and semantic value s k The semantic value s of the pixel corresponding to the ray is obtained using the volume rendering method. i :transparency The loss value during the optimization phase is calculated. in The semantics of the point obtained from the volume rendering are the probabilities of category l. The loss value represents the probability that the edited semantic graph belongs to class l at this point. Then, the parameters in the MLP are updated using the gradient backpropagation method to complete the editing of the implicit 3D scene.

[0017] (4) During the rendering stage, after the user provides an arbitrary camera pose, the neural coding cube is sampled using the new camera pose to obtain the feature value f′. k The feature values ​​f′ of the sampling points k The position encoding result of the three-dimensional coordinates of the sampling point γ(x′) k ), the position encoding result of the ray direction γ(d′) kInput the RGB values ​​of the scene image to be edited into the MLP, and output the volume density value σ′ of the scene after editing. k and color value r′ k Using volume rendering formula This will give you the RGB value c′ of a specific pixel in the RGB image after scene editing at the user-specified camera pose. i After performing the above operations on all pixels of the image, you will obtain the RGB image of the entire scene after editing.

[0018] Furthermore, in step (1), the generalization training scenario can be different from the editing scenario. Experiments have shown that the structure of the 3D scene semantic editing network can significantly improve the training speed of subsequent functions.

[0019] Furthermore, the method for extracting image features using a 2D convolution module in step (1.1) is F. i =T(I i ), where T represents a 2D CNN network with a 4x downsampling structure, I i The representative dimension is H. i ×W i ×3 RGB image, F i The representative dimension is H. i / 4×W i A 2D feature map of / 4×C, where H i and W i These represent the length and width of the image, respectively. C represents the number of channels in the 2D feature map and also the number of convolutional kernels in the 2D CNN network.

[0020] Further, the homography transformation method in step (1.2) involves obtaining the parameter matrix Φ = [K, R, t] from the known intrinsic and extrinsic parameters of the camera, where K represents the intrinsic parameter matrix, and R and t represent the camera's rotation and translation, respectively. These two matrices together form the extrinsic parameter matrix. The homography transformation matrix can be obtained using the following formula: in K represents the homography transformation matrix at depth z from viewpoint i to the reference viewpoint. i R represents the intrinsic parameter matrix corresponding to viewpoint i. i t represents the camera rotation matrix corresponding to viewpoint i. i The n1 represents the camera translation vector corresponding to viewpoint i, and the n1 represents the camera optical axis corresponding to the viewpoint image; all feature maps F i The reference perspective can be changed in the following ways: Where (u, v) represent the pixel coordinates of the reference viewpoint, F i，z The feature map corresponding to viewpoint i is transformed to the feature map at depth z of the reference viewpoint.

[0021] Furthermore, the method for calculating the variance between feature maps from different viewpoints to obtain the loss cube in step (1.2) is P(u, v, z) = Var(F i，z (u, v)), where P(u, v, z) represents the value of the loss cube at point (u, v, z), F i，z (u, v) represents the value of the feature map of view i at depth z of the reference view point (u, v) after homography transformation, and Var represents the variance between multiple view feature maps.

[0022] Furthermore, in step (1.4), the formula for the position encoding function γ(·) is as follows:

[0023] γ(p)=(sin(2 0 πp), cos(2 0 πp), …, sin(2 Q-1 πp), cos(2 Q-1 πp))

[0024] For three-dimensional coordinates x k When performing position encoding, Q is set to 10, and the ray direction d is... k When performing position encoding, Q is set to 4.

[0025] Furthermore, in step (1.5), the RGB value of the input image refers to the high-dimensional RGB vector obtained at the pixel point by superimposing all the input images together, where the pixel corresponds to the pixel point of the image obtained by volume rendering.

[0026] Furthermore, the semantic graphs of the scenes in steps (1) and (2) are obtained by semantically segmenting the RGB images using PSPNet semantic segmentation technology.

[0027] Furthermore, in step (3), the edited semantic graph is used as the supervision of the network, and gradient backpropagation only updates the parameters in the MLP to achieve the editing of the implicit 3D scene.

[0028] Furthermore, in step (3), the user uses a visual script program or Photoshop software to edit the semantic map colors.

[0029] Furthermore, in step (4), the user can obtain a two-dimensional rendered image of the implicit three-dimensional scene after editing from any viewpoint.

[0030] The beneficial effects of this invention are: the method allows users to edit implicit 3D scenes by editing a 2D semantic graph, and render the edited 2D photo from any new perspective. This method mainly includes four steps: generalization training, fine-tuning, optimization, and rendering. During generalization training, this method uses data from multiple scenes for training, improving the network's ability to learn scene geometry, thereby accelerating the convergence speed of training on new scenes. During fine-tuning, this method uses the RGB image and semantic graph of the new scene as supervision to train the MLP in the network, improving the network's ability to represent new scenes. During optimization, this method uses the user-edited semantic graph as supervision to train the MLP in the network, completing the 3D scene editing. During rendering, this method can render a 2D RGB photo of the scene after editing based on any input camera pose. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A flowchart illustrating a method for editing a 3D scene using a semantic graph, as provided in one embodiment;

[0033] Figure 2 This is a 2D convolutional network structure provided in one embodiment;

[0034] Figure 3 This is a 3D convolutional network structure provided in one embodiment;

[0035] Figure 4 An MLP network structure is provided as an example. Detailed Implementation

[0036] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0037] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0038] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0039] This invention provides a method for editing 3D scenes using semantic graphs, such as... Figure 1 As shown, the method includes the following steps:

[0040] (1) In the generalization training phase, a 3D scene semantic editing network consisting of 2D convolutional modules, 3D convolutional modules, and an MLP is constructed. The input consists of RGB images of the scene, semantic maps, and the camera poses corresponding to each image. The 2D convolutional modules, 3D convolutional modules, and the MLP are trained to improve the network's ability to learn scene geometric information, thereby accelerating the training speed of subsequent steps, including:

[0041] (1.1) Obtain the 2D feature map of the input RGB image through a 2D convolution module. The structure of the 2D convolution module is as follows: Figure 2 As shown;

[0042] (1.2) Use homography transformation to convert 2D feature maps from different perspectives to the reference perspective, and then obtain the loss cube by calculating the variance between 2D feature maps from different perspectives.

[0043] (1.3) The neural coding cube is obtained by extracting the features of the loss cube through a 3D convolutional module; in this embodiment, the specific structure of the 3D convolutional module is that the model uses the loss cube f′ 2d The input image I is passed through seven ConvBnReLU layers, denoted as CBR0-CBR6. CBR0 outputs a vector with a dimension of 8, CBR1 and CBR2 with a dimension of 16, CBR3 and CBR4 with a dimension of 32, and CBR5 and CBR6 with a dimension of 64. The resulting feature vector is then processed by three ConvTransposeBn layers, denoted as CTB0-CBT2. CBT0, with an output dimension of 32, is added to the output of CBR4 and used as input to CBT1. CBT1 outputs a 16-dimensional vector, which is then added to the output of CBR2 and passed to CTB2. CTB2 outputs an 8-dimensional vector, which is added to the output of CBR1 to form the feature vector f of the neural coding cube. 3d , specifically Figure 3 As shown;

[0044] (1.4) Based on the camera pose corresponding to a certain image, a ray is formed, and the neural coding cube is uniformly sampled along the ray to obtain the feature value f of each sampling point. k and three-dimensional coordinates xk For the three-dimensional coordinates x of the sampling point k , Ray direction d k Position encoding is performed, specifically as follows: γ(p) = (sin(2 0 πp), cos(2 0 πp), …, sin(2 Q-1 πp), cos(2 Q-1 πp)), where for three-dimensional coordinates x k When performing position encoding, Q is set to 10, and the ray direction d is... k Q is set to 4 when performing position encoding;

[0045] In this embodiment, the sampling details are as follows: First, the sampling range [near, far] is determined according to the scene type: near = 0, far = 100 for outdoor scenes, and near = 0, far = 10 for indoor scenes; then, 128 points are uniformly sampled from the ray formed by the camera pose within the range [near, far] to obtain the three-dimensional coordinates x of the 128 sampling points. k Based on the three-dimensional coordinates x k Find the 8 nearest points in the neural coding cube, and perform bilinear interpolation on the feature values ​​of these points to obtain the feature value f of the sampling point. k .

[0046] (1.5) The feature values ​​f of the sampling points k The position encoding result of the three-dimensional coordinates of the sampling point γ(x) k ), the position encoding result of the ray direction γ(d) k The input image's RGB values ​​are used as input to the multilayer perceptron (MLP), and the output volume density value σ is... k Color value r k and semantic value s kIn this embodiment, the specific structure of the Multilayer Perceptron (MLP) mainly consists of seven parts, denoted as bias_linears, pts_linears, feature_linears, views_linears, rgb_linears, alpha_linears, and semantic_linears. Specifically, bias_linears consists of a single linear layer that maps the feature values ​​of the sampling points to a 256-dimensional bias vector bias; pts_linears consists of six sequentially connected linear layers. The first layer takes the position encoding of the three-dimensional coordinates of the sampling points as input, the fifth layer takes the output of the previous layer multiplied by the bias plus the position encoding of the three-dimensional coordinates of the sampling points as input, and the remaining layers take the output of the previous layer multiplied by the bias as input. The output dimension of all layers is 256; feature_linears consists of one linear layer, whose input is the vector obtained from pts_linears and the bias. The product of s, and the output 256-dimensional vector will be encoded with the position of the ray direction, input into views_linears consisting of one linear layer, to obtain a 128-dimensional output, which is then mapped to 3-dimensional RGB color values ​​by an rgb_linears layer; alpha_linears has the same input as feature_linears and outputs a 1-dimensional volume density value; similarly, semantic_linears passes the input through 6 linear layers, with output dimensions of 256 / 256 / 256 / 256 / 128 / 45 respectively, to obtain the final semantic output, as detailed below. Figure 4 As shown;

[0047] (1.6) Based on the volume density value σ output by the MLP k Color value r k and semantic value s k The color value c of a 2D pixel corresponding to a ray is obtained using volume rendering methods. t and semantic value s t :transparency Regarding the color value c t Compared to the true color value The L2 norm is used to calculate the color loss value: Regarding the semantic value s t With true semantic value The semantic loss value is obtained by calculating the cross-entropy loss function. in The semantics of the point obtained from the volume rendering are the probabilities of category l. The probability that the true semantic graph belongs to category l at this point is represented by L, where L is the total number of semantic categories; the total loss value is obtained by weighted summing of the color loss value and the semantic loss value: L total =Lc +λL s λ is the semantic loss weight; in this embodiment, the true semantic value is obtained by semantic segmentation of the RGB image using PSPNet technology, and λ is 0.04.

[0048] (1.7) Update the parameters of the 2D convolutional module, 3D convolutional module and MLP using the total loss value through gradient backpropagation;

[0049] (1.8) Use the multi-scene data in the dataset to repeat the above steps to train the network, improve the network's ability to learn scene geometric information, and thus improve the scene generalization ability;

[0050] In this embodiment, the Kitti-360 open-source dataset is used for generalization training for outdoor editing scenarios, and the Replica open-source dataset is used for generalization training for indoor editing scenarios.

[0051] (2) In the fine-tuning stage, input the RGB image, semantic map and corresponding camera pose of the scene to be edited, repeat steps (1.1) to (1.6) to obtain the total loss value, but do not update the parameters in the 2D convolutional module and 3D convolutional module during gradient backpropagation, only update the parameters in the MLP to improve the expressive ability of the 3D scene semantic editing network for the scene to be edited.

[0052] (3) Optimization stage, specifically: users use visual scripts or software such as Photoshop to edit the colors of the semantic map; the feature values ​​f of the sampling points are... k The position encoding result of the three-dimensional coordinates of the sampling point γ(x) k ), the position encoding result of the ray direction γ(d) k The MLP takes the RGB values ​​of the input image as input and the volume density value σ as output. k and semantic value s k The semantic value s of the pixel corresponding to the ray is obtained using the volume rendering method. t : The loss value during the optimization phase is calculated. in The semantics of the point obtained from the volume rendering are the probabilities of category l. The loss value represents the probability that the edited semantic graph belongs to class l at this point. Then, the parameters in the MLP are updated using the gradient backpropagation method to complete the editing of the implicit 3D scene.

[0053] (4) During the rendering stage, after the user provides an arbitrary camera pose, the neural coding cube is sampled using the new camera pose to obtain the feature value f′. k The feature values ​​f′ of the sampling points kThe position encoding result of the three-dimensional coordinates of the sampling point γ(x′) k ), the position encoding result of the ray direction γ(d′) k Input the RGB values ​​of the scene image to be edited into the MLP, and output the volume density value σ′ of the scene after editing. k and color value r′ k Using volume rendering formula This will give you the RGB value c′ of a specific pixel in the RGB image after scene editing at the user-specified camera pose. t After performing the above operations on all pixels of the image, you will obtain the RGB image of the entire scene after editing.

[0054] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0055] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0056] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0057] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "in response to a determination".

[0058] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A method for editing 3D scenes using semantic graphs, characterized in that, Includes the following steps: (1) In the generalization training phase, a 3D scene semantic editing network consisting of 2D convolutional modules, 3D convolutional modules, and an MLP is constructed. The input consists of RGB images of the scene, semantic maps, and the camera poses corresponding to each image. The 2D convolutional modules, 3D convolutional modules, and the MLP are trained, including: (1.1) Obtain the 2D feature map of the input RGB image through a 2D convolution module; (1.2) Use homography transformation to convert 2D feature maps from different perspectives to the reference perspective, and then obtain the loss cube by calculating the variance between 2D feature maps from different perspectives; (1.3) The neural coding cube is obtained by extracting the features of the loss cube through a 3D convolution module; (1.4) Based on the camera pose corresponding to a certain image, a ray is formed, and the neural coding cube is uniformly sampled along the ray to obtain the feature value of each sampling point. and three-dimensional coordinates , for the three-dimensional coordinates of the sampling points ray direction Perform position encoding; (1.5) The feature values ​​of the sampling points Location encoding results of three-dimensional coordinates of sampling points Position coding results of ray direction The RGB values ​​of the input image are used as input to the multilayer perceptron (MLP), where This is a position encoding function that outputs volume density values. Color value and semantic value ; (1.6) Based on the volume density value output by the MLP Color value and semantic value The color value of a 2D pixel corresponding to a ray is obtained using volume rendering methods. and semantic value :transparency , , ; Regarding this color value Compared to the true color value The L2 norm is used to calculate the color loss value: Regarding this semantic value With true semantic value The semantic loss value is obtained by calculating the cross-entropy loss function. ,in The semantics of the point obtained by the representation rendering is The probability of the category, The representative semantic graph at this point is The probability of each category, where L is the total number of semantic categories; the total loss value is obtained by weighted summation of the color loss value and the semantic loss value. , These are the semantic loss weights; (1.7) Update the parameters of the 2D convolutional module, 3D convolutional module and MLP using the total loss value through gradient backpropagation; (1.8) Use multi-scene data to repeat the above steps to train the network, improve the network's ability to learn scene geometric information, and thus improve scene generalization ability; (2) In the fine-tuning stage, input the RGB image, semantic map and corresponding camera pose of the scene to be edited, repeat steps (1.1) to (1.6) to obtain the total loss value, but do not update the parameters in the 2D convolutional module and 3D convolutional module during gradient backpropagation, only update the parameters in the MLP to improve the expressive ability of the 3D scene semantic editing network for the scene to be edited. (3) Optimization stage, specifically: the user edits the semantic graph; the feature values ​​of the sampling points are... Location encoding results of three-dimensional coordinates of sampling points Position coding results of ray direction The RGB values ​​of the input image are used as input to the MLP, and the volume density value is obtained from the output of the MLP. and semantic value The semantic values ​​of the pixels corresponding to the rays are obtained using volume rendering methods. :transparency , The loss value in the optimization stage is calculated. ,in The semantics of the point obtained by the representation rendering is The probability of the category, The semantic graph at this point represents the edited semantic graph. The probability of each category is used, and then the loss value is used to update the parameters in the MLP through gradient backpropagation to complete the editing of the implicit 3D scene; (4) During the rendering stage, after the user provides an arbitrary camera pose, the neural coding cube is sampled using the new camera pose to obtain feature values. , the feature values ​​of the sampling points Location encoding results of three-dimensional coordinates of sampling points Position coding results of ray direction Input the RGB values ​​of the scene image to be edited into the MLP, and output the volume density values ​​of the scene after editing. and color value Using volume rendering formula , This will give you the RGB value of a specific pixel in the RGB image of the scene after editing, under the user-specified camera pose. After performing the above operations on all pixels of the image, you will obtain the RGB image of the entire scene after editing.

2. The method for editing a 3D scene using a semantic graph according to claim 1, characterized in that, In step (1), the generalization training scenario is different from the editing scenario.

3. The method for editing a 3D scene using a semantic graph according to claim 1, characterized in that, In step (1.4), the position encoding function The formula is as follows: For three-dimensional coordinates When performing position encoding Set to 10, for the ray direction When performing position encoding Set it to 4.

4. The method for editing a 3D scene using a semantic graph according to claim 1, characterized in that, In step (1.5), the RGB values ​​of the input image refer to the values ​​obtained by overlaying all the input images together at the pixel level. The high-dimensional RGB vector obtained at point, where The pixels correspond to the pixels of the image obtained from volume rendering.

5. The method for editing a 3D scene using a semantic graph according to claim 1, characterized in that, The semantic graphs of the scenes in steps (1) and (2) are obtained by semantic segmentation of RGB images using PSPNet semantic segmentation technology.

6. The method for editing a 3D scene using a semantic graph according to claim 1, characterized in that, In step (3), the edited semantic graph is used as the supervision of the network, and gradient backpropagation only updates the parameters in the MLP to achieve the editing of the implicit 3D scene.

7. The method for editing a 3D scene using a semantic graph according to claim 1, characterized in that, In step (3), the user uses a visual script or Photoshop software to edit the semantic map colors.

8. The method for editing a 3D scene using a semantic graph according to claim 1, characterized in that, In step (4), the user can obtain a two-dimensional rendered image of the implicit three-dimensional scene after editing from any viewpoint.

Images