A neural-radiance-field-based explicit scene composition method

Abstract

This invention discloses an explicit scene composition method based on neural radiation fields, belonging to the field of computer vision. The method includes: acquiring multi-view data of a source scene and a target scene; pre-training a source neural radiation field and a target neural radiation field using the acquired multi-view data; generating multi-view data from arbitrary perspectives using the trained source neural radiation field; jointly training an occlusion field with the pre-trained source neural radiation field; using a combination rendering equation to combine the source neural radiation field and the target neural radiation field and render an image of the combined scene; and using the trained occlusion field to achieve scene editing. This invention utilizes an occlusion field to combine two scenes represented by neural radiation fields in a self-supervised mode, rendering realistic images of the combined scene from arbitrary perspectives with natural object edge transitions, while also providing scene editing capabilities.

Description

Technical Field

[0001] This invention belongs to the field of computer vision. Specifically, this invention relates to an explicit scene combination method based on neural radiation fields. Background Technology

[0002] With the development of computer vision, 3D reconstruction technology has been widely applied in various fields, such as virtual reality, film animation, and medical image reconstruction. Recently, Neural Radiation Fields (NeRF) implicitly encode the geometry and appearance of a scene within a multilayer perceptron, demonstrating impressive performance in free-viewpoint synthesis tasks. Numerous works based on NeRF aim to improve its rendering quality, speed, and flexibility. However, no work has yet explored how to directly combine two pre-trained NeRFs, which could be used in interactive applications and has significant implications and value. The following section will introduce three parts: implicit 3D neural representation, NeRF-based combination, and NeRF-based decoupling.

[0003] Implicit 3D Neural Representations: With the rapid development of deep learning in 3D vision, implicit neural representations have attracted widespread attention from researchers and have demonstrated excellent performance in tasks such as 3D shape prediction and novel view synthesis. Common 3D representations include: signed distance fields, occupancy fields, and neural radiance fields (Ben Mildenhall, Pratul PSrinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV, 65(1): 99–106, 2021.). Neural radiance fields (NeRF) and their variants synthesize novel views by optimizing low-level continuous volumetric scene functions learned from multi-view images and have shown impressive performance in free view synthesis. However, these implicit representations, including neural radiance fields, share some common problems, such as: 3D objects represented by implicit models are difficult to synthesize spatially, while explicit models can do so easily. This is precisely the problem that this invention addresses.

[0004] Combinatorial Neural Radiation Fields: Combinatorial neural radiation fields aim to model multiple editable objects or scenes in an integrated implicit function. Guo et al. (Michelle Guo, Alireza Fathi, Jiajun Wu, and Thomas Funkhouser. Object-centric neural scene rendering. arXiv preprint arXiv:2012.08503,2020.) proposed learning an object-centric scattering function based on neural radiation fields, which implicitly models the light transport of each object using a neural network dependent on lighting and view. This framework can render occlusion, specular reflection, shadows, and indirect lighting as specific objects move. Yang et al. (Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In ICCV, pages 13779–13788, 2021.) proposed a dual-path architecture where the scene and objects are encoded by two branches, with each independent object encoded by a learnable latent variable. This work introduces a scene-guided training strategy to resolve 3D spatial ambiguity and learn clear boundaries for each object. Niemeyer et al. (Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In CVPR, pages 11453–11464, 2021.) proposed a compositional generative neural feature field that learns from unstructured multi-view images in a self-supervised manner, thereby separating objects from the background. The model supports translation and rotation of learned editable objects within the scene and achieves impressive rendering quality.

[0005] Decoupled Neural Radiation Fields: Decoupled neural radiation fields focus on resolving different factors within a scene. Rebain et al. (Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. In CVPR, pages 14153–14161, 2021.) proposed spatially decomposing the scene and assigning a smaller network to each decomposed part to achieve efficient and GPU-friendly rendering. During the rendering stage, each sub-network renders a portion of the image, which is then combined into a complete image. Guo et al. (Yuan-Chen Guo, Di Kang, Linchao Bao, Yu He, and Song-Hai Zhang. Nerfren: Neural radiance fields with reflections. In CVPR, pages 18409–18418, 2022.) modified neural radiation fields to model scenes with reflection properties, dividing the scene into transmission and reflection components and modeling these two components using separate neural radiation fields. Martin et al. (Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photocollections. In CVPR, pages 7210–7219, 2021.) proposed rendering static scenes and transient elements separately to model high-fidelity static scenes. An uncertainty map is introduced and simultaneously optimized during the training phase to identify and eliminate uncertain regions in the image. Summary of the Invention

[0006] To address the limitations and defects of the existing methods and to balance the consumption of training resources with the rationality and realism of scene combination, the present invention aims to provide a method for combining pre-trained neural radiation fields. Based on the geometric and appearance information of the pre-trained source neural radiation fields, the foreground object and the background are decoupled, thereby combining the foreground object with the target neural radiation field.

[0007] To achieve the above-mentioned objectives, the technical solution adopted by the method of the present invention is as follows:

[0008] An explicit scene composition method based on neural radiation fields, comprising the following steps:

[0009] S1: Collect data and construct a dataset; the dataset includes two parts: virtual scene data and real scene data; for virtual scene data, render the corresponding multi-view images in the rendering engine and export the corresponding camera parameters; for real scene data, use the camera to capture multi-view images and use the structure of motion reconstruction algorithm to obtain the corresponding camera parameters.

[0010] S2: Pre-training of source neural radiation field and target neural radiation field: Build a model and train the source neural radiation field and target neural radiation field respectively using multi-view image data collected in the source scene and target scene;

[0011] S3: Using the source neural radiation field pre-trained in step S2, generate scene images from different viewing angles by setting different observation angles;

[0012] S4: Construct a loss function, jointly train the occlusion field and the source neural radiation field pre-trained in step S2, and use the multi-view scene images generated in step S3 for supervision.

[0013] S5: Using the occlusion field trained in step S4, combined with the rendering equation, the foreground objects in the source neural radiation field are extracted and combined with the target neural radiation field. Then, the combined scene image of the source scene and the target scene from any viewpoint is rendered.

[0014] Compared with existing technologies, this invention is the first to achieve scene composition based on pre-trained neural radiation fields, and also the first to achieve scene composition without ground truth supervision of the composed scene. It also proposes a scene composition rendering formula based on pre-trained neural radiation fields. The method of this invention starts from the pre-trained source neural radiation field, constructs an occlusion field, which can better decouple the foreground objects and background in the source neural radiation field. Then, combined with the scene composition rendering formula, the foreground objects are combined with the target neural radiation field for rendering, resulting in realistic combined scene images with natural object edge transitions from any perspective, while also having scene editing capabilities. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method of the present invention;

[0016] Figure 2 This is a flowchart of the operation phase in an embodiment of the present invention;

[0017] Figure 3 This is a visualization of the intermediate results and combined rendering results output by the network in this embodiment of the invention;

[0018] Figure 4This is a visualization of the ablation experiment in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] Unlike previous methods, this invention focuses on synthesizing realistic boundaries and complex occlusion relationships when merging two pre-trained neural radiation fields. An occlusion field, learned from the pre-trained source neural radiation fields, helps synthesize anti-aliased boundaries between foreground and occluded objects. Once the occlusion field is learned, the objects to be merged can be edited (moved, rotated, scaled) while maintaining high rendering quality. Figure 1 As shown, an explicit scene combination method based on neural radiation fields according to the present invention includes the following steps:

[0021] S1: Data Acquisition and Dataset Construction. The dataset consists of two parts: virtual scene data and real scene data. For virtual scene data, the models to be rendered (source scene model and target scene model) are imported into the virtual engine. Lighting, camera, and other parameters are adjusted. Pitch angles are set to [-30, -15, 0, 15, 30], and yaw angles are uniformly selected from [-180, 180], resulting in 180 viewpoint images. The camera pose matrix is ​​then transformed from the coordinate system defined by the rendering engine to the coordinate system of the neural radiation field. For real scene data, the subject is placed in a simple photography studio, and multi-viewpoint data is captured using a Canon 5D3 DSLR camera. Approximately 20 images are taken for each scene. The camera pose matrix is ​​then obtained using the Structure for Motion Reconstruction (SMR) algorithm and transformed back to the coordinate system defined by the neural radiation field. At this point, the data required for network training is complete, including multi-viewpoint images of the scene and their corresponding camera pose matrices.

[0022] S2: Pre-training of Source and Target Neural Radiation Fields: Models are constructed using multi-view image data from the collected source and target scenes to train two neural radiation fields respectively. In this embodiment, the target scene is defined as the main scene, and the source scene is defined as a component. As the main scene, its background cannot be changed during composition. Therefore, to ensure that the background of the composed scene is consistent with the background of the target scene, the target neural radiation field models both the objects and the background in the target scene; while the source neural radiation field only models the objects in the source scene and does not model the background.

[0023] S3: Generate multi-view images using the trained source neural radiation field: Using the source neural radiation field pre-trained in step S2, scene images are generated from different viewing angles. This multi-view data will be used for the joint training of the occlusion field and the source neural radiation field in step S4. Its purpose is to ensure that the rendering result of the source neural radiation field with updated parameters is not excessively changed, which is the basis for scene combination in step S5. A batch of multi-view images are rendered using the source neural radiation field trained in step S2 to supervise the joint training in step S4 and prevent the source neural radiation field from being excessively changed during the joint training.

[0024] S4: Joint Training of Occlusion Field and Source Neural Radiation Field: The occlusion field and the pre-trained source neural radiation field are jointly trained. Through the loss function and rendering equation designed in this invention, at the end of training, the occlusion field can extract foreground objects from the source neural radiation field, which is the basis for S5 scene composition rendering. aliasing loss Geometric loss Geometric loss regularization term Four loss functions.

[0025] S5: Combine the source neural radiation field and the target neural radiation field, and render a realistic combined scene image: Using the occlusion field trained in step S4, combined with the combined rendering equation proposed in this invention, the foreground objects in the source neural radiation field can be extracted and perfectly combined with the target neural radiation field, and a scene image combining the source scene and the target scene from any viewpoint can be rendered.

[0026] Furthermore, it also includes S6: Scene Editing: By manipulating the source neural radiation field and occlusion field, the combined scene image can be edited, and then the edited combined scene image can be obtained using the rendering equation.

[0027] The steps, principles, and effects of the method of the present invention are described in detail below.

[0028] (1) Neural Radiation Field (NeRF) uses a multilayer perceptron to represent a scene as a continuous volumetric field, which is based on 3D position x = (x, y, z) and 2D viewing angle. As input, the network outputs the density value σ and color value c = (r, g, b) for each sample point. To calculate the color of a given pixel, the network first predicts the density and color values ​​of N sample points on the camera-emitted ray r, and then uses a discretized volumetric rendering method to calculate the opacity of that pixel. and color value

[0029]

[0030]

[0031] Cumulative transmittance Opacity α i =1-exp(-σ i δ i ), δ i =t i+1 -t i This is the distance between adjacent sampling points. The contribution of each sampling point to the final cumulative color is defined as:

[0032] w i =T i α i

[0033] To improve the network's performance in representing high-resolution and complex scenes, positional encoding is used to map the input to a higher dimension. An image-based loss function Li is employed. photo To optimize the neural radiation field:

[0034]

[0035] Where C(r) represents the true color value of the image pixel corresponding to that ray. It represents a beam of light.

[0036] (2) Due to diffraction, natural images contain blurred transitions at the boundaries between foreground objects and the background, which is unavoidable in the human eye or optical imaging devices (such as cameras). Although the blurred boundaries can be mitigated by adjusting the camera's aperture and focus parameters, they cannot be completely eliminated, especially when the foreground consists of slender objects such as hair and leaves. Previous work has attempted to synthesize multiple neural radiation field models, but the aliasing problem at occlusion edges has been neglected. Furthermore, the underconstraint of the density field reconstructed from the neural radiation field makes the combination of neural radiation field models even more unstable.

[0037] To address this issue, this invention proposes an Occlusion Field (OCF), specifically designed for more accurate modeling of the occupancy attributes of objects in a scene. Specifically, given the position x of a 3D point and the viewing angle d, the occlusion field... Predict an occlusion value o at each 3D point x. i :

[0038]

[0039] (3) Obtain the occlusion value o at each sampling point iSubsequently, this invention proposes a novel composite rendering equation to generate more realistic composite scene images. The foreground object in the composite scene is defined as S, the background as T, and the corresponding source neural radiation field... and target neural radiation field They can be represented by the following equations:

[0040]

[0041]

[0042] Where x is a three-dimensional coordinate point, and d is the viewing angle. The color of the source neural radiation field is defined as the color of the volume rendering integral. and background color The weighted fusion yielded:

[0043]

[0044] in, The color value rendered by the source neural radiation field, integral color This represents the color value of the foreground object rendered by the source neural radiation field. This represents the background color of the scene. With the help of this decoupled scene representation, the colors of the combined scene can be calculated using the following formula:

[0045]

[0046] in:

[0047]

[0048] in This represents the cumulative transmittance of the sampling points in the combined scene. These represent the opacities of sampling points in the source neural radiation field and the target neural radiation field, respectively. These are the color values ​​of the sampling points in the source neural radiation field and the target neural radiation field, respectively. This represents the contribution of each sampling point to the final integrated color of the light.

[0049] The method of this invention supports manipulation of objects in a composite scene. To render an image of the composite scene manipulated by the user, it is necessary to transform the 3D sampling points and the 2D viewing angle, which can be represented as:

[0050] x'=s·Rx+t

[0051] d′=Rd

[0052] Where s is the scaling factor for relative pose, R is the rotation matrix, and t is the translation vector. Then, using x′ and d′ as inputs to the source neural radiation field and the target neural radiation field, the density value σ and color value c are predicted, and the manipulated, composite scene image is rendered through a combined rendering equation.

[0053] (4) Figure 2 As shown, an occlusion field (OCF) is trained to synthesize the source and target neural radiation fields using pre-trained source and target neural radiation fields as inputs. The occlusion field is a multilayer perceptron consisting of 10 fully connected layers. The first nine layers each have 256 channels followed by a ReLU activation function; the last layer has 128 channels followed by a Sigmoid activation function to ensure that the output o value is between 0 and 1. Skip links are introduced into the network, and positional encoding is used to map the input x and d to a higher dimension.

[0054] Training an occlusion field without the supervision of combined scene images is challenging. While the transparency T can be derived from the output σ of the neural radiation field, it is only effective for single scenes and cannot handle combinations of multiple scenes. Furthermore, using a neural radiation field to predict σ and c for a given scene is an under-constrained problem, meaning that rendering an accurate scene image does not guarantee accurate geometric information about the scene. Therefore, this invention designs a loss function suitable for self-supervised mode, including: photometric loss. aliasing loss Geometric loss Geometric regularization loss

[0055] Photometric measurement loss During the training phase, photometric loss is applied to prevent excessive alteration of the constantly updated source neural radiation field, which is crucial for scene composition and rendering. The definition of photometric loss is as follows:

[0056]

[0057] Where C S (r) represents the actual color value of the image.

[0058] aliasing loss Aliasing loss is designed to extract objects from the source neural radiation field. The challenge in achieving this lies in decoupling the blurred transitions at occlusion boundaries, which is essential for seamless synthesis of the source and target neural radiation fields. This invention introduces an occlusion map, which is used to extract objects through volume rendering based on the geometry of the source neural radiation field.

[0059]

[0060] in Masks for foreground objects rendered in the occlusion field.

[0061] Finally, a novel combined rendering equation is used, which integrates color by incorporating geometric information from the occlusion field:

[0062]

[0063] in

[0064]

[0065] Then, aliasing loss is used to enable the occlusion field to extract objects with semi-transparent and edge transition properties from the source neural radiation field:

[0066]

[0067] in The color value of the foreground object extracted from the occlusion field.

[0068] Geometric loss Considering that both the occlusion field and the source neural radiation field contain geometric information of the source object, the occlusion mask and source mask are obtained using the corresponding rendering equations, and then L2 loss is used to ensure consistency between them. Geometric loss ensures that the object shape represented by the occlusion field is consistent with the object shape represented by the source neural radiation field.

[0069]

[0070] Geometric regularization loss In the experiment, it was found that the neural radiation field tends to predict low-density σ values ​​at some sampling points, thus causing cloud-like artifacts. The root cause of this phenomenon is that the neural radiation field predicts an incorrect σ distribution in the background region. When the source neural radiation field and the target neural radiation field are combined, noise in the σ field causes cloud-like artifacts and unnatural transitions at occlusion boundaries.

[0071] To address this issue, it's necessary to ensure that no impurities are introduced into the neural radiation field of the composite scene. Observation reveals that only a small fraction of the sampled points along the rendered rays actually hit objects or the background in the scene, while the remaining points are mostly noise. Noise can be largely suppressed by minimizing the sum of σ along each ray. Based on this observation, a geometric regularization loss term is introduced:

[0072]

[0073] The complete loss function is as follows:

[0074]

[0075] Where, λ geo , λ mat , λ reg These are the weights of each loss function.

[0076] (5) Input the camera poses corresponding to the multi-view images of two neural radiation fields in the test set. The network predicts the σ and c values ​​of each sampling point under the corresponding viewpoint, and then uses the volume rendering equation to obtain the final combined scene image. The reconstruction results are evaluated using three indicators: peak signal-to-noise ratio, structural similarity, and learned perceptual image patch similarity. The results are shown in Table 1, and some visualization results are shown in Table 2. Figure 3 As shown.

[0077] Table 1: Quantitative Indicator Evaluation Results

[0078] Dataset Peak signal-to-noise ratio ↑ Structural similarity ↑ Learning to perceive image patch similarity ↓ Hair - Black 23.04 0.918 0.203 Hair - Brown 22.10 0.877 0.231 Boxes - Fruit 32.32 0.967 0.152 Horse-saddle 33.46 0.981 0.128 Table - Bowl 26.13 0.880 0.182

[0079] Experimental results show that this method can combine any two pre-trained neural radiation field models in a self-supervised mode and render realistic combined scene images, even in complex scenes, it performs very well.

[0080] The following ablation experiments will be conducted to verify the effectiveness of this method. The experiments include...

[0081] 1. Only remove aliasing loss

[0082] 2. Only remove geometric loss

[0083] 3. Remove only the geometric loss regularization term

[0084] 4. Adjust the prediction of occlusion field to not be based on the 2D viewing angle d.

[0085] 5. Full method, that is, the full method described in this invention.

[0086] Table 2 shows the results of the five ablation experiments on the test set.

[0087] Table 2: Quantitative Evaluation Results of Ablation Experiments

[0088]

[0089]

[0090] By comparing the experimental results, the aliasing loss To ensure that the occlusion field can effectively learn the geometric information of the object; geometric loss. Ensures that the scene images are rendered to be basically correct; geometric loss regularization term This allows for further optimization of details in the scene, such as semi-transparent areas and object occlusion boundaries. Experiments demonstrate that all loss functions designed in this method improve the final combination and rendering results in different ways, and are essential conditions for combining the two pre-trained neural radiation fields effectively. The visualization results are as follows: Figure 4 As shown.

Claims

1. An explicit scene composition method based on neural radiation fields, characterized in that, Includes the following steps: S1: Collect data and construct a dataset; the dataset includes two parts: virtual scene data and real scene data; for virtual scene data, render the corresponding multi-view images in the rendering engine and export the corresponding camera parameters; for real scene data, use the camera to capture multi-view images and use the structure of motion reconstruction algorithm to obtain the corresponding camera parameters. S2: Pre-training of source neural radiation field and target neural radiation field: Build a model and train the source neural radiation field and target neural radiation field respectively using multi-view image data collected in the source scene and target scene; S3: Generate multi-view images using the pre-trained source neural radiation field: Using the pre-trained source neural radiation field from step S2, generate scene images from different viewing angles by setting different viewing angles. S4: Joint training of occlusion field and source neural radiation field: Jointly train the occlusion field and the pre-trained source neural radiation field. Through the designed loss function and rendering equation, the occlusion field can extract the foreground object from the source neural radiation field at the end of the training. S5: Combine the source neural radiation field and the target neural radiation field, and render a realistic combined scene image: Using the occlusion field trained in step S4, combined with the rendering equation, the foreground objects in the source neural radiation field are extracted and perfectly combined with the target neural radiation field, and a scene image combining the source scene and the target scene from any viewpoint is rendered.

2. The explicit scene combination method based on neural radiation fields according to claim 1, characterized in that, In step S2, the target neural radiation field models both the objects and the background in the target scene; the source neural radiation field only models the objects in the source scene and does not model the background.

3. The explicit scene combination method based on neural radiation fields according to claim 1, characterized in that, In step S4, the loss function includes photometric measurement loss. aliasing loss Geometric loss Geometric loss regularization term The four loss functions are as follows: in For a ray of light, For light beams, The color values ​​rendered by the source neural radiation field. The actual color values ​​of the image; in The number of upsampling points for the light ray. The color value of the foreground object extracted from the occlusion field. The color value of the foreground object rendered by the source neural radiation field; in To mask the foreground objects rendered by the occlusion field. A mask for the foreground object rendered from the source neural radiation field; in This is the occlusion value.

4. The explicit scene combination method based on neural radiation fields according to claim 1, characterized in that, In step S4, the occlusion field specifically refers to: in The location of the three-dimensional sampling point. It is a two-dimensional observation perspective. This represents the occlusion value of the sampling points predicted for the occlusion field.

5. The explicit scene combination method based on neural radiation fields according to claim 1, characterized in that, In step S5, the rendering equation is specifically as follows: in This represents the cumulative transmittance of sampled points in the combined source and target scenes. , These represent the opacities of sampling points in the source neural radiation field and the target neural radiation field, respectively. , These are the color values ​​of the sampling points in the source neural radiation field and the target neural radiation field, respectively.

6. The explicit scene combination method based on neural radiation fields according to claim 1, characterized in that, After obtaining the combined scene image in step S5, the scene can be edited by manipulating the source neural radiation field and occlusion field. Then, the edited combined scene image can be obtained using the rendering equation.

Images