A Novel Perspective on Volumetric Cloud Synthesis and 3D Reconstruction Based on Neural Radiation Fields

Abstract

This invention discloses a novel perspective synthesis and 3D reconstruction method for volumetric clouds based on neural radiation fields. The method includes a data preprocessing step, a network model forward propagation step, a rendering step, and a backpropagation step. The data preprocessing step obtains a mask dataset MASKS from the image dataset IMAGES containing surrounding clouds. The model forward propagation step performs ray stepping sampling on the camera pose corresponding to each image, and the encoded sampling points and single light source directions are used by the network model forward propagation to calculate density and phase. The rendering step renders the density, phase, and other variables according to a simplified radiative transfer equation to obtain the predicted image and cloud mask. The backpropagation step uses the predicted image, the predicted cloud mask, IMAGES, and MASKS to calculate the error loss Loss for variable and parameter optimization. This method can reconstruct densities that conform to the cloud shape and achieves good rendering results for both novel and input perspectives.

Description

Technical Field

[0001] This invention relates to the field of computer vision and graphics, and in particular to a novel method for volumetric cloud synthesis and 3D reconstruction based on neural radiation fields. Background Technology

[0002] Neural Radiance Fields (NeRF) is a deep learning model for 3D scene reconstruction. It generates realistic 3D scenes from multi-view 2D images by optimizing neural networks and is widely used in fields such as computer vision and virtual reality.

[0003] In the paper Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM 65.1(2021):99-106., a method using neural radiance fields is proposed. This method generates realistic 3D scenes from multi-view 2D images by optimizing neural networks, achieving efficient view synthesis and reconstruction.

[0004] However, using volume rendering equations in a typical neural radiation field makes it difficult to reconstruct a satisfactory volumetric cloud model and also produces incorrect new perspective images. Specifically, some perspectives deviate significantly from the target cloud image; density exists not only in the cloud but also throughout the entire bounding box. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this disclosure provides a novel perspective synthesis and 3D reconstruction method for volumetric clouds based on neural radiation fields. The method includes data preprocessing, forward propagation of the network model, rendering, and backpropagation to optimize variables and parameters. This method can reconstruct densities that conform to the cloud shape and achieves good rendering results for both the novel and input perspectives. The aim is to achieve more accurate modeling and synthesis of volumetric clouds from novel perspectives.

[0006] This invention is achieved through the following technical solution: a novel perspective synthesis and 3D reconstruction method for volumetric clouds based on neural radiation fields, the method comprising the following steps:

[0007] (1) Masking data preprocessing: The input cloud image data IMAGES is calculated to generate cloud masking image data MASKS;

[0008] (2) Forward propagation of the network model: The camera pose of each input cloud image data IMAGES is sampled by light stepping. The encoded sampling points and the direction of a single light source are used to calculate the cloud density and cloud phase through forward propagation of the network model.

[0009] (3) Rendering of simplified volumetric cloud radiative transfer equation: The cloud density, cloud phase and other variables are rendered according to the simplified radiative transfer equation to obtain the predicted image IMAGES_pred and cloud mask MASKS_pred.

[0010] (4) Backpropagation: For the input cloud image data IMAGES, the input cloud mask image data MASKS, the predicted image IMAGES_pred, and the predicted cloud mask MASKS_pred, calculate the error loss to optimize variables and parameters.

[0011] Further, step (1) specifically involves: for an input image cloud with a black background or a background chroma greater than 0.5, if the background of the cloud in the input image is black, then the corresponding mask image pixel value for input image pixels with a chroma greater than 0.1 is white; otherwise, it is black. If the background of the input image has a chroma greater than 0.5, then the chroma is calculated for each pixel I(p), and the calculation expression is as follows:

[0012] S(p)=(Imax(p)-Imin(p)) / Imax(p);

[0013] Where Imax and Imin are the maximum and minimum values ​​of the RGB components of pixel I(p). If S(p) is greater than 0.1, the corresponding pixel of the masked image is white; otherwise, it is black.

[0014] Further, step (2) specifically involves: performing light step sampling based on the camera pose corresponding to each input image, performing multi-resolution hash grid encoding on the position x of each sampling point (x,d), and encoding the position d of the direction d. The input is then fed into the density MLP_density network to obtain the density σ and latent vector geo_feat of each sampling point. The encoded viewing direction, encoded light source direction, encoded position, and latent vector geo_feat output by the density network MLP_density are then fed into the phase function network MLP_phase to obtain the phase ρ.

[0015] Furthermore, the phase function network MLP_phase includes inputs of (x, w) i ,w o ,geo_feat), where x is the hash-encoded value of the sample point, w i With w oThe value represents the positional encoding of the light input and output directions after 4 layers, and geo_feat represents the latent vector output from the density network MLP_density; the phase ρ is output after passing through a 4-layer, 128-channel multilayer perceptron.

[0016] Further, step (3) specifically involves rendering the density σ obtained from the density network MLP_density, the phase ρ obtained from the phase function network MLP_phase, the light source direction, albedo, extinction coefficient, and light source intensity according to a simplified radiative transfer equation to obtain the predicted image dataset IMAGES_pred and the predicted cloud mask MASKS_pred.

[0017] Specifically, the simplified radiative transfer equation is obtained by ignoring the background, simplifying the scattering term to the light source intensity, and ignoring the cumulative transmittance term from the current point to the light source:

[0018]

[0019] S(x,w)=p(x,w,I,geo_feat)I intensity ;

[0020]

[0021] μ s =Sigma s *σ;

[0022] μ t =Sigma t *σ;

[0023] albedo = Sigma s / Sigma t ;

[0024] Where L(x, w) is the radiation at point x along the w direction, S(x, w) is the scattered radiation at point x along the w direction, T(x, w) is the cumulative transmittance at point x along the w direction, and μ s The scattering value is μ. t The extinction value is p(x, wi, I, geo_feat), which is the phase function. x is the hash-encoded sampling point position, wi is the viewing direction of the sampling point after 4 layers of position encoding, geo_feat is the latent vector output by the density network, and I is the direction of the single light source. intensity Let σ be the intensity of a single light source, and σ be the density of the current sampling point. s Sigma is the scattering coefficient. t is the extinction coefficient, and albedo is the albedo.

[0025] Further, step (4) specifically involves processing each of the input cloud image data IMAGES i Image data IMAGES_pred corresponding to the predicted clouds i Calculate the loss (Loss_image) i Masked image data for each of the input clouds. i The corresponding predicted cloud masking image IMAGES_pred i Calculate the error loss Loss_mask i Where i represents the i-th image in the data, and the loss calculation method is to calculate the mean squared error loss:

[0026]

[0027] Among them, y j This represents the i-th input image data (the input cloud image IMAGES). i Or the input mask image MASKS i The j-th pixel value of y_pred j This represents the j-th pixel value of the i-th predicted image data, i.e., the predicted cloud image IMAGES_pred. i Or the predicted cloud masking image MASKS_pred i ); n is the total number of pixels in an image participating in the calculation. The total loss Loss is obtained by combining the Loss_image and the Loss_mask. The Loss is backpropagated to optimize variables and parameters.

[0028] Furthermore, the optimization of variables and parameters specifically involves: optimizing the multi-resolution grid density values, the density network module parameters and phase network module parameters, the light source direction, the light source intensity, the albedo, and the extinction coefficient.

[0029] Specifically, the expression for the total loss is as follows:

[0030]

[0031] Where N is the total number of images, and Loss_image i Loss_mask is the error of the i-th image. i Let be the error of the image of the i-th mask, where β = 0.1.

[0032] The present invention has the following beneficial effects:

[0033] 1. The volume rendering equation method disclosed herein adopts a simplified radiative transfer equation. Based on this, the difference between steps (2) and (3) and the general neural radiation field can solve the problem that the cloud image rendered by the neural radiation field is seriously deviated from the target image at some viewpoints.

[0034] 2. The cloud masking loss optimization added in step (4) of this disclosure can solve the problem that the density of clouds in general neural radiation fields is spread throughout the entire bounding box, and can make the density concentrated only in the cloud shape. Attached Figure Description

[0035] Figure 1 This is a flowchart of the present disclosure;

[0036] Figure 2 This is one set of graphs from the input dataset of this disclosure;

[0037] Figure 3 This is a comparison diagram between the present disclosure and the instant-ngp (Instant Neural Graphics Primitives with a Multiresolution Hash Encoding) method. Detailed Implementation

[0038] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0039] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0040] The overall flowchart of this invention is shown below. Figure 1 .

[0041] (1) Data collection and preprocessing. Collect an image dataset of a cloud called IMAGES, extract the mask of the cloud part as a mask dataset MASKS, extract camera pose transforms from IMAGES, and combine them to obtain the dataset IMAGES+MASKS+transforms.

[0042] (1-1) For input image clouds with a black background or a high background chroma (e.g., blue), the pixel values ​​are determined based on the background type. If the background of the clouds in the input image is black, the corresponding mask image pixel value for input image pixels with a chroma greater than 0.1 is white; otherwise, it is black. If the background chroma of the input image is greater than 0.5, the chroma S(p) = (Imax(p) - Imin(p)) / Imax(p) is calculated for each pixel I(p), where Imax and Imin are the maximum and minimum values ​​of the RGB components of pixel I(p). If S(p) is greater than 0.1, the corresponding pixel in the mask image is white; otherwise, it is black. This yields the cloud mask dataset MASKS. A processed dataset is shown below. Figure 2 .

[0043] (1-2) For each input cloud image in the image dataset IMAGES, use colmap to estimate the pose.

[0044] (2) Forward propagation of the network model. The camera pose of each input cloud image data IMAGES is sampled by light stepping. The encoded sampling points and the direction of a single light source are used to calculate the cloud density and cloud phase through forward propagation of the network model.

[0045] (2-1) Perform light step sampling based on the camera pose corresponding to each input image.

[0046] (2-2) For each sampling point (x,d), the position x is encoded using the multi-resolution hash grid encoding mentioned in instant-ngp (Müller T, Evans A, Schied C, et al. Instant neural graphics primitives with a multiresolution hashencoding[J]. ACM transactions on graphics(TOG),2022,41(4):1-15.), and the direction d is encoded using a 4-layer position encoding, and the light source direction is encoded using a 4-layer position encoding.

[0047] (2-3) Input the encoded position and direction into the density MLP_density network mentioned in instant-ngp (Müller T, Evans A, Schied C, et al. Instant neural graphics primitives with a multiresolution hash encoding[J]. ACM transactions on graphics(TOG),2022,41(4):1-15.) to obtain the density σ and hidden vector geo_feat of each sampling point.

[0048] (2-4) Input the encoded position, encoded gaze direction, encoded light source direction, and the latent vector geo_feat output by the density network MLP_density to obtain the phase function ρ in the phase function network MLP_phase. The phase function network MLP_phase is a 4-layer, 128-channel multilayer perceptron.

[0049] (3) Rendering. The density, phase and other variables are rendered according to the simplified radiative transfer equation to obtain the predicted image IMAGES_pred and the cloud mask MASKS_pred.

[0050] (3-1) The density σ and phase ρ obtained in step (2), as well as the light source intensity, albedo, and extinction coefficient Sigma s and light source intensity I intensity The predicted image dataset IMAGES_pred is obtained by rendering according to the simplified radiative transfer equation. The simplified radiative transfer equation is as follows:

[0051]

[0052] S(x,w)=p(x,w,I,geo_feat)I intensity ;

[0053]

[0054] μ s =Sigma s *σ;

[0055] μ t =Sigma t *σ;

[0056] albedo = Sigma s / Sigma t ;

[0057] Where L(x, w) is the radiation at point x along the w direction, S(x, w) is the scattered radiation at point x along the w direction, T(x, w) is the cumulative transmittance at point x along the w direction, and μ s The scattering value is μ. t The extinction value is p(x, w, I, geo_feat), which is the phase function described in step (2), that is, the phase ρ obtained by the output of the phase network MLP_phase described in step (2). x is the position of the encoded sampling point, w is the viewpoint of the encoded sampling point, geo_feat is the latent vector output by the density network described in step (2), and I is the direction of the single light source. intensity Let σ be the intensity of a single light source, and σ be the density of the current sampling point output by the density network described in step (2). s Sigma is the scattering coefficient. t is the extinction coefficient, and albedo is the albedo.

[0058] (3-2) Determine each pixel of the predicted image IMAGES_pred. If it is greater than 1*10... -4 Then set this pixel to 0, otherwise set it to 1, to obtain the predicted cloud's mask MASKS_pred.

[0059] (4) Backpropagation. The error loss is calculated for the input cloud image data IMAGES, the input cloud mask image data MASKS, the predicted image IMAGES_pred, and the predicted cloud mask MASKS_pred, in order to optimize the variables and parameters.

[0060] (4-1) For each of the input cloud image data IMAGES i Image data IMAGES_pred corresponding to the predicted clouds i Calculate the loss (Loss_image) i For each input cloud masking image data, the corresponding predicted cloud masking image IMAGES_pred i Calculate the error loss Loss_mask i Where i represents the i-th image in the data, and the loss calculation method is to calculate the mean squared error loss:

[0061]

[0062] Among them, y j This represents the i-th input image data (the input cloud image IMAGES). i Or the input mask image MASKS i The j-th pixel value of y_pred jThis represents the i-th predicted image data (the predicted cloud image IMAGES_pred). i Or the predicted cloud masking image MASKS_pred i The j-th pixel value of ), where n is the total number of pixels in the i-th image involved in the calculation.

[0063] (4-2) Calculate the total loss:

[0064]

[0065] Where N is the total number of images, and Loss_image i Loss_mask is the error of the i-th image. i Let be the error of the image of the i-th mask, where β = 0.1

[0066] (4-3) Backpropagation is performed using the total loss to optimize the multi-resolution grid density value, the density network module parameters and phase network module parameters, the light source direction, the light source intensity, the albedo, and the extinction coefficient.

[0067] (4-4) Repeat step (2) until the specified number of iterations is reached. After optimization, a new perspective image can be obtained through steps (2) and (3).

[0068] The volume rendering equation method disclosed herein uses a simplified radiative transfer equation. Based on this, the difference between steps (2) and (3) and the general neural radiation field can solve the problem of severe deviation between cloud images and target images rendered by the neural radiation field at some viewpoints. Figure 3 This method is compared with the rendering image of Instant-ngp.

[0069] The loss optimization for cloud masking added in step (4) of this disclosure can solve the problem that the density of clouds in general neural radiation fields is distributed throughout the entire bounding box, allowing the density to be concentrated only within the cloud shape. See Figure 3 This method is compared with the depth map of Instant-ngp.

[0070] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0071] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A novel perspective synthesis and 3D reconstruction method for volumetric clouds based on neural radiation fields, characterized in that, The method includes the following steps: (1) Masking data preprocessing: preprocessing the input cloud image data Perform calculations to generate cloud masking image data Specifically: for an input image where the cloud background is black or the background chroma is greater than 0.5, if the cloud background is black, the corresponding mask image pixel value for input image pixels with a chroma greater than 0.1 is white; otherwise, it is black. If the input image background has a chroma greater than 0.5, each pixel... The chromaticity is calculated using the following expression: ; in , It is a pixel The maximum and minimum values ​​of the RGB components, if If the value is greater than 0.1, the corresponding pixel in the mask image is white; otherwise, it is black. (2) Forward propagation of the network model: For each input cloud image data The camera pose is used to perform light step sampling. The encoded sampling points and the direction of a single light source are used to calculate the cloud density and cloud phase through forward propagation of the network model. (3) Rendering using the simplified volumetric cloud radiative transfer equation: The cloud density, cloud phase, and other variables are rendered according to the simplified radiative transfer equation to obtain the predicted image. and the predicted cloud cover Specifically, for density networks Output density and phase function network Output phase The predicted image is obtained by rendering the light source direction, albedo, extinction coefficient, and light source intensity according to the simplified volumetric cloud radiative transfer equation. and the predicted cloud cover ; The simplified volumetric cloud radiative transfer equation is obtained by ignoring the background, the scattering term (which is then simplified to the light source intensity), and the cumulative transmittance term from the current point to the light source: ； ； ； ； ； ； in for Along the point Radiation in the direction. for Along the point The amount of scattered radiation in the direction. for Along the point Cumulative transmittance in the direction, This is the scattering value. Extinction value, ) is the phase function. The location of the sampling point is hash-encoded. The line-of-sight direction of the sampling points after 4 layers of position coding. These are the hidden vectors output by the density network. For a single light source direction, The intensity of a single light source, The density of the current sampling point, The scattering coefficient is... Extinction coefficient, Albedo; (4) Backpropagation: the image data of the input cloud Input cloud masking image data Predicted images and the predicted cloud cover ;Calculate the error loss to optimize variables and parameters.

2. The method for novel perspective synthesis and 3D reconstruction of volumetric clouds based on neural radiation fields according to claim 1, characterized in that, Step (2) specifically involves: performing light step sampling based on the camera pose corresponding to each input image, and sampling each sampling point. Location Perform multi-resolution hash grid encoding to achieve direction. Perform position encoding and input density. The network obtains the density of each sampling point. With latent vectors Input the encoded view direction, encoded light source direction, encoded position and density network Output latent vector To phase function network obtain phase .

3. The method for novel perspective synthesis and 3D reconstruction of volumetric clouds based on neural radiation fields according to claim 2, characterized in that, The phase function network Including, the input is ,in The value of the sample point after hash encoding. and The value represents the positional encoding of the light input and output directions after four layers. Represents the density network The output latent vector; the phase output after passing through a 4-layer, 128-channel multilayer perceptron. .

4. The method for novel perspective synthesis and 3D reconstruction of volumetric clouds based on neural radiation fields according to claim 1, characterized in that, Step (4) specifically involves processing the image data of each input cloud. Image data corresponding to the predicted clouds Calculate loss For each of the input cloud masking image data The corresponding predicted cloud masking image Calculate error loss Where i represents the i-th image in the data, and the loss calculation method is to calculate the mean squared error loss: ； in, This represents the j-th pixel value of the i-th input image data; the input image data is the input cloud image data. Or input masking image data , This represents the j-th pixel value of the i-th predicted image data, i.e., the predicted cloud image data. Or a predicted cloud cover image. ; The total number of pixels in an image used in the calculation, combined and Total loss Backpropagation In order to optimize variables and parameters.

5. The method for novel perspective synthesis and 3D reconstruction of volumetric clouds based on neural radiation fields according to claim 4, characterized in that, The total loss The expression is as follows: ； Where N is the total number of images, Let be the error of the i-th image. Let be the error of the image of the i-th mask, where =0.1.

Images