A generalizable scene rendering method based on NeRF

Abstract

The present invention discloses a generalizable scene rendering method based on NeRF, which adopts an image appearance information encoder, an image geometry information encoder, a low-frequency forward reasoning module, and a high-frequency forward reasoning module, and performs sampling in space along the light under the target viewing angle. In order to reduce the spectrum aliasing problem that is easily caused in the sampling process, each sampling point parameter is transformed into an isotropic Gaussian distribution under a certain scale, and the sampling point is projected to other known viewing angles. The color value and density value of the corresponding point can be obtained by using the above modules. After obtaining the radiation field and the density field, the pixel value of the corresponding pixel point is obtained along the entire light ray by using the volume rendering technology. Compared with the prior art, the new viewing angle synthesis method proposed by the present invention can obtain a better new viewing angle synthesis effect when a small number of pictures with camera internal and external parameters are input; at the same time, it has a better cross-scene rendering capability.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a generalizable scene rendering method based on NeRF. Background Art

[0002] Neural Radiance Fields (NeRF) is a new type of 3D scene representation. It takes a series of photos of the scene with camera internal and external parameters at a known perspective, and uses a neural network to implicitly encode the density and color of spatial points. The three-dimensional space can be represented as a set of learnable and continuous radiation fields. After the network training is completed, a realistic new perspective view can be rendered for the scene at any perspective.

[0003] However, many new perspective synthesis methods based on NeRF usually require a large number of images and long-term training and optimization based on specific scenes. When the scene changes, the entire training process needs to be repeated, which is very unfavorable for practical application. In addition, due to the imbalance in details and scale of objects near and far from the camera, many existing NeRF models usually produce blurry or low-resolution renderings. Summary of the invention

[0004] The technical solution of the present invention is: to overcome the shortcomings of the prior art and design a generalizable scene rendering method based on NeRF; to take into account the spectrum anti-aliasing effect and effectively improve the rendering quality; and to design low-frequency and high-frequency forward reasoning modules respectively to reasonably allocate network capacity. When only a small number of pictures under sparse perspectives are available, realistic new perspective views can still be rendered; at the same time, the model has the ability to generalize across scenes, that is, the model trained under the previous scene can be fully utilized by the new scene, and after migrating to the new scene, the previous prior knowledge can be used for direct reasoning.

[0005] To achieve the above object, the technical solution of the present invention is specifically:

[0006] The present invention provides a generalizable scene rendering method based on NeRF, which includes an image appearance information encoder, an image geometry information encoder, a low-frequency forward reasoning module, and a high-frequency forward reasoning module. The method realizes reconstruction under sparse perspective, has cross-scene rendering capabilities, and can take into account the rendering quality of both near and far objects. Specifically, the method includes the following steps:

[0007] Step 1: Photograph the scene to obtain a sparse color image, obtain the camera intrinsic parameters through colmap and other methods, as well as the camera extrinsic parameters when taking each image, randomly select one of them as the target perspective image, and the rest as source images.

[0008] Step 2: Pass each source image through an image appearance information encoder with shared parameters and an image geometry information encoder with shared parameters to obtain an appearance feature map and a geometry feature map that are aligned with the source image pixels, and bilinearly interpolate these two feature maps to the same size as the input image.

[0009] Step 3: Send out light from the target perspective, perform rough sampling in space along the light, that is, uniform sampling, to obtain the direction d, coordinate x, and distance t from the optical center of the target camera of the sampling point tgt The sampling points are parameterized into an isotropic three-dimensional Gaussian distribution, whose mean is the spatial position coordinate of the sampling point and the variance is rt tgt λ, where r is the actual width of a pixel in world coordinates and λ is a manually tuned hyperparameter.

[0010] During the sampling process, the sampling points on each ray will be continuously resampled as the model is trained, so that the sampling points are gradually concentrated on the surface of the object.

[0011] Step 4: Parameterize the sampling points along a certain ray under the target camera with a three-dimensional isotropic Gaussian distribution, project it onto the appearance feature map and geometric feature map of each source view, and obtain the feature query range. The weight distribution on this range follows a two-dimensional Gaussian distribution, whose mean is the position of the sampling point projected onto the feature map, and its variance is rt src ·η, where η is a hyperparameter, t src is the distance between the sampling point and the source camera. The weight is used to weight the features within the query range to obtain the appearance and geometric feature information of the sampling point in each source view.

[0012] Step 5: Parameterize the coordinates of the coarse sampling points into an isotropic three-dimensional Gaussian distribution, and perform Fourier encoding on the direction of the sampling points and the position of the parameterized Gaussian distribution. The Fourier encoding formula is:

[0013] γ(x)=[sin(x),cos(x),…,sin(2 L-1 x),cos(2 L-1 x)] T

[0014] x is an arbitrary variable; L is a hyperparameter.

[0015] The geometric feature information corresponding to the coarse sampling points on each source image is averaged and pooled to obtain the synthetic geometric features, which are sent to the low-frequency forward reasoning module together with the Fourier code to obtain the density value of the coarse sampling points.

[0016] Step 6: According to the density value of the coarse sampling point and the interval between the sampling points, the weight corresponding to each sampling point can be calculated (the weight here is the weight coefficient before the color value during volume rendering). Divide the weight of the coarse sampling point by the weight sum, and normalize it to obtain the probability density function of the piecewise constant formula; then according to this probability density function, perform the next fine sampling. After the fine sampling point is subjected to the same Fourier encoding as in step 5, the appearance features of the fine sampling point corresponding to each source view feature map are obtained according to the method of step 4, and the synthetic appearance features are obtained by average pooling, which are sent to the high-frequency forward reasoning module together with the Fourier encoding of the fine sampling point to obtain the density value and color value of the fine sampling point.

[0017] Step 7: The density and color values ​​of all fine sampling points on the light are used to obtain the predicted pixel value through volume rendering. Finally, the predicted pixel value is compared with the true value of the image at the target perspective to calculate the loss function

[0018] Step 8: Calculate the weight distribution of all fine sampling points along the light, i.e. the probability density function, as in step 6. Set the loss function The weight distribution of the fine sampling points is used to guide the update of the weight distribution of the coarse sampling points, and the two loss functions are and The sum is used as the final loss function, and the model parameters are optimized using the back propagation algorithm until convergence.

[0019] Compared with the prior art, the advantages of the present invention are: (1) the appearance information and geometric information in the image are decoupled, and the encoder can be used as a general scene feature extractor to encode the appearance features and geometric features that conform to the new data based on the input image, making full use of the two-dimensional prior information of the input image. (2) The network capacity is reasonably allocated. The low-frequency forward reasoning module uses a very small MLP for the prediction of geometric contours; the high-frequency forward reasoning module uses a slightly larger MLP for learning geometric detail information and appearance color information. (3) Considering the scale of the sampling points, on the one hand, the sampling points are parameterized into an isotropic three-dimensional Gaussian distribution; on the other hand, when projected to the feature map, it also corresponds to an area, and the features of this area are weighted. This achieves a certain adaptive filtering function, so that both near and far objects can have a good rendering quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is the overall framework diagram of the present invention;

[0021] Figure 2 A schematic diagram of a specific implementation when seeking synthesis features for the present invention;

[0022] Figure 3 It is a schematic diagram of the low-frequency forward reasoning module of the present invention;

[0023] Figure 4 This is a schematic diagram of a high-frequency forward reasoning module of the present invention;

[0024] Figure 5 This is the effect diagram synthesized from the new perspective of the present invention. DETAILED DESCRIPTION

[0025] In order to more clearly illustrate the purpose, technical solutions and advantages of the present invention, the specific implementation methods of the present invention are further described below in conjunction with the accompanying drawings.

[0026] like Figure 1 As shown in FIG. 1 , the overall structure of a NeRF-based generalizable scene rendering method of the present invention is composed of an image appearance information encoder, an image geometry information encoder, a low-frequency forward reasoning module, and a high-frequency forward reasoning module. Two or more images are required for training, and only one image can be input for reasoning.

[0027] In the description of the present invention, the color value is an attribute of the sampling point, and the color on the image that we usually refer to is referred to as the pixel value in this article. If the feature map below does not specify whether it is an appearance feature map or a geometric feature map, it refers to both and the same operation is performed on them.

[0028] The method comprises the following steps in sequence:

[0029] 1. First, obtain a small number of images with known camera intrinsic parameters, and use tools such as colmap to obtain the camera pose when each photo was taken. Before training, you need to randomly select a picture as the ground truth of the target image to be reconstructed, and the remaining pictures as source images, i.e., the input data of the model. The camera corresponding to the target image is called the target camera, and light is emitted from the luminous center of the target camera and passes through the pixel position on the target image.

[0030] 2. First, make uniform and large-interval coarse sampling on the light, and perform Fourier encoding on the coordinates and directions of the coarse sampling points. The Fourier encoding formula is:

[0031] γ(x)=[sin(x),cos(x),…,sin(2 L-1 x),cos(2 L-1 x)] T

[0032] x is an arbitrary variable. For the direction d, the direction encoding can be obtained by directly substituting each dimension of the direction vector into the formula; the position encoding can be obtained by substituting each dimension of the position coordinate into the formula; L is a hyperparameter.

[0033] For the position coordinates parameterized by the Gaussian distribution,

[0034]

[0035] Among them, μ and ∑ are the mean and variance of the Gaussian distribution in the world coordinate system, are the variance along the light direction and the variance perpendicular to the light direction, μ t is the distance between the sampling point and the optical center. t =σ r = rμ t λ, r is the actual width of a pixel in the world coordinate system, λ is a manually adjusted hyperparameter, and d is the direction vector from the optical center of the target camera to a pixel point on the image plane.

[0036] Then find the expectation of the Gaussian distribution position encoding γ(x):

[0037]

[0038] μ γ =Pμ,Σ γ =PΣP T

[0039] Get the mean and variance under the basis P. Then after sine and cosine modulation, it is known that:

[0040]

[0041] Finally, the position-encoded expectation of the multivariate Gaussian is obtained.

[0042]

[0043] in Represents element-wise multiplication.

[0044] 3. The image appearance information encoder and geometric information encoder both use resnet50 (removing the last layer) and load its pre-trained weights after imagenet training. All source images are sent to obtain the corresponding appearance feature maps and geometric feature maps.

[0045] 4. Project the parameterized three-dimensional Gaussian distribution of the sampling points onto the feature maps of each source camera. The projection range is the 3σ boundary of the three-dimensional Gaussian distribution. This range is called the feature query range. Then, the features in the query range are weighted to obtain the expectation. The weights of the features follow a two-dimensional Gaussian distribution. The mean of this two-dimensional Gaussian is the position of the sampling point projected onto the feature map, and its variance is rt src η, where r is the pixel width, η is a hyperparameter, and t srcis the distance between the spatial sampling point and the source camera. The specific implementation of this weighted process is to perform plane sampling in the query area. In order to distinguish it from the previous spatial sampling points, the sampling points obtained by plane sampling here are called plane sampling points; the inverse transformation method is used to generate the sampling of the two-dimensional Gaussian distribution, and the features of the plane sampling points are added and then divided by the total number of plane sampling points n. Round up, α is a hyperparameter, R is the radius of the feature query range; if a plane sampling point exceeds the feature map boundary under the source camera, the plane sampling point is considered invalid, that is, the feature vector where it is located is considered to be 0 and is not counted in the total number of plane sampling points n. This process is as follows Figure 2 As shown in the figure, the grid is the feature map, and the points are the sampling points on the feature map. After obtaining the feature vectors corresponding to the sampling points on the feature maps of each source view, average pooling is performed to obtain the synthetic features.

[0046] 5. The position code and direction code in step 2, together with the geometric synthesis features, are sent to the low-frequency forward reasoning module, which is as follows Figure 3 As shown, each box represents a layer, the box is a neuron, and the number represents the number of neurons; the density σ of the output sampling point is then calculated to obtain the weight w corresponding to the color value of the spatial sampling point. The calculation process is as follows:

[0047] Here we need to explain volume rendering in advance, that is, when the density value and color value of the spatial sampling point are obtained, the following integral is performed along all the spatial sampling points on a certain ray to obtain the predicted pixel value corresponding to this ray r This integral formula is the formula for volume rendering.

[0048]

[0049] in, represents the cumulative transmittance along the light; t n and t f They represent the near and far planes of the camera when sampling along the ray, and are used to determine the range of the integral. Discrete the integral into a sum to obtain:

[0050]

[0051] Among them, δ i =t i+1 -t i is the spacing between sampling points;

[0052] This formula can be regarded as a weighted sum of colors, and the weight is defined as w(t) = T(t)σ(t). Then, after obtaining the density value of the coarse sampling, the weight w corresponding to each point can be obtained. i .

[0053] 6. Normalize these weights w This generates a piecewise constant probability density function along the ray, N c is the number of coarse sampling points on each ray, and we use the inverse transform to extract fine sampling points from this distribution.

[0054] 7. Repeat process 4, and send the position and direction codes of the fine sampling points, together with the appearance synthesis features, to the high-frequency forward reasoning module. Figure 4 As shown in the figure, it is a multi-layer perceptron (MLP), each box represents a layer, the number of neurons in the box represents the number of neurons; the density value σ and color value c corresponding to the fine sampling point are obtained. The above volume rendering formula is used to obtain the predicted pixel value of the corresponding light.

[0055] 8. Calculate the loss function by comparing the predicted pixel value corresponding to each ray r with the true image value C(r) at the target viewing angle

[0056]

[0057] 9. Calculate the weight distribution of all fine sampling points along the light, that is, the probability density function. Set the loss function To use the weight distribution of fine sampling points to guide the update of the weight distribution of coarse sampling points, is the KL divergence between coarse and fine sampling points:

[0058]

[0059] Among them, P is the normalized weight distribution corresponding to the fine sampling point, and Q is the normalized weight distribution corresponding to the coarse sampling point. One thing to note here is: When updating weights, the gradient is only passed back to the branch where the low-frequency forward reasoning module is located, and will not affect the parameters of the high-frequency forward reasoning module.

[0060] The two loss functions and Add them as the final loss function and apply the back propagation algorithm to optimize the model parameters. Figure 5 As shown, this is the effect diagram of the new perspective synthesis of the present invention.

[0061] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A generalizable scene rendering method based on NeRF, characterized in that: The steps include: Step 1: Get the color images sparsely collected by the camera containing the camera's internal and external parameters, randomly select one of the images as the target image, and the rest as the source images; Step 2: Pass each source image through an appearance information encoder and a geometric information encoder with shared parameters to obtain an appearance feature map and a geometric feature map that are aligned with the source image pixels; Step 3: Send out light from the target perspective, perform rough sampling in space along the light, that is, uniform sampling, to obtain the direction of the sampling point d , location coordinates x The distance from the optical center of the target camera ; During the sampling process, the sampling points on each ray will be continuously resampled as the model is trained, so that the sampling points are gradually concentrated on the surface of the object; Step 4: After the coarse sampling points are parameterized with a three-dimensional isotropic Gaussian distribution, they are projected onto the appearance feature map and geometric feature map of each source image to obtain the feature query range. The weight distribution of the features in this range obeys a two-dimensional Gaussian distribution, whose mean is the position coordinate of the sampling point projected onto the feature map, and its variance is ,in is a hyperparameter, is the distance of the sampling point from the source camera, It is the actual width of the pixel in the world coordinate system. The weight is used to weight the features within the feature query range to obtain the appearance feature information and geometric feature information corresponding to the sampling point in each source image. Step 5: Set the position coordinates of the coarse sampling points x Parameterized into an isotropic three-dimensional Gaussian distribution, the direction and position coordinates of the sampling points are Fourier encoded to obtain the direction code and position code, the geometric feature information corresponding to the coarse sampling points on each source image is averaged and pooled to obtain the synthetic geometric features, which are sent to the low-frequency forward reasoning module together with the direction code and position code to obtain the density value of the coarse sampling points; Step 6: According to the density value of the coarse sampling point and the interval between the sampling points, the weight before the color value of each sampling point corresponding to the volume rendering can be calculated, and the weight before the color value of the coarse sampling point is divided by the weight before the color value and normalized to obtain the probability density function of the piecewise constant formula; then according to this probability density function, the next fine sampling is performed, and the fine sampling point is subjected to the same Fourier encoding as in step 5 to obtain the direction encoding and position encoding of the fine sampling point, and then the appearance feature information corresponding to the fine sampling point on the appearance feature map of each source image is obtained according to the method of step 4, and the synthetic appearance feature is obtained by average pooling, and sent to the high-frequency forward reasoning module together with the direction encoding and position encoding of the fine sampling point to obtain the density value and color value of the fine sampling point; Step 7: The density and color values ​​of all fine sampling points on the light are used to obtain the predicted pixel values ​​through volume rendering. Finally, the loss function is calculated by comparing the predicted pixel values ​​with the true value of the target image at the target perspective. ; Step 8: Calculate the weight distribution of all fine sampling points along the light ray, i.e. the probability density function, and set the loss function as in step 6 , which uses the weight distribution of fine sampling points to guide the update of the weight distribution of coarse sampling points, and transforms the two loss functions and The sum is used as the final loss function, and the model parameters are optimized using the back propagation algorithm until convergence.

2. The generalizable scene rendering method according to claim 1, characterized in that: In step 2, the image appearance information encoder and the image geometry information encoder both use resnet50 without the last layer and load its pre-trained weights after ImageNet training. The i-th image is recorded as , encoding to obtain the feature maps of each source image .

3. The generalizable scene rendering method according to claim 1, characterized in that: In step 4, the i-th picture is , the transformation matrix from the world coordinate system to its corresponding camera coordinate system is recorded as , is the rotation matrix, is the translation vector for the sampling point along the ray from the target perspective , converted to each source camera coordinate system, the transformed coordinates are Then multiply the intrinsic parameter matrix to obtain the image coordinates of each source image coordinate system , based on which the corresponding position is found on the feature map, which is the mean of the two-dimensional Gaussian distribution.

4. The generalizable scene rendering method according to claim 1, characterized in that: The feature query range is determined by connecting the 3σ boundary of the three-dimensional Gaussian distribution parameterized by the sampling points to the optical center of the source camera. The intercepted range on the feature map is the feature query range in step 4.

Images