A scene new view generation method based on local space aggregated neural radiance field

Abstract

The application provides a scene new view generation method based on local space fusion neural radiation field, containing the following steps: S1, using COLMAP to calibrate the camera of the input picture, and digitally representing the rays corresponding to each pixel on the image; S2, constructing a network structure, containing a neural radiation field network model and a local space information aggregation model; S3, selecting pixels as the input of each network iteration according to an optimized sampling mode, and outputting the predicted density and predicted color of each sampling point; S4, establishing a volume rendering model to calculate the color of the pixel corresponding to each ray; S5, constructing a loss function according to the actual color and predicted color of the pixel, considering the consistency of the local space color information, and updating the parameters of the network structure of S2; the application realizes the generation of a new view with higher quality by acquiring and fusing the three-dimensional point information of the local space, and interacting with the common features in the three-dimensional adjacent space.

Description

[Technical Field]

[0001] This invention relates to the field of computer graphics, specifically to a method for generating new scene views based on local spatial aggregated neural radiation fields. [Background Technology]

[0002] In the fields of computer graphics and 3D reconstruction, generating realistic 3D models has always been an important research direction. Over the past few decades, researchers have proposed many methods and algorithms to capture and render 3D scenes.

[0003] Traditional methods include structured light scanning, multi-view stereo photography, and triangulation. These methods acquire geometric information about a scene from different perspectives, then reconstruct and texture map it to generate a 3D model. However, these methods typically require complex data processing and manual operation, and often demand significant time and computational resources.

[0004] In recent years, with the development of deep learning technology, neural network-based methods have made significant progress in the field of computer graphics. Among them, Neural Radiation Field (NeRF) is an emerging technique that models the propagation of light in a scene by training a deep neural network, thereby achieving high-quality image rendering.

[0005] Compared to traditional methods, NeRF has the following advantages: it can generate realistic images, including realistic lighting effects and details; it does not require an explicit geometric reconstruction process, thus avoiding complex geometric processing; and it can handle complex lighting and occlusion conditions, generating more realistic results.

[0006] However, current NeRF methods still face some limitations and challenges. For example, they require high-quality training data, including a large amount of high-quality images and corresponding geometric information. Furthermore, NeRF networks take 3D points as input and output information about those points; during training, each 3D point is completely independent and does not interact with other points. This means they do not effectively utilize shared features from neighboring regions in 3D space and learn more information through interaction, thus leaving room for improvement in the quality of generated new views. [Summary of the Invention]

[0007] To address the aforementioned technical problems in related technologies, this invention aims to provide an improved NeRF method to overcome the limitations of existing technologies and further improve the quality of synthesized views. This method extracts and aggregates ray information from local space to fuse adjacent density and color information on a cross-section of a 3D scene, thereby achieving higher-quality new view generation. The method includes the following steps:

[0008] Step 1: Digitally represent the rays corresponding to each pixel in the input image.

[0009] Step 2: During each round of iterative training, select several rays corresponding to adjacent pixels as local spaces (patch), and sample three-dimensional points on these rays. Use their pose information as input data for the neural radiation field network, and output the density and color of each sampled point.

[0010] Step 3: Use a convolutional neural network to aggregate local information about the density and color of adjacent sampling points in each patch.

[0011] Step 4: Based on the volume rendering model, each sampling point is treated as a non-rigid object. Its density information is converted into radiation intensity, and the influence of background light is ignored. The predicted color of the corresponding pixel can be obtained by integrating within the sampling point range.

[0012] Step 5: Calculate the loss function by comprehensively considering the mean and variance of the color of each pixel and the color of each patch, and update the parameters of the neural radiation field network and the convolutional neural network with the goal of minimizing this value.

[0013] Step 6: After each network is trained, for a given viewpoint, the corresponding camera pose parameters are given to obtain the ray digital description of the pixel to be rendered. Inputting this description into the network will calculate the color of the corresponding pixel, and the combination will produce a new view image.

[0014] Preferably, step (1) specifically involves: inputting the existing viewpoint image into the COLMAP tool for camera calibration to obtain camera pose information, and then combining the normalized coordinates (direction vector) of each pixel in the camera coordinate system with the rotation matrix in the pose information to obtain the digital representation of the ray (starting point is the camera and ending point is infinity) corresponding to each pixel.

[0015] Preferably, step (2) specifically involves: The optimized sampling method is to randomly select several pixels as the center point of the patch, and then select all pixels within a k*k pixel range around it. The determination of sampling points on each ray uses a layered sampling method, i.e., training two networks separately, one for coarse rendering and the other for fine rendering. During coarse rendering, sampling points are uniformly selected within the sampling range, while during fine rendering, sampling points are proportionally allocated on the rays based on the density prediction results of the coarse rendering network. The sampling formula is as follows:

[0016]

[0017] Where t i Represents the sampling point, t n and t f The distance unit representing the sampling range (nearest and farthest points) is obtained during camera calibration, where N represents the number of sampling points.

[0018] As a preferred embodiment, step (3) specifically involves: using separate convolutional neural networks to aggregate local information for the predicted density and color of the sampling points, while ensuring that the dimension and size of the tensor remain unchanged before and after aggregation (i.e., equal-width convolution).

[0019] As a preferred embodiment, the volume rendering model in step (4) is as follows: after inputting the sampling points on the sampling light into the neural radiation field model described above, the predicted density and color of each point are obtained. Then, a volume rendering model is established, the matter is abstracted into a particle swarm, and the probability collision of light with particles when passing through the object is simulated according to the density and color. The final color formula is obtained by combining the four parts of absorption, emission, external scattering and internal scattering as follows:

[0020]

[0021] Where σ(u) is the predicted density, C(t) is the predicted color, and t represents the distance from the nearest point in the sampling range. n Distance to each sampling point.

[0022] As a preferred approach, in practical calculations, the continuous integral equation needs to be converted into a discrete summation formula, as shown below:

[0023]

[0024] Where δ n This represents the distance between adjacent sampling points, and is used to obtain the predicted color of the selected pixel during the iteration process.

[0025] Preferably, the loss function in step (5) is as follows: the first part is the mean squared error (MSE) between the predicted color and the actual color, the second part is the L2 norm of the mean of the predicted color and the mean of the actual color in each local space (patch), and the third part is the L2 norm of the variance of the predicted color and the variance of the actual color in each patch, as shown in the following formula:

[0026] L total =ρ1L color +ρ2L mean +ρ3L variance

[0027]

[0028]

[0029]

[0030] Where Q represents the set of all sampled pixels, and P represents the set of all patches. Let C(p) represent the predicted color at a certain pixel, C(p) represent the true color, M(·) represent the average color of pixels in a certain patch, V(·) represent the variance of pixel colors in a certain patch, and ρ1, ρ2, and ρ3 are the weight hyperparameters of the three loss functions.

[0031] This invention optimizes the sampling method during iterative training, sampling rays from adjacent spaces as patch inputs to the neural radiation field network, and adding a spatial aggregation module to use a convolutional neural network to fuse lateral spatial information. This allows the entire network to pay attention to surrounding information when outputting the density and color of each sampling point, achieving a more accurate new view synthesis effect. [Attached Image Description]

[0032] Figure 1 This is a flowchart of the overall process framework of the present invention.

[0033] Figure 2 This is a schematic diagram of the neural network structure of the present invention.

[0034] Figure 3 This is a schematic diagram of the sampling method designed in this invention.

Detailed Implementation Methods

[0035] The specific embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0036] refer to Figure 1 , Figure 2 as well as Figure 3 This implementation provides a method for generating new scene views based on local spatial aggregated neural radiation fields, such as... Figure 1 The process includes the following steps:

[0037] S1. Obtain the input image of the scene to be reconstructed, use the COLMAP tool to obtain the pose information of each camera, and obtain the digital representation of the rays from the camera to each pixel on each image.

[0038] First, several photos (20 or more, including non-repeating viewpoints, 30 in this case) are taken using a camera tool to reconstruct the physical objects. These photos are then input into the COLMAP tool, where feature extraction, feature matching, and sparse reconstruction are performed to obtain the camera pose information. Next, the `imgs2poses.py` script from the LLFF project is used to convert the data into a Local Light Field Fusion (LLFF) dataset. This dataset consists of a single `.npy` file containing 17-dimensional pose parameters for 30 images. The original images are read and their resolution is reduced according to the downsampling factor in the parameters. Finally, the pose parameters are converted into a 3x3 rotation transformation matrix and a 3x1 translation transformation matrix, along with the image's width, height, focal length, and a 2D boundary tensor. If scaling parameters are present, the rotation transformation matrix and boundary tensor are multiplied by a scaling factor to scale the data, resulting in the camera-to-world coordinate system transformation matrix and the pose matrix along the rendering path. The pose information, rendered pose, and test index for each image are returned.

[0039] The camera's intrinsic parameter matrix is ​​calculated based on the image's width, height, and focal length information, using the following formula:

[0040]

[0041] Where f is the camera focal length and H is the image height; based on the height and width of each image and the camera intrinsic parameter matrix, the ray representation originating from the camera corresponding to each pixel is calculated, including R. o That is, the translation transformation matrix in the pose information is expanded into a 3D size and R d That is, the direction vector obtained by multiplying the expression of each pixel's direction unit vector in the camera coordinate system by the rotation transformation matrix, as shown in the following formula:

[0042] R o =E(M c2w [:3,-1],R d )

[0043] R d [i, j] = ∑M dirs [i, j]·M c2w [:3,:3]

[0044] Among them, M c2w It is a 3x4 matrix containing rotation and translation transformation matrices, M. dirs This represents the unit vector of direction for each pixel in the camera coordinate system.

[0045] S2 constructs a network structure, including a neural radiation field network model and a local spatial information aggregation model.

[0046] Creating a neural radiation field model: (1) A position encoding structure maps the 3D position representation and 3D direction representation of the sampling points to a high-dimensional space through sine and cosine encoding to describe the high-frequency information in the scene, as shown in the following formula:

[0047] γ(p)={sin(2 0 πp), cos(2 0 πp), …, sin(2 L-1 πp), cos(2 L-1 πp)}

[0048] Where p represents the position or orientation information of each sampling point, sin(·) and cos(·) represent the sine and cosine calculations respectively, L represents the coding frequency, and π represents pi; (2) A multilayer perceptron (MLP) network predicts the sampling point density and color through the sampling point information, and references Figure 2 As shown in the network structure diagram, the density network consists of 8 fully connected layers with an output dimension of 256. The input dimension of the first layer is the dimension of the sampling point position vector (63 dimensions in this example). The input dimension of the fifth layer is 319 dimensions (256-dimensional output of the previous layer + 63-dimensional position vector). The input dimensions of the second, third, fourth, sixth, seventh, and eighth layers are all 256-dimensional. Then, a fully connected layer with an input of 256 dimensions and an output of 1 dimension is connected to obtain the predicted density of the sampling points. The color network consists of a fully connected layer with an input of 283 dimensions (256-dimensional output of the eighth layer of the density network + 27-dimensional orientation vector) and an output of 128 dimensions to extract features. A fully connected layer with an input of 128 dimensions and an output of 3 dimensions converts the features into the predicted RGB values ​​of the sampling points. The calculation formula for the fully connected layer is shown below:

[0049] Y = Activation(X·W+b)

[0050] X = [x1, x2, ..., x] n ] T

[0051]

[0052] b = [b1, b2, ..., b m ] T

[0053] Y = [y1, y2, ..., y] m ] T

[0054] Where Y represents the output vector of the fully connected layer, X represents the input vector of the fully connected layer, W represents the weight matrix, b represents the bias vector, n represents the input dimension, m represents the output dimension, represents matrix multiplication, + represents the addition between vectors, and Activation(·) represents the activation function, which can be a linear activation function (such as the identity function) or a non-linear activation function (such as ReLU, Sigmoid, tanh, etc.).

[0055] Creating a Local Spatial Information Aggregation Model: Based on the predicted density and color of each sampling point obtained from the neural radiation field network, this invention uses a convolutional neural network to aggregate local spatial information for both parts separately. Specifically, a single-layer convolutional layer with the number of input channels being the number of sampling points on a single ray * 1 (density) or * 3 (color), and the number of output channels being the same as the number of input channels, with a 3*3 kernel size and padding of 1, is applied to the predicted density and predicted color, realizing the interaction of horizontal information in the local space.

[0056]

[0057]

[0058] in This represents the input data, where H is the height of the input image, W is the width of the input image, and C is the number of channels in the input image. Each element X... i，j，k This represents the value of the input data at position (i, j) and channel k. This represents the filters in the convolutional layer, where K is the kernel size and N is the number of filters, i.e., the number of channels in the output feature map. Each filter F... p，q，r，s The weights of the p-th row, q-th column, r-th input channel, and s-th output channel are represented by the following formula:

[0059]

[0060] in Y represents the output of the convolutional layer. i，j，s This represents the value of the output data at position (i, j) and channel s.

[0061] S3 selects pixels as input for each network iteration based on the optimized sampling method, and performs coarse rendering and fine rendering in sequence using a hierarchical sampling method, outputting the predicted density and predicted color of each sampling point.

[0062] In each iteration, several pixels from a photograph are randomly selected. To acquire and fuse the horizontal information between pixels, this invention designs a sampling mode that uses adjacent pixels as patches. Specifically, in a complete image, several center points are randomly selected (512 in this example). Pixels radiating outwards from these 512 pixels within a 3x3 area are included in the sampling range. These 512 patches serve as input to the neural radiation field network, allowing the network to fuse and learn the 3D information representation of each physical point in 3D space and its surrounding local space. It is important to note that to obtain reasonable and effective patches after the center points are determined, the outermost ring of pixels in the image should be excluded when selecting the center points (two rings for a 5x5 area, three rings for a 7x7 area, and so on). Figure 3 As shown.

[0063] After selecting the ray, three-dimensional points need to be sampled on each ray. This invention adopts a layered sampling method, that is, the first round performs uniform sampling at equal intervals on the ray, and after inputting it into the neural radiation field network to obtain the predicted density, dense sampling with non-equal intervals is performed in the region with high density according to the predicted density. In this way, more scene 3D information can be learned under the same computational cost.

[0064] S4, the three-dimensional rendering model calculates the color of the corresponding pixel point by predicting the density and color of each sampling point on each ray according to probability.

[0065] After inputting the sampling points on the sampling light beam into the neural radiation field model described above, the predicted density and color of each point are obtained. Then, a volume rendering model is built, abstracting matter into a particle swarm. Based on the density and color, the probability collisions between light and particles when passing through objects are simulated. The final color formula is obtained by combining the four parts of absorption, emission, external scattering, and internal scattering as follows:

[0066] Where σ(u) is the predicted density and C(t) is the predicted color, in actual calculations, the continuous integral equation needs to be converted into a discrete summation formula as shown below:

[0067]

[0068] The predicted color of the selected pixel is obtained during the iteration process.

[0069] S5: Based on the actual color and predicted color of the pixel, a loss function is constructed considering the consistency of local spatial color information, and the parameters of the S2 network structure are updated accordingly.

[0070] After obtaining the predicted color of each pixel, the predicted color is compared with the true color of the input image data to calculate the loss function. To further aggregate information in the local space, the loss function designed in this invention consists of three parts: the first part is the mean squared error (MSE) between the predicted color and the actual color; the second part is the L2 norm of the mean of the predicted color and the mean of the actual color in each local space (patch); and the third part is the L2 norm of the variance of the predicted color and the variance of the actual color in each patch. The formula is shown below:

[0071] L total =p1L color +p2L mean +p3L oariance

[0072]

[0073]

[0074]

[0075] Where Q represents the set of all sampled pixels, and P represents the set of all patches. Let C(p) represent the predicted color at a certain pixel, C(p) represent the true color, M(·) represent the average color of pixels in a certain patch, and V(·) represent the variance of pixel colors in a certain patch. ρ1, ρ2, and ρ3 are the weight hyperparameters of the three loss functions, which are set to 1, 5e-2, and 1e-3 respectively in this example. Based on this loss function, the backpropagation method is used to update and iterate the parameters of the neural radiation field network and the local spatial information aggregation network.

[0076] After training for several rounds (100,000 rounds in this embodiment), based on the rendering pose given when reading data, the ray representation of each pixel is obtained using the same method. This representation is then input into the trained neural radiation field network and the volume rendering model is used to obtain the pixel colors of the new perspective rendering image. Arranging these colors in the corresponding order yields the new perspective rendering image, thus completing the 3D reconstruction of the scene.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating new scene views based on local spatial aggregated neural radiation fields, characterized in that, Includes the following steps: S1. Obtain images and construct a dataset; S2. Digitally represent the ray corresponding to each pixel in the image; The specific method for step S2 is as follows: S2-1. Input the existing viewpoint image into the COLMAP tool to calibrate the camera and obtain the camera pose information; S2-2. Then, by combining the normalized coordinates of each pixel in the camera coordinate system with the rotation matrix in the pose information, we obtain the digital representation of the ray corresponding to each pixel. S3. Construct the network structure, including a neural radiation field network model and a local spatial information aggregation model. S4. During each iteration of training, the neural radiation field network model selects several rays corresponding to adjacent pixels as patches, and samples three-dimensional points on these rays. The pose information of the sampled three-dimensional points is used as the input data of the neural radiation field network, and the predicted density and color of each sampled point are output. A local spatial information aggregation model is used to aggregate the local information of the density and color of adjacent sampling points in each patch; S5. Based on the volume rendering model, each sampling point is regarded as a non-rigid object. Its density information is converted into radiation intensity, and the influence of background light is omitted. The predicted color of the corresponding pixel is obtained by integrating within the sampling point range. S6. Taking into account the mean and variance of the color of each pixel and the color of each patch, calculate the loss function, and update the parameters of the neural radiation field network model and the local spatial information aggregation model with the goal of minimizing this value. The loss function in step S6 is specifically as follows: This includes the mean squared error (MSE) between the predicted and actual colors, the L2 norm of the mean of the predicted colors and the mean of the actual colors in each patch, and the L2 norm of the variance of the predicted colors and the variance of the actual colors in each patch. S7. After each network is trained, for a given viewpoint, the corresponding camera pose parameters are given to obtain the digital description of the ray corresponding to the pixel to be rendered. The network is then input to calculate the color of the corresponding pixel, and the combination is used to obtain the new view generated image.

2. The method for generating new scene views based on local spatial aggregated neural radiation fields according to claim 1, characterized in that, The normalized coordinates refer to the direction vector of each pixel in the camera coordinate system.

3. The method for generating new scene views based on local spatial aggregated neural radiation fields according to claim 1, characterized in that, The ray corresponding to each pixel refers to the ray in the camera coordinate system that starts from the camera and ends at infinity.

4. The method for generating new scene views based on local spatial aggregated neural radiation fields according to claim 1, characterized in that, In step S3, two neural radiation field networks are built for coarse rendering and fine rendering, respectively. The sampling method for the three-dimensional points is as follows: randomly select a number of pixels as the center point of the patch, and select all pixels within a k*k pixel range around it.

5. The method for generating new scene views based on local spatial aggregated neural radiation fields according to claim 1, characterized in that, The sampling points on each ray are determined using a layered sampling method. Two neural radiation field networks are trained separately, one for coarse rendering and the other for fine rendering. During coarse rendering, sampling points are selected uniformly within the sampling range. During fine rendering, sampling points are distributed proportionally on the rays based on the density prediction results of the coarse rendering network.

6. The method for generating new scene views based on local spatial aggregated neural radiation fields according to claim 1, characterized in that, In step S3, the dimensions and size of the tensors remain unchanged before and after aggregation.

Images