An efficient neural radiance field rendering method based on depth-guided sampling

Abstract

The present invention discloses an efficient neural radiation field rendering method based on depth-guided sampling. First, a target view is obtained, and the image features of the target view are obtained; before the target view, several depth planes are sampled, and a cost volume with variance as the cost is constructed on each depth plane using the extracted image features; the cost volume is processed using a neural network based on 3D convolution to obtain a probability volume, and the depth of each pixel is obtained; for each pixel point, a small number of points are sampled in the interval where the depth may be located, and the neural network is used to calculate the neural radiation field; the color of each pixel point is calculated using a volume rendering formula to obtain the final rendered image. The present invention can support high-speed rendering based on neural radiation fields. High-speed rendering based on neural radiation fields can support related applications of new perspective synthesis in large scenes and dynamic scenes, including free viewpoint video live broadcast and other applications.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to an efficient neural radiation field rendering method based on depth-guided sampling. Background Art

[0002] Recently, some rendering techniques based on neural radiance fields have achieved excellent results in new perspective synthesis, but this technology requires hours of training and a long rendering process. In order to avoid hours of training, some methods use image features to reconstruct neural radiance fields, which enables this technology to generalize to new scenes and avoid training, but it still requires a long rendering process. To solve this problem, some methods use efficient data structures to pre-calculate and cache the neural radiance fields of the scene to achieve acceleration. However, pre-calculation takes a long time and caching requires a lot of storage space. These shortcomings prevent new perspective synthesis technology based on neural radiance application in large scenes and dynamic scenes. Summary of the invention

[0003] The purpose of the present invention is to address the deficiencies of the prior art and to propose an efficient neural radiation field rendering method based on depth-guided sampling.

[0004] The object of the present invention is achieved through the following technical solution: an efficient neural radiation field rendering method based on depth-guided sampling, the specific steps of the method are as follows:

[0005] (1) Obtain M images near the perspective to be rendered, and extract the features of the M images through a deep neural network {F i |i=1,…,N}, where F i is the feature of the i-th picture; the picture of the perspective to be rendered is called the target view, and the picture near the perspective to be rendered is called the given view.

[0006] (2) Use the given view to calculate the depth image D of the target view, and use D(u,v) to represent the depth value of the pixel point (u,v) on the target view.

[0007] (3) For a pixel (u, v) on the target view, the possible depth interval is [D(u, v)-ΔD(u, v), D(u, v)+ΔD(u, v)], and N samples are taken. k Points {x k |k=1,…,N k}, where ΔD(u,v) is the maximum error between the true depth value and the depth value calculated in step (2), x k Normalize the coordinates of the device coordinate space for the target view;

[0008] (4) For the point x sampled in step (3) k, the neural network Φ is used to calculate the neural radiation field of the sampling point (c, σ) = Φ (d, f). Where c and σ are the values ​​of point x respectively. k The radiation and volume density of the camera are: d is the direction of the ray from the camera center through the pixel (u, v) in 3D space; f is x k The corresponding image features are projected onto the deep neural network input image in step (1);

[0009] (5) Use the volume rendering formula to aggregate the neural radiation fields (c, σ) of all sampling points at each pixel to obtain the final rendered image.

[0010] Furthermore, the mean square error loss function is used to supervise the deep neural network in step (1) and the neural network Φ in step (3).

[0011] Furthermore, the process of calculating the target view depth image D in step (2) is to construct two cost volumes in succession; the resolution of the second cost volume is higher than that of the first cost volume; the construction process is as follows: before the target view, a range containing scene objects is given, and M′ planes {L j ′|j=1,…,M′},L′ j is the depth of the jth plane; the cost volume with variance as the cost is constructed on each depth plane using the image features extracted in step (1); specifically, given the i-th view I i The camera parameters [K i ,R i ,t i ] and the camera parameters [K, R, t] of the target view define a homography: Where n represents the camera center axis of the target view, this homography transformation H i (z) Convert a pixel (u, v) on the target view to the i-th given view I at depth z i On the top, interpolate to find the feature F′ corresponding to the converted pixel point i,z (u,v)=F i (H i (z)[u,v,1] T ), where F i (H i (z)[u,v,1] T )) is the image feature F i Remove H i (z)[u,v,1] T The features of the corresponding pixel points; based on the converted features, the value of each element in the cost volume is defined as the feature {F′ on a given view i,zThe variance of (u,v}|i=1,…,N}; this cost body is input into the neural network based on 3D convolution to obtain a deep probability body P′, and the jth plane P′ of the probability body P′ j The probability at a pixel point (u, v) represents the probability that the depth of the corresponding pixel point falls on this plane; the depth of a pixel point (u, v) is defined as the expectation of the probability distribution on M′ depth planes According to the depth D′ calculated in the above steps, in the depth range [D′(u,v)-ΔD′(u,v),D′(u,v)+ΔD′(u,v)], sample M depth planes {L j |j=1,…,M},L j (u, v) is the depth of the pixel (u, v) in the jth plane. The image features extracted in step (1) are used to construct a cost volume with variance as the cost on each depth plane. This cost volume is input into a 3D convolution-based neural network to obtain a depth probability volume P. The jth plane P of the probability volume P is j The probability at a pixel point (u, v) represents the probability that the depth of the corresponding pixel point falls on this plane; the depth of a pixel point (u, v) is defined as the expectation of the probability distribution on the M depth planes

[0012] Furthermore, the process of calculating ΔD(u,v) in step (3) is as follows:

[0013] Furthermore, the process of calculating f in step (4) is to aggregate the features of N images using the average pooling operation φ to obtain the average feature f on N images = φ(F1,…,F N ).

[0014] Beneficial effects of the present invention: The present invention can support high-speed neural radiation field-based rendering without additional overhead. High-speed neural radiation field-based rendering can support related applications of new perspective synthesis in large scenes and dynamic scenes, including free viewpoint video live broadcast and other applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 is a schematic diagram of efficient neural radiation field rendering based on depth-guided sampling according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0018] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0019] like Figure 1 As shown, the present invention provides an efficient neural radiation field rendering method based on depth-guided sampling, which estimates the depth based on a multi-viewpoint method and uses this depth to guide the sampling of the neural radiation field. The specific steps are:

[0020] 1. Get N images near the rendering perspective, and use an existing deep neural network, such as 2D U-Net, to extract N image features {F i |i=1,…,N}, and use the mean square error loss function to supervise the deep neural network. i is the feature of the i-th picture; the picture of the perspective to be rendered is called the target view, and the picture near the perspective to be rendered is called the given view.

[0021] 2. Use the given view to calculate the depth image D of the target view, and use D(u,v) to represent the depth value of the pixel point (u,v) on the target view. The process of calculating the depth image D of the target view is as follows:

[0022] Two cost volumes are constructed successively; the resolution of the second cost volume is higher than that of the first cost volume; the construction process is as follows: before the target view, a range containing scene objects is given, and M′ planes {L j ′|j= 1,…,M′},L′ j is the depth of the jth plane; the cost volume with variance as the cost is constructed on each depth plane using the image features extracted in step (1); specifically, given the i-th view I i The camera parameters [K i ,R i ,t i ] and the camera parameters [K, R, t] of the target view define a homography: Where n represents the camera center axis of the target view, this homography transformation H i (z) Convert a pixel (u, v) on the target view to the i-th given view I at depth z iOn the top, interpolate to find the feature F′ corresponding to the converted pixel point i,z (u,v)=F i (H i (z)[u,v,1] T ), where F i (H i (z)[u,v,1] T )) is the image feature F i Remove H i (z)[u,v,1] T The features of the corresponding pixel points; based on the converted features, the value of each element in the cost volume is defined as the feature {F′ on a given view i,z The variance of (u,v}|i=1,…,N}; this cost body is input into the neural network based on 3D convolution to obtain a deep probability body P′, and the jth plane P′ of the probability body P′ j The probability at a pixel point (u, v) represents the probability that the depth of the corresponding pixel point falls on this plane; the depth of a pixel point (u, v) is defined as the expectation of the probability distribution on M′ depth planes According to the depth D′ calculated in the above steps, in the depth range [D′(u,v)-ΔD′(u,v),D′(u,v)+ΔD′(u,v)], sample M depth planes {L j |j=1,…,M},L j (u, v) is the depth of the pixel (u, v) in the jth plane. The image features extracted in step (1) are used to construct a cost volume with variance as the cost on each depth plane. This cost volume is input into a 3D convolution-based neural network to obtain a depth probability volume P. The jth plane P of the probability volume P is j The probability at a pixel point (u, v) represents the probability that the depth of the corresponding pixel point falls on this plane; the depth of a pixel point (u, v) is defined as the expectation of the probability distribution on the M depth planes

[0023] 3. For a pixel point (u, v) on the target view, calculate the maximum error ΔD(u, v) between the true depth value and the depth value calculated in step 2. The specific process is: In the interval where its depth may be [D(u,v)-ΔD(u,v),D(u,v)+ΔD(u,v)], sample N k Points {x k |k=1,…,N k}, K ranges from 2 to 8, where x kNormalize the coordinates of the device coordinate space for the target view; for the sampling point, use a neural network Φ, such as a multi-layer perceptron with a depth of 8 and a width of 256, to calculate the neural radiation field (c, σ) = Φ(d, f), and use the mean square error loss function to supervise the neural network Φ. Where d is x k The direction in the target view's normalized device coordinate space. f is the x k In step 1, the corresponding image features of the deep neural network input image are aggregated. Specifically, the average pooling operation φ is used to aggregate the features of N images to obtain the average feature f=φ(F1,…,F N ).

[0024] 4. Use the volume rendering formula to aggregate the neural radiation fields (c, σ) of all sampling points at each pixel to obtain the final rendered image. Specific embodiment:

[0026] On the DTU dataset, given a target view, three neighboring views are obtained as given views. A 2D U-Net is used to obtain three image features; before the target view, the depth D is estimated in combination with the process described in step (2); using the process described in step (3), 2 points are sampled for each pixel, and a multi-layer perceptron with a depth of 8 and a width of 256 is used to calculate the color and volume density of each point; the color of each pixel is calculated using the volume rendering formula, and the mean square error loss function is used to complete the supervision of the 2D U-Net and the multi-layer perceptron. The traditional method does not use the specific process of step (2) of the present invention. The present invention innovatively uses the probability volume P to process and predict the depth of the pixel point, and calculates the neural radiation field. The traditional method needs to sample multiple points (128 to 256 points) from the range of the object included in step (1) in the process of step (3) of the present invention to achieve the same level of rendering quality as the method of sampling a small number of points (2 points) near the depth obtained in step (3). Therefore, the present invention can support high-speed rendering based on neural radiation fields without additional overhead.

[0027] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0028] The above is only a preferred embodiment of the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any technician familiar with the art can make many possible changes and modifications to the technical solution of the present invention by using the above disclosed methods and technical contents without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. An efficient neural radiance field rendering method based on depth-guided sampling, characterized in that: The specific steps of this method are as follows: (1) Obtain N images near the perspective to be rendered, and extract the features of N images through a deep neural network {F i |i=1,...,N},where F i is the feature of the i-th picture; The image of the perspective to be rendered is recorded as the target view, and the image near the perspective to be rendered is recorded as the given view; (2) Using the given view, calculate the depth image D of the target view, and use D(u, v) to represent the depth value of the pixel point (u, v) on the target view; (3) For a pixel (u, v) on the target view, the possible depth interval is [D(u, v)-ΔD(u, v), D(u, v)+ΔD(u, v)], and N samples are collected. k Points {x k |k=1,...,N k }, where ΔD(u, v) is the maximum error between the true depth value and the depth value calculated in step (2), x k is the point uniformly sampled in the above depth interval by the ray passing through the pixel point (u, v) from the center of the camera; (4) For the point x sampled in step (3) k , the neural network Φ is used to calculate the neural radiation field (c, σ) = Φ(d, f) of the sampling point; where c and σ are the k The radiation and volume density of the camera are: d is the direction of the ray from the camera center through the pixel (u, v) in 3D space; f is x k The corresponding image features are projected onto the deep neural network input image in step (1); (5) Use the volume rendering formula to aggregate the neural radiation fields (c, σ) of all sampling points at each pixel to obtain the final rendered image.

2. According to claim 1, a method for efficient neural radiation field rendering based on depth-guided sampling is characterized in that: The mean square error loss function is used to supervise the deep neural network in step (1) and the neural network Φ in step (4).

3. The method for efficient neural radiation field rendering based on depth-guided sampling according to claim 1, characterized in that: The process of calculating the target view depth image D in step (2) is to construct two cost volumes in succession; the resolution of the second cost volume is higher than that of the first cost volume; the construction process is as follows: Before the target view, given a range containing scene objects, M′ planes {L j ′|j=1,...,M′},L′ j is the depth of the jth plane; the cost volume with variance as the cost is constructed on each depth plane using the image features extracted in step (1); specifically, given the i-th view I i The camera parameters [K i , R i , t i ] and the camera parameters [K, R, t] of the target view define a homography: Where n represents the camera center axis of the target view, this homography transformation H i (z) Convert a pixel (u, v) on the target view to the i-th given view I at depth z i On the top, interpolate to find the feature F′ corresponding to the converted pixel point i，z (u, v) = F i (H i (z)[u,v,1] T ), where F i (H i (z)[u,v,1] T )) is the image feature F i Remove H i (z)[u,v,1] T The features of the corresponding pixel points; based on the converted features, the value of each element in the cost volume is defined as the feature {F′ on a given view i，z The variance of (u, v}|i=1,...,N}; this cost body is input into the neural network based on 3D convolution to obtain a deep probability body P′, and the jth plane P′ of the probability body P′ j The probability at the pixel point (u, v) represents the probability that the depth of the corresponding pixel point falls on this plane; the depth of the pixel point (u, v) is defined as the expectation of the probability distribution on M′ depth planes According to the depth D′ calculated in the above steps, in the depth range [D′(u, v)-ΔD′(u, v), D′(u, v)+ΔD′(u, v)], M depth planes {L j |j=1,...,M},L j (u, v) is the depth of the pixel (u, v) in the jth plane; the image features extracted in step (1) are used to construct a cost volume with variance as the cost on each depth plane; this cost volume is input into a 3D convolution-based neural network for processing to obtain a depth probability volume P. The jth plane P of the probability volume P is j The probability at the pixel point (u, v) represents the probability that the depth of the corresponding pixel point falls on this plane; the depth of the pixel point (u, v) is defined as the expectation of the probability distribution on the M depth planes 4. The method for efficient neural radiation field rendering based on depth-guided sampling according to claim 1, characterized in that: The process of calculating ΔD(u, v) in step (3) is:

5. The method for efficient neural radiation field rendering based on depth-guided sampling according to claim 1, characterized in that: The process of calculating f in step (4) is to aggregate the features of N images using the average pooling operation φ to obtain the average feature f = φ(F1, ..., F N ).

Images