FPGA accelerator for high-quality novel view synthesis based on neural radiance fields

Abstract

The application discloses an FPGA accelerator for realizing high-quality new view synthesis based on a neural radiation field; the FPGA accelerator comprises an FPGA and three operation modules, namely, a position coding module, an MLP module and a ray stepping module, and the operation modules are arranged on the FPGA; the MLP module is split into two-stage pipelines; a front-stage pipeline is used to derive body density and a hidden layer, and a rear-stage pipeline is used to derive color information; the front-stage pipeline has eight fully-connected layers, and there are ReLu activation functions between the fully-connected layers; a coordinate signal is subjected to the front-stage pipeline to obtain the body density; the rear-stage pipeline has two fully-connected layers, and input hidden layer information and a view direction, and then is subjected to a sigmoid activation function to obtain the color information. Experimental results show that the hardware architecture designed in the application can accelerate the neural radiation field calculation, and compared with the same algorithm running on a GPU, the running power consumption is far less than that of the GPU, and the generated image quality is similar to that of the GPU.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image synthesis, and particularly relates to an FPGA accelerator for realizing high-quality new view synthesis based on a neural radiation field. BACKGROUND

[0002] For decades, extending human visual perception to 3D has been an important research direction in the field of computer vision, and related technologies are widely used in 3D televisions or virtual reality games. Traditional computer vision technology uses multiple 2D images to reconstruct a 3D geometric model, and then reprojects it to obtain virtual views of different perspectives in the same scene. However, the time consumed by traditional methods such as vertex network, polygon triangle segmentation, and surface subdivision to reconstruct a 3D model is proportional to the complexity of the scene; the more complex the geometric relationship of the objects in the scene, the longer the modeling time, and the more likely modeling errors.

[0003] However, the recent neural radiation field (NeRF) method proposes a completely different idea from traditional modeling methods. Neural radiation field is a fully connected deep neural network, in which the input is a single continuous 5D coordinate (spatial position (x, y, z) and perspective direction ), and the output is the volume density of the spatial position and the color related to the perspective direction. Neural radiation field is used to represent a scene, and new views are synthesized by querying the 5D coordinates on the camera ray, and the output color and density are projected onto the image using classic volume rendering techniques. The new views synthesized using this deep learning method have better quality than previous works, and the rendering speed is accelerated and the model storage size is reduced.

[0004] However, synthesizing new views through neural radiation field is a time-consuming process, and it is difficult to achieve real-time synthesis speed; the biggest drawback of the neural radiation field technology is that it takes a long time to train and render. During rendering, for a million-level number of pixels, the multi-layer perception MLP network needs to be queried hundreds of times for each generated pixel. The MLP used by the neural radiation field is relatively deep and wide, resulting in a long processing time. At the same time, the neural radiation field technology relies on a ray marching-based volume rendering algorithm, and the currently widely deployed graphics hardware has not been adapted.

[0005] To solve the above problems, the main existing technologies are as follows:

[0006] In "Baking Neural Radiance Fields for Real-Time View Synthesis" 1A new representation, Sparse Neural Radiance Grid (SNeRG), is proposed to enable real-time rendering of Neural Radiance Fields by pre-computing and storing.

[0007] In the paper MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures 2 A new texture-mapped polygon-based representation of Neural Radiance Fields is proposed, which can effectively synthesize new images with the standard rendering pipeline.

[0008] The above two techniques are to improve the Neural Radiance Field at the algorithm level to speed up. These two techniques increase the model training cost and model size, while reducing the quality of the synthesized new view image and the similarity to the real scene.

[0009] Therefore, in order to make the Neural Radiance Field technology land in the application end, it is necessary to accelerate it. However, the computing power of Digital Signal Processing DSP is limited, and it is difficult to guarantee the real-time performance of the system when the image resolution is improved; the computing power of the Graphics Processing Unit GPU is very powerful, but the power consumption is high, the size is large, and it is not convenient to integrate in the mobile terminal. Special hardware such as Field Programmable Gate Array FPGA or Application Specific Integrated Circuit ASIC can provide hardware acceleration for Neural Radiance Field while greatly reducing the operation power consumption compared with GPU, achieving real-time rendering effect. The above adapts the Neural Radiance Field algorithm from the perspective of hardware architecture, which can avoid the problems of existing technologies while accelerating the calculation of Neural Radiance Field. SUMMARY

[0010] In view of the deficiencies of the prior art, the purpose of the present application is to provide a hardware architecture specially used for Neural Radiance Field rendering. Since the process of calculating the color of each pixel point is independent, multiple pixel values can be calculated in parallel.

[0011] The technical solutions of the present application are specifically introduced as follows.

[0012] An FPGA accelerator for realizing high-quality new view synthesis based on Neural Radiance Field comprises an FPGA and three operation modules: a position encoding module, an MLP (Multi-Layer Perceptron) module, and a ray marching module, and the operation modules are deployed on the FPGA; wherein:

[0013] The position encoding module is used for encoding the input position information; before the input is transmitted to the MLP module, the input is mapped to a higher dimensional space with a high frequency function as the position encoding function, so as to better fit the data containing high frequency changes. The position encoding function used is:

[0014] γ(p) = (sin(2 0 πp), cos(2 0 πp), …, sin(2 L-1 πp), cos(2 L-1 πp),

[0015] Where L = 10 is the number of diffusion terms for processing coordinates, and L = 4 is the number of viewing direction processing.

[0016] The MLP module splits into two stages of pipeline: the first stage pipeline derives the body density and the hidden layer, and the second stage pipeline derives the color information; the first stage pipeline has a total of 8 fully connected layers, and there is a ReLu activation function between the fully connected layers. The coordinate signal obtains the body density after passing through the first stage pipeline; the second stage pipeline has a total of two fully connected layers, and the input is the hidden layer information obtained by the first stage pipeline and the viewing direction, and then the sigmoid activation function is used to obtain the color information.

[0017] The ray marching module is used for integral operation of stereoscopic rendering of points on a ray; in the region of the upper and lower bounds t f and t n on the ray r(t) = o + td, the expected color C(r) obtained is:

[0018]

[0019] Where σ(r(t)) is the body density of the ray at t,

[0020] In specific implementation, [t n , t f ] is uniformly divided into N segments, and an average random sampling is performed in each segment, that is,

[0021]

[0022] Therefore, the stereoscopic rendering formula is rewritten as:

[0023]

[0024] Where σ i = σ(r(t i )),

[0025] On this basis, the formula is rewritten as:

[0026]

[0027] wherein

[0028] Thus, the T obtained at each stage is saved at the time of calculation i , for calculating T i+1 , improve resource reuse rate.

[0029] In the present application, a direct memory access module, an I / O module for communication between the FPGA and the host, a central routing module for controlling the transmission of data flow and a data routing module are further included.

[0030] In the present application, a plurality of levels of cache are respectively arranged in the direct memory access module, the data routing module, the position encoding module and the MLP module; and a level of cache is arranged in the output device.

[0031] Compared with the prior art, the present application has the beneficial effects that: in the present application, a plurality of repeated MLP modules can be deployed on the FPGA, the process of calculating the color of each pixel point by different MLP modules is independent, a plurality of pixel values can be calculated in parallel, and the operation rate is improved. In addition, experimental results show that the hardware architecture designed in the present application can accelerate the neural radiation field calculation, and compared with running the same algorithm on a GPU, the running power consumption is much smaller than that of the GPU, and the generated image quality is similar to that of the GPU. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a structure diagram of a previous stage pipeline in the MLP module.

[0033] Figure 2 is a structure diagram of a subsequent stage pipeline in the MLP module.

[0034] Figure 3 is the overall framework of the FPGA accelerator. DETAILED DESCRIPTION

[0035] The technical solutions of the present application will be described in detail below with reference to the drawings and embodiments.

[0036] The present application provides an FPGA accelerator for realizing high-quality new view synthesis based on a neural radiation field.

[0037] In the operation process of the neural radiation field, in addition to the MLP part, there are two key parts, which are position encoding (Positional Encoding, PE) and ray marching.

[0038] The application designs three modules: position encoding module, MLP module and ray marching module from the basic operation process of neural radiance field.

[0039] The position encoding module encodes the input position information. Before the input is transmitted to the MLP module, the input is mapped to a higher dimensional space using a high frequency function, which can better fit the data containing high frequency changes. The position encoding function used is:

[0040] γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp)),

[0041] Where p represents the input position information. Where L=10 for processing coordinates and L=4 for processing viewing direction.

[0042] In order to reduce the waste of operation resources, the MLP module splits into two stages of pipeline: the first stage of pipeline derives the volume density and hidden layer, and the second stage of pipeline derives the color information. The first stage of pipeline has a total of 8 fully connected layers, and there are ReLu activation functions between the fully connected layers. After the coordinate signal passes through the first stage of pipeline, the volume density is obtained through ReLU. The second stage of pipeline has a total of two fully connected layers, and the hidden layer information and viewing direction obtained by the first stage of pipeline are input, and then the color information is obtained through the sigmoid activation function.

[0043] The ray marching module is an integral operation for the stereoscopic rendering of points on a ray. In the area of the upper and lower limits of t f and t n on the ray r(t)=o+td, the expected color C(r) is:

[0044]

[0045] Where σ(r(t)) is the volume density of the ray at t,

[0046] In specific implementation, the area method is adopted, and [t n , t f ] is uniformly divided into N segments, and in each segment, it is randomly sampled on average, that is,

[0047]

[0048] Therefore, the stereoscopic rendering formula can be rewritten as:

[0049]

[0050] Where σi = σ(r(t i )),

[0051] On this basis, the formula can also be rewritten as:

[0052]

[0053] where

[0054] Thus, the T i obtained at each level is saved during calculation i+1 , to improve resource reuse.

[0055] Figure 3 is the overall framework of the FPGA accelerator. When implemented on the FPGA, in addition to the three operation modules mentioned above, there are a direct memory access module (DMA), an I / O module for communication between the FPGA and the host, a central routing module and a data routing module for controlling the transmission of data streams. In order to ensure that the overall pipeline of the system can run efficiently and smoothly, multi-level cache RAM is set in the DMA module, the data routing module, the position encoding module and the MLP module, and a level of cache is designed for output.

[0056] References

[0057] 1. Hedman P, Srinivasan P P, Mildenhall B, et al. Baking neural radiance fields for real-time view synthesis [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 5875-5884.

[0058] 2 , Chen Z, Funkhouser T, Hedman P, et al. MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures [J]. arXiv preprint arXiv:2208.00277, 2022.

Claims

1. An FPGA accelerator for high-quality novel view synthesis based on neural radiance fields, characterized in that, It includes FPGA and three operation modules: position encoding module, MLP module and ray marching module, and the operation modules are deployed on the FPGA; wherein: The position encoding module encodes the input position information including coordinates and perspective direction; it uses a high-frequency function as the position encoding function to map the input to a higher dimensional space to better fit the data containing high-frequency changes, and the position encoding function γ(p) used is: y(p) = (sin(2 0 πp), cos(2 0 πp),..., sin(2 L-1 πp), cos(2 L-1 πp)), Where p represents the input position information, when processing coordinate information, the number of expansion terms L=10, and when processing perspective direction information, the number of expansion terms L=4; The MLP module processes the position information encoded data to obtain volume density and color information; the MLP module splits into two-stage pipeline: the first stage pipeline obtains volume density and hidden layer, and the second stage pipeline obtains color information; wherein: the first stage pipeline has a total of 8 fully connected layers, and the fully connected layers are connected by ReLu activation function, and the coordinate information obtains the volume density of the space position after passing through the first stage pipeline; the second stage pipeline has a total of two fully connected layers, and the input is perspective direction information and hidden layer information obtained by the first stage pipeline, and then the sigmoid activation function is used to obtain the color information of the perspective direction; The ray marching module performs integral operation on the points on a ray for solid rendering, and projects the obtained volume density and color information onto the image.

2. The FPGA accelerator for high-quality novel view synthesis based on neural radiance fields of claim 1, wherein, In the ray marching module, It is known that on a light ray r(t) = o + td, with t f and t n the region of the upper and lower bounds respectively, the resulting desired color C(r) is: where σ(r(t)) is the volume density of the light at t, In a specific implementation, the area method is used to divide [t n ,t f ] into N segments uniformly, and average random sampling is performed in each segment, that is, Thus the solid rendering formula is rewritten as: where σ i = σ(r(t i )), On this basis, the formula is rewritten as: wherein Thus, the T obtained at each stage is saved at the time of calculation i , for calculating T i+1 , to improve resource reuse rate.

3. The FPGA accelerator for high-quality novel view synthesis based on neural radiance fields of claim 1, wherein, It also includes a direct memory access module for communication between the FPGA and the host, an I / O module, a central routing module for controlling the transmission of data flow, and a data routing module.

4. The FPGA accelerator for high-quality novel view synthesis based on neural radiance fields of claim 3, wherein, The direct memory access module, the data routing module, the position encoding module and the MLP module are respectively provided with multi-level cache; and the output device is provided with a one-level cache.

Images