A NeRF renderer and rendering method based on FPGA and multi-resolution hash coding algorithm

Abstract

The present invention discloses a NeRF renderer and a rendering method based on FPGA and a multi-resolution hash coding algorithm, relating to the technical field of image rendering. The NeRF renderer is deployed on the FPGA, and the NeRF renderer includes a feature reader, a color and opacity reasoner, a ray renderer, and a memory; the feature reader calculates the light direction of a sampling point and the distance between the sampling points according to input parameters, and sends them to the color and opacity reasoner and the ray renderer respectively; the address of the required feature in the memory is calculated according to the coordinates of the calculated sampling point, and after reading the feature, interpolation processing is performed and the result is sent to the color and opacity reasoner; the color and opacity reasoner calculates the color of the sampling point and the opacity of the sampling point and sends them to the ray renderer; the ray renderer calculates the rendering result according to the distance between the sampling points, the color of the sampling points, and the opacity information; the target features are distinguished, thereby reducing memory usage and simplifying the rendering steps.

Description

Technical Field

[0001] The present invention relates to the field of image rendering technology, and more specifically, to a NeRF renderer and rendering method based on FPGA and a multi-resolution hash coding algorithm. Background Art

[0002] Neural Radiance Fields (NeRF) is a technology that constructs complete 3D scenes from a small number of 2D images. Since its introduction, it has garnered widespread attention. This technology effectively extracts 3D scene information from 2D images and uses this information to construct 3D scenes. It is foreseeable that this technology will have significant applications in areas such as 3D reconstruction and image rendering.

[0003] Early NeRF scene construction relied entirely on neural networks. This approach involved upscaling the sampled points and the directions of light rays passing through them, feeding them directly into the neural network. The neural network's output provided the color and opacity information for each point. This approach had the following drawbacks: 1. The neural network was large, requiring long training times and computational complexity. 2. Increasing the scene's resolution required further scaling up the neural network and requiring even longer training times. 3. Even after training, rendering a single image took a significant amount of time.

[0004] Later, some faster NeRF scene construction technologies emerged, represented by the table lookup method. This method does not use neural networks at all, but directly stores the opacity of the sampling points in space and the spherical harmonic function coefficients of the color in a table. When calculating the color of a certain point, the corresponding features are directly read from the table according to the position of the point for interpolation, and then the color of the point is calculated using the light direction and the spherical harmonic function coefficients. This method is much faster than the early NeRF in both training and rendering, but the disadvantage of this method is also very obvious, that is, it consumes a huge amount of memory to store this table, and as the scene resolution increases, the size of this table will continue to grow. Therefore, this method is not suitable for building scenes that are too detailed.

[0005] In order to solve these problems, the multi-resolution lookup table method came into being. The principle of this method is to take into account that for a point, some features will not have a very high frequency, while some features will have a higher frequency. Then, by lowering the resolution of the table of those low-frequency features and increasing the resolution of the table of those high-frequency features, the memory usage can be reduced while ensuring the overall scene resolution. This method requires the reintroduction of the neural network. After reading the features of a certain point in the tables of different resolutions, these features and the direction of the light passing through the point are sent to the neural network to obtain the corresponding color and opacity information. The scale of the neural network used in this method is much smaller than that used in the early NeRF. More importantly, its scale does not need to increase with the improvement of the scene resolution, so its memory usage is negligible compared to the table.

[0006] Although the multi-resolution lookup table method effectively reduces memory usage, the storage of high-resolution feature tables still consumes a huge amount of memory.

[0007] The prior art discloses an efficient neural radiation field rendering method based on depth-guided sampling, and the specific steps of the method are as follows: (1) obtain N pictures near the perspective to be rendered, and extract N picture features {Fi|i=1, ..., N} through a deep neural network, where Fi is the feature of the i-th picture; the picture of the perspective to be rendered is recorded as the target view, and the picture near the perspective to be rendered is recorded as the given view; (2) the depth image D of the target view is calculated using the given view, and D(u, v) is used to represent the depth value of the pixel point (u, v) on the target view; (3) for a pixel point (u, v) on the target view, in the interval [D(u, v)-ΔD(u, v), D(u, v)+ΔD(u, v)] where its depth may be located, sample Nk points {xk|k=1, ..., Nk}, where ΔD(u, v) is the difference between the true depth value and the depth value in step (2 ) is the maximum error of the depth value calculated, xk is the point uniformly sampled in the above depth interval by the ray from the camera center through the pixel point (u, v); (4) for the point xk sampled in step (3), the neural network Φ is used to calculate the neural radiation field (c, σ) = Φ(d, f) of the sampling point; wherein c and σ are the radiation and volume density of point xk respectively, d is the direction of the ray from the camera center through the pixel point (u, v) in 3D space; f is the aggregation of the corresponding image features of xk projected on the deep neural network input picture in step (1); (5) the neural radiation fields (c, σ) of all sampling points on each pixel are aggregated using the volume rendering formula to obtain the final rendered picture; the shortcomings of the existing technology are that the extracted target features and related parameters occupy a large memory capacity, the target features are not distinguished, and the cost volume needs to be constructed, which increases the workload of processing information. Summary of the Invention

[0008] In order to solve the problems in the existing technology that the extraction of target features occupies a large amount of memory, the target features are not differentiated, and the image rendering steps are cumbersome, the present invention proposes a NeRF renderer and rendering method based on FPGA and multi-resolution hash coding algorithm. The corresponding features are extracted by a feature reader, the corresponding feature storage location is found in the memory and read, the target features are differentiated, and the rendering results are calculated based on the distance between sampling points, the color and opacity information of the sampling points, thereby simplifying the image rendering steps.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0010] A NeRF renderer based on an FPGA and a multi-resolution hash coding algorithm is deployed on the FPGA. The NeRF renderer includes a feature reader, a color and opacity inference unit, a light renderer, and a memory pre-stored with tables of different resolutions; the tables of different resolutions correspondingly store features of feature points at different frequencies.

[0011] The feature reader calculates the light direction of the sampling points and the distance between the sampling points according to the camera parameters and image parameters of the image to be rendered;

[0012] The feature reader sends the calculated light direction of the sampling point to the color and opacity reasoner, and sends the calculated distance between the sampling points to the light renderer; calculates the required features based on the calculated coordinates of the sampling points, reads the table of different frequencies in the memory based on the multi-resolution hash coding table lookup method, and obtains the corresponding features; and interpolates the obtained features and sends them to the color and opacity reasoner;

[0013] The color and opacity reasoner calculates the color and opacity of the sampling point according to the obtained light direction of the sampling point and the features after interpolation processing, and sends them to the light renderer;

[0014] The light renderer calculates the rendering result according to the distance between the obtained sampling points, the color of the sampling points and the opacity information.

[0015] The working principle of the present invention is as follows:

[0016] The feature reader calculates the light direction of the sampling point, the distance between the sampling points, and the features after interpolation processing; the color and opacity reasoner calculates the color of the sampling point and the opacity of the sampling point; the light renderer uses a rendering equation to calculate the rendering result based on the distance between the sampling points, the color of the sampling points and the opacity information.

[0017] Preferably, the tables of different resolutions in the memory are read based on the multi-resolution hash coding table lookup method, as follows:

[0018] Set a limit value for the size of a table and determine whether the size of all tables is greater than the limit value. If the size of the table is less than the limit value, direct addressing of the three-dimensional coordinates is used when addressing the contents of the table; if the size of the table is greater than the limit value, a hash function is used to address the hash encoding result of the three-dimensional coordinates, and the hash algorithm is used to compress the table.

[0019] Furthermore, the feature reader includes a light parameter calculator, a light sampler, an address generator, a memory controller, and an interpolator;

[0020] The light parameter calculator obtains the light direction, light starting point, near-end distance, and far-end distance of the sampling point based on the camera parameters and image parameters of the image to be rendered; and determines whether rendering is required based on the size relationship between the near-end distance and the far-end distance;

[0021] The light parameter calculator sends the light direction of the sampling point to the color and opacity reasoner; after determining that rendering is required, the light direction, light starting point, near end distance, and far end distance of the sampling point are sent to the light sampler;

[0022] The light sampler calculates the distance between the sampling points and sends it to the light renderer, and calculates the coordinates of the sampling points and sends them to the address generator;

[0023] The address generator calculates the address of the sampling point coordinates in the memory and sends it to the memory controller;

[0024] The memory controller reads the corresponding features from the memory and sends them to the interpolator;

[0025] The interpolator performs interpolation processing and sends the interpolated features to the color and opacity reasoner.

[0026] Furthermore, the specific steps of the light parameter calculator for calculating the light direction, light starting point, near end distance, and far end distance of the sampling point are as follows:

[0027] S101: Consider the camera position as the starting point of the light;

[0028] S102: Calculate the direction of the pixel point in the camera coordinate system;

[0029] S103: using matrix multiplication to transform the direction into a direction in the real space;

[0030] S104: The transformed result is normalized to obtain the light direction of the sampling point;

[0031] S105: Calculate the distance from the starting point of the light to the six planes where the sampling area boundaries are located in the direction of the light;

[0032] S106: among the three groups of opposite planes, the three with smaller distances are divided into a proximal group, and the three with larger distances are divided into a distal group;

[0033] S107: The maximum value between the distances in the proximal group and 0 is the proximal distance, and the minimum value among the distances in the distal group is the distal distance;

[0034] S108: Determine whether the collected light needs to be rendered by determining the size relationship between the near end distance and the far end distance; if the near end distance is greater than the far end distance, it indicates that the light needs to be rendered; if the far end distance is greater than the near end distance, it indicates that the light does not need to be rendered.

[0035] Furthermore, after the address generator calculates the coordinates of the sampling point, it reads the features of the eight feature points in the table that are closest to the sampling point from the memory, and uses trilinear interpolation to obtain the features of the sampling point.

[0036] Preferably, the color and opacity reasoner includes a direction cache module, a spherical harmonic function encoder, and a fully connected neural network module;

[0037] The direction buffer module receives the light direction of the sampling point sent from the feature reader;

[0038] The spherical harmonic function encoder takes out the light direction of the sampling point stored earliest in the direction cache module from the direction cache module, performs spherical harmonic function encoding on it, and then sends it to the fully connected neural network module;

[0039] The fully connected neural network module calculates the color and opacity of the sampling point based on the interpolated features sent by the feature reader and the light direction of the sampling point encoded by the spherical harmonic function encoder, and sends them to the light renderer.

[0040] Furthermore, the calculation of the fully connected neural network module is completed based on the multiplication and accumulation unit constructed by the DSP on the FPGA; the fully connected neural network module is a structure with optional parallelism.

[0041] The fully connected neural network module includes an input layer, an output layer, a first hidden layer, a second hidden layer, a third hidden layer, and a fourth hidden layer; the number of input features of the input layer is 32, the number of hidden features of the first hidden layer is 64, and the activation function is ReLU; the number of features of the second hidden layer is 16, and the activation function is ReLU, the first feature is taken as the opacity output, and the remaining 15 features;

[0042] The fully connected neural network module uses a resolution level of 16, each resolution provides two features, so the input feature dimension is 32; the spherical harmonic function encoding used is 4th order, so the input direction dimension after encoding is 16; except for the output layer, each layer uses the ReLU function to provide nonlinearity;

[0043] The opacity of the sampling point is adjusted to the range of [0, +∞) using the ReLU function before output, and the color range of the sampling point is adjusted to the range of [0, 1] using the piecewise broken line function before output; the piecewise broken line function is as follows:

[0044]

[0045] In the formula, x is the independent variable of the piecewise broken line function, and f(x) is the function value of the piecewise broken line function; the piecewise broken line function compresses the values between -8 and 8 to 0 to 1 first, compresses the values less than -8 to 0, and compresses the values greater than 8 to 1.

[0046] Preferably, the ray renderer includes a distance cache module, a weight calculator, a weight cache module, and a multiplier-accumulator;

[0047] The distance buffer receives the distance between the sampling points sent by the feature reader;

[0048] The weight calculator takes out the earliest distance stored in the distance cache module from the distance cache module, receives the opacity of the sampling point sent from the color and opacity reasoner, calculates the earliest distance stored in the distance cache module and the opacity of the sampling point to obtain a weight, and sends it to the weight cache module;

[0049] The multiplication and accumulation device takes out the weight stored in the multiplication and accumulation device earliest from the weight cache module, accumulates the weight with the color of the sampling point sent by the color and opacity inference device, and outputs the accumulated result as the rendering result.

[0050] Furthermore, the light renderer completes the rendering of light through the following formula:

[0051]

[0052]

[0053] Where C i is the color of the sampling point sent by the color and opacity inference engine, σ i is the opacity sent by the color and opacity reasoner, δ i is the distance between the sampling points sent by the feature reader, C(r) is the color of the light r, i and j are the subscripts of the accumulation process, T i It has no practical meaning for process quantities.

[0054] A rendering method for a NeRF renderer based on FPGA and a multi-resolution hash coding algorithm, the method being as follows:

[0055] According to the camera parameters and image parameters of the input image to be rendered, the light direction of the sampling point and the distance between the sampling points are calculated;

[0056] Calculate the required features based on the calculated coordinates of the sampling points, read the tables of different frequencies in the memory based on the multi-resolution hash coding table lookup method to obtain the corresponding features; and interpolate the obtained features. The memory pre-stores tables of different resolutions; the tables of different resolutions correspondingly store features of different frequencies of feature points;

[0057] The color and opacity of the sampling point are calculated based on the light direction of the sampling point and the features after interpolation processing;

[0058] The rendering result is calculated based on the distance between the obtained sampling points, the color of the sampling points, and the opacity information.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. Use multi-resolution hash coding lookup table method to read tables of different frequencies. Using multi-resolution hash coding lookup table method to read is beneficial to reduce memory usage and achieve good rendering effect.

[0061] 2. Deploying the NeRF model to the FPGA solves the problem of NeRF-based 3D reconstruction and image rendering being separated from the GPU. This makes NeRF deployment no longer limited to GPUs and can be applied on lightweight computing devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Schematic diagram of the structure of the NeRF renderer based on FPGA and multi-resolution hash coding algorithm.

[0063] Figure 2 Schematic diagram of the structure of the fully connected neural network module described in the embodiment.

[0064] Figure 3 Schematic diagram of the structure of the multiplier-accumulator. DETAILED DESCRIPTION

[0065] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] Example 1

[0067] In this embodiment, if Figure 1 As shown, a NeRF renderer based on FPGA and multi-resolution hash coding algorithm is deployed on the FPGA, wherein the NeRF renderer includes a feature reader, a color and opacity reasoner, a light renderer, and a memory pre-stored with tables of different resolutions; the tables of different resolutions correspondingly store features of feature points of different frequencies;

[0068] The feature reader calculates the light direction of the sampling points and the distance between the sampling points according to the camera parameters and image parameters of the image to be rendered;

[0069] The feature reader sends the calculated light direction of the sampling point to the color and opacity reasoner, and sends the calculated distance between the sampling points to the light renderer; calculates the required features based on the calculated coordinates of the sampling points, reads the table of different frequencies in the memory based on the multi-resolution hash coding table lookup method, and obtains the corresponding features; and interpolates the obtained features and sends them to the color and opacity reasoner;

[0070] The color and opacity reasoner calculates the color and opacity of the sampling point according to the obtained light direction of the sampling point and the features after interpolation processing, and sends them to the light renderer;

[0071] The light renderer calculates the rendering result according to the distance between the obtained sampling points, the color of the sampling points and the opacity information.

[0072] The working principle of the present invention is as follows:

[0073] The feature reader calculates the light direction of the sampling point, the distance between the sampling points, and the features after interpolation processing; the color and opacity reasoner calculates the color of the sampling point and the opacity of the sampling point; the light renderer uses a rendering equation to calculate the rendering result based on the distance between the sampling points, the color of the sampling points and the opacity information.

[0074] In this embodiment, a multi-resolution hash coding table lookup method is used to read tables of different resolutions in memory, as follows:

[0075] Set a limit value for the size of a table and determine whether the size of all tables is greater than the limit value. If the size of the table is less than the limit value, direct addressing of the three-dimensional coordinates is used when addressing the contents of the table; if the size of the table is greater than the limit value, a hash function is used to address the hash encoding result of the three-dimensional coordinates, and the hash algorithm is used to compress the table.

[0076] More specifically, the feature reader includes a light parameter calculator, a light sampler, an address generator, a memory controller, and an interpolator;

[0077] The light parameter calculator obtains the light direction, light starting point, near-end distance, and far-end distance of the sampling point based on the camera parameters and image parameters of the image to be rendered; and determines whether rendering is required based on the size relationship between the near-end distance and the far-end distance;

[0078] The light parameter calculator sends the light direction of the sampling point to the color and opacity reasoner; after determining that rendering is required, the light direction, light starting point, near end distance, and far end distance of the sampling point are sent to the light sampler;

[0079] The light sampler calculates the distance between the sampling points and sends it to the light renderer, and calculates the coordinates of the sampling points and sends them to the address generator;

[0080] The address generator calculates the address of the sampling point coordinates in the memory and sends it to the memory controller;

[0081] The memory controller reads the corresponding features from the memory and sends them to the interpolator;

[0082] The interpolator performs interpolation processing and sends the interpolated features to the color and opacity reasoner.

[0083] More specifically, the light parameter calculator calculates the light direction, light starting point, near end distance, and far end distance of the sampling point in the following steps:

[0084] S101: Consider the camera position as the starting point of the light;

[0085] S102: Calculate the direction of the pixel point in the camera coordinate system;

[0086] S103: using matrix multiplication to transform the direction into a direction in the real space;

[0087] S104: The transformed result is normalized to obtain the light direction of the sampling point;

[0088] S105: Calculate the distance from the starting point of the light to the six planes where the sampling area boundaries are located in the direction of the light;

[0089] S106: among the three groups of opposite planes, the three with smaller distances are divided into a proximal group, and the three with larger distances are divided into a distal group;

[0090] S107: The maximum value between the distances in the proximal group and 0 is the proximal distance, and the minimum value among the distances in the distal group is the distal distance;

[0091] S108: Determine whether the collected light needs to be rendered by determining the size relationship between the near end distance and the far end distance; if the near end distance is greater than the far end distance, it indicates that the light needs to be rendered; if the far end distance is greater than the near end distance, it indicates that the light does not need to be rendered.

[0092] More specifically, after the address generator calculates the coordinates of the sampling point, it reads the features of the eight feature points in the table that are closest to the sampling point from the memory, and uses trilinear interpolation to obtain the features of the sampling point.

[0093] More specifically, the color and opacity reasoner includes a direction cache module, a spherical harmonic function encoder, and a fully connected neural network module;

[0094] The direction buffer module receives the light direction of the sampling point sent from the feature reader;

[0095] The spherical harmonic function encoder takes out the light direction of the sampling point stored earliest in the direction cache module from the direction cache module, performs spherical harmonic function encoding on it, and then sends it to the fully connected neural network module;

[0096] The fully connected neural network module calculates the color and opacity of the sampling point based on the interpolated features sent by the feature reader and the light direction of the sampling point encoded by the spherical harmonic function encoder, and sends them to the light renderer.

[0097] More specifically, the calculation of the fully connected neural network module is completed based on the multiplication and accumulation unit constructed by the DSP on the FPGA; the fully connected neural network module has an optional parallelism structure and is adaptable to FPGAs of different models and different numbers of available resources.

[0098] More specifically, the ray renderer includes a distance cache module, a weight calculator, a weight cache module, and a multiply-accumulator;

[0099] The distance buffer receives the distance between the sampling points sent by the feature reader;

[0100] The weight calculator takes out the earliest distance stored in the distance cache module from the distance cache module, receives the opacity of the sampling point sent from the color and opacity reasoner, calculates the earliest distance stored in the distance cache module and the opacity of the sampling point to obtain a weight, and sends it to the weight cache module;

[0101] The multiplication and accumulation device takes out the weight stored in the multiplication and accumulation device earliest from the weight cache module, accumulates the weight with the color of the sampling point sent by the color and opacity inference device, and outputs the accumulated result as the rendering result.

[0102] The light renderer uses the following formula to render the light:

[0103]

[0104]

[0105] Where C i is the color of the sampling point sent by the color and opacity inference engine, σ i is the opacity sent by the color and opacity reasoner, δ i is the distance between the sampling points sent by the feature reader, C(r) is the color of the light r, i and j are the subscripts of the accumulation process, T i It has no practical meaning for process quantities.

[0106] Example 2

[0107] Based on the NeRF renderer based on FPGA and multi-resolution hash coding algorithm described in Example 1, in this embodiment, the light parameter calculator calculates the light starting point, light direction, near-end distance and far-end distance of all pixels that may need to be rendered based on the camera position, direction, angle, lens focal length, photosensitive element size and image resolution information of the image to be rendered.

[0108] All calculations involved are fixed-point calculations. Using fixed-point calculations can effectively utilize the DSP resources on the FPGA while ensuring accuracy. In special computing scenarios, when the dynamic range of certain values is confirmed in advance, the precision can be improved by modifying the decimal point position of the fixed-point number, thereby obtaining more accurate calculation results.

[0109] In this embodiment, if Figure 2As shown, the fully connected neural network module includes an input layer, an output layer, a first hidden layer, a second hidden layer, a third hidden layer, and a fourth hidden layer; the number of input features of the input layer is 32, the number of hidden features of the first hidden layer is 64, and the activation function is ReLU; the number of features of the second hidden layer is 16, and the activation function is ReLU, and the first feature is taken as the opacity output, leaving 15 features;

[0110] The fully connected neural network module uses a resolution level of 16, each resolution provides two features, so the input feature dimension is 32; the spherical harmonic function encoding used is 4th order, so the input direction dimension after encoding is 16; except for the output layer, each layer uses the ReLU function to provide nonlinearity;

[0111] The opacity of the sampling point is adjusted to the range of [0, +∞) using the ReLU function before output, and the color range of the sampling point is adjusted to the range of [0, 1] using the piecewise broken line function before output; the piecewise broken line function is as follows:

[0112]

[0113] In the formula, x is the independent variable of the piecewise broken line function, and f(x) is the function value of the piecewise broken line function; the piecewise broken line function compresses the values between -8 and 8 to 0 to 1 first, compresses the values less than -8 to 0, and compresses the values greater than 8 to 1.

[0114] In this embodiment, the calculation of the fully connected neural network module is performed based on the multiplication and accumulation unit of the DSP, and this structure can be completely implemented by the internal circuit of the DSP; the multiplication and accumulation unit in the fully connected neural network module is the same as the multiplication and accumulation unit in the ray rendering, and both are circuits that accumulate the results of multiple multiplications.

[0115] In this embodiment, if Figure 3 As shown, a and b are the two factors of a product to be accumulated. After multiplication and the L1 register cache, they are accumulated. If the product is the first number in the accumulation, first is pulled high to add the product to 0. Otherwise, first is pulled low to add the product to the previous accumulation result, thus achieving the accumulation effect.

[0116] When using this structure to implement a fully connected neural network module, first assume that the calculation time of each layer is 64 cycles. For the first layer, 32 multiplication and accumulation units can be used to perform two rounds of calculations within 64 cycles. Each round of calculation performs 32 logarithmic multiplications and accumulations, and outputs 32 results of the hidden layer, for a total of 64 results. This fills the work of the 32 multiplication and accumulation units. For the second layer, 16 multiplication and accumulation units can be used to take out the two rounds of results calculated in the previous layer, a total of 64 values, in two times, and complete one round of calculation within 64 cycles, thereby obtaining the 16 results required for the second layer. The subsequent layers also use this method for calculation, thereby achieving a pipeline that calculates one point every 64 cycles. When transferring results between layers, saturation overflow is used and rounded down to simplify the circuit. For each pipeline, the number of accumulators in each layer is fixed, because the fully connected neural network module in this model is fixed.

[0117] In this structure, the direction input is about one pipeline level later than the feature input, which is why there is a direction cache module; the opacity output is three pipeline levels earlier than the color output, which is why there is a weight cache module. The first pipeline level refers to the time required from the input of a layer to its output to the next layer; according to this design structure, the specific number of first-level pipeline cycles is different for each layer. The pipeline from the input layer to the first hidden layer is 32 cycles, the pipeline from the first hidden layer to the second hidden layer is 64 cycles; the pipeline from the second hidden layer to the third hidden layer is 32 cycles; the pipeline from the third hidden layer to the fourth hidden layer is 64 cycles; the pipeline from the fourth hidden layer to the output layer is 64 cycles. Generally speaking, the direction input is later than the feature input, and the opacity output is earlier than the color output.

[0118] Throughput refers to the average number of clock cycles required to process each point. The average processing cycle of each pipeline is 64 cycles, which means that the calculation of one point can be completed every 64 cycles. By replicating 64 pipelines at the same time, it is possible to complete the calculation of one point per cycle on average, thereby matching the processing speed of its front-end and back-end.

[0119] Pipelining refers to dividing a large task into multiple components according to stages and assigning them to different modules. Registers are inserted between modules so that the circuit can handle multiple tasks at a time, and each task is in a different processing stage or module in each clock cycle.

[0120] When replicating pipelines to increase throughput, different pipelines can simultaneously use the same control logic and network parameters, reducing bandwidth requirements and simplifying the circuit. This is the basis for the selectable parallelism structure in this design. Based on the maximum throughput provided by the upper and lower levels, the pipeline can be replicated up to 64 times, achieving a throughput of one point per cycle.

[0121] Example 3

[0122] In this embodiment, the weight calculator calculates the weight using the following formula:

[0123] w i =T i (1-exp(-σ i δ i ))

[0124]

[0125] Where σ i is the opacity sent by the color and opacity reasoner, δ i is the distance between the sampling points sent by the feature reader, w i is the weight; the weight and color are added together to get the rendering result; the weight calculation process requires the use of the exp function, the domain of which is (-∞, 0]; the hardware implementation of the function is the cumulative multiplication method, and the calculated value, in the case of a 16-bit fixed-point number, e -2^n The effective precision range of n in the equation is [-16, 3]. That is, the bits of the number to be calculated are taken, and whether the bit is 1 determines whether it needs to participate in the cumulative multiplication, and finally completes the calculation of the exp function. In the actual design, in order to ensure high-speed operation of the circuit, the circuit is designed as a fully pipelined structure. After having the exp calculation circuit, the weight calculation steps are as follows:

[0126] S301: After the fully connected neural network module sends the opacity, take out the earliest distance stored in the distance cache module from the distance cache module and multiply it by the distance;

[0127] S302: using the exp calculation circuit to calculate the exponential function value;

[0128] S303: Subtract the calculated exponential function value from 1 and multiply it by the value in the T register to obtain a weight, which is then stored in the weight cache.

[0129] S304: Update the T register. If the point is the last point on a ray, update the T register to 1; otherwise, update the T register to the value multiplied by the exp calculation circuit output.

[0130] The entire computational process is designed to be fully pipelined, providing lower-level circuits for up to 64 fully connected neural network pipelines.

[0131] A rendering method of a NeRF renderer based on the above-mentioned FPGA and multi-resolution hash coding algorithm, the method being as follows: according to camera parameters and image parameters of an input image to be rendered, calculating the light direction of a sampling point and the distance between the sampling points;

[0132] Calculate the required features based on the calculated coordinates of the sampling points, read the tables of different frequencies in the memory based on the multi-resolution hash coding table lookup method to obtain the corresponding features; and interpolate the obtained features. The memory pre-stores tables of different resolutions; the tables of different resolutions correspondingly store features of different frequencies of feature points;

[0133] The color and opacity of the sampling point are calculated based on the light direction of the sampling point and the features after interpolation processing;

[0134] The rendering result is calculated based on the distance between the obtained sampling points, the color of the sampling points, and the opacity information.

[0135] The NeRF renderer based on FPGA and multi-resolution hash coding algorithm described in Examples 1 and 2 can be deployed on the FPGA of the Zynq-7000. For example, on the XC7Z020CLG484-2 chip, a fully connected neural network pipeline can be implemented and run at 2000MHz. Under this deployment condition, if each ray collects 128 points, then the time required to render a 400×300 image is about 5 seconds; if each ray collects 512 points, then the time required to render a 1920×1080 image is about 5 minutes and 40 seconds; 64 fully connected neural network pipelines can achieve 512 points per ray at 200MHz, and the time required to render a 1920×1080 image is about 5 seconds.

[0136] Obviously, the above embodiments of the present invention are merely examples for the purpose of illustrating the present invention, and are not intended to limit the embodiments of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A NeRF renderer based on FPGA and multi-resolution hash coding algorithm, characterized in that Deploy a NeRF renderer on the FPGA, wherein the NeRF renderer includes a feature reader, a color and opacity reasoner, a light renderer, and a memory pre-stored with tables of different resolutions; the tables of different resolutions correspondingly store features of different frequencies of feature points; The feature reader calculates the light direction of the sampling points and the distance between the sampling points based on the camera parameters and image parameters of the image to be rendered; The feature reader sends the calculated light direction of the sampling point to the color and opacity reasoner, and sends the calculated distance between the sampling points to the light renderer; calculates the required features based on the calculated coordinates of the sampling points, and reads the table of different frequencies in the memory based on the multi-resolution hash coding table lookup method to obtain the corresponding features; The obtained features are interpolated and sent to the color and opacity reasoner; The color and opacity reasoner calculates the color and opacity of the sampling point according to the obtained light direction of the sampling point and the features after interpolation processing, and sends them to the light renderer; The light renderer calculates the rendering result according to the distance between the obtained sampling points, the color of the sampling points and the opacity information.

2. A NeRF renderer based on FPGA and multi-resolution hash coding algorithm according to claim 1, characterized in that: The table with different resolutions in memory is read based on the multi-resolution hash coding table lookup method, as follows: Set a limit value for the size of a table and determine whether the size of all tables is greater than the limit value. If the size of the table is less than the limit value, direct addressing of the three-dimensional coordinates is used when addressing the contents of the table; if the size of the table is greater than the limit value, a hash function is used to address the hash encoding result of the three-dimensional coordinates, and the hash algorithm is used to compress the table.

3. A NeRF renderer based on FPGA and multi-resolution hash coding algorithm according to claim 2, characterized in that: The feature reader includes a light parameter calculator, a light sampler, an address generator, a memory controller, and an interpolator; The light parameter calculator obtains the light direction, light starting point, near-end distance, and far-end distance of the sampling point based on the camera parameters and image parameters of the image to be rendered; and determines whether rendering is required based on the size relationship between the near-end distance and the far-end distance; The light parameter calculator sends the light direction of the sampling point to the color and opacity reasoner; after determining that rendering is required, the light direction, light starting point, near end distance, and far end distance of the sampling point are sent to the light sampler; The light sampler calculates the distance between the sampling points and sends it to the light renderer, and calculates the coordinates of the sampling points and sends them to the address generator; The address generator calculates the address of the sampling point coordinates in the memory and sends it to the memory controller; The memory controller reads the corresponding features from the memory and sends them to the interpolator; The interpolator performs interpolation processing and sends the interpolated features to the color and opacity reasoner.

4. A NeRF renderer based on FPGA and multi-resolution hash coding algorithm according to claim 3, characterized in that: The specific steps of the light parameter calculator for calculating the light direction, light starting point, near end distance, and far end distance of the sampling point are as follows: S101: Consider the camera position as the starting point of the light; S102: Calculate the direction of the pixel point in the camera coordinate system; S103: using matrix multiplication to transform the direction into a direction in the real space; S104: The transformed result is normalized to obtain the light direction of the sampling point; S105: Calculate the distance from the starting point of the light to the six planes where the sampling area boundaries are located in the direction of the light; S106: among the three groups of opposite planes, the three with smaller distances are divided into a proximal group, and the three with larger distances are divided into a distal group; S107: The maximum value between the distances in the proximal group and 0 is the proximal distance, and the minimum value among the distances in the distal group is the distal distance; S108: Determine whether the collected light needs to be rendered by determining the size relationship between the near end distance and the far end distance; if the near end distance is greater than the far end distance, it indicates that the light needs to be rendered; if the far end distance is greater than the near end distance, it indicates that the light does not need to be rendered.

5. The NeRF renderer based on FPGA and multi-resolution hash coding algorithm according to claim 3, characterized in that: After the address generator calculates the coordinates of the sampling point, it reads the features of the eight feature points in the table that are closest to the sampling point from the memory, and uses trilinear interpolation to obtain the features of the sampling point.

6. The NeRF renderer based on FPGA and multi-resolution hash coding algorithm according to claim 1, characterized in that: The color and opacity reasoner includes a direction cache module, a spherical harmonic function encoder, and a fully connected neural network module; The direction buffer module receives the light direction of the sampling point sent from the feature reader; The spherical harmonic function encoder takes out the light direction of the sampling point stored earliest in the direction cache module from the direction cache module, performs spherical harmonic function encoding on it, and then sends it to the fully connected neural network module; The fully connected neural network module calculates the color and opacity of the sampling point based on the interpolated features sent by the feature reader and the light direction of the sampling point encoded by the spherical harmonic function encoder, and sends them to the light renderer.

7. The NeRF renderer based on FPGA and multi-resolution hash coding algorithm according to claim 6, characterized in that: The calculation of the fully connected neural network module is completed based on the multiplication and accumulation unit constructed by the DSP on the FPGA; the fully connected neural network module has an optional parallelism structure; The fully connected neural network module includes an input layer, an output layer, a first hidden layer, a second hidden layer, a third hidden layer, and a fourth hidden layer; the number of input features of the input layer is 32, the number of hidden features of the first hidden layer is 64, and the activation function is ReLU; the number of features of the second hidden layer is 16, and the activation function is ReLU, the first feature is taken as the opacity output, and the remaining 15 features; The fully connected neural network module uses a resolution level of 16, each resolution provides two features, so the input feature dimension is 32; the spherical harmonic function encoding used is 4th order, so the input direction dimension after encoding is 16; except for the output layer, each layer uses the ReLU function to provide nonlinearity; The opacity of the sampling point is adjusted to the range of [0, +∞) using the ReLU function before output, and the color range of the sampling point is adjusted to the range of [0, 1] using the piecewise broken line function before output; the piecewise broken line function is as follows: In the formula, x is the independent variable of the piecewise broken line function, and f(x) is the function value of the piecewise broken line function; the piecewise broken line function compresses the values between -8 and 8 to 0 to 1 first, compresses the values less than -8 to 0, and compresses the values greater than 8 to 1.

8. The NeRF renderer based on FPGA and multi-resolution hash coding algorithm according to claim 1, characterized in that: The ray renderer includes a distance cache module, a weight calculator, a weight cache module, and a multiplier-accumulator; The distance buffer receives the distance between the sampling points sent by the feature reader; The weight calculator takes out the earliest distance stored in the distance cache module from the distance cache module, receives the opacity of the sampling point sent from the color and opacity reasoner, calculates the earliest distance stored in the distance cache module and the opacity of the sampling point to obtain a weight, and sends it to the weight cache module; The multiplication and accumulation device takes out the weight stored in the multiplication and accumulation device earliest from the weight cache module, accumulates the weight with the color of the sampling point sent by the color and opacity inference device, and outputs the accumulated result as the rendering result.

9. The NeRF renderer based on FPGA and multi-resolution hash coding algorithm according to claim 8, characterized in that: The light renderer uses the following formula to render the light: Where C i is the color of the sampling point sent by the color and opacity inference engine, σ i is the opacity sent by the color and opacity reasoner, δ i is the distance between the sampling points sent by the feature reader, C(r) is the color of the light r, i and j are the subscripts of the accumulation process, T i It has no practical meaning for process quantities.

10. The rendering method of the NeRF renderer based on FPGA and multi-resolution hash coding algorithm according to any one of claims 1 to 9, characterized in that: The method described is as follows: According to the camera parameters and image parameters of the input image to be rendered, the light direction of the sampling point and the distance between the sampling points are calculated; The required features are calculated based on the coordinates of the calculated sampling points, and the tables of different frequencies in the memory are read based on the multi-resolution hash coding table lookup method to obtain the corresponding features; The obtained features are interpolated. The memory is pre-stored with tables of different resolutions; the tables of different resolutions correspondingly store features of different frequencies of feature points; The color and opacity of the sampling point are calculated based on the light direction of the sampling point and the features after interpolation processing; The rendering result is calculated based on the distance between the obtained sampling points, the color of the sampling points, and the opacity information.

Images