Three-dimensional scene neural rendering device, method and chip

Abstract

The invention relates to a three-dimensional scene neural rendering device, method and chip, the device is used for rendering a to-be-rendered target image according to a preset scene grid structure, and the device comprises a control module, a light sampling module, a neural field representation module and a voxel rendering module; the method comprises the following steps: compressing a scene grid structure in a light sampling module to obtain a compact scene boundary, and outputting a first termination signal to terminate rendering operation of a target pixel in advance when a target light does not have an intersection point with the compact scene boundary; when the sampling position exceeds the interval where the target light and the boundary of the compact scene intersect, a second termination signal is output, and the number of iterations of the light sampling module is reduced; meanwhile, in the calculation process of the voxel rendering module, when the accumulated weight is smaller than a preset weight threshold value, a third termination signal is output, rendering on the coordinates of the target pixel point is terminated in advance, the current rendering color is output, and the time needed by rendering is effectively shortened; the device is high in efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of computer graphics and vision, and in particular to a three-dimensional scene neural rendering device, method and chip. Background Art

[0002] In the field of computer graphics and vision, new perspective synthesis has always been a hot research topic. Its main goal is to generate corresponding pictures from any new perspective from a limited set of observed images. In the fields of 3D scene reconstruction and AR (Augmented Reality) / VR (Virtual Reality), new perspective synthesis has a wide range of applications. The key issue is how to use the information of these input images to express the 3D scene.

[0003] In related technologies, a 3D reconstruction method based on Neural Radiance Fields (NeRF) is used to generate realistic 3D scenes using a small number of 2D images, and images can be generated in any position and direction. This greatly simplifies the process of synthesizing new perspectives and also reduces the cost of data collection and processing.

[0004] However, the efficiency of the device for implementing the NeRF-based three-dimensional rendering technology in the related art is low. Summary of the invention

[0005] Based on this, it is necessary to provide a more efficient three-dimensional scene neural rendering device, method and chip to address the above technical problems.

[0006] In a first aspect, the present application provides a three-dimensional scene neural rendering device. The device is used to render a target image to be rendered according to a preset scene grid structure, and the device includes: A control module, used for receiving pixel point coordinates of each pixel in the target image, and for transmitting the target pixel point coordinates to the light sampling module under the control of the start signal, and for ending the rendering process of the target pixel point coordinates according to the first termination signal, the second termination signal or the third termination signal; a light sampling module, for determining a target light corresponding to the target pixel coordinates according to the target pixel coordinates and camera parameters, for sampling the target light according to a compact scene boundary to determine the sampling point coordinates corresponding to the target pixel coordinates, for outputting a first termination signal when there is no intersection between the target light and the compact scene boundary, and for outputting a second termination signal when the sampling position exceeds an interval where the target light intersects with the compact scene boundary, wherein the compact scene boundary is obtained by compressing a scene grid structure; A neural field representation module is used to input the features calculated based on the position encoding of the sampling point coordinates into a preset neural network model to obtain the color value and density value corresponding to the sampling point coordinates; The voxel rendering module is used to perform stereoscopic rendering operation on the color value and density value corresponding to the sampling point coordinates to obtain the rendering color corresponding to the target pixel point coordinates, and to output a third termination signal when the accumulated weight is less than a preset weight threshold during the stereoscopic rendering operation.

[0007] In one embodiment, the light sampling module includes: A light generating unit is used to transform the coordinates of the target pixel point into world coordinates according to the camera parameters to obtain the target light; A memory unit, used to store a scene grid structure, and to output a sampling parameter corresponding to the address information according to the received address information, wherein if the position indicated by the address information exists in the scene grid structure, the sampling parameter is a first parameter, and if the position indicated by the address information does not exist in the scene grid structure, the sampling parameter is a second parameter; A grid compression unit is used to compress the scene grid structure to obtain a compact scene boundary, and output the scene boundary to the intersection calculation unit; an intersection calculation unit, configured to perform intersection calculation processing on the target ray and the compact scene boundary, determine a first intersection point and a second intersection point between the target ray and the compact scene boundary, and output a first termination signal when a result of the intersection calculation processing indicates that there is no intersection point between the target ray and the compact scene boundary; The sampling unit is used to control the sampling position to iteratively advance from the first intersection point along the target light ray to the second intersection point according to a preset step length to obtain the sampling point coordinates, and is used to transmit the current sampling position as address information to the memory unit in each iteration process, receive the sampling parameters fed back by the memory unit, determine whether to use the current sampling position as the sampling point coordinates according to the sampling parameters, and output a second termination signal when the current sampling position exceeds the interval where the target light ray intersects with the boundary of the compact scene.

[0008] In one of the embodiments, the grid compression unit is specifically used to crop the positions corresponding to invalid voxels in the scene grid structure to obtain a compact scene boundary, where the invalid voxels are voxels that do not have data in the corresponding positions in the scene grid structure.

[0009] In one of the embodiments, the grid compression unit comprises: A first AND gate tree structure is used to compress the scene grid structure from a first dimension to obtain a first grid structure; The second AND gate tree structure is used to compress the first grid structure from a second dimension to obtain a second network structure; The third AND gate tree structure is used to compress the second grid structure from a third dimension to obtain a compact scene boundary.

[0010] In one embodiment, the neural field representation module includes: A position encoding unit is used to receive the sampling point coordinates and perform position encoding calculation on the sampling point coordinates to obtain encoding features corresponding to the sampling point coordinates; The neural network unit is used to perform calculations based on the coding features and the preset neural network model to obtain the color value and density value corresponding to the sampling point coordinates.

[0011] In one of the embodiments, the position encoding unit is specifically configured to perform multi-resolution hash encoding calculation on the sampling point coordinates based on a preset hash table to obtain encoding features corresponding to the sampling point coordinates.

[0012] In one of the embodiments, the size of the hash table is preset to be a power of 2.

[0013] In one embodiment, the voxel rendering module is used to perform stereoscopic rendering operation processing according to the following formula:

[0014] in, r represents the target light, C(r) Indicates the rendering color corresponding to the target light. N is the number of sampling point coordinates, T i Indicates i The cumulative weight corresponding to the coordinates of the sampling points is T i+1 Indicates i+ The cumulative weight corresponding to the coordinates of a sampling point, σ i For the i The opacity corresponding to the sampling point coordinates, δ is the preset step size, c i For the i The color value corresponding to the coordinates of the sampling points; Among them, the voxel rendering module is used in the current T i When it is less than a preset weight threshold, a third termination signal is output.

[0015] In a second aspect, the present application also provides a three-dimensional scene neural rendering method. The method is used to render a target image to be rendered according to a preset scene grid structure, comprising: Determine the target light corresponding to the target pixel coordinates according to the target pixel coordinates and camera parameters; The target light is sampled according to the compact scene boundary to determine the sampling point coordinates corresponding to the target pixel point coordinates. The compact scene boundary is obtained by compressing the scene grid structure. Input the features calculated based on the position encoding of the sampling point coordinates into a preset neural network model to obtain the color value and density value corresponding to the sampling point coordinates; The color value and density value corresponding to the sampling point coordinates are processed by stereo rendering operation to obtain the rendering color corresponding to the target pixel point coordinates; Among them, when there is no intersection between the target light and the compact scene boundary, or when the sampling position exceeds the interval where the target light intersects the compact scene boundary, or when the cumulative weight is less than the preset weight threshold during the stereoscopic rendering operation, the rendering process of the target pixel point is terminated.

[0016] In a third aspect, the present application further provides a chip, wherein the chip includes the three-dimensional scene neural rendering device provided in the first aspect.

[0017] The above-mentioned three-dimensional scene neural rendering device, method and chip, the device is used to render the target image to be rendered according to a preset scene grid structure, including: a control module, a light sampling module, a neural field representation module and a voxel rendering module; wherein, a compact scene boundary is obtained by compressing the scene grid structure in the light sampling module, and a first termination signal is output when there is no intersection between the target light and the compact scene boundary to terminate the rendering operation of the target pixel in advance, and the time required for the camera light to reach the first occupied grid unit is reduced by providing a compact scene boundary and adopting an early termination strategy of the first termination signal; at the same time, when the sampling position exceeds the interval where the target light intersects the compact scene boundary, a second termination signal is output to reduce the number of iterations of the light sampling module; at the same time, during the calculation process of the voxel rendering module, when the cumulative weight is less than the preset weight threshold, a third termination signal is output to terminate the rendering of the target pixel point coordinates in advance, and the current rendering color is output, effectively reducing the time required for rendering; the three-dimensional scene neural rendering device provided by the present application is highly efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a structural block diagram of a three-dimensional scene neural rendering device in one embodiment; Figure 2 A schematic diagram of the timing sequence of control signals of a three-dimensional scene neural rendering device in one embodiment; Figure 3 It is a structural block diagram of a three-dimensional scene neural rendering device in another embodiment. Implementation

[0019] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0020] In one embodiment, Figure 1 As shown, a three-dimensional scene neural rendering device is provided, which is used to render a target image to be rendered according to a preset scene grid structure. The device includes: a control module, a light sampling module, a neural scene representation module and a voxel rendering module.

[0021] A control module is used to receive the pixel point coordinates of each pixel in the target image, and to transmit the target pixel point coordinates to the light sampling module under the control of the start signal, and to end the rendering process of the target pixel point coordinates according to the first termination signal, the second termination signal or the third termination signal.

[0022] Among them, the three-dimensional scene neural rendering device performs three-dimensional scene rendering on each pixel in the target image, the target pixel point coordinates are the pixel point coordinates corresponding to the target pixel, and the target pixel is any pixel in the target image.

[0023] In a possible implementation, when the start signal line corresponding to the start signal is set to a high level, the control module sends the coordinates of the target pixel point to be rendered to the light sampling module to start the light sampling module to calculate the sampling point coordinates; exemplarily, the signal line corresponding to the start signal is set to a high level, which can be achieved by setting the start signal to "1" in the hardware description language VerilogHDL by blocking assignment.

[0024] The control module ends the rendering process of the target pixel point coordinates according to the first termination signal, the second termination signal and the third termination signal. At this time, the voxel rendering module outputs the rendering color for the target pixel point coordinates.

[0025] In a possible implementation, an output signal is set in the control module, and the output signal can be obtained according to the "OR" result of the first termination signal, the second termination signal and the third termination signal. Exemplarily, the above three termination signals all correspond to the signal value "1", and the process of outputting the above three termination signals is to set the corresponding signal line to a high level; when any one of the above three termination signals is output, the output signal determined according to the "OR" result of the above three termination signals is assigned a value of "1", that is, it is set to a high level, and the rendering process of the target pixel coordinates is ended.

[0026] In a possible implementation, a three-dimensional scene neural rendering device can determine whether to start rendering a target image through a start signal; if the start signal is set to a high level, the device starts rendering the target image, sets the start signal corresponding to the first clock cycle to a high level, and sets the start signals corresponding to other clock cycles to a low level, and at the same time passes the pixel point coordinates of the first pixel of the target image to the light sampling module. In the next clock cycle when the output signal is set to a high level, the start signal is set to a high level to perform the rendering process of the next pixel point coordinates.

[0027] In a possible implementation, the three-dimensional scene neural rendering device can determine whether to end the rendering process of the target image through an end signal. In a possible implementation, if the end signal is set to a high level, the device ends the rendering process of the target image. Exemplarily, when the pixel point coordinate rendering of the last pixel in the target image is completed, the end signal is set to a high level.

[0028] A light sampling module is used to determine the target light corresponding to the target pixel coordinates according to the target pixel coordinates and camera parameters, and to sample the target light according to a compact scene boundary to determine the sampling point coordinates corresponding to the target pixel coordinates, and to output the first termination signal when there is no intersection between the target light and the compact scene boundary, and to output the second termination signal when the sampling position exceeds the interval where the target light intersects with the compact scene boundary, wherein the compact scene boundary is obtained by compressing the scene grid structure.

[0029] The camera parameters include the transformation matrix W from the image coordinate system to the world coordinate system and the origin O of the observation point.

[0030] The process of the light sampling module for determining the target light corresponding to the target pixel coordinates according to the target pixel coordinates and the camera parameters mainly involves matrix multiplication operations; among them, the target pixel coordinates in the image coordinate system are converted to the light direction of the target light in the world coordinate system using the transformation matrix W in the camera parameters, and the position origin O in the camera parameters is used as the origin of the target light.

[0031] The light sampling module is used to sample the target light according to the compact scene boundary to determine the sampling point coordinates corresponding to the target pixel coordinates. The process mainly includes two parts. One part is to compress the scene grid structure to obtain a compact scene boundary, perform intersection calculation processing on the target light and the compact scene boundary, and determine the intersection information between the target light and the compact scene boundary. When the result of the intersection calculation processing indicates that the target light and the compact scene boundary have no intersection, it is considered that the pixel corresponding to the target light has no color and no subsequent rendering processing is required. At this time, a first termination signal is output to end the rendering process of the target pixel coordinates. Exemplarily, outputting the first termination signal can be achieved by setting the first signal line corresponding to the first termination signal to a high level. Exemplarily, if there is an intersection between the target light and the compact scene boundary, the first signal line remains at a low level.

[0032] The other part is to sample the target light based on the intersection information to determine the sampling point coordinates corresponding to the target pixel coordinates, wherein during the sampling process, if the sampling position exceeds the interval where the target light intersects with the compact scene boundary, a second termination signal is output to end the rendering process of the target pixel coordinates. Exemplarily, the output of the second termination signal can be achieved by setting the second signal line corresponding to the second termination signal from a low level to a high level. Exemplarily, if the sampling position is within the interval where the target light intersects with the compact scene boundary, the second signal line remains at a low level.

[0033] In terms of timing, the output time of the first termination signal is before the second termination signal. If the light sampling module outputs the first termination signal, the current rendering process is terminated in advance, and subsequent processes such as sampling point coordinate calculation are not performed.

[0034] The number of sampling point coordinates corresponding to the target pixel point coordinates may be multiple, 1, or 0.

[0035] The neural field representation module is used to input the features obtained based on the position encoding calculation of the sampling point coordinates into a preset neural network model to obtain the color value and density value corresponding to the sampling point coordinates.

[0036] Among them, in the field of computer vision, the neural field is the process of using spatial coordinates or other dimensions (time, camera pose, etc.) as input, simulating the target function through a neural network model, and generating a target scalar (color, depth, etc.).

[0037] The density value is the density value of the voxel color corresponding to the sampling point coordinates; illustratively, the density value may be opacity or transparency, which may be selected according to a specific implementation method.

[0038] The voxel rendering module is used to perform stereoscopic rendering operation on the color value and density value corresponding to the sampling point coordinates to obtain the rendering color corresponding to the target pixel point coordinates, and to set the third termination signal to a high level when the accumulated weight is less than the weight threshold during the stereoscopic rendering operation.

[0039] Among them, the processing process of voxel rendering is the process of accumulating the color values ​​and density values ​​corresponding to the coordinates of all sampling points corresponding to the target pixel point coordinates. In the embodiment of the present application, a preset weight threshold is set in the voxel rendering module. During the stereo rendering operation, when the cumulative weight is less than the preset weight threshold, it means that the target light will no longer continue to participate in the rendering calculation. At this time, the voxel rendering module outputs a third termination signal. Exemplarily, the output of the third termination signal can be achieved by setting the third signal line corresponding to the third termination signal to a high level. If the cumulative weight is greater than or equal to the preset weight threshold, the third signal line corresponding to the third termination signal remains at a low level.

[0040] Please refer to Figure 2 , is an exemplary timing diagram of the control signal of the three-dimensional scene neural rendering device provided in this embodiment. Figure 2 As shown, when the start signal line corresponding to the start signal is set to a high level, the rendering of the target image begins, and then the start signal line corresponding to the start signal is set to a high level, and the rendering calculation process for the first pixel begins; when the first signal line corresponding to the first end signal is set to a high level, the output signal line corresponding to the output signal is also set to a high level, and the rendering process of the first pixel ends at this time; in the next clock cycle when the output signal line corresponding to the output signal is set to a high level, the start signal line corresponding to the start signal is set to a high level, and the rendering calculation process of the second pixel begins. In the next cycle when the start signal line is set to a high level, the signal lines corresponding to the three end signals are all set to a low level, and the corresponding output signal lines are also set to a low level. And so on, until the rendering of the last pixel is completed, the signal line corresponding to the end signal is set to a high level, and the rendering of the target image ends.

[0041] The three-dimensional scene neural rendering device provided by the above embodiment obtains a compact scene boundary by compressing the scene grid structure in the light sampling module, and outputs a first termination signal to terminate the rendering operation of the target pixel in advance when the target light has no intersection with the compact scene boundary. By providing a compact scene boundary and adopting an early termination strategy of the first termination signal, the time required for the camera light to reach the first occupied grid unit is reduced; at the same time, when the sampling position exceeds the interval where the target light intersects the compact scene boundary, a second termination signal is output to reduce the number of iterations of the light sampling module; at the same time, during the calculation process of the voxel rendering module, when the cumulative weight is less than the preset weight threshold, a third termination signal is output to terminate the rendering of the target pixel point coordinates in advance, and the current rendering color is output, effectively reducing the time required for rendering; the three-dimensional scene neural rendering device provided by the present application is highly efficient.

[0042] Furthermore, the above-mentioned three-dimensional scene neural rendering device only needs to input camera parameters and the pixel coordinates corresponding to each pixel of the target image to obtain the three-dimensional rendering result corresponding to the preset scene grid structure. There is no need for CPU / GPU to participate in the calculation throughout the process, thereby achieving low power consumption and high energy efficiency.

[0043] In one embodiment, please refer to Figure 3 ,The ray sampling module includes a ray generation unit, an intersection calculation unit, a memory unit and a sampling unit.

[0044] The light generating unit is used to transform the coordinates of the target pixel point into world coordinates according to the camera parameters to obtain the target light.

[0045] A memory unit is used to store a scene grid structure and to output a sampling parameter corresponding to the address information according to the received address information, wherein if the position indicated by the address information exists in the scene grid structure, the sampling parameter is a first parameter; if the position indicated by the address information does not exist in the scene grid structure, the sampling parameter is a second parameter.

[0046] The memory unit may be an on-chip memory. The memory unit stores the three-dimensional scene to be rendered. In the present application, the rendered three-dimensional scene is divided into small voxels to obtain a scene grid structure, wherein each voxel in the scene grid structure records whether there is content at the corresponding position.

[0047] The grid compression unit is used to compress the scene grid structure to obtain a compact scene boundary, and output the compact scene boundary to the intersection calculation unit.

[0048] Among them, the grid compression unit is responsible for receiving the scene grid structure in the memory unit and compressing the scene grid structure to compress the grid structure that effectively represents the three-dimensional scene to the minimum to obtain a compact scene boundary.

[0049] In a possible implementation, the grid compression unit is specifically used to crop positions corresponding to invalid voxels in the scene grid structure to obtain a compact scene boundary, where invalid voxels are voxels with no data in corresponding positions in the scene grid structure.

[0050] Optionally, the grid compression unit includes a first AND gate tree structure, a second AND gate tree structure and a third AND gate tree structure, which compress the scene grid interface layer by layer from three dimensions of the three-dimensional space. Specifically, the first AND gate tree structure is used to compress the scene grid structure from the first dimension to obtain a first grid structure; the second AND gate tree structure is used to compress the first grid structure from the second dimension to obtain a second network structure; the third AND gate tree structure is used to compress the second grid structure from the third dimension to obtain a compact scene boundary.

[0051] In another possible implementation, the mesh compression unit may also use other compression methods to compress the scene mesh structure. For example, the compression process is achieved by reducing the number of vertices in the mesh structure.

[0052] An intersection calculation unit is used to perform intersection calculation processing on a target light ray and a compact scene boundary, determine a first intersection point and a second intersection point between the target light ray and the compact scene boundary, and output a first termination signal when a result of the intersection calculation processing indicates that there is no intersection point between the target light ray and the compact scene boundary.

[0053] Exemplarily, the first intersection point is used to refer to the near-end intersection point that is closer to the origin O of the target light, and the second intersection point is used to refer to the far-end intersection point that is farther from the origin O of the target light. It should be noted that in other examples, the first intersection point may also refer to the far-end intersection point, and the second intersection point may also refer to the near-end intersection point. For ease of description, in this embodiment, the first intersection point is taken as the near-end intersection point and the second intersection point is taken as the far-end intersection point.

[0054] In a possible implementation, the process of determining the first intersection and the second intersection by the intersection calculation unit includes: taking the compact scene boundary as a bounding box, and calculating the distances between the origin O of the target light and the two intersections of the bounding box based on the axis aligned bounding box (AABB) algorithm: the distance from the near-end intersection to the origin O (which may be referred to as the near-end intersection distance T1 in this embodiment) and the distance from the far-end intersection to the origin O (which may be referred to as the far-end intersection distance T2 in this embodiment). If there is no intersection between the target light and the compact scene boundary, that is, the near-end intersection distance T1 is greater than the far-end intersection distance T2, it is considered that the pixel coordinates corresponding to the target light have no color, and no subsequent rendering processing is required. At this time, a first termination signal is output.

[0055] The sampling unit is used to control the sampling position to iteratively advance from the first intersection point along the target light ray to the second intersection point according to a preset step length to obtain the sampling point coordinates, and is used to transmit the current sampling position as address information to the memory unit in each iteration process, receive the sampling parameters fed back by the memory unit, determine whether to use the current sampling position as the sampling point coordinates according to the sampling parameters, and output a second termination signal when the current sampling position exceeds the interval where the target light ray intersects with the boundary of the compact scene.

[0056] In each iteration process, if the sampling parameter fed back by the memory unit is the first sampling parameter, the current sampling position is output as the sampling point coordinate; if the sampling parameter fed back by the memory unit is the second sampling parameter, it is considered that the current sampling position does not exist in the scene grid structure, and the sampling position is not output.

[0057] In a possible implementation, the intersection calculation unit outputs the proximal intersection distance T1 and the distal intersection distance T2, and the sampling unit takes the proximal intersection as the first sampling position, that is, the input of the first iterative process is the proximal intersection distance T1, and obtains the coordinate data of the proximal intersection as address information and transmits it to the memory unit, and receives the sampling parameters fed back by the memory unit; at the same time, according to the preset step size, obtains the distance between the second sampling position and the origin O as the input of the next iterative process; until the distance between the sampling position and the origin O is greater than the distal intersection distance T2, a second termination signal is output to end the iterative process.

[0058] In one embodiment, Figure 3 As shown, the neural field representation module includes: a position encoding unit and a neural network unit.

[0059] The position encoding unit is used to receive the sampling point coordinates and perform position encoding calculation on the sampling point coordinates to obtain the encoding features corresponding to the sampling point coordinates.

[0060] In a possible implementation, the position encoding unit is specifically configured to perform multi-resolution hash encoding calculation on the sampling point coordinates based on a preset hash table to obtain encoding features corresponding to the sampling point coordinates.

[0061] Among them, the multi-resolution hash coding method is used to encode the sampling point coordinates, which can improve the representation ability of the candidate neural network and the effect of the overall neural field representation module.

[0062] Optionally, the size of the hash table is preset to be a power of 2. By adjusting the size of the hash table to a power of 2, the speed of the modulo operation in the position encoding process is accelerated.

[0063] In another possible implementation, the position encoding unit is specifically configured to encode and calculate the sampling point coordinates based on a frequency encoding method, a parameter encoding method, or a sparse parameter encoding method to obtain encoding features corresponding to the sampling point coordinates.

[0064] The neural network unit is used to perform calculations based on the coding features and the preset neural network model to obtain the color value and density value corresponding to the sampling point coordinates.

[0065] In a possible implementation, the preset neural network model adopts a three-layer multi-level perceptron structure. In a possible implementation, the preset neural network model adopts a convolutional neural network model structure.

[0066] In one embodiment, the voxel rendering module in the provided three-dimensional scene neural rendering device is used to perform stereoscopic rendering operation processing according to the following formula:

[0067] in, r represents the target light, C(r) Indicates the rendering color corresponding to the target light. N is the number of sampling point coordinates, T i Indicates i The cumulative weight corresponding to the coordinates of the sampling points is T i+1 Indicates i+ The cumulative weight corresponding to the coordinates of a sampling point, σ i For the i The opacity corresponding to the sampling point coordinates, δ is the preset step size, c i For the i The color value corresponding to the coordinates of the sampling points; that is, in this embodiment, the density value specifically adopts opacity.

[0068] Among them, the voxel rendering module is used in the current Ti When it is less than a preset weight threshold, a third termination signal is output.

[0069] For example, in order to verify the beneficial effects of the three-dimensional rendering device provided by the present application, SystemVerilog is used as the hardware description language to Figure 3 The device shown in the figure is used for implementation; taking the rendering of an 800*800 image in a Lego scene as an example, the device is first deployed on the FPGA (Field Programmable Gate Array) of Alveo-U280, and the scene grid structure information representing the three-dimensional scene to be rendered obtained by training is transmitted to the on-chip memory (i.e., memory unit) through the data bus in advance. Under this deployment condition, the camera parameters to be observed and the image parameters of the target image are input, and the Lego image corresponding to the angle is returned through the processing of this device. Compared with the results of the software test, the peak signal-to-noise ratio (PSNR) is not much different.

[0070] For example, the entire computing architecture is synthesized into ASIC (Application Specific Integrated Circuit) using the TSMC 28nm COMS process library and compared with Instant-NGP (Instant neuralgraphics primitives). Table 1 shows the comparative experimental data of the two.

[0071] Table 1: Comparative experimental data of the device provided in this application and Instant-NGP

[0072] Generally, the real-time frame rate requirement in AR / VR technology is 25FPS. It can be seen from the experimental data shown in Table 1 that the device provided by this application can process a photo with a pixel size of 800*800 at a speed of 72 frames per second (72 FPS), which is sufficient to meet the real-time requirements. Furthermore, the power consumption of the device provided by this application is 3.57W, and the energy consumption is 0.049J per frame, achieving low power consumption and high energy efficiency, which has a huge application potential in some devices with limited resources and requirements for delay.

[0073] At the same time, under the synthetic field data set, the rendering time of the device provided in this application is 22ms on average when the grid compression unit is not enabled, and the rendering time is 14ms on average after the grid compression unit is enabled, achieving a 1.6-fold acceleration, shortening the rendering time and improving rendering efficiency.

[0074] In one embodiment, a three-dimensional scene rendering method is provided, the method being used to render a target image to be rendered according to a preset scene grid structure, comprising: Step 101: Determine the target light corresponding to the target pixel coordinates according to the target pixel coordinates and camera parameters.

[0075] Step 102 , sampling the target light rays according to the compact scene boundary to determine the sampling point coordinates corresponding to the target pixel point coordinates. The compact scene boundary is obtained by compressing the scene grid structure.

[0076] Step 103, inputting the features obtained by calculating the position encoding of the sampling point coordinates into a preset neural network model to obtain the color value and density value corresponding to the sampling point coordinates.

[0077] Step 104 , performing a three-dimensional rendering operation on the color value and density value corresponding to the sampling point coordinates to obtain a rendering color corresponding to the target pixel point coordinates.

[0078] Among them, during the execution of the above steps 102 to 104, when there is no intersection between the target light and the compact scene boundary, or when the sampling position exceeds the interval where the target light intersects the compact scene boundary, or when the cumulative weight is less than the preset weight threshold during the stereoscopic rendering operation, the rendering process of the target pixel point is terminated.

[0079] The execution processes of step 102, step 103 and step 104 may have overlapping parts in terms of timing; in a possible implementation manner, the number of sampling point coordinates corresponding to the target pixel point coordinates is multiple, and when the color value and density value corresponding to the first sampling point coordinate are output to step 104, step 103 continues to calculate the color value and density value of the second sampling point coordinate, and at the same time, step 102 continues to calculate the third sampling point coordinate.

[0080] In one embodiment, a chip is provided, which includes the three-dimensional scene neural rendering device provided in the above embodiment.

[0081] It can be understood by those skilled in the art that the devices, modules, units and chips provided in the above embodiments can be designed using hardware description languages ​​such as SystemVerilog, VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) or Verilog HDL, and implemented in hardware in FPGA or ASIC.

[0082] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A three-dimensional scene neural rendering device, characterized in that: The device is used to render a target image to be rendered according to a preset scene grid structure, and the device comprises: A control module, used for receiving the pixel point coordinates of each pixel in the target image, and for transmitting the target pixel point coordinates to the light sampling module under the control of the start signal, and for ending the rendering process of the target pixel point coordinates according to the first termination signal, the second termination signal or the third termination signal; The light sampling module is used to determine the target light corresponding to the target pixel coordinates according to the target pixel coordinates and camera parameters, and to sample the target light according to a compact scene boundary to determine the sampling point coordinates corresponding to the target pixel coordinates, and to output the first termination signal when there is no intersection between the target light and the compact scene boundary, and to output the second termination signal when the sampling position exceeds the interval where the target light intersects with the compact scene boundary, wherein the compact scene boundary is obtained by compressing the scene grid structure; A neural field representation module, used to input the features obtained by position encoding calculation of the sampling point coordinates into a preset neural network model to obtain the color value and density value corresponding to the sampling point coordinates; The voxel rendering module is used to perform stereoscopic rendering operation on the color value and density value corresponding to the sampling point coordinates to obtain the rendering color corresponding to the target pixel point coordinates, and to output the third termination signal when the accumulated weight is less than a preset weight threshold during the stereoscopic rendering operation.

2. The device according to claim 1, characterized in that The light sampling module comprises: A light generating unit, used for performing a world coordinate transformation process on the target pixel point coordinates according to the camera parameters to obtain the target light; a memory unit, used to store the scene grid structure, and to output a sampling parameter corresponding to the address information according to the received address information, wherein if the position indicated by the address information exists in the scene grid structure, the sampling parameter is a first parameter, and if the position indicated by the address information does not exist in the scene grid structure, the sampling parameter is a second parameter; A grid compression unit, configured to compress the scene grid structure to obtain the compact scene boundary, and output the compact scene boundary to the intersection calculation unit; an intersection calculation unit, configured to perform intersection calculation processing on the target light ray and the compact scene boundary, determine a first intersection point and a second intersection point between the target light ray and the compact scene boundary, and output the first termination signal when a result of the intersection calculation processing indicates that there is no intersection point between the target light ray and the compact scene boundary; A sampling unit is used to control the sampling position to iteratively advance from the first intersection point along the target light ray to the second intersection point according to a preset step length to obtain the sampling point coordinates, and is used to transmit the current sampling position as the address information to the memory unit in each iteration process, receive the sampling parameters fed back by the memory unit, determine whether to use the current sampling position as the sampling point coordinates according to the sampling parameters, and output the second termination signal when the current sampling position exceeds the interval where the target light ray intersects the boundary of the compact scene.

3. The device according to claim 2, characterized in that The grid compression unit is specifically used to crop the positions corresponding to invalid voxels in the scene grid structure to obtain the compact scene boundary, and the invalid voxels are voxels in which no data exists at the corresponding positions in the scene grid structure.

4. The device according to claim 3, characterized in that The grid compression unit comprises: A first AND gate tree structure is used to compress the scene grid structure from a first dimension to obtain a first grid structure; A second AND gate tree structure is used to compress the first grid structure from a second dimension to obtain a second network structure; The third AND gate tree structure is used to compress the second grid structure from a third dimension to obtain the compact scene boundary.

5. The device according to claim 1, characterized in that The neural field representation module comprises: A position coding unit, used to receive the sampling point coordinates, and perform position coding calculation on the sampling point coordinates to obtain coding features corresponding to the sampling point coordinates; A neural network unit is used to perform calculations based on the coding features and the preset neural network model to obtain the color value and density value corresponding to the sampling point coordinates.

6. The device according to claim 5, characterized in that The position encoding unit is specifically used to perform multi-resolution hash encoding calculation on the sampling point coordinates based on a preset hash table to obtain encoding features corresponding to the sampling point coordinates.

7. The device according to claim 6, characterized in that The size of the preset hash table is a power of 2.

8. The device according to claim 2, characterized in that The voxel rendering module is used to perform stereo rendering operation processing according to the following formula: ， in, r represents the target ray, C(r) Indicates the rendering color corresponding to the target light, N is the number of sampling point coordinates, T i Indicates i The cumulative weight corresponding to the coordinates of the sampling points, T i+1 Indicates i+ The cumulative weight corresponding to the coordinates of 1 sampling point is i The opacity corresponding to the sampling point coordinates is the preset step size, c i For the i The color value corresponding to the coordinates of the sampling points; Wherein, the voxel rendering module is used for T i When it is less than the preset weight threshold, the third termination signal is output.

9. A three-dimensional scene neural rendering method, characterized in that: The method is used to render a target image to be rendered according to a preset scene grid structure, and includes: Determine the target light corresponding to the target pixel coordinates according to the target pixel coordinates and camera parameters; Sampling the target light according to a compact scene boundary to determine sampling point coordinates corresponding to the target pixel point coordinates, wherein the compact scene boundary is obtained by compressing the scene grid structure; Inputting the features obtained by calculating the position encoding of the sampling point coordinates into a preset neural network model to obtain the color value and density value corresponding to the sampling point coordinates; Performing a three-dimensional rendering operation on the color value and density value corresponding to the sampling point coordinates to obtain a rendering color corresponding to the target pixel point coordinates; Among them, when there is no intersection between the target light and the compact scene boundary, or when the sampling position exceeds the interval where the target light intersects the compact scene boundary, or when the accumulated weight is less than a preset weight threshold during the stereoscopic rendering operation, the rendering process of the target pixel point is terminated.

10. A chip, characterized in that: The chip comprises the device according to any one of claims 1-8.

Images