Method and apparatus for real-time rendering of neural radiance fields in a rasterization rendering pipeline

Abstract

The present invention discloses a method and apparatus for real-time rendering of a neural radiance field in a rasterization rendering pipeline, including the following steps: while the rasterization rendering pipeline performs rendering for each frame to obtain scene rendering data, screen space data for step sampling of the neural radiance field is dynamically acquired in units of neural primitives; in the rasterization rendering pipeline, real-time overlay rendering of the neural primitives is performed based on the scene rendering data and the screen space data, including occlusion processing and relighting processing of the neural primitives in the scene rendering data. In this way, the rendering of the neural radiance field is cleverly integrated into the rasterization rendering pipeline, achieving real-time rendering and ensuring rendering efficiency and rendering quality.

Description

Technical Field

[0001] The present invention belongs to the technical field of combining novel view synthesis and real-time rendering, and particularly relates to a method and device for real-time rendering of neural radiance fields in a rasterization rendering pipeline. Background Art

[0002] 3D reconstruction has always been an important task in the fields of computer vision and computer graphics, and there is a great demand for high-quality and high-fidelity 3D scene reconstruction in fields such as movies, games, and virtual reality. Traditional manual modeling methods require a large amount of manpower and time, and it is difficult to handle complex detailed features. Therefore, 3D reconstruction technologies based on real-shot images have received great attention and have achieved many breakthroughs with the rapid development of deep learning technologies in recent years.

[0003] Neural Radiance Field (NeRF) is a technology proposed in 2020. This technology proposes to use an optimized neural network to implicitly represent various information of a 3D scene, which can handle the task of novel view synthesis and render high-fidelity images containing complex details.

[0004] However, existing neural radiance fields are difficult to achieve real-time rendering. In this regard, Instant-NGP developed by Nvidia has made many optimizations based on neural radiance fields, and proposes to use multi-resolution hash encoding, enabling both training and inference to be carried out quickly and achieving real-time rendering.

[0005] A patent application with the publication number CN 117911633 A discloses a method for rendering neural radiance fields based on the Unreal Engine. The method is applied to a neural radiance field rendering framework, and the method includes: extracting key frames from the captured video of the scene to be reconstructed to obtain an image sequence; estimating the 3D scene structure and camera parameters based on the image sequence to obtain an estimation result; training the neural radiance field rendering framework according to the estimation result and the image sequence, and exporting rendering materials, where the rendering materials include Unreal Engine blueprint assets; sending the Unreal Engine blueprint assets into the Unreal Engine for automatic rendering to obtain a rendering result. In this technical solution, through the training of the neural radiance field rendering framework, the automatic export of rendering materials, and automatic real-time and high-quality rendering in the Unreal Engine, a high-precision 3D model can be obtained quickly.

[0006] The above technical solution imports the rendering materials of the neural radiance field into rasterization for sampling and drawing, but does not implement the specific details of combining with the rendering results of the original rasterized scene. Summary of the Invention

[0007] To solve the combination and application of neural radiance fields in the rasterization rendering pipeline, the object of the present invention is to provide a method and device for real-time rendering of neural radiance fields in the rasterization rendering pipeline, realizing the correct transformation, occlusion, and relighting of neural radiance fields.

[0008] To achieve the above object of the invention, an embodiment provides a method for real-time rendering of neural radiance fields in the rasterization rendering pipeline, including the following steps:

[0009] While the rasterization rendering pipeline renders each frame to obtain scene rendering data, screen space data of step sampling of the neural radiance field is dynamically obtained in units of neural primitives;

[0010] In the rasterization rendering pipeline, real-time overlay rendering of neural primitives is performed based on the scene rendering data and the screen space data, including occlusion processing and relighting processing of neural primitives in the scene rendering data.

[0011] Preferably, dynamically obtaining screen space data of the neural radiance field in units of neural primitives includes:

[0012] When the rasterization rendering pipeline renders each frame, a camera matrix of the neural radiance field is calculated according to the rasterized camera parameters, resolution, and affine transformation of the neural primitive;

[0013] When it is determined that the camera matrix and resolution of the neural radiance field of the current frame and the previous frame have changed, screen space data for rendering the neural primitive based on the camera matrix of the neural radiance field corresponding to the current frame is dynamically obtained.

[0014] Preferably, the method further includes: after it is determined that the camera parameters and resolution of the neural radiance field of the current frame and the previous frame have changed, frustum culling is also performed, specifically determining whether the bounding structure of the neural primitive of the previous frame is within the frustum of the current frame. If not, the screen space data of the previous frame is cleared, and the scene rendering data is directly drawn into the frame buffer. If so, screen space data of the neural radiance field of the current frame is dynamically obtained.

[0015] Preferably, the occlusion processing of the neural primitive in the scene rendering data includes:

[0016] Each neural primitive is configured with at least one bounding sphere with an adjustable radius to fit the volume of the neural radiance field. The depth buffer data included in the screen space data of each neural primitive is the closest intersection distance of ray marching, and a maximum distance among the closest intersection distances is recorded. This maximum distance is the distance of the background that does not need to be drawn;

[0017] First, determine whether the current nearest intersection distance of each pixel in the scene rendering data is less than the recorded maximum distance. If it is less, it means that there is a neural radiance field for the current neural primitive fragment, and the neural primitive needs to be drawn. Otherwise, directly output the current scene color;

[0018] Then, find the intersection of the projection ray corresponding to the nearest intersection distance of the current neural primitive fragment with the bounding sphere of the neural primitive to obtain the world space coordinate X, and restore the world space coordinate X` according to the scene depth included in the scene rendering data. Compare the distances from the world space coordinate X and the world space coordinate X` to the camera. If X is closer, the neural primitive needs to be drawn. Otherwise, directly output the scene color.

[0019] Preferably, the relighting process of the neural primitive in the scene rendering data includes:

[0020] Perform relighting on the neural primitive that needs to be drawn after occlusion processing. Specifically, perform configurable attenuation on the color buffer data included in the screen space data of the neural primitive, and then traverse all light sources in the rasterization rendering pipeline, and calculate the superimposed illumination on the attenuated color based on the normal buffer data included in the screen space data to achieve the relighting of the neural primitive.

[0021] Preferably, the method further includes: after dynamically obtaining the screen space data of the neural radiance field, filter the normal buffer data included in the screen space data, and then use the filtered normal buffer data for relighting.

[0022] Preferably, when performing relighting on the neural primitive, when there is environmental buffer data, calculate the ambient illumination based on the normal buffer data and superimpose the calculation result on the relighting result of the neural primitive.

[0023] Preferably, the screen space data of the step sampling of the dynamically obtained neural radiance field comes from Instant-NGP. Specifically, make modifications to Instant-NGP, including specifying cache memory for the color buffer data output by Instant-NGP, modifying the depth buffer data output by Instant-NGP to the nearest intersection distance of ray stepping, modifying the normal buffer data output by Instant-NGP and specifying the corresponding cache memory, encapsulating the output data based on the modifications as the screen space data of the neural primitive as an instance object, and also adding training switch control, camera synchronization, and load and save functions, and compiling them into a dynamic link library, and designing interfaces for each function at the same time;

[0024] The rasterization rendering pipeline is correspondingly encapsulated as a proxy object, which manages the pointers exposed by the instance object and provides a proxy for a series of interfaces.

[0025] To achieve the above-mentioned invention objective, the embodiment further provides a computing device, including a memory and one or more processors. An executable code is stored in the memory. When the one or more processors execute the executable code, it is used to implement the above method for real-time rendering of a neural radiance field in a rasterization rendering pipeline.

[0026] To achieve the above-mentioned invention objective, the embodiment further provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it implements the above method for real-time rendering of a neural radiance field in a rasterization rendering pipeline.

[0027] Compared with the prior art, the beneficial effects of the present invention at least include:

[0028] The present invention can integrate the rendering of the neural radiance field into the rasterization rendering pipeline. Specifically, the sampling process is left outside the rasterization rendering pipeline, that is, sampling is implemented in the neural radiance field to form screen space data, and the screen space data is combined with the original rasterization rendering scene of the rasterization rendering pipeline, so as to achieve a real-time rendering frame rate. At the same time, occlusion is considered during the combination process, and re-illumination based on experience of the drawn object is supported, ensuring the rendering efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0030] Figure 1 is a flowchart of the method for real-time rendering of a neural radiance field in a rasterization rendering pipeline provided by the embodiment;

[0031] Figure 2 is a specific flowchart of the method for real-time rendering of a neural radiance field in a rasterization rendering pipeline provided by the embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] To make the objectives, technical solutions and advantages of the present invention clearer, the following will further describe the present invention in detail with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the protection scope of the present invention.

[0033] As Figure 1 shown, a method for real-time rendering of a neural radiance field in a rasterization rendering pipeline provided by the embodiment includes the following steps:

[0034] S1, while the rasterization rendering pipeline renders each frame to obtain scene rendering data, it dynamically obtains screen space data of the neural radiance field step sampling in units of neural primitives.

[0035] In the embodiment, the rasterization rendering pipeline executes the rasterization rendering pipeline process for real-time rendering of each frame, completes the scene drawing, and obtains the scene rendering data of each frame, specifically including color buffer data, depth buffer data, and geometry buffer (G-Buffer). After completion, operations such as post-processing and output are not performed temporarily.

[0036] In the embodiment, while the rasterization rendering pipeline renders each frame, it also needs to dynamically obtain the screen space data of the neural radiance field to update each rendering frame, so as to superimpose the rendering result of the neural radiance field on the rendering frame of the rasterization rendering pipeline. In the update process of each rendering frame, if there is an update in the camera matrix or the transformation of the neural primitive, the drawing interface of the neural primitive will be called to perform data synchronization and the drawing of the neural radiance field, and the screen space data will be dynamically obtained. Specifically, when obtaining the data, the screen space data corresponding to the neural primitive in the neural radiance field drawing is obtained in units of neural primitives, and the screen space data is returned in the form of a frame data structure, specifically including color buffer data, depth buffer data, and normal buffer data in the screen space. The depth buffer data contains the nearest intersection distance of the ray step, and a maximum distance in the nearest intersection distance is recorded at the same time. The maximum distance is the distance of the background that does not need to be drawn.

[0037] For the update of the camera view transformation, first, when the rasterization rendering pipeline renders each frame, it calculates the camera matrix of the neural radiance field according to the rasterized camera parameters, resolution, and affine transformation of the neural primitive. Among them, the rasterized camera parameters include (position, orientation, field of view, etc.), and the resolution determines the field of view angle of the x-axis. Specifically, when calculating, first calculate the camera parameters of the neural radiance field using the rasterized camera parameters and resolution. Since the view information in the neural radiance field is expressed by the camera matrix composed of camera parameters, the inverse transformation of the affine transformation of the neural primitive in the pipeline is superimposed on the result of the camera matrix of the neural radiance field to achieve the effects of displacement, rotation, and translation.

[0038] Then, it is judged whether the camera matrix of the neural radiance field of the current frame and the previous frame has changed. When it has changed, the screen space data of the neural primitive sampled and drawn based on the camera matrix and resolution of the neural radiance field corresponding to the current frame is dynamically obtained.

[0039] In the embodiment, after it is determined that the camera parameters and resolution of the neural radiance field of the current frame and the previous frame have changed, frustum culling is also performed. Specifically, it is determined whether the bounding structure (which can be a bounding sphere) of the neural primitives of the previous frame is within the frustum of the current frame. If not, the screen space data of the previous frame is cleared, and the scene rendering data is directly drawn into the frame buffer. If so, the screen space data of the neural radiance field of the current frame is dynamically obtained.

[0040] In the embodiment, after the screen space data of the neural radiance field is dynamically obtained, the normal buffer data included in the screen space data is filtered and smoothed, and the filtered normal buffer data is used for relighting processing. The filtering algorithm used for specific filtering can be bilateral filtering, and its parameters and the number of iterations are both configured by the user in real time.

[0041] S2. Based on the scene rendering data and the screen space data, the neural primitives are overlaid and drawn in real time in the rasterization rendering pipeline, including occlusion handling and relighting processing of the neural primitives in the scene rendering data.

[0042] In the embodiment, based on the scene color buffer data and the scene depth buffer data of the rasterization rendering pipeline, the drawing of the neural primitives is realized based on the screen space data of the neural radiance field. The drawing here is a screen space operation based on the rendered scene data and the screen space data, specifically including occlusion handling and relighting processing.

[0043] In the occlusion handling, the culling or covering of screen space pixels is required. Each neural primitive is configured with a series of bounding spheres with adjustable radii to represent the volume of the fitted neural radiance field. Before shading, for the pixels with valid texture values in the screen space data of each frame, rays are projected, and after obtaining the intersection point coordinates by intersecting with the bounding sphere structure of the neural primitive, the intersection point coordinates are compared with the world space coordinates restored from the corresponding point scene depth information. According to the depth order, the current neural primitive fragment is discarded or the scene fragment is covered.

[0044] When specifically implemented, first, it is determined whether the current nearest intersection distance of each pixel in the scene rendering data is less than the recorded maximum distance. If so, it means that there is a neural radiance field for the current neural primitive fragment, and the neural primitive needs to be drawn. Otherwise, the current scene color is directly output. Then, the ray corresponding to the nearest intersection distance of the current neural primitive fragment intersects with the bounding sphere of the neural primitive to obtain the world space coordinate X, and the world space coordinate X` is restored according to the scene depth included in the scene rendering data. The distances from the world space coordinate X and the world space coordinate X` to the camera are compared. If X is closer, the neural primitive needs to be drawn. Otherwise, the scene color is directly output.

[0045] In the relighting process, all neural primitive fragments that have not been culled after occlusion processing are relighted and shaded. First, the color buffer data contained in the screen space data of the neural primitive is configurably attenuated, and then all light sources of the rasterization rendering pipeline are traversed, and based on the filtered normal buffer data contained in the screen space data, the superimposed lighting on the attenuated color is calculated, which includes the shadows transmitted by the receiving scene objects. When there is environmental buffer data, the ambient lighting calculation (IBL) is performed based on the filtered normal buffer data, and the calculation result is superimposed on the relighting processing result of the neural primitive. It should be noted that all relighting can have an adjustable variable to control the amplitude.

[0046] In specific implementation, when relighting the neural primitive, the base color color_base can directly sample the color buffer data in the screen space data. If light source lighting and ambient lighting need to be superimposed, follow the following formula:

[0047] color_final = color_base * decay_ratio+ direct_light * light_influence_ratio + env_light * env_influence_ratio;

[0048] Among them, the adjustable parameter decay_ratio represents the degree of light attenuation, which attenuates the basic color to a certain extent. direct_light represents the direct lighting of the rendering pipeline light source, and uses the previously obtained bounding sphere intersection position X and the normal N sampled from the corresponding coordinates of the normal texture to calculate the contribution of each light source, including the shadows of the rasterized scene. light_influence_ratio represents an adjustable weight. env_light represents ambient lighting, and also uses normal N to sample the radiance of the environment buffer data to calculate the image-based ambient lighting. env_influence_ratio also represents an adjustable weight.

[0049] After relighting, the color texture is output to the frame buffer, and the overall rendering of a frame of image combining rasterization rendering and neural radiation field rendering is completed.

[0050] The above method realizes the real-time rendering of neural radiation field in the rasterization pipeline. When the camera perspective does not change, the frame rate can be maintained at the original frame rate of the rasterization pipeline; when the camera perspective switches, the frame rate can reach 15-30 frames per second. In addition,

[0051] When the screen space data of the dynamically obtained neural radiance field contains geometric data, it can also be compatible with other screen space algorithms as needed, such as screen space ambient occlusion, screen space reflection, etc., which can be applied according to specific scenarios.

[0052] Based on the above method, the screen space data of the step sampling of the dynamically obtained neural radiance field can come from Instant-NGP. Modify Instant-NGP to support the above method of the present invention. The specific modifications include the call form and the resource acquisition interface, etc.

[0053] Specifically, for the resources required by the screen space data dynamically obtained for each frame, the following modifications need to be made to adapt to the above method of the present invention, including:

[0054] (1) For the color texture generated by Instant-NGP, when modifying, specify the cache memory for the color buffer data output by Instant-NGP;

[0055] (2) For the depth texture generated by Instant-NGP, modify the depth buffer data output by Instant-NGP to the nearest intersection distance of the ray step, and at the same time record the maximum distance in the nearest intersection distance;

[0056] (3) For the normal texture generated by Instant-NGP, although there is already calculation in the source code, it is only displayed in the debug mode and not output. When modifying, modify Instant-NGP to enforce the normal for each frame and output the normal buffer data, and at the same time specify the cache memory corresponding to the normal buffer data;

[0057] (4) Optionally, the alpha value representing transparency can be output on the a channel of the normal texture for other screen space algorithms.

[0058] In addition to the modifications to the output data recorded in (1)-(4) above, it is also necessary to cancel the visualization window, and add function operations such as training switch control, camera synchronization, loading and saving, etc. Corresponding interfaces are added respectively, which can be called by the pipeline rasterization rendering pipeline.

[0059] The above designs are all for a neural radiance field object. Such an object can be understood as the neural primitive mentioned above. The present invention encapsulates a neural radiance field at the Instant-NGP end as an instance object, and correspondingly encapsulates it as a proxy object at the rendering pipeline end. It manages the pointer exposed by the instance object and provides a proxy for a series of interfaces. The modified Instant-NGP is recompiled into a dynamic link library that only performs calculations and does not display. Through the designed interface, the creation, training, and drawing of neural primitives can be performed to obtain screen space data.

[0060] Among them, the input of the neural radiance field object can be of two types: (i) a series of images and a json file recording the pose information of the image camera, and Instant-NGP will perform real-time training and rendering on it. (ii) The.ingp file saved after training by Instant-NGP.

[0061] By configuring the file path, the rasterization rendering pipeline creates a proxy object and dynamically loads the corresponding screen space data (image or cache), that is, an instance corresponding to a neural radiance field, and performs subsequent control and drawing based on the instance. Based on the scenario where the instance has been created and initialized, as Figure 2 shown, combining Instant-NGP and the rasterization rendering pipeline, the method for real-time drawing of a neural radiance field in the rasterization rendering pipeline includes the following steps:

[0062] S01, the rasterization rendering pipeline of the renderer executes normally, and the color buffer and depth buffer of the scene are obtained.

[0063] S02, calculate the camera matrix in the neural radiance field space according to the affine transformation of the neural primitives and the rasterized camera parameters;

[0064] Step S03, analyze whether the camera matrix, resolution, and previous frame have changed. If there is no change, the previously obtained screen space data is used and directly enter S07 for drawing; if there is a change, it is necessary to enter S04 for the next judgment;

[0065] Step S04, perform frustum culling, and judge whether the neural primitives are in the changed frustum according to the bounding structure of the neural primitives. If not in the frustum, no drawing is performed, the content of the screen space data is cleared, and the color and depth of the scene are directly drawn to the frame buffer to end the frame.

[0066] Step S05, call the interface in the dynamic link library to update the camera parameters, and use Instant-NGP to perform a sampling of the neural radiance field once, and synchronize the sampling data to the frame data structure, that is, update the screen space data. Among them, when updating the depth buffer data, it is necessary to record the maximum depth in a frame, that is, the farthest distance of the light ray, for judging whether it intersects with the neural primitives.

[0067] Step S06, perform a filtering operation on the normal buffer data in the screen space data.

[0068] Step S07, use the bounding sphere structure of the neural primitives to perform a depth test by comparing with the depth buffer of the rasterization rendering pipeline to determine the neural primitives that need to be re-illuminated.

[0069] Determine the occlusion relationship between the neural primitive and the scene objects, and determine the final pixel color accordingly. Since all transformations in Instant-NGP are implemented by converting to camera transformations, the AABB bounding box of the neural primitive itself will not change with the affine transformation and needs to be maintained according to the information in the pipeline. On the other hand, the depth given by the Instant-NGP projection result is not the standard spatial depth of the device, and the output depth is discontinuous and at different scales, making it difficult to align directly for comparison. Therefore, this method proposes to use the nearest distance of ray step intersection instead of the standardized depth output, and at the same time use one or several bounding spheres with adjustable radius to fit the range of the neural radiation field.

[0070] Step S08: draw the neural primitives, and perform re-illumination by sampling the world space normal and light source parameters according to the neural radiation field.

[0071] Step S09: writing the rendering result into the buffer frame, and the rendering of this frame is completed.

[0072] Based on the same inventive concept, an embodiment further provides a computing device including a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, the device is used to implement the above-mentioned method of real-time rendering of a neural radiance field in a rasterization rendering pipeline. The method specifically includes the following steps:

[0073] S1, the rasterization rendering pipeline renders each frame to obtain scene rendering data, while dynamically obtaining the screen space data of the neural radiation field step sampling in units of neural primitives;

[0074] S2, real-time overlay rendering of neural primitives based on scene rendering data and screen space data in the rasterization rendering pipeline, including occlusion processing and relighting processing of neural primitives in scene rendering data.

[0075] The computing device provided in the embodiment, at the hardware level, includes not only a processor and memory, but also hardware required for other services such as an internal bus, a network interface, and memory. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the method of real-time drawing of the neural radiation field in the rasterization rendering pipeline described in S1-S2 above. Of course, in addition to software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0076] Based on the same inventive concept, the embodiment also provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the method for real-time rendering of a neural radiance field in a rasterization rendering pipeline is implemented, which specifically includes the following steps:

[0077] S1, while the rasterization rendering pipeline performs rendering for each frame to obtain scene rendering data, screen space data for step sampling of the neural radiance field is dynamically obtained in units of neural primitives;

[0078] S2, in the rasterization rendering pipeline, real-time overlay rendering of neural primitives is performed based on the scene rendering data and the screen space data, including occlusion processing and relighting processing of neural primitives in the scene rendering data.

[0079] In the embodiment, the computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data.

[0080] The above specific embodiments have detailed the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, supplements, equivalent replacements, etc. made within the scope of the principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for real-time rendering of neural radiance fields in a rasterization rendering pipeline, characterized in that Including the following steps: While the rasterization rendering pipeline renders each frame to obtain scene rendering data, screen space data of the neural radiance field step sampling is dynamically obtained in units of neural primitives; In the rasterization rendering pipeline, the neural primitives are rendered in real time and superimposed based on the scene rendering data and the screen space data. Here, the rendering is a screen space operation based on the rendered scene data and the screen space data, including occlusion processing and relighting processing of the neural primitives in the scene rendering data. Among them, in the occlusion processing, it is necessary to perform culling or covering of screen space pixels. In the relighting processing, all the neural primitive fragments that have not been culled after the occlusion processing are relit and shaded.

2. The method for real-time rendering of neural radiance fields in a rasterization rendering pipeline according to claim 1, wherein Dynamically obtaining screen space data of the neural radiance field in units of neural primitives includes: When the rasterization rendering pipeline renders each frame, it calculates the camera matrix of the neural radiance field according to the rasterized camera parameters, resolution, and the affine transformation of the neural primitives; When it is determined that the camera matrix of the neural radiance field of the current frame has changed compared with the previous frame or there is an update in the transformation of the neural primitives, screen space data of the neural primitives sampled and rendered based on the camera matrix and resolution of the neural radiance field corresponding to the current frame is dynamically obtained.

3. The method for real-time rendering of a neural radiance field in a rasterization rendering pipeline according to claim 2, wherein It also includes: When it is determined that the camera parameters and resolution of the neural radiance field of the current frame have changed compared with the previous frame, frustum culling is also performed. Specifically, it is determined whether the bounding structure of the neural primitives in the previous frame is within the frustum of the current frame. If not, the screen space data of the previous frame is cleared, and the scene rendering data is directly drawn into the frame buffer. If so, the screen space data of the neural radiance field of the current frame is dynamically obtained.

4. The method for real-time rendering of a neural radiance field in a rasterization rendering pipeline according to claim 1, wherein The occlusion processing of the neural primitives in the scene rendering data includes: Each neural primitive is configured with at least one bounding sphere with an adjustable radius to fit the volume of the neural radiance field. The depth buffer data included in the screen space data of each neural primitive is the nearest intersection distance of the ray step. At the same time, a maximum distance in the nearest intersection distances is recorded, and this maximum distance is the distance of the background that does not need to be drawn; First, it is determined whether the current nearest intersection distance of each pixel in the scene rendering data is less than the recorded maximum distance. If it is less, it means that there is a neural radiance field for the current neural primitive fragment, and the neural primitive needs to be drawn. Otherwise, the current scene color is directly output; Then, the intersection of the projection ray corresponding to the nearest intersection distance of the current neural primitive fragment and the bounding sphere of the neural primitive is calculated to obtain the world space coordinate X, and the world space coordinate X` is restored according to the scene depth included in the scene rendering data. The distances from the world space coordinate X and the world space coordinate X` to the camera are compared. If X is closer, the neural primitive needs to be drawn. Otherwise, the scene color is directly output.

5. The method for real-time rendering of a neural radiance field in a rasterization rendering pipeline according to claim 1, wherein The relighting processing of the neural primitives in the scene rendering data includes: The neural primitives that need to be drawn after the occlusion processing are relit. Specifically, the color buffer data included in the screen space data of the neural primitives is configured for attenuation, and then all the light sources in the rasterization rendering pipeline are traversed, and based on the normal buffer data included in the screen space data, the superimposed illumination on the attenuated color is calculated to achieve the relighting processing of the neural primitives.

6. The method for real-time rendering of a neural radiance field in a rasterization rendering pipeline according to claim 5, wherein, It also includes: After dynamically obtaining the screen space data of the neural radiance field, filter the normal buffer data included in the screen space data, and then use the filtered normal buffer data for relighting processing.

7. The method for real-time rendering of a neural radiance field in a rasterization rendering pipeline according to claim 5, wherein When performing relighting processing on neural primitives, when there is environment buffer data, calculate the ambient lighting based on the normal buffer data, and superimpose the calculation result on the relighting processing result of the neural primitives.

8. The method for real-time rendering of a neural radiance field in a rasterization rendering pipeline according to claim 1, characterized in that, The screen space data of the stepped sampling of the dynamically obtained neural radiance field comes from Instant-NGP. Specifically, modifications are made to Instant-NGP, including specifying cache memory for the color buffer data output by Instant-NGP, modifying the depth buffer data output by Instant-NGP to the nearest intersection distance of ray stepping, modifying the normal buffer data output by Instant-NGP and specifying the corresponding cache memory, encapsulating the output data based on the modifications as the screen space data of the neural primitives as an instance object, adding training switch control, camera synchronization, and load / save functions, compiling them into a dynamic link library, and designing interfaces for each function. The rasterization rendering pipeline is correspondingly encapsulated as a proxy object, which manages the pointers exposed by the instance object and provides a proxy for a series of interfaces.

9. A computing device, comprising a memory and one or more processors, wherein executable code is stored in the memory, characterized in that, When the one or more processors execute the executable code, it is used to implement the method of real-time rendering of the neural radiance field in the rasterization rendering pipeline according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A program is stored thereon, and when the program is executed by a processor, it implements the method of real-time rendering of the neural radiance field in the rasterization rendering pipeline according to any one of claims 1-7.

Images