Method and device for drawing neural radiation field in rasterization rendering pipeline in real time

Abstract

The invention discloses a method and device for drawing a neural radiation field in a rasterization rendering pipeline in real time, and the method comprises the following steps: carrying out the rendering of each frame through the rasterization rendering pipeline, obtaining the scene rendering data, and dynamically obtaining the stepping sampling screen space data of the neural radiation field through taking a neural primitive as a unit; real-time superposition drawing is carried out on the neural primitives based on the scene rendering data and the screen space data in the rasterization rendering pipeline, including shielding processing and relighting processing of the neural primitives in the scene rendering data, and neural radiation field drawing is ingeniously combined into the rasterization rendering pipeline in this way, so that real-time drawing is achieved, and the rendering efficiency is improved. And the drawing efficiency and the drawing quality can be ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of combining new perspective synthesis with real-time rendering, and specifically relates to a method and device for real-time rendering of a neural radiation field in a rasterization rendering pipeline. Background Art

[0002] 3D reconstruction has always been an important task in the field of computer vision and computer graphics. There is a great demand for high-quality and highly realistic 3D scene reconstruction in the fields of movies, games, virtual reality, etc. Traditional manual modeling requires a lot of manpower and time, and it is difficult to handle complex details. Therefore, 3D reconstruction technology based on real images has received great attention, and with the rapid development of deep learning technology in recent years, many breakthroughs have been made.

[0003] Neural Radiance Field (NeRF) is a technology proposed in 2020. This technology proposes to use an optimized neural network to implicitly represent various types of information of a 3D scene. It can handle the task of synthesizing new perspectives and render highly realistic images with complex details.

[0004] However, the existing neural radiation field is difficult to achieve real-time rendering. In response to this, Instant-NGP developed by Nvidia has made many optimizations based on the neural radiation field and proposed the use of multi-resolution hash coding, so that both training and inference can be carried out quickly and real-time rendering can be achieved.

[0005] The patent application with publication number CN 117911633 A discloses a neural radiation field rendering method based on Unreal Engine, which is applied to the neural radiation field rendering framework, and includes: extracting key frames from the video of the scene to be reconstructed to obtain an image sequence; estimating the three-dimensional scene structure and camera parameters according to the image sequence to obtain an estimated result; training the neural radiation field rendering framework according to the estimated result and the image sequence, and exporting rendering materials, which include Unreal Engine blueprint assets; sending the Unreal Engine blueprint assets to the Unreal Engine for automatic rendering to obtain rendering results. In this technical solution, a high-precision 3D model can be quickly obtained by training the neural radiation field rendering framework, automatically exporting rendering materials, and automatically performing real-time and high-quality rendering in the Unreal Engine.

[0006] The above technical solution imports the rendering material of the neural radiation field into rasterization for sampling and drawing, but does not implement the specific details of combining it with the original rasterization scene rendering results. Summary of the invention

[0007] In order to solve the combination and application of neural radiation fields in rasterization rendering pipelines, the purpose of the present invention is to provide a method and device for real-time drawing of neural radiation fields in rasterization rendering pipelines, so as to achieve correct transformation, occlusion, and re-lighting of neural radiation fields.

[0008] To achieve the above-mentioned object of the invention, an embodiment provides a method for real-time rendering of a neural radiation field in a rasterization rendering pipeline, comprising the following steps: While the rasterization rendering pipeline renders each frame to obtain scene rendering data, it dynamically obtains the screen space data of the neural radiation field step sampling in units of neural primitives; In the rasterization rendering pipeline, neural primitives are overlaid and drawn in real time based on scene rendering data and screen space data, including occlusion processing and re-lighting processing of neural primitives in scene rendering data.

[0009] Preferably, dynamically acquiring screen space data of the neural radiation field in units of neural graph elements includes: When rendering each frame, the rasterization rendering pipeline calculates the camera matrix of the neural radiance field based on the rasterization camera parameters, resolution, and affine transformation of the neural primitives; When it is determined that the camera matrix and resolution of the neural radiation field of the current frame and the previous frame have changed, the screen space data of the neural primitive drawn based on the camera matrix of the neural radiation field corresponding to the current frame is dynamically obtained.

[0010] Preferably, the method also includes: when it is determined that the camera parameters and resolution of the neural radiation field of the current frame and the previous frame have changed, frustum clipping is also performed to specifically determine whether the enclosing structure of the neural primitive of the previous frame is within the frustum of the current frame. If not, the screen space data of the previous frame is cleared and the scene rendering data is directly drawn to the frame buffer. If so, the screen space data of the neural radiation field of the current frame is dynamically obtained.

[0011] Preferably, the occlusion processing of the neural primitive in the scene rendering data includes: Each neural primitive is configured with at least one bounding sphere with an adjustable radius to fit the volume of the neural radiation field. The depth buffer data included in the screen space data of each neural primitive is the nearest intersection distance of the ray step, and a maximum distance among the nearest intersection distances is recorded, which is the distance of the background that does not need to be drawn; First, determine whether the current nearest intersection distance of each pixel in the scene rendering data is less than the maximum distance recorded. If so, it means that there is a neural radiation field in the current neural primitive fragment, and the neural primitive needs to be drawn. Otherwise, the current scene color is directly output; Then, the projection ray corresponding to the nearest intersection distance of the current neural primitive fragment is intersected with the bounding sphere of the neural primitive to obtain the world space coordinate X, and the world space coordinate X' is restored according to the scene depth contained in the scene rendering data. The distance from the world space coordinate X to the camera is compared with the world space coordinate X'. If X is closer, the neural primitive needs to be drawn, otherwise the scene color is directly output.

[0012] Preferably, the relighting processing of the neural primitive in the scene rendering data includes: The neural primitives that have been confirmed to need to be drawn after occlusion processing are re-lighted. Specifically, the color buffer data contained in the screen space data of the neural primitive is configurably attenuated. Then all light sources in the rasterization rendering pipeline are traversed, and based on the normal buffer data contained in the screen space data, the superimposed lighting on the attenuated color is calculated to achieve re-lighting of the neural primitive.

[0013] Preferably, the method further comprises: after dynamically acquiring the screen space data of the neural radiation field, filtering the normal buffer data contained in the screen space data, and then performing re-lighting processing using the filtered normal buffer data.

[0014] Preferably, when relighting the neural primitive, when there is environment buffer data, the environment lighting calculation is performed based on the normal buffer data, and the calculation result is superimposed on the relighting result of the neural primitive.

[0015] Preferably, the screen space data of the neural radiation field step sampling dynamically acquired comes from the Instant-NGP, and the Instant-NGP is modified specifically, including specifying cache memory for the color buffer data output by the Instant-NGP, modifying the depth buffer data output by the Instant-NGP to the nearest intersection distance of the ray step, modifying the normal buffer data output by the Instant-NGP and specifying the corresponding cache memory, encapsulating the modified output data into the screen space data of the neural primitive as an instance object, adding training switch control, camera synchronization, and loading and saving functions, and compiling into a dynamic link library, and designing an interface for each function; The rasterization rendering pipeline is encapsulated as a proxy object, which manages the pointers exposed by the instance object and provides a series of interface proxies.

[0016] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned method of real-time drawing of neural radiation fields in a rasterization rendering pipeline.

[0017] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned method of real-time rendering of a neural radiation field in a rasterization rendering pipeline is implemented.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention can integrate neural radiation field drawing into the rasterization rendering pipeline, specifically leaving the sampling process outside the rasterization rendering pipeline, that is, implementing sampling in the neural radiation field and forming screen space data, and combining the screen space data with the original rasterization rendering scene of the rasterization rendering pipeline, so as to achieve real-time rendering frame rate, and at the same time consider occlusion in the process of combining, and support experience-based re-illumination of the drawing object, so as to ensure drawing efficiency and drawing quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a flowchart of a method for real-time rendering of a neural radiation field in a rasterization rendering pipeline provided by an embodiment; Figure 2 It is a specific flow chart of a method for real-time rendering of a neural radiation field in a rasterization rendering pipeline provided by an embodiment. DETAILED DESCRIPTION

[0021] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0022] like Figure 1 As shown, an embodiment provides a method for real-time rendering of a neural radiation field in a rasterization rendering pipeline, comprising the following steps: S1, while the rasterization rendering pipeline renders each frame to obtain scene rendering data, it dynamically obtains the screen space data of the neural radiation field step sampling in units of neural primitives.

[0023] In the embodiment, the rasterization rendering pipeline executes the rasterization rendering pipeline process to perform real-time rendering of each frame, completes scene drawing, and obtains scene rendering data for each frame of the scene, specifically including color buffer data and depth buffer data, as well as geometry buffer (G-Buffer). After completion, post-processing and output operations are temporarily not performed.

[0024] In an embodiment, the rasterization rendering pipeline needs to dynamically obtain the screen space data of the neural radiation field to update each rendering frame while rendering each frame, so as to superimpose the drawing result of the neural radiation field into the rendering frame of the rasterization rendering pipeline. In the update process of each rendering frame, if there is an update to the transformation of the camera matrix or the neural primitive, the drawing interface of the neural primitive will be called to synchronize the data and draw the neural radiation field, and dynamically obtain the screen space data. Specifically, when obtaining the data, the screen space data corresponding to the drawing of the neural primitive in the neural radiation field is obtained in units of neural primitives. The screen space data is returned in the form of a frame data structure, which specifically includes color buffer data, depth buffer data, and normal buffer data of the screen space, wherein the depth buffer data contains the nearest intersection distance of the ray step, and at the same time records a maximum distance in the nearest intersection distance, which is the distance of the background that does not need to be drawn; There are updates to the camera perspective transformation. First, when rendering each frame, the rasterization rendering pipeline calculates the camera matrix of the neural radiation field based on the rasterized camera parameters, resolution, and affine transformation of the neural primitives. The rasterized camera parameters include (position, orientation, field of view, etc.), and the resolution determines the field of view angle of the x-axis. In the specific calculation, the camera parameters of the neural radiation field are first calculated using the rasterized camera parameters and resolution. Since the perspective information in the neural radiation field is expressed by the camera matrix composed of camera parameters, the inverse transformation of the affine transformation of the neural primitives in the pipeline is superimposed on the result of the camera matrix of the neural radiation field to achieve the effects of displacement, rotation, and translation.

[0025] Then determine whether the camera matrix of the neural radiation field of the current frame and the previous frame has changed. If it has changed, dynamically obtain the screen space data of the neural primitive sampled and drawn based on the camera matrix and resolution of the neural radiation field corresponding to the current frame.

[0026] In an embodiment, after determining that the camera parameters and resolution of the neural radiation field of the current frame and the previous frame have changed, a visual cone clipping is also performed to specifically determine whether the enclosing structure (which may be an enclosing sphere) of the neural primitive of the previous frame is within the visual cone of the current frame. If not, the screen space data of the previous frame is cleared and the scene rendering data is directly drawn to the frame buffer. If so, the screen space data of the neural radiation field of the current frame is dynamically acquired.

[0027] In an embodiment, after dynamically acquiring the screen space data of the neural radiation field, filtering and smoothing are performed on the normal buffer data contained in the screen space data, and the filtered normal buffer data is used for re-illumination processing. The filtering algorithm used in the specific filtering may be bilateral filtering, and its parameters and number of iterations are configured by the user in real time.

[0028] S2, real-time overlay drawing of neural primitives based on scene rendering data and screen space data in the rasterization rendering pipeline, including occlusion processing and re-lighting processing of neural primitives in the scene rendering data.

[0029] In an embodiment, based on the scene color buffer data and scene depth buffer data of the rasterization rendering pipeline, the neural primitives are drawn based on the screen space data of the neural radiation field. The drawing here is a screen space operation based on the rendering scene data and the screen space data, specifically including occlusion processing and re-lighting processing.

[0030] In the occlusion process, it is necessary to remove or cover the pixels in the screen space. Each neural primitive is configured with a series of bounding spheres with adjustable radius to represent the volume of the fitted neural radiation field. Before shading, rays are cast on the pixels with valid texture values ​​in the screen space data of each frame. After the intersection coordinates are obtained by intersecting with the neural primitive bounding sphere structure, the intersection coordinates are compared with the world space coordinates restored from the scene depth information of the corresponding point, and the decision is made to discard the current neural primitive fragment or cover the scene fragment based on the depth.

[0031] In the specific implementation, first determine whether the current nearest intersection distance of each pixel in the scene rendering data is less than the recorded maximum distance. If it is less, it means that there is a neural radiation field in the current neural primitive fragment, and the neural primitive needs to be drawn, otherwise the current scene color is directly output; then the projection light corresponding to the current neural primitive fragment's nearest intersection distance is intersected with the bounding sphere of the neural primitive to obtain the world space coordinate X, and the world space coordinate X' is restored according to the scene depth contained in the scene rendering data, and the world space coordinate X is compared with the distance from the world space coordinate X' to the camera. If X is closer, the neural primitive needs to be drawn, otherwise the scene color is directly output.

[0032] In the relighting process, all neural primitive fragments that have not been culled after occlusion processing are relit and shaded. First, the color buffer data contained in the screen space data of the neural primitive is configurably attenuated, and then all light sources of the rasterization rendering pipeline are traversed, and based on the filtered normal buffer data contained in the screen space data, the superimposed lighting on the attenuated color is calculated, including the shadows transmitted by the receiving scene objects. When there is environmental buffer data, the ambient lighting calculation (IBL) is performed based on the filtered normal buffer data, and the calculation result is superimposed on the relighting processing result of the neural primitive. It should be noted that all relighting can have an adjustable variable to control the amplitude.

[0033] In specific implementation, when relighting the neural primitive, the base color color_base can directly sample the color buffer data in the screen space data. If you need to superimpose light source illumination and ambient illumination, follow the following formula: color_final = color_base * decay_ratio+ direct_light * light_influence_ratio + env_light * env_influence_ratio; Among them, the adjustable parameter decay_ratio represents the degree of light attenuation, which attenuates the basic color to a certain extent. direct_light represents the direct lighting of the rendering pipeline light source. The previously obtained bounding sphere intersection position X and the normal N sampled from the corresponding coordinates of the normal texture are used to calculate the contribution of each light source, including the receiving rasterized scene shadows. light_influence_ratio represents an adjustable weight. env_light represents ambient lighting. Normal N is also used to sample the radiance of the ambient buffer data to calculate image-based ambient lighting. env_influence_ratio also represents an adjustable weight.

[0034] After re-lighting, the color texture is output to the frame buffer, and the overall drawing of a frame of image combining rasterization rendering and neural radiation field drawing is completed.

[0035] The above method realizes the real-time rendering of neural radiation field in the rasterization pipeline. When the camera perspective does not change, the frame rate can be maintained at the original frame rate of the rasterization pipeline; when the camera perspective switches, the frame rate can reach 15-30 frames per second. In addition, When the screen space data of the dynamically acquired neural radiation field contains geometric data, it can also be compatible with other screen space algorithms as needed, such as screen space ambient occlusion, screen space reflection, etc., and can be applied according to specific scenarios.

[0036] Based on the above method, the screen space data of the dynamically acquired neural radiation field step sampling can come from Instant-NGP. The Instant-NGP is modified to support the above method of the present invention. The specific modifications include the calling form and the resource acquisition interface.

[0037] Specifically, the resources required for the screen space data dynamically obtained for each frame need to be modified as follows to adapt to the above method of the present invention, including: (1) For color textures generated by Instant-NGP, specify cache memory for color buffer data output by Instant-NGP when modifying; (2) For the depth texture generated by Instant-NGP, modify the depth buffer data output by Instant-NGP to the closest intersection distance of the ray step, and record the maximum distance among the closest intersection distances at the same time; (3) Although the normal texture generated by Instant-NGP has been calculated in the source code, it is only displayed in debug mode and is not output. When modifying, modify Instant-NGP to enforce normal execution and output normal buffer data for each frame, and specify the cache memory corresponding to the normal buffer data; (4) You can also choose to output the alpha value representing transparency in the a channel of the normal texture for use in other screen space algorithms.

[0038] In addition to the modifications to the output data described in (1)-(4) above, it is also necessary to cancel the visualization window, add training switch control, camera synchronization, load and save functions, and add corresponding interfaces for the pipeline rasterization rendering pipeline to call.

[0039] The above designs are all for a neural radiation field object. Such an object can be understood as the neural primitive mentioned above. The present invention encapsulates a neural radiation field on the Instant-NGP side as an instance object, and correspondingly encapsulates it as a proxy object on the rendering pipeline side, which manages the pointers exposed by the instance object and provides a series of interface proxies. The modified Instant-NGP is recompiled into a dynamic link library that only performs calculations but not displays. Through the designed interface, the neural primitives can be created, trained, and drawn to obtain screen space data.

[0040] The input of the neural radiation field object can be two kinds: (i) a series of images and a json file that records the image camera pose information, which Instant-NGP will train and render in real time. (ii) a .ingp file saved after Instant-NGP training.

[0041] Through the configuration file path, the rasterization rendering pipeline creates a proxy object and dynamically loads the corresponding screen space data (image or cache), which corresponds to an instance of the neural radiation field, and performs subsequent control and drawing based on the instance. Figure 2 As shown, the method for real-time rendering of neural radiation fields in a rasterization rendering pipeline by combining Instant-NGP and a rasterization rendering pipeline includes the following steps: S01, the rasterization rendering pipeline of the renderer executes normally, and obtains the color buffer and depth buffer of the scene.

[0042] S02, calculating the camera matrix of the neural radiation field space according to the affine transformation of the neural primitive and the rasterized camera parameters; Step S03, analyzing whether the camera matrix and resolution have changed from the previous frame. If not, the previously obtained screen space data is used and the process directly proceeds to S07 for drawing. If there is a change, the process proceeds to S04 for the next step of judgment. Step S04, perform frustum clipping, and determine whether the neural primitive is in the changed frustum based on its enclosing structure. If it is not in the frustum, no drawing is performed, the screen space data content is cleared, and the color and depth of the scene are directly drawn to the frame buffer to end the frame.

[0043] Step S05, call the interface in the dynamic link library, update the camera parameters, and use Instant-NGP to sample the neural radiation field, synchronize the sampled data to the frame data structure, that is, update the screen space data. When updating the depth buffer data, it is necessary to record the maximum depth in a frame, that is, the farthest distance of the light, to determine whether it intersects with the neural primitive.

[0044] Step S06: performing a filtering operation on the normal buffer data in the screen space data.

[0045] Step S07, using the bounding sphere structure of the neural primitive, and comparing it with the depth buffer of the rasterization rendering pipeline to perform a depth test, and determine the neural primitive that needs to be re-illuminated.

[0046] Determine the occlusion relationship between the neural primitive and the scene objects, and determine the final pixel color accordingly. Since all transformations in Instant-NGP are achieved by converting to camera transformations, the AABB bounding box of the neural primitive itself will not change with the affine transformation and needs to be maintained according to the information in the pipeline. On the other hand, the depth given by the Instant-NGP projection result is not the standard spatial depth of the device, and the output depth is discontinuous and at different scales, making it difficult to align directly for comparison. Therefore, this method proposes to use the nearest distance of ray step intersection instead of the standardized depth output, and use one or several bounding spheres with adjustable radius to fit the range of the neural radiation field.

[0047] Step S08, draw the neural primitives, and perform re-illumination by sampling the world space normal and light source parameters according to the neural radiation field.

[0048] Step S09, writing the rendering result into the buffer frame, and the drawing of this frame is completed.

[0049] Based on the same inventive concept, an embodiment further provides a computing device, including a memory and one or more processors, wherein executable code is stored in the memory, and when the one or more processors execute the executable code, the method for real-time rendering of a neural radiation field in a rasterization rendering pipeline is implemented, specifically including the following steps: S1, the rasterization rendering pipeline renders each frame to obtain scene rendering data, and dynamically obtains the screen space data of the neural radiation field step sampling in units of neural primitives; S2, real-time overlay drawing of neural primitives based on scene rendering data and screen space data in the rasterization rendering pipeline, including occlusion processing and re-lighting processing of neural primitives in the scene rendering data.

[0050] The computing device provided in the embodiment, at the hardware level, includes not only a processor and a memory, but also hardware required for other services such as an internal bus, a network interface, and a memory. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the method of drawing a neural radiation field in real time in a rasterization rendering pipeline as described in S1-S2 above. Of course, in addition to software implementations, the present invention does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0051] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method for real-time rendering of a neural radiation field in a rasterization rendering pipeline is implemented, specifically including the following steps: S1, the rasterization rendering pipeline renders each frame to obtain scene rendering data, and dynamically obtains the screen space data of the neural radiation field step sampling in units of neural primitives; S2, real-time overlay drawing of neural primitives based on scene rendering data and screen space data in the rasterization rendering pipeline, including occlusion processing and re-lighting processing of neural primitives in the scene rendering data.

[0052] In the embodiment, computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0053] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for real-time rendering of neural radiance fields in a rasterized rendering pipeline, It is characterized in that The following steps are involved: While the rasterization rendering pipeline renders each frame to obtain scene rendering data, it dynamically obtains the screen space data of the neural radiation field step sampling in units of neural primitives; In the rasterization rendering pipeline, neural primitives are overlaid and drawn in real time based on scene rendering data and screen space data, including occlusion processing and re-lighting processing of neural primitives in scene rendering data.

2. The method for real-time rendering of neural radiation fields in a rasterization rendering pipeline according to claim 1, It is characterized in that Dynamically obtain screen space data of neural radiation fields in units of neural primitives, including: When rendering each frame, the rasterization rendering pipeline calculates the camera matrix of the neural radiance field based on the rasterization camera parameters, resolution, and affine transformation of the neural primitives; When it is determined that the camera matrix of the neural radiation field of the current frame and the previous frame has changed or the transformation of the neural primitive has been updated, the screen space data of the neural primitive sampled and drawn based on the camera matrix and resolution of the neural radiation field corresponding to the current frame is dynamically obtained.

3. The method for real-time rendering of neural radiation fields in a rasterization rendering pipeline according to claim 2, It is characterized in that Also includes: When it is determined that the camera parameters and resolution of the neural radiation field of the current frame and the previous frame have changed, frustum clipping is also performed to specifically determine whether the enclosing structure of the neural primitives of the previous frame is within the frustum of the current frame. If not, the screen space data of the previous frame is cleared and the scene rendering data is directly drawn to the frame buffer. If so, the screen space data of the neural radiation field of the current frame is dynamically obtained.

4. The method for real-time rendering of neural radiation fields in a rasterization rendering pipeline according to claim 1, It is characterized in that The occlusion processing of the neural primitive in the scene rendering data includes: Each neural primitive is configured with at least one bounding sphere with an adjustable radius to fit the volume of the neural radiation field. The depth buffer data included in the screen space data of each neural primitive is the nearest intersection distance of the ray step, and a maximum distance among the nearest intersection distances is recorded, which is the distance of the background that does not need to be drawn; First, determine whether the current nearest intersection distance of each pixel in the scene rendering data is less than the maximum distance recorded. If so, it means that there is a neural radiation field in the current neural primitive fragment, and the neural primitive needs to be drawn. Otherwise, the current scene color is directly output; Then, the projection ray corresponding to the nearest intersection distance of the current neural primitive fragment is intersected with the bounding sphere of the neural primitive to obtain the world space coordinate X, and the world space coordinate X' is restored according to the scene depth contained in the scene rendering data. The distance from the world space coordinate X to the camera is compared with the world space coordinate X'. If X is closer, the neural primitive needs to be drawn, otherwise the scene color is directly output.

5. The method for real-time rendering of neural radiation fields in a rasterization rendering pipeline according to claim 1, It is characterized in that Neural primitives relighting in scene rendering data, including: The neural primitives that have been confirmed to need to be drawn after occlusion processing are re-lighted. Specifically, the color buffer data contained in the screen space data of the neural primitive is configurably attenuated. Then all light sources in the rasterization rendering pipeline are traversed, and based on the normal buffer data contained in the screen space data, the superimposed lighting on the attenuated color is calculated to achieve re-lighting of the neural primitive.

6. The method for real-time rendering of neural radiation fields in a rasterization rendering pipeline according to claim 5, It is characterized in that Also includes: After dynamically acquiring the screen space data of the neural radiation field, the normal buffer data contained in the screen space data is filtered, and then the filtered normal buffer data is used for re-lighting.

7. The method for real-time rendering of neural radiation fields in a rasterization rendering pipeline according to claim 5, It is characterized in that When relighting a neural primitive, when there is environment buffer data, the environment lighting calculation is performed based on the normal buffer data, and the calculation result is superimposed on the relighting result of the neural primitive.

8. The method for real-time rendering of neural radiation fields in a rasterization rendering pipeline according to claim 1, It is characterized in that The screen space data of the dynamically acquired neural radiation field step sampling comes from Instant-NGP. The specific changes to Instant-NGP include specifying cache memory for the color buffer data output by Instant-NGP, modifying the depth buffer data output by Instant-NGP to the nearest intersection distance of the ray step, modifying the normal buffer data output by Instant-NGP and specifying the corresponding cache memory, encapsulating the modified output data into the screen space data of the neural primitive as an instance object, adding training switch control, camera synchronization, and load and save functions, and compiling it into a dynamic link library, and designing interfaces for each function; The rasterization rendering pipeline is encapsulated as a proxy object, which manages the pointers exposed by the instance object and provides a series of interface proxies.

9. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, It is characterized in that When the one or more processors execute the executable code, they are used to implement the method for real-time rendering of a neural radiation field in a rasterization rendering pipeline according to any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that A program is stored thereon, and when the program is executed by a processor, the method for real-time rendering of a neural radiation field in a rasterization rendering pipeline as described in any one of claims 1 to 7 is implemented.

Images