Global neural rendering method and system based on controllable neural light transport sparse inference

Abstract

The application discloses a global neural rendering method and system based on controllable neural light transmission sparse inference, which can be deployed in a real-time graphics system supporting neural rendering inference, such as a game engine, an interactive visualization system or a hybrid rendering framework. In the frame-by-frame rendering process, combined with the current frame geometry attribute information, the historical intermediate features and the historical rendering frame as the output result, adaptive hierarchical feature query, geometry-constrained intermediate feature super-resolution reconstruction, time error-driven sparse decoding and time synthesis based on fusion intermediate features are sequentially executed, the complete current rendering frame image is output, and the related historical state is saved for the next frame. Through joint modeling of spatial detail complexity and time reuse risk, the computing resources are centrally allocated to the areas which have greater impact on the quality and time stability of the current frame, so that continuous adjustment of quality and overhead is realized under the limited real-time budget.

Claims

1. A global neural rendering method based on controllable neural light transport sparse inference, characterized in that, Includes the following steps: Adaptive hierarchical feature query: Based on the geometric attribute information of the current frame in screen space, the region complexity level is determined and the corresponding query density and query level are configured accordingly. The intermediate features of the current frame for low-cost query are obtained from the underlying neural illumination representation according to the query density and query level. Super-resolution reconstruction of intermediate features with geometric constraints: Using the geometric attribute information of the current frame as geometric constraints, super-resolution reconstruction is performed on the intermediate features of the current frame to improve feature resolution, restore the constraints of geometric boundaries, and fuse features at different levels to obtain high-resolution intermediate features of the current frame. Temporal error-driven sparse decoding: After temporal feature fusion of the high-resolution intermediate features of the current frame and the aligned historical intermediate features, the error risk distribution of each local region is calculated. This includes: aligning the historical intermediate features to the corresponding positions in the current frame based on motion information, so that the historical content can establish a consistent spatial correspondence with the high-resolution intermediate features of the current frame; then performing temporal feature aggregation of the high-resolution intermediate features of the current frame and the historical intermediate features; and outputting the error risk estimation results of each pixel position in the current frame based on the temporal feature aggregation, thereby obtaining the error risk distribution of the local region. Here, the error risk reflects the relative distortion risk when the current position continues to depend on the historical content, and is used to characterize the update priority of the current frame. The local update regions with high error risk are selected for neural decoding to obtain local decoding results. This includes: sorting screen spatial positions according to error risk and selecting positions with high error risk as local update regions, where local update regions can be determined by pixel granularity, tile, block, or local area granularity; extracting high-resolution intermediate features corresponding to the local update regions and inputting them into a local neural decoder with shared parameters to obtain local decoding results. Temporal synthesis based on fused intermediate features: The local decoding results, high-resolution intermediate features and historical intermediate features are fused in a temporal manner to obtain fused intermediate features. Motion information is recovered based on the fused intermediate features. The historical rendering frames are then fused with the local decoding results based on the recovered motion information to obtain the current rendering frame.

2. The global neural rendering method based on controllable neural light transport sparse inference according to claim 1, characterized in that, The method for determining the region complexity level based on the current frame geometric attribute information of the screen space includes: dividing the screen space into local regions, and then analyzing the region complexity of the local regions based on the current frame geometric attribute information. The analysis is based on geometric boundaries, occlusion changes, texture detail changes, and lighting intensity changes to obtain the region complexity level that includes complex and simple regions.

3. The method of claim 1, wherein, Configure the corresponding query density and query level according to the region complexity level, including: Configure high query density and fine-grained neural illumination representation layers for complex regions; Configure low query density and coarse neural lighting representation hierarchy for simple regions.

4. The method of claim 1, wherein, Using the geometric attribute information of the current frame as geometric constraints, super-resolution reconstruction is performed on the intermediate features of the current frame, including: The geometric attribute information of the current frame includes the depth map, normal map, and albedo map; The intermediate features from different query levels of the current frame are fused together and input into the feature reconstruction module along with the geometric attribute information of the current frame. Under geometric constraints, resolution enhancement and detail restoration are performed. The depth map, normal map, and albedo map are used to constrain the feature restoration direction and boundary propagation range, so that the features remain separated at depth discontinuity boundaries, normal abrupt boundaries, and material boundaries, and suppress cross-surface feature mixing, forming high-resolution intermediate features for the current frame.

5. The global neural mapping method based on controllable neural optical transmission sparse inference according to claim 1, characterized in that, The current frame is obtained by fusing historical rendering frames with local decoding results based on the recovered motion information, including: The restored motion information is used to map the historical rendering frames to the current frame position to obtain the aligned historical content. Then, the aligned historical content is fused with the local decoding result of the current frame.

6. The global neural rendering method based on controllable neural optical transmission sparse inference according to claim 1, characterized in that, The method also includes current drawing frame output and history state update: output the current drawing frame and update the current drawing frame with the high-resolution intermediate features, fused intermediate features and the current drawing frame as history state.

7. A global neural mapping system based on controllable neural optical transmission sparse inference, characterized in that, include: The adaptive hierarchical feature query module is used to determine the region complexity level based on the current frame geometric attribute information in the screen space and configure the corresponding query density and query level accordingly. It obtains the current frame intermediate features for low-cost query from the underlying neural illumination representation according to the query density and query level. The geometrically constrained intermediate feature super-resolution reconstruction module is used to perform super-resolution reconstruction of the intermediate features of the current frame using the geometric attribute information of the current frame as geometric constraints, thereby improving feature resolution, restoring the constraints of geometric boundaries, and fusing features at different levels to obtain high-resolution intermediate features of the current frame. The temporal error-driven sparse decoding module is used to calculate the error risk distribution of each local region by temporally fusing the high-resolution intermediate features of the current frame with the aligned historical intermediate features. This includes: aligning the historical intermediate features to the corresponding positions in the current frame based on motion information, so that the historical content can establish a consistent spatial correspondence with the high-resolution intermediate features of the current frame; then performing temporal feature aggregation on the high-resolution intermediate features of the current frame and the historical intermediate features; and outputting the error risk estimation results of each pixel position in the current frame based on the temporal feature aggregation, thereby obtaining the error risk distribution of the local region. Here, the error risk reflects the relative distortion risk when the current position continues to depend on the historical content, and is used to characterize the update priority of the current frame. The local update regions with high error risk are selected for neural decoding to obtain local decoding results. This includes: sorting screen spatial positions according to error risk and selecting positions with high error risk as local update regions, where local update regions can be determined by pixel granularity, tile, block, or local area granularity; extracting high-resolution intermediate features corresponding to the local update regions and inputting them into a local neural decoder with shared parameters to obtain local decoding results. The temporal synthesis module based on fused intermediate features is used to temporally fuse local decoding results, high-resolution intermediate features, and historical intermediate features to obtain fused intermediate features. After recovering motion information based on the fused intermediate features, the historical rendering frames and local decoding results are fused based on the recovered motion information to obtain the current rendering frame.

8. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the global neural rendering method based on sparse inference of controllable neural optical transmission as described in any one of claims 1-6.

Images