Global neural rendering method and system based on controllable neural light transport sparse inference

Abstract

This invention discloses a global neural rendering method and system based on controllable neural optical transmission sparse inference, which can be deployed in real-time graphics systems that support neural rendering inference, such as game engines, interactive visualization systems, or hybrid rendering frameworks. During frame-by-frame rendering, the method combines the geometric attribute information of the current frame, historical intermediate features, and historical rendering frames as output results. It sequentially performs adaptive hierarchical feature querying, geometrically constrained intermediate feature super-resolution reconstruction, temporal error-driven sparse decoding, and temporal synthesis based on fused intermediate features to output a complete current rendering frame image and saves relevant historical states for use in the next frame. By jointly modeling spatial detail complexity and temporal reuse risks, computational resources are concentrated on regions with a greater impact on the quality and temporal stability of the current frame, achieving continuous adjustment of quality and overhead under a limited real-time budget.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics technology, specifically relating to a global neural rendering method and system based on sparse inference of controllable neural optical transmission. Background Technology

[0002] With the development of applications such as game rendering, virtual reality, augmented reality, and interactive simulation, real-time graphics systems are increasingly demanding higher levels of realism and rendering efficiency. Especially in scenes involving indirect lighting, shadow bounces, specular reflection, and complex material responses, traditional local shading models are no longer sufficient to meet the requirements of high-quality real-time rendering. Achieving high-fidelity global illumination rendering within a strict frame time budget has become a crucial technical challenge in the field of computer graphics.

[0003] Existing solutions for accelerating global illumination mainly fall into two categories: one is the rendering pipeline based on ray tracing and denoising, which, by reducing the number of samples per pixel and combining temporal or spatial denoising, can approximate high-quality results to some extent. However, under dynamic scenes, high-resolution output, or low sampling budget conditions, it is still prone to problems such as noise residue, flickering, ghosting, and insufficient recovery of high-frequency details. Moreover, its computational overhead usually increases rapidly with resolution and sampling scale, making it difficult to meet the real-time requirements of high frame rate interactive applications in the long term. The other category is to improve the apparent frame rate or output resolution through screen space post-processing techniques such as frame generation, frame interpolation, frame extrapolation, and super-resolution reconstruction. This type of method mainly operates in the image post-processing stage and cannot directly control the computational allocation in the underlying light transmission solution process. Therefore, it is difficult to fundamentally reduce the cost of global illumination computation itself. When there are changes in occlusion, rapid viewpoint movement, or dynamic lighting disturbances in the scene, it is also prone to problems such as loss of detail, reprojection distortion, and temporal instability.

[0004] In recent years, neural rendering and neural phototransfer technologies have provided new pathways for achieving high-quality real-time rendering. These methods typically model complex lighting transport relationships through scene-dependent implicit feature fields or neural decoders, thereby generating near-offline rendering quality results with fewer explicit ray tracing processes. However, most existing neural phototransfer methods employ a dense, uniform inference pattern, performing fixed-structure, fixed-cost feature queries and color decoding on the entire frame or a large number of pixels. They lack the ability to differentiate computational allocation based on the intensity of spatiotemporal variations and error risks in different regions, easily leading to computational redundancy in low-variety areas and insufficient budget in high-risk areas.

[0005] Furthermore, existing neural optical transmission schemes face significant inference cost inflation under high-resolution conditions and typically lack phased, interpretable budget control interfaces, making it difficult to continuously adjust feature acquisition, pixel decoding, and temporal fusion based on scene complexity and hardware performance. Directly reducing the sampling rate or network computation can easily lead to image quality degradation, loss of detail, and increased artifacts. Meanwhile, when relying on historical results for temporal reuse, inaccurate motion estimation or significant local illumination changes can easily cause flickering, ghosting, and error accumulation. Therefore, current technologies lack a global optical transmission neural rendering scheme that can simultaneously balance computational controllability, sparse inference efficiency, and temporal stability. Summary of the Invention

[0006] In view of the above, this paper aims to address the problems of high inference overhead, uneven computational allocation, and insufficient temporal stability in existing global illumination rendering methods. It also addresses the shortcomings of existing neural optical transmission methods, which generally employ a dense, uniform inference mode and struggle to differentiate computational allocation based on regional error risk and spatiotemporal variation intensity. Therefore, a global neural rendering method and system based on controllable neural optical transmission sparse inference is proposed. This allows for controllable adjustment of features acquisition, temporal error modeling, pixel decoding, and temporal synthesis, improving global illumination rendering quality and reducing redundant computation while ensuring real-time performance.

[0007] To achieve the above-mentioned objectives, an embodiment provides a global neural mapping method based on controllable neural optical transmission sparse inference, comprising the following steps: Adaptive hierarchical feature query: Based on the geometric attribute information of the current frame in screen space, the region complexity level is determined and the corresponding query density and query level are configured accordingly. The intermediate features of the current frame for low-cost query are obtained from the underlying neural illumination representation according to the query density and query level. Super-resolution reconstruction of intermediate features with geometric constraints: Using the geometric attribute information of the current frame as geometric constraints, super-resolution reconstruction is performed on the intermediate features of the current frame to improve feature resolution, restore the constraints of geometric boundaries, and fuse features at different levels to obtain high-resolution intermediate features of the current frame. Temporal error-driven sparse decoding: After temporal feature fusion of the high-resolution intermediate features of the current frame and the aligned historical intermediate features, the error risk distribution of each local region is calculated. Local update regions with high error risk are selected for neural decoding to obtain local decoding results. Temporal synthesis based on fused intermediate features: The local decoding results, high-resolution intermediate features and historical intermediate features are fused in a temporal manner to obtain fused intermediate features. Motion information is recovered based on the fused intermediate features. The historical rendering frames are then fused with the local decoding results based on the recovered motion information to obtain the current rendering frame.

[0008] Preferably, determining the region complexity level based on the current frame geometric attribute information of the screen space includes: dividing the screen space into local regions, and then analyzing the region complexity of the local regions based on the current frame geometric attribute information. The analysis is based on geometric boundaries, occlusion changes, texture detail changes, and illumination change intensity to obtain the region complexity level that includes complex regions and simple regions.

[0009] Preferably, the query density and query level are configured according to the region complexity level, including: Configure high query density and fine-grained neural illumination representation layers for complex regions; Configure low query density and coarse neural lighting representation hierarchy for simple regions.

[0010] Preferably, super-resolution reconstruction is performed on the intermediate features of the current frame using the geometric attribute information of the current frame as geometric constraints, including: The geometric attribute information of the current frame includes the depth map, normal map, and albedo map; The intermediate features from different query levels of the current frame are fused together and input into the feature reconstruction module along with the geometric attribute information of the current frame. Under geometric constraints, resolution enhancement and detail restoration are performed. The depth map, normal map, and albedo map are used to constrain the feature restoration direction and boundary propagation range, so that the features remain separated at depth discontinuity boundaries, normal abrupt boundaries, and material boundaries, and suppress cross-surface feature mixing, forming high-resolution intermediate features for the current frame.

[0011] Preferably, the error risk distribution of each local region is calculated by temporally fusing the high-resolution intermediate features of the current frame with the aligned historical intermediate features, including: Based on motion information, historical intermediate features are aligned to the corresponding positions in the current frame, so that historical content can establish a consistent spatial correspondence with the high-resolution intermediate features of the current frame. Then, the high-resolution intermediate features of the current frame and the historical intermediate features are aggregated in a temporal sequence. Based on temporal features, the error risk estimation results of each pixel position in the current frame are aggregated and output, thereby obtaining the error risk distribution of the local region. The error risk reflects the relative distortion risk when the current position continues to depend on historical content, and is used to characterize the update priority of the current frame.

[0012] Preferably, neural decoding is performed on local update regions with high error risk to obtain local decoding results, including: The screen spatial locations are sorted according to error risk, and locations with high error risk are selected as local update areas. The local update areas can be determined by pixel granularity, tile, block, or local area granularity. High-resolution intermediate features corresponding to the locally updated region are extracted and input into a local neural decoder with shared parameters to obtain the local decoding result.

[0013] Preferably, the current frame is obtained by fusing historical rendering frames with local decoding results based on the recovered motion information, including: The restored motion information is used to map the historical rendering frames to the current frame position to obtain the aligned historical content. Then, the aligned historical content is fused with the local decoding result of the current frame.

[0014] Preferably, the method further includes current drawing frame output and historical state update: outputting the current drawing frame and updating the current frame as the high-resolution intermediate features, fused intermediate features, and the current drawing frame as the historical state.

[0015] To achieve the above-mentioned objectives, the embodiments also provide a global neural mapping system based on controllable neural optical transmission sparse inference, comprising: The adaptive hierarchical feature query module is used to determine the region complexity level based on the current frame geometric attribute information in the screen space and configure the corresponding query density and query level accordingly. It obtains the current frame intermediate features for low-cost query from the underlying neural illumination representation according to the query density and query level. The geometrically constrained intermediate feature super-resolution reconstruction module is used to perform super-resolution reconstruction of the intermediate features of the current frame using the geometric attribute information of the current frame as geometric constraints, thereby improving feature resolution, restoring the constraints of geometric boundaries, and fusing features at different levels to obtain high-resolution intermediate features of the current frame. The temporal error-driven sparse decoding module is used to perform temporal feature fusion of the high-resolution intermediate features of the current frame and the aligned historical intermediate features, calculate the error risk distribution of each local region, select local update regions with high error risk for neural decoding, and obtain local decoding results. The temporal synthesis module based on fused intermediate features is used to temporally fuse local decoding results, high-resolution intermediate features, and historical intermediate features to obtain fused intermediate features. After recovering motion information based on the fused intermediate features, the historical rendering frames and local decoding results are fused based on the recovered motion information to obtain the current rendering frame.

[0016] To achieve the above-mentioned objectives, the embodiments also provide a computer-readable storage medium storing a program that, when executed by a processor, implements the above-mentioned global neural rendering method based on sparse inference of controllable neural optical transmission.

[0017] Compared with the prior art, the beneficial effects of the present invention include at least the following: The global neural rendering method and system proposed in this invention, based on controllable neural optical transmission sparse inference, is characterized by its ability to adaptively allocate computational resources during the neural rendering process according to the spatiotemporal variation characteristics and error risk distribution of scene regions. By introducing a controllable sparse inference mechanism, this invention prioritizes neural updates for regions of significant perceptual quality while reusing historical content for regions with low change, thereby reducing the redundant computational overhead caused by uniform dense inference across the entire frame. This method improves the quality and temporal stability of global illumination rendering while ensuring real-time performance, and supports the same model running under different budget configurations, demonstrating good deployment flexibility and practical engineering value. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the global neural rendering method based on sparse inference with controllable neural optical transmission provided in the embodiment; Figure 2 This is a detailed breakdown diagram of each step in the global neural rendering method based on sparse inference with controllable neural optical transmission provided in the embodiment; Figure 3 This is a schematic diagram of the structure of the global neural rendering system based on sparse inference with controllable neural optical transmission provided in the embodiment. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0021] The inventive concept of this invention is to propose a global neural rendering scheme based on controllable neural optical transmission sparse inference for real-time surface rendering. This scheme can be deployed in real-time graphics systems that support neural rendering inference, such as game engines, interactive visualization systems, or hybrid rendering frameworks. During frame-by-frame rendering, the scheme combines the geometric attribute information of the current frame, historical intermediate features, and historical rendering frames as output results. It sequentially performs adaptive hierarchical feature querying, geometrically constrained intermediate feature super-resolution reconstruction, temporal error-driven sparse decoding, and temporal synthesis based on fused intermediate features to output a complete current rendering frame image and saves relevant historical states for use in the next frame. By jointly modeling spatial detail complexity and temporal reuse risks, computational resources are concentrated on regions with a greater impact on the quality and temporal stability of the current frame, achieving continuous adjustment of quality and overhead within a limited real-time budget.

[0022] Before implementing the global neural rendering method based on sparse inference of controllable neural optical transmission proposed in this invention, it is necessary to generate the geometric attribute information of the current frame and prepare the historical state.

[0023] The rendering system performs a conventional geometric rendering process based on the camera parameters, scene geometry data, and material data of the current frame, generating screen-space geometric attribute information for the current frame. This geometric attribute information includes a depth map, a normal map, an albedo map, and motion vectors representing motion information. The depth map represents the distance relationship between pixels in the current frame and the corresponding surfaces; the normal map represents the surface orientation; the albedo map represents the base color of the material; and the motion vectors represent the screen-space displacement relationship between the current frame and previous frames.

[0024] Simultaneously, the system reads historical intermediate features and historical rendering frames saved from the previous frame or several previous frames from the history buffer. Historical intermediate features serve as historical input for subsequent temporal modeling, while historical rendering frames serve as input for the complete restoration of historical content in the current frame. The historical state, composed of historical intermediate features and historical rendering frames, can be saved using multi-channel textures, feature buffers, or other equivalent forms and updated at the end of each frame.

[0025] This section outputs two types of data: the geometric attribute information of the current frame and the historical state. The geometric attribute information of the current frame is fed into the intermediate feature super-resolution reconstruction part of adaptive hierarchical feature query and geometric constraint, as guiding information for spatial structure analysis and intermediate feature recovery; historical intermediate features and historical drawn frames are fed into the temporal error-driven sparse decoding and temporal synthesis part based on fused intermediate features, as input for the historical content of the current frame temporal modeling, local update and complete recovery.

[0026] Figure 1 This is a schematic diagram of the data flow for a global neural rendering method based on controllable neural optical transmission sparse inference according to the present invention. Figure 1The diagram illustrates the process of taking the current frame's geometric attributes, low-level neural lighting representation, historical intermediate features, and historical rendering frames as input. These are then processed sequentially through adaptive hierarchical feature querying, geometrically constrained intermediate feature super-resolution reconstruction, temporal error-driven sparse decoding, and temporal synthesis based on fused intermediate features to obtain the current rendering frame. Finally, historical state updates are used to form the overall data flow required for the next frame. Specifically, the detailed steps include the following: S1, Adaptive hierarchical feature query: Based on the current frame's geometric attribute information in screen space, determine the region complexity level and configure the corresponding query density and query level accordingly. Obtain the current frame's intermediate features for low-cost query from the underlying neural illumination representation according to the query density and query level.

[0027] like Figure 2 As shown, the adaptive hierarchical feature query steps mainly include structural complexity analysis, query density allocation, and query level allocation.

[0028] For structural complexity analysis: Within the screen space of the current frame, the structural complexity of local regions is first analyzed. The analysis is based on geometric boundaries, occlusion variations, texture detail variations, and the intensity of lighting changes. Complex regions correspond to locations with dense boundaries, significant changes in occlusion relationships, rich local details, or significant shading variations, while simple regions correspond to locations with smooth surfaces, continuous structures, and gradual changes. The region complexity analysis results, including both complex and simple regions, form a hierarchical distribution of the screen space, which is used for subsequent query strategy configuration.

[0029] Regarding query density allocation: Based on regional complexity analysis, different feature query densities are configured for different regions. Complex regions use higher query densities to obtain more comprehensive local feature sampling; simple regions use lower query densities to reduce redundant access to underlying representations. In this way, the feature query overhead adaptively changes with the distribution of screen space content, allowing the limited budget to be prioritized for high-impact regions.

[0030] For query hierarchy allocation: different query hierarchies are further configured for different regions. Complex regions prioritize access to finer neural illumination representation hierarchies to preserve local high-frequency information and detailed structure; simple regions prioritize access to coarser neural illumination representation hierarchies to obtain a more compact contextual representation. Query density and query hierarchy jointly determine the feature acquisition intensity of the current region, enabling the construction of intermediate features to possess both spatial and representational hierarchies.

[0031] Through the above design, the access method of the underlying neural illumination representation in the screen space is transformed from a uniform fixed-density query to a hierarchical query structure that varies with the complexity of the region, providing input features with budget control capabilities for subsequent intermediate feature reconstruction.

[0032] In practice, the inputs are the depth map, normal map, albedo map, and low-level neural lighting representation of the current frame. First, the screen space is divided into local regions. Then, based on the depth, normal, and albedo variations and local structural complexity of each region, the region complexity level is determined, and the corresponding query density and query level are configured accordingly. Regions with higher complexity use higher query density and finer representation levels, while regions with lower complexity use lower query density and coarser representation levels.

[0033] After query control is determined, the intermediate features required for the current frame are extracted from the underlying neural illumination representation. Complex regions perform denser queries at finer-grained locations and access finer-level underlying features; simple regions perform sparser queries at coarser-grained locations and access coarser-level underlying features. This results in intermediate features for the current frame obtained with lower average query overhead. These intermediate features are then fed into a geometrically constrained intermediate feature super-resolution reconstruction step, serving as direct input for subsequent resolution upscaling and detail restoration.

[0034] S2, Super-resolution reconstruction of intermediate features with geometric constraints: Using the geometric attribute information of the current frame as geometric constraints, super-resolution reconstruction is performed on the intermediate features of the current frame to improve resolution, restore geometric boundaries, and fuse features at different levels to obtain high-resolution intermediate features of the current frame.

[0035] The aforementioned adaptive hierarchical feature query yields intermediate screen-space features constructed at a low query density. These intermediate features cover the overall spatial distribution of the current frame, but their high-frequency expressive power is limited by query density and query level, resulting in relatively insufficient feature details for local boundaries, fine structures, and rapidly changing regions. Therefore, a geometrically constrained intermediate feature reconstruction process is introduced in subsequent processing to restore details and enhance the resolution of low-frequency intermediate features, forming high-resolution intermediate features suitable for error prediction, local neural decoding, and temporal synthesis. Figure 2 As shown, the intermediate feature super-resolution reconstruction steps of geometric constraints mainly include feature resolution enhancement, geometric boundary constraint recovery, and feature fusion at different levels.

[0036] During intermediate feature reconstruction, the geometric attribute information of the current frame is used as guiding information to perform feature super-resolution recovery on the intermediate features obtained under low query density. The reconstruction process revolves around enhancing the details of screen space features, gradually restoring the coarse-grained representations in the original intermediate features to higher resolution and higher fidelity feature representations. The object being restored is not the final color result, but rather the intermediate feature layer in neural optical transmission inference; therefore, this process is essentially feature enhancement and structural restoration geared towards subsequent inference stages.

[0037] During the reconstruction process, intermediate feature information from different levels is further integrated to unify and fuse low-frequency query results and higher-level local information. Lower-level features provide richer local structural clues, while higher-level features provide more stable contextual semantics. Both participate in feature recovery under geometric constraints, enabling the final intermediate representation to possess local high-frequency details, global consistency, and boundary stability. This intermediate feature representation can better support subsequent temporal error modeling and local decoding processes, reducing information loss caused by low query density at the front end.

[0038] After the above reconstruction, a high-resolution intermediate feature for the current frame is formed. This feature, with a low low-level query cost, improves the detail representation capability of the intermediate layer through geometrically guided feature recovery, enabling local neural decoding to update the current frame based on more reliable features, and providing structurally stable feature support for temporal synthesis.

[0039] In practice, the intermediate features from the low-cost query output of the adaptive hierarchical feature query step, along with geometric attribute information such as the depth map, normal map, and albedo map of the current frame, are input to the feature reconstruction module. Resolution enhancement and detail restoration are performed under geometric constraints. The depth map, normal map, and albedo map are used to constrain the feature restoration direction and boundary propagation range, ensuring feature separation at depth discontinuities, abrupt normal changes, and material boundaries, and suppressing cross-surface feature mixing. Simultaneously, intermediate features obtained from different query levels are fused to form high-resolution intermediate features for the current frame. These high-resolution intermediate features are then fed into both the temporal error-driven sparse decoding step and the temporal synthesis step based on the fused intermediate features, serving as the feature basis for local update selection, local neural decoding, and complete restoration of the current frame, respectively.

[0040] S3, Temporal Error-Driven Sparse Decoding: After temporal feature fusion of the high-resolution intermediate features of the current frame and the aligned historical intermediate features, the error risk distribution of each local region is calculated, and local regions with high error risk are selected for neural decoding to obtain the local decoding result.

[0041] After forming the intermediate feature representation of the current frame, historical states (historical intermediate features and historical rendered frames) and motion information are introduced to model the reliability of historical reuse, and the error distribution is directly used for the local neural decoding of the current frame. Through this design, the decoding computation of the current frame is concentrated on regions that more urgently need updating. Figure 2 As shown, the time-series error-driven sparse decoding steps mainly include time-series feature fusion, error risk ranking, and local neural decoding.

[0042] For temporal feature fusion: Based on motion information, historical intermediate features are aligned to their corresponding positions in the current frame, enabling a consistent spatial correspondence between historical content and current frame intermediate features. Subsequently, the current frame intermediate features and historical intermediate features are temporally aggregated to form a compact representation containing temporal context, and the error risk estimation result for each pixel position in the current frame is output accordingly. This error risk reflects the relative confidence that the current position continues to depend on historical content, and is used to characterize the update priority of the current frame.

[0043] Error risk prioritization: After obtaining the error risk distribution, pixels or regions in the screen space are sorted according to error risk to determine the priority position for neural decoding in the current frame. Regions with higher error risk are added to the local update set of the current frame, while regions with lower error risk retain a higher proportion of historical state reuse. In this way, the range of neural decoding in the current frame is directly determined by the temporal error distribution, making the decoding overhead vary with the actual update requirements.

[0044] For local neural decoding: For a selected local region, the corresponding high-resolution intermediate features are entered into the local neural decoding process to generate the current frame color or illumination update results for these locations. The decoder adopts a position-by-position decoding form with shared parameters, enabling different screen locations to complete the current frame recovery under a unified decoding structure. Since neural decoding only operates on the selected local region, the computational cost of the color generation stage is directly related to the local update range, thus supporting continuous reduction of inference cost under the same model structure by dynamically adjusting the local update ratio according to the runtime budget or target frame time.

[0045] In practice, the input consists of high-resolution intermediate features of the current frame, historical intermediate features, and motion vectors. The system first aligns the historical intermediate features to their corresponding positions in the current frame based on the motion vectors, resulting in historical intermediate features that are consistent with the spatial position of the current frame. Then, the aligned historical intermediate features are combined with the high-resolution intermediate feature representation of the current frame to form a temporal feature representation that includes the temporal context.

[0046] After obtaining the temporal feature representation, the error risk distribution of each pixel position or local region in the current frame is output. The screen spatial positions are sorted according to the error risk to determine the local update region for which neural decoding is prioritized in the current frame. The local update region can be determined at the pixel level, or at the tile, block, or local area level.

[0047] After the local update region is determined, the high-resolution intermediate features of the current frame at the corresponding position are extracted and input into the local neural decoder with shared parameters to generate the local color or illumination update result of the current frame as the local decoding result.

[0048] The output of this step includes the error risk distribution or the selected local update region, as well as the local neural decoding result of the current frame, which are then fed into the temporal synthesis step based on the fusion intermediate features. The error risk distribution participates in the subsequent temporal fusion control, and the local neural decoding result is added as new content of the current frame to participate in the final complete restoration.

[0049] S4, Temporal synthesis based on fused intermediate features: The local decoding result, high-resolution intermediate features and historical intermediate features are temporally fused to obtain fused intermediate features. Motion information is recovered based on the fused intermediate features. The historical rendering frame and the local decoding result are fused based on the recovered motion information to obtain the current rendering frame.

[0050] After completing local neural decoding, historical intermediate features and high-resolution intermediate features of the current frame are temporally fused in a unified feature space. Motion information for temporal synthesis is then directly recovered based on the fused intermediate features. This recovered motion information is not presented as an independent pre-estimation result but is formed concurrently with the current frame feature representation during the temporal fusion process, describing the effective mapping relationship between historical content and the current frame. For example... Figure 2 As shown, the temporal synthesis steps based on fused intermediate features mainly include the construction of temporal fusion intermediate features, motion recovery based on fused intermediate features, and fusion of historical results with current results.

[0051] For the construction of intermediate features for temporal fusion: the high-resolution intermediate features of the current frame, historical intermediate features, and local decoding results jointly participate in the temporal fusion of the feature layer, forming a fused intermediate feature that simultaneously contains current structural information and historical context information. This fused intermediate feature retains the spatial structure and local update content of the current frame on the one hand, and the continuous information (i.e., context information) of historically drawn frames in the temporal dimension on the other hand, thus providing a unified foundation for subsequent temporal recovery.

[0052] Motion recovery based on fused intermediate features: Based on fused intermediate features, motion information used for temporal synthesis is further recovered, and combined with the local decoding results of the current frame, historical content is temporally aligned and fused. Since the recovered motion information comes directly from the fused intermediate features, it has higher consistency with the current frame structure, historical context, and local update regions, and can more accurately reflect the mapping relationship required for temporal recovery of the current frame.

[0053] After obtaining the recovered motion information, the historical content in the previous rendered frames is mapped to the current frame and fused with the local decoding results of the current frame to reconstruct the complete current rendered frame. For locations with strong temporal continuity and reliable historical content, the reuse of historical content is enhanced, meaning that historical content accounts for a high proportion in the current rendered frame. For locations with significant local updates or changes, the contribution of the local decoding results of the current frame is enhanced, meaning that the local decoding results of the current frame account for a high proportion in the current rendered frame. In this way, the local neural decoding results are expanded into the complete current frame image while maintaining good temporal continuity and structural stability.

[0054] In practice, the inputs are the high-resolution intermediate features of the current frame, historical rendered frames, historical intermediate features, local decoding results, and motion vectors. First, the high-resolution intermediate features of the current frame, historical intermediate features, and local decoding results are temporally fused in a unified feature space to form fused intermediate features. Then, motion information for temporal synthesis is directly recovered based on the fused intermediate features, and this motion information is used to map the historical rendered frames to the current frame position, obtaining aligned historical content. Next, the aligned historical content is fused with the local decoding results of the current frame to recover the current rendered frame (i.e., the complete image of the current frame). Finally, the complete current rendered frame image is output, along with the fused intermediate features or auxiliary states related to temporal recovery. The current rendered frame image is sent to the display buffer or subsequent rendering stage; the fused intermediate features and auxiliary states are sent to the current rendered frame output and historical state update steps for writing back to the historical buffer.

[0055] S5, Current Frame Output and History State Update: Output the current frame and update the history state by taking the high-resolution intermediate features, fused intermediate features, and the current frame as the current frame.

[0056] Step S5 takes as input the complete current frame image, the high-resolution intermediate features of the current frame, the fused intermediate features, and necessary timing auxiliary states. The current frame image is written to the display buffer as the final rendering output for this frame; simultaneously, the high-resolution intermediate features, fused intermediate features, and the current frame image are written to the history buffer to form the historical states required for the next frame. Then, the updated historical state set is output.

[0057] The historical state set is re-inputted at the beginning of the next frame to complete the current frame's geometric information generation and historical state preparation steps, forming a frame-by-frame cyclic data stream. Thus, the method of this invention forms a closed-loop processing flow of "current frame geometric attribute information generation and historical state preparation, adaptive hierarchical feature query, geometrically constrained intermediate feature super-resolution reconstruction, temporal error-driven sparse decoding, temporal synthesis based on fused intermediate features, and current frame output and historical state update."

[0058] like Figure 3 As shown, the embodiment also provides a global neural rendering system based on controllable neural optical transmission sparse inference, including: an adaptive hierarchical feature query module, a geometrically constrained intermediate feature super-resolution reconstruction module, a temporal error-driven sparse decoding module, a temporal synthesis module based on fused intermediate features, and a current rendering frame output and historical state update module. Among them, the adaptive hierarchical feature query module is used to determine the region complexity level based on the current frame geometric attribute information in the screen space and configure the corresponding query density and query level accordingly, and obtain the current frame intermediate features for low-cost query from the underlying neural illumination representation according to the query density and query level. The geometrically constrained intermediate feature super-resolution reconstruction module is used to perform super-resolution reconstruction of the intermediate features of the current frame using the geometric attribute information of the current frame as geometric constraints, thereby improving feature resolution, restoring the constraints of geometric boundaries, and fusing features at different levels to obtain high-resolution intermediate features of the current frame. Temporal error-driven sparse decoding is used to perform temporal feature fusion of the high-resolution intermediate features of the current frame and the aligned historical intermediate features, calculate the error risk distribution of each local region, and select local update regions with high error risk for neural decoding to obtain local decoding results. The temporal synthesis module based on fused intermediate features is used to temporally fuse local decoding results, high-resolution intermediate features, and historical intermediate features to obtain fused intermediate features. After recovering motion information based on the fused intermediate features, the historical rendering frames are fused with the local decoding results based on the recovered motion information to obtain the current rendering frame. The Current Frame Output and History State Update module is used to output the current frame and update the history state by taking the high-resolution intermediate features, fused intermediate features, and the current frame as the current frame.

[0059] It should be noted that the global neural mapping system device based on controllable neural optical transmission sparse inference provided in the above embodiments should be illustrated using the above-described functional module division as an example when performing global optical transmission neural mapping. The functions described above can be assigned to different functional modules as needed, that is, the internal structure of the terminal or server can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the global neural mapping system based on controllable neural optical transmission sparse inference provided in the above embodiments and the global neural mapping method embodiment based on controllable neural optical transmission sparse inference belong to the same concept. For details of its implementation process, please refer to the global neural mapping method embodiment based on controllable neural optical transmission sparse inference, which will not be repeated here.

[0060] Based on the same inventive concept, the embodiments also provide a computer-readable storage medium storing a program that, when executed by a processor, implements the above-described global neural rendering method based on sparse inference using controllable neural optical transmission, specifically including the following steps: S1, Adaptive hierarchical feature query: Based on the current frame geometric attribute information in the screen space, determine the region complexity level and configure the corresponding query density and query level accordingly. Obtain the current frame intermediate features for low-cost query from the underlying neural illumination representation according to the query density and query level. S2, Super-resolution reconstruction of intermediate features with geometric constraints: Using the geometric attribute information of the current frame as geometric constraints, super-resolution reconstruction is performed on the intermediate features of the current frame to improve feature resolution, restore the constraints of geometric boundaries, and fuse features at different levels to obtain high-resolution intermediate features of the current frame. S3, Temporal Error-Driven Sparse Decoding: After temporal feature fusion of the high-resolution intermediate features of the current frame and the aligned historical intermediate features, the error risk distribution of each local region is calculated, and the local update regions with high error risk are selected for neural decoding to obtain the local decoding result. S4, Temporal synthesis based on fused intermediate features: The local decoding result, high-resolution intermediate features and historical intermediate features are temporally fused to obtain fused intermediate features. Motion information is recovered based on the fused intermediate features. The historical rendering frame and the local decoding result are fused based on the recovered motion information to obtain the current rendering frame.

[0061] S5, Current Frame Output and History State Update: Output the current frame and update the history state by taking the high-resolution intermediate features, fused intermediate features, and the current frame as the current frame.

[0062] In this embodiment, the computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data.

[0063] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A global neural rendering method based on controllable neural optical transmission sparse inference, characterized in that, Includes the following steps: Adaptive hierarchical feature query: Based on the geometric attribute information of the current frame in screen space, the region complexity level is determined and the corresponding query density and query level are configured accordingly. The intermediate features of the current frame for low-cost query are obtained from the underlying neural illumination representation according to the query density and query level. Super-resolution reconstruction of intermediate features with geometric constraints: Using the geometric attribute information of the current frame as geometric constraints, super-resolution reconstruction is performed on the intermediate features of the current frame to improve feature resolution, restore the constraints of geometric boundaries, and fuse features at different levels to obtain high-resolution intermediate features of the current frame. Temporal error-driven sparse decoding: After temporal feature fusion of the high-resolution intermediate features of the current frame and the aligned historical intermediate features, the error risk distribution of each local region is calculated. Local update regions with high error risk are selected for neural decoding to obtain local decoding results. Temporal synthesis based on fused intermediate features: The local decoding results, high-resolution intermediate features and historical intermediate features are fused in a temporal manner to obtain fused intermediate features. Motion information is recovered based on the fused intermediate features. The historical rendering frames are then fused with the local decoding results based on the recovered motion information to obtain the current rendering frame.

2. The global neural rendering method based on controllable neural optical transmission sparse inference according to claim 1, characterized in that, The method for determining the region complexity level based on the current frame geometric attribute information of the screen space includes: dividing the screen space into local regions, and then analyzing the region complexity of the local regions based on the current frame geometric attribute information. The analysis is based on geometric boundaries, occlusion changes, texture detail changes, and lighting intensity changes to obtain the region complexity level that includes complex and simple regions.

3. The global neural rendering method based on controllable neural optical transmission sparse inference according to claim 1, characterized in that, Configure the corresponding query density and query level according to the region complexity level, including: Configure high query density and fine-grained neural illumination representation layers for complex regions; Configure low query density and coarse neural lighting representation hierarchy for simple regions.

4. The global neural mapping method based on controllable neural optical transmission sparse inference according to claim 1, characterized in that, Using the geometric attribute information of the current frame as geometric constraints, super-resolution reconstruction is performed on the intermediate features of the current frame, including: The geometric attribute information of the current frame includes the depth map, normal map, and albedo map; The intermediate features from different query levels of the current frame are fused together and input into the feature reconstruction module along with the geometric attribute information of the current frame. Under geometric constraints, resolution enhancement and detail restoration are performed. The depth map, normal map, and albedo map are used to constrain the feature restoration direction and boundary propagation range, so that the features remain separated at depth discontinuity boundaries, normal abrupt boundaries, and material boundaries, and suppress cross-surface feature mixing, forming high-resolution intermediate features for the current frame.

5. The global neural mapping method based on controllable neural optical transmission sparse inference according to claim 1, characterized in that, The error risk distribution of each local region is calculated by temporally fusing the high-resolution intermediate features of the current frame with the aligned historical intermediate features, including: Based on motion information, historical intermediate features are aligned to the corresponding positions in the current frame, so that historical content can establish a consistent spatial correspondence with the high-resolution intermediate features of the current frame. Then, the high-resolution intermediate features of the current frame and the historical intermediate features are aggregated in a temporal sequence. Based on temporal features, the error risk estimation results of each pixel position in the current frame are aggregated and output, thereby obtaining the error risk distribution of the local region. The error risk reflects the relative distortion risk when the current position continues to depend on historical content, and is used to characterize the update priority of the current frame.

6. The global neural rendering method based on controllable neural optical transmission sparse inference according to claim 1, characterized in that, High-risk local update regions are selected for neural decoding to obtain local decoding results, including: The screen spatial locations are sorted according to error risk, and locations with high error risk are selected as local update areas. The local update areas can be determined by pixel granularity, tile, block, or local area granularity. High-resolution intermediate features corresponding to the locally updated region are extracted and input into a local neural decoder with shared parameters to obtain the local decoding result.

7. The global neural rendering method based on sparse inference using controllable neural optical transmission according to claim 1, characterized in that, The current frame is obtained by fusing historical rendering frames with local decoding results based on the recovered motion information, including: The restored motion information is used to map the historical rendering frames to the current frame position to obtain the aligned historical content. Then, the aligned historical content is fused with the local decoding result of the current frame.

8. The global neural rendering method based on controllable neural optical transmission sparse inference according to claim 1, characterized in that, The method also includes current drawing frame output and history state update: output the current drawing frame and update the current drawing frame with the high-resolution intermediate features, fused intermediate features and the current drawing frame as history state.

9. A global neural mapping system based on controllable neural optical transmission sparse inference, characterized in that, include: The adaptive hierarchical feature query module is used to determine the region complexity level based on the current frame geometric attribute information in the screen space and configure the corresponding query density and query level accordingly. It obtains the current frame intermediate features for low-cost query from the underlying neural illumination representation according to the query density and query level. The geometrically constrained intermediate feature super-resolution reconstruction module is used to perform super-resolution reconstruction of the intermediate features of the current frame using the geometric attribute information of the current frame as geometric constraints, thereby improving feature resolution, restoring the constraints of geometric boundaries, and fusing features at different levels to obtain high-resolution intermediate features of the current frame. The temporal error-driven sparse decoding module is used to perform temporal feature fusion of the high-resolution intermediate features of the current frame and the aligned historical intermediate features, calculate the error risk distribution of each local region, select local update regions with high error risk for neural decoding, and obtain local decoding results. The temporal synthesis module based on fused intermediate features is used to temporally fuse local decoding results, high-resolution intermediate features, and historical intermediate features to obtain fused intermediate features. After recovering motion information based on the fused intermediate features, the historical rendering frames and local decoding results are fused based on the recovered motion information to obtain the current rendering frame.

10. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the global neural rendering method based on sparse inference of controllable neural optical transmission as described in any one of claims 1-8.

Images