Complex material-oriented multi-path full-frequency global illumination neural rendering method and system

Abstract

The invention discloses a complex-material-oriented multi-path full-frequency global illumination neural rendering method and system, and belongs to the technical field of computer graphics, and the method comprises the steps: carrying out the scene query of light emitted by pixels, and obtaining a primary geometric buffer and a multi-path mixed geometric buffer of the multi-time ejection of the light; respectively extracting main frequency features and multi-path sub-frequency features in a neural feature field based on the two features; performing directed graph structured modeling based on the multi-path sub-frequency features to obtain propagation features representing light path propagation information and global features representing overall light path information; performing multi-path frequency joint feature fusion based on the dominant frequency feature, the propagation feature and the global feature to generate a fusion feature with a full-frequency feature; and estimating a final radiation brightness value of the pixel based on the fusion feature, and completing neural rendering of global illumination. According to the method, efficient modeling and high-quality drawing of multi-path full-frequency global illumination under a complex material can be realized, and the sense of reality and dynamic consistency of a rendered image are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics technology, specifically relating to a neural rendering method and system for multi-path full-frequency global illumination of complex materials. Background Technology

[0002] In recent years, neural rendering technology has become a research hotspot in computer graphics, especially in achieving high-quality global illumination (GI), paving an important path to improve the rendering efficiency and realism of complex scenes. This type of method embeds neural networks into the rendering process, using learnable feature representations to fit complex lighting phenomena such as indirect lighting, reflection, and refraction, thereby avoiding the high computational overhead of traditional path tracing strategies. Current mainstream techniques include using spatial feature grids or neural radiance fields (NeRF) to construct a global illumination representation of the scene, thereby achieving learning-based rendering prediction for dynamic scenes.

[0003] The paper "Dual-Band Feature Fusion for Neural Global Illumination with Multi-Frequency Reflections" (Mo et al., SIGGRAPH 2025) utilizes the geometric features of single reflections to query deformable feature fields, thereby learning the effects of multi-frequency single reflections in a scene. Specifically, this method employs multiple deformable feature fields for dynamic objects and superimposes them at different spatial locations to capture the indirect lighting effects between objects; it obtains high-frequency reflection information by emitting single-reflection rays to query the feature fields, uses a feature fusion module for interpolation, and finally predicts the final radiance through a neural network. This method achieves high-quality global illumination modeling in dynamic scenes and, to some extent, solves the problem of fitting multi-frequency reflection effects. However, despite achieving a certain balance between efficiency and quality, this method still faces the following key technical bottlenecks: (1) Insufficient multi-frequency lighting effect processing mechanism: When dealing with the multi-frequency reflection characteristics of complex materials, existing methods are limited to the simple case of single reflection and do not clearly distinguish and integrate the multi-frequency global lighting components of multiple ejections. This leads to unstable performance in scenes with more complex materials, such as transparent materials, and it is difficult to simultaneously take into account the low frequency of rough surfaces and the high frequency details caused by mirror or smooth materials under high-order ejection conditions. As a result, it is impossible to restore the overall lighting accuracy and high frequency detail reproduction. In complex scenes where multiple frequency components coexist, it often leads to problems such as low frequency deviation, high frequency blurring or over-sharpening, and inconsistency of details.

[0004] (2) Lack of structured modeling for different optical paths: The unified direct surface feature field strategy is used to model all optical paths in the scene. The distinct light transmission behaviors such as specular reflection, rough reflection, refraction, and even multiple ejections are mixed in the same representation space for learning and inference. When representing multiple complex optical path types, due to the lack of differentiation and structured modeling of different optical path characteristics, the model is difficult to take into account the differences in the expression of various optical paths. This leads to problems such as blurred details, inaccurate boundaries, and even local distortion in areas with overlapping multiple paths or complex optical behaviors. This is particularly evident in scenes containing transparent materials, strong refractive interfaces, or high-order ejected rays.

[0005] Furthermore, the paper "Active Exploration for Neural Global Illumination of Variable Scenes" (Kwon et al., TOG 2022) improves the adaptability of neural rendering systems to complex conditions such as spatial occlusion and lighting changes by introducing guided sampling and active exploration mechanisms. Although this method shows certain advantages at the sampling strategy level, its feature representation is still mainly based on a single channel, focusing on improving the convergence efficiency of global illumination. It does not propose a systematic multipath lighting processing scheme for modeling and fusing full-frequency lighting components, thus having a significant limitation in its ability to express indirect lighting interactions between dynamic objects in complex scenes.

[0006] Therefore, existing technologies generally suffer from the following problems: a lack of structured modeling and fusion mechanisms for different optical path types; difficulty in simultaneously taking into account low-frequency global illumination with frequency differences and high-frequency specular reflection or refraction details; and susceptibility to artifacts or loss of detail in dynamic scenes. Against this backdrop, there is an urgent need for a neural global illumination method that can model different optical path types, clearly separate and fuse multi-frequency illumination features, and support high-quality rendering of dynamic scenes. Summary of the Invention

[0007] In view of the above, the purpose of this invention is to provide a neural rendering method and system for multi-path full-frequency global illumination for complex materials. By explicitly modeling the intrinsic structural relationship between light paths, it distinguishes and efficiently compresses multiple ejected light paths involving both reflection and refraction, as well as lighting characteristics with different frequency distributions such as diffuse reflection, glossy reflection, and specular reflection. This achieves accurate modeling and stable rendering of complex multi-path full-frequency global illumination, improving the detail realism and dynamic consistency of global illumination neural rendering in complex dynamic scenes.

[0008] To achieve the above-mentioned objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a neural rendering method for multi-path full-frequency global illumination of complex materials, comprising the following steps: The initial geometric buffer and the multi-path hybrid geometric buffer of multiple ray bounces are obtained by scene query for the emitted light of the pixel. Based on the two, the main frequency feature and the multi-path secondary frequency feature are extracted in the neural feature field respectively. Directed graph structure modeling is performed based on multi-path secondary frequency characteristics to characterize the propagation relationship of multiple ejection optical paths, thereby obtaining propagation characteristics that characterize optical path propagation information and global characteristics that characterize overall optical path information. Multi-path frequency joint feature fusion is performed based on the main frequency feature, propagation feature and global feature to generate fused feature with full frequency characteristics; The final radiance value of the pixel is estimated based on the fused features, and the neural rendering of global illumination is completed.

[0009] Preferably, the step of obtaining the initial geometric buffer and the multi-path hybrid geometric buffer for multiple ray bounces by performing scene query for pixel-emitted rays includes: The emitted ray intersects with the scene, and the initial intersection point and the corresponding initial geometric buffer are obtained; Based on the material information at the initial intersection point, light rays are emitted along the direction of mirror reflection or refraction to intersect with the scene, obtaining one or more ejection intersection points and the corresponding multipath hybrid geometry buffer.

[0010] Preferably, the extraction of primary frequency features and secondary frequency features from the neural feature field based on primary geometric buffering and multi-path hybrid geometric buffering includes: Based on the initial intersection position of the initial geometric buffer in the scene, the dominant frequency feature used to characterize the low-frequency global illumination effect is obtained by sampling in the neural feature field. Based on the sampling of the locations of each ejection intersection point in the scene using the multipath hybrid geometric buffer, multipath secondary frequency features for characterizing high-frequency reflection or refraction effects are obtained in the neural feature field.

[0011] Preferably, the directed graph structure modeling based on multi-path secondary frequency features to characterize the propagation relationship of multiple ejected optical paths, obtaining propagation features characterizing optical path propagation information and global features characterizing overall optical path information, includes: The initial intersection point of the initial geometry buffer in the scene is taken as the root node, and the ejection intersection points of the multipath hybrid geometry buffer in the scene are taken as child nodes. Based on the reflection or transmission events in the light transmission process, a directed graph structure is constructed to characterize the propagation relationship of multiple ejection light paths. Based on the directed graph structure, the features of child nodes are aggregated to the parent node layer by layer to obtain the propagation features; Based on the directed graph structure, global features are obtained by aggregating the features of all nodes globally.

[0012] Preferably, the directed graph structure is specifically constructed as a path tree structure, where each node represents an interaction between light and the surface of an object in the scene, each directed edge represents the process of light propagating from the parent node to the child node, and all paths together constitute a hierarchical branch sequence starting from the root node.

[0013] Preferably, the step of fusing multi-path frequency joint features based on the main frequency feature, propagation feature, and global feature to generate a fused feature with full-frequency characteristics includes: The main frequency features, propagation features and global features are input into the fusion network to generate filter convolution kernels and weighted interpolation weights at multiple resolutions. The propagation features are filtered using a multi-resolution convolution kernel and then weighted and mixed to obtain the filtered multipath secondary frequency features. The primary frequency feature and the filtered multipath secondary frequency feature are weighted and interpolated according to the weighted interpolation weights to generate the fused feature.

[0014] Preferably, the fusion network includes: The encoder is used to encode the main frequency features, propagation features and global features of the input at multiple scales. The decoder, which skips connections with the encoder, is used to fuse multi-scale features and dynamically generate filter convolution kernels and weighted interpolation weights at multiple resolutions. The weighted interpolation weights are three-channel weights, which correspond to the filtered multipath secondary frequency features, primary frequency features, and zero vector, respectively. After normalization, they are used to perform weighted interpolation on the filtered multipath secondary frequency features and primary frequency features.

[0015] Secondly, embodiments of the present invention provide a neural rendering system for multi-path full-frequency global illumination of complex materials, which is implemented using the aforementioned neural rendering method for multi-path full-frequency global illumination of complex materials, including: a geometric buffer query and feature extraction module, a multi-path feature structured modeling module, a multi-path frequency joint feature fusion module, and a radiance prediction and neural rendering module. The geometric buffer query and feature extraction module is used to perform scene query for the emitted light from the pixel to obtain the initial geometric buffer and the multi-path hybrid geometric buffer of the light ricocheting multiple times, and extract the main frequency feature and the multi-path secondary frequency feature in the neural feature field based on the two respectively. The multi-path feature structured modeling module is used to perform directed graph structured modeling based on multi-path secondary frequency features to characterize the propagation relationship of multiple ejection optical paths, and obtain the propagation features characterizing the propagation information of the optical path and the global features characterizing the overall optical path information. The multi-path frequency joint feature fusion module is used to perform multi-path frequency joint feature fusion based on the main frequency feature, propagation feature and global feature to generate fused features with full frequency characteristics; The radiance prediction and neural rendering module is used to estimate the final radiance value of the pixel based on the fusion features, and to complete the neural rendering of global illumination.

[0016] Thirdly, embodiments of the present invention provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described neural rendering method for multipath full-frequency global illumination of complex materials when executing the computer program.

[0017] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the aforementioned neural rendering method for multipath full-frequency global illumination of complex materials.

[0018] Compared with the prior art, the beneficial effects of the present invention include at least the following: (1) Enhance the ability of multi-path full-frequency global illumination modeling: By separating, extracting and fusing the main frequency features and the secondary frequency features of multiple paths, this invention can effectively capture and reconstruct low-frequency illumination phenomena (such as diffuse reflection and rough reflection) and high-frequency illumination phenomena (such as specular reflection and fine specular highlights) that coexist in complex scenes in a variety of different light paths, significantly improving the ability of the neural rendering system to express multi-frequency effects under complex light paths, and effectively improving the shortcomings of existing methods that are difficult to take into account different frequency features under various light paths.

[0019] (2) Improved visual quality and detail representation of rendered images: After multi-path frequency joint feature fusion, the fidelity of the generated image is significantly improved in detail edges, mirror areas, and high-contrast reflection and refraction surfaces, which can effectively reduce common artifacts such as high-frequency distortion, blurring, and aliasing. Experiments in multiple dynamic scenes (such as moving objects and changing viewpoints) show that compared with similar neural rendering methods, it improves the PSNR index by about 6.01dB and the SSIM index by about 5.4%, and visually retains more realistic reflection and refraction details, thus improving the overall visual quality of the rendered image.

[0020] (3) Enhanced dynamic consistency and temporal stability: Through the synergistic effect of structured modeling and multi-path frequency joint feature fusion, this invention can effectively reduce instability in the feature learning process, reduce mutual interference between different frequency illumination components, and enable the fused features to have good temporal consistency between consecutive frames. In dynamic scenes, this invention can significantly suppress temporal noise fluctuations and improve the stability of video output.

[0021] (4) Good balance between efficiency and accuracy: This invention achieves accurate modeling of complex global illumination effects under the premise of limited number of ray queries. Compared with the traditional path tracing combined with neural post-processing, it can significantly reduce the consumption of computing resources. Under the same time overhead, the PSNR index is improved by an average of 2.53dB, which has good practicality and scalability.

[0022] (5) Broad application prospects: This invention can be widely applied to real-time rendering, virtual reality, film special effects, industrial simulation and new neural rendering platforms, especially suitable for scenarios that require high-quality reflection effects under limited computing power. Combined with current mainstream graphics hardware, this method has achieved real-time rendering speed on the NVIDIA RTX 4090 platform, and has good engineering feasibility and commercial potential. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the neural rendering method for multi-path full-frequency global illumination of complex materials provided in this embodiment of the invention. Figure 2 This is a schematic diagram of the overall framework of the neural rendering method for multi-path full-frequency global illumination for complex materials provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of multi-path feature structured modeling provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the multi-path frequency joint feature fusion module provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a neural rendering system for multi-path full-frequency global illumination of complex materials provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0026] Existing methods generally rely on single-band feature modeling strategies, making it difficult to effectively capture the differences between indirect lighting on rough surfaces and multi-frequency reflection phenomena such as specular reflection under multiple ejected light paths. This leads to problems such as blurriness, missing details, or inconsistencies in the rendering results, especially with unstable effects under dynamic viewpoint changes and complex light paths. For example, our research team's previous research (publication number CN120726204A, hereinafter referred to as Patent Document 1) provides a neural rendering method for global full-frequency lighting of complex materials. It only uses the initial geometric buffer and single-reflection geometric buffer to extract scene feature information, and then mixes and predicts the final rendering result through a dual-frequency fusion module. This method supports simple single-reflection light transmission, and the feature fusion task only involves two features: the dominant frequency and the secondary frequency. The processing difficulty is relatively low, but its applicable scenarios are greatly limited. It only supports reflective materials, and the number of ejected reflected light is limited to a single time. Therefore, the global lighting effect is very limited (for example, the effect of two metal spheres reflecting each other cannot be predicted). However, due to the presence of complex optical path phenomena such as refraction, transmission, multiple reflections, and transparent materials (such as glass) in real dynamic scenes, Patent Document 1 is unable to cope with the new challenges brought about by these complex scenes: it cannot handle the query of light rays in the direction of refraction, cannot model the optical path dependency relationship of multiple ejections, and cannot effectively fuse complex features of multiple paths and multiple frequencies.

[0027] In view of the above, the inventive concept of this invention is as follows: Addressing the shortcomings of existing neural global illumination methods in multi-path full-frequency reflection modeling, such as insufficient expressive power, inadequate detail restoration, and poor dynamic consistency, this invention provides a neural rendering method and system for multi-path full-frequency global illumination of complex materials. Based on the overall framework of Patent Document 1, it expands ray querying from single reflection to multi-path hybrid querying. Furthermore, focusing on the unique technical challenge of structured modeling of multi-path secondary frequency features, it proposes a structured encoding and fusion technology based on directed graphs / path trees. By explicitly modeling the intrinsic structural relationships between ray paths, it distinguishes and expresses multiple ejected ray paths involving both reflection and refraction, as well as illumination characteristics with different frequency distributions such as diffuse reflection, glossy reflection, and specular reflection. This constructs an efficient representation and compression mechanism for the multi-path feature space, thereby improving the accuracy and stability of global illumination simulation in complex dynamic scenes.

[0028] like Figure 1 and Figure 2 As shown, the embodiment provides a neural rendering method for multi-path full-frequency global illumination of complex materials, including the following steps: S1 performs scene query to obtain the initial geometric buffer and the multi-path hybrid geometric buffer for multiple ray bounces for the pixel emitted light, and extracts the main frequency feature and multi-path secondary frequency feature in the neural feature field based on the two respectively.

[0029] In this embodiment, scene configuration and neural feature field construction are performed first. A dynamic 3D scene is used as input, containing multiple moving objects, complex geometry, and various materials (including reflective materials such as diffuse, glossy, and specular reflections, as well as refractive materials with different roughness). All data is preprocessed to generate training samples, including camera and object dynamic parameters, multiple geometric buffers (including the initial and subsequent multipath hybrid geometric buffers), and realistically rendered images. Each object in the scene is assigned a unique identifier, and a corresponding neural feature field is constructed centered on each object to implicitly represent the global illumination characteristics in the 3D scene. The feature field consists of multi-resolution three-plane encoding structures of different sizes, used to store the learned spatial feature representations. This feature field is parameterized as a set of learnable tensors, with initial values ​​of random noise satisfying a certain distribution, and subsequently optimized through backpropagation training. The feature mesh transforms along with the object's movement, thereby maintaining the relative position of external queries aligned in object space.

[0030] Then, geometry buffer lookup and feature extraction are performed. In each training or inference step, a ray originating from the camera is emitted for each rendered pixel, its initial intersection with the scene is calculated, and the ray is emitted again based on the intersection material information, and the intersection is calculated again. This process is repeated multiple times to form different multiple bounce paths. The path from the initial intersection is used to collect the initial geometry buffer, while the paths from subsequent bounces collect multi-path hybrid geometry buffers (including reflection geometry buffers or refraction geometry buffers, or results from emitting light from the intersection based on specular reflection or refraction directions and querying scene information using a similar method). Figure 2 As shown on the left, the intersection positions of each path are transformed to the local coordinate space of the dynamic object, and spatial features are sampled in the corresponding feature field. The results of multi-object query feature fields are superimposed and encoded to generate features corresponding to various different paths: initial geometric buffer. Corresponding main frequency characteristics Designed to sense low-frequency spatial changes on a large scale, capturing indirect lighting from diffuse reflection and rough materials; the remaining multipath hybrid geometry buffers Corresponding multi-path secondary frequency features The design aims to perceive sparsely distributed high-frequency details along various paths, focusing on local details captured by multiple ejections, such as reflective edges and specular highlights.

[0031] This step uses an initial geometric buffer and a multi-path hybrid geometric buffer to emit secondary rays in the direction of reflection and refraction, and recursively performs multiple ejection queries to obtain rich optical path information with different depths and ejection types. This is the key to solving the problem that Patent Document 1 cannot predict more complex optical paths (such as multiple reflections, multiple refractions, and total internal reflection inside transparent objects).

[0032] S2, based on the multi-path secondary frequency characteristics, a directed graph structure model is performed to characterize the propagation relationship of multiple ejected optical paths, thereby obtaining the propagation characteristics that characterize the propagation information of the optical path and the global characteristics that characterize the overall optical path information.

[0033] Introducing a multi-path hybrid geometric buffer introduces a new problem entirely absent in Patent Document 1: multiple secondary frequency features originate from different depths and types of ejection paths, exhibiting complex dependencies and hierarchical structures. Simple dual-frequency weighted fusion cannot effectively utilize this structured information. This invention proposes a specific solution to this new problem: structured modeling of multi-path secondary frequency features.

[0034] In the embodiments, such as Figure 3 As shown on the left, the initial geometry buffer obtained from the previous query is... and multipath hybrid geometry buffer As a complex optical path propagation clue for multiple ejections, the secondary frequency characteristics of the multi-path are analyzed. Structured coding modeling is performed. First, the initial intersection point of the initial geometry buffer in the scene is taken as the root node, and the ejection intersection points of the multi-path hybrid geometry buffer in the scene are taken as child nodes. Based on the laws of light transmission, ray paths are recursively generated in the scene. For each surface interaction, only two basic scattering events, reflection and transmission, are allowed. All possible ray paths can be organized into a hierarchical structure during propagation, where each path corresponds to a branch sequence starting from the root node.

[0035] By recording the above propagation process, a hierarchical set of paths is constructed. By structurally connecting the path nodes and their propagation relationships, a directed graph structure can be obtained. Each node represents an interaction between a ray and a surface; each directed edge represents the propagation of a ray from its parent node to its child node; all paths together form a path tree structure with the first intersection point as the root node. Subsequently, a bottom-up aggregation encoding process is used to propagate information along the tree structure to integrate the dependencies between paths. Specifically, for each node in the path tree… First, extract the local geometric properties of the node and its corresponding secondary frequency features, and then combine them with the path features of its reflected child nodes. Path features of refraction sub-nodes Input a lightweight propagation coding unit to predict the path features of the current node. This process aggregates child node features layer by layer towards the parent node to encode the impact of subsequent paths on the current node. This ensures that each node's features not only contain local information but also encode its contextual relationship within the overall path structure. After propagation to the root node, the predicted output features are... This can represent the result information of the current optical path propagation. For example... Figure 2 As shown on the right, after completing node-level encoding, the entire path tree needs to be globally aggregated to obtain a unified path feature representation. Preferably, this can be achieved by introducing virtual global nodes, accumulating all node features, and inputting them into the global encoder to obtain global features. This represents the overall optical path transmission information.

[0036] This step, by encoding the tree structure, can achieve a compact representation of multi-path optical transmission information while maintaining the dependencies between paths, thereby improving the efficiency and accuracy of subsequent feature modeling and reconstruction processes.

[0037] S3 performs multi-path frequency joint feature fusion based on the main frequency feature, propagation feature and global feature to generate fused features with full frequency characteristics.

[0038] Based on the above-mentioned main frequency characteristics, propagation characteristics and global characteristics, this invention further proposes a multi-path frequency joint feature fusion strategy to overcome the limitations of the traditional dual-frequency fusion method in Patent Document 1 when processing multi-path and multi-frequency mixed illumination information, and finally generate full-frequency fusion features that have both low-frequency illumination consistency and high-frequency reflection and refraction details.

[0039] In the embodiments, such as Figure 4 As shown, the main frequency characteristics with lower frequency band features are... The global features obtained from the above multi-path feature structure modeling and propagation characteristics The input is fed into the multipath frequency joint feature fusion module. The structural design of this module is as follows: The backbone employs a U-Net-style convolutional neural network structure to encode and decode all input features. It uses geometric buffers and features with different path characteristics and frequencies as input to predict convolutional kernels at three different resolutions. These kernels are then used to filter and weightedly mix the propagation features at the three different resolutions to obtain the filtered multi-path secondary frequency features. Furthermore, the aforementioned convolutional neural network also predicts a three-channel interpolation weight, representing the weights of the second-frequency features of the filtered multi-path when fusing the filtered features. Weights of the main frequency feature and zero vector weights These weights, after normalization, are used to interpolate between the primary frequency features and the filtered multipath secondary frequency features to obtain the final fused features. The calculation method is as follows: This fusion feature contains multi-path full-frequency information, which can simultaneously maintain the consistency of high-frequency details and low-frequency illumination, thereby significantly improving the realism and detail reproduction of the final image.

[0040] S4 estimates the final radiance value of the pixel based on the fused features, and completes the neural rendering of global illumination.

[0041] In the embodiment, the fusion features With the initial geometric buffer The albedo in the image is input together into a lightweight final decoder. This is used to predict the final radiance value of the pixel.

[0042] Furthermore, considering the semantic and distributional differences between dominant and secondary frequency features, a two-stage training strategy is adopted during training. The training objective function is a multi-component loss function, including mean absolute error L1 and structure-aware error SSIM. The training phase is divided into two consecutive stages: The first stage is low-frequency training, which uses only networks with low-frequency characteristics related to the main frequency features to train all neural feature fields and the final decoder. The goal is to accurately reproduce the low-frequency lighting structure of the scene, stabilize network convergence, and stably learn the overall lighting distribution. The second stage involves multi-frequency training. A series of multi-path secondary frequency features with high-frequency characteristics are introduced to form a multi-frequency dataset. Simultaneously, the parameters of the propagation coding unit and multi-path feature fusion module in structured modeling are unlocked to enhance the multi-path secondary frequency features. These are then used to jointly train and fine-tune the entire network, enabling it to gradually learn high-frequency details and their dynamic consistency across different perspectives. This strategy significantly reduces gradient instability in the early stages of training, improving the final rendering quality and robustness.

[0043] In summary, the neural rendering method for multi-path full-frequency global illumination of complex materials provided in this invention has significant advantages over existing technologies. Most existing technologies rely solely on initial geometric buffers and single-reflection geometric buffers, limiting themselves to low-frequency and high-frequency reflection modeling of single ejections. They cannot effectively handle complex optical path scenes involving refraction, transmission, multiple reflections, and even transparent materials, and lack modeling capabilities for transparent materials with refractive behavior, such as glass, and global illumination effects involving total internal reflection. This invention introduces a multi-path hybrid geometric buffer, systematically collecting multiple ejection optical path information with different depths and ejection types (reflection and refraction), performing unified structured modeling of complex light transmission processes, and achieving deep fusion of multi-path full-frequency features. This enables the invention to accurately predict complex global illumination effects that existing methods (Patent Document 1) cannot handle, such as multiple reflections, multiple refractions, and total internal reflection within transparent objects. It is particularly suitable for dynamic 3D scenes containing transparent materials such as glass, refractive interfaces, and high-order ejected rays.

[0044] Based on the same inventive concept, such as Figure 5 As shown, this embodiment of the invention also provides a neural rendering system 500 for multi-path full-frequency global illumination of complex materials, including: a geometric buffer query and feature extraction module 510, a multi-path feature structured modeling module 520, a multi-path frequency joint feature fusion module 530, and a radiance prediction and neural rendering module 540.

[0045] The geometric buffer query and feature extraction module 510 is used to perform scene query for the emitted light from the pixel to obtain the initial geometric buffer and the multi-path hybrid geometric buffer of the light ricocheting multiple times, and extract the main frequency feature and the multi-path secondary frequency feature in the neural feature field based on the two respectively.

[0046] The multi-path feature structured modeling module 520 is used to perform directed graph structured modeling based on multi-path secondary frequency features to characterize the propagation relationship of multiple ejected optical paths, and obtain the propagation features that characterize the propagation information of the optical path and the global features that characterize the overall optical path information.

[0047] The multi-path frequency joint feature fusion module 530 is used to perform multi-path frequency joint feature fusion based on the main frequency feature, propagation feature and global feature to generate fused features with full frequency characteristics.

[0048] The radiance prediction and neural rendering module 540 is used to estimate the final radiance value of the pixel based on the fusion features, and complete the neural rendering of global illumination.

[0049] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described neural rendering method for multipath full-frequency global illumination of complex materials when executing the computer program.

[0050] Based on the same inventive concept, embodiments of the present invention provide a computer-readable storage medium storing a computer program. When the computer program is executed by a computer, the above-described neural rendering method for multipath full-frequency global illumination of complex materials is implemented.

[0051] It should be noted that the neural rendering system, electronic device, and computer-readable storage medium for multi-path full-frequency global illumination of complex materials provided in the above embodiments all belong to the same inventive concept as the neural rendering method for multi-path full-frequency global illumination of complex materials. The specific implementation process is detailed in the embodiments of the neural rendering method for multi-path full-frequency global illumination of complex materials, and will not be repeated here.

[0052] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A neural rendering method for multi-path full-frequency global illumination of complex materials, characterized in that, Includes the following steps: The initial geometric buffer and the multi-path hybrid geometric buffer of multiple ray bounces are obtained by scene query for the emitted light of the pixel. Based on the two, the main frequency feature and the multi-path secondary frequency feature are extracted in the neural feature field respectively. Directed graph structure modeling is performed based on multi-path secondary frequency characteristics to characterize the propagation relationship of multiple ejection optical paths, thereby obtaining propagation characteristics that characterize optical path propagation information and global characteristics that characterize overall optical path information. Multi-path frequency joint feature fusion is performed based on the main frequency feature, propagation feature and global feature to generate fused feature with full frequency characteristics; The final radiance value of the pixel is estimated based on the fused features, and the neural rendering of global illumination is completed.

2. The neural rendering method for multi-path full-frequency global illumination of complex materials according to claim 1, characterized in that, The process of obtaining the initial geometric buffer and the multi-path hybrid geometric buffer for multiple ray bounces by pixel-emitted rays through scene querying includes: The emitted ray intersects with the scene, and the initial intersection point and the corresponding initial geometric buffer are obtained; Based on the material information at the initial intersection point, light rays are emitted along the direction of mirror reflection or refraction to intersect with the scene, obtaining one or more ejection intersection points and the corresponding multipath hybrid geometry buffer.

3. The neural rendering method for multi-path full-frequency global illumination of complex materials according to claim 1 or 2, characterized in that, The extraction of dominant frequency features and multi-path secondary frequency features from the neural feature field based on primary geometric buffering and multi-path hybrid geometric buffering includes: Based on the initial intersection position of the initial geometric buffer in the scene, the dominant frequency feature used to characterize the low-frequency global illumination effect is obtained by sampling in the neural feature field. Based on the sampling of the locations of each ejection intersection point in the scene using the multipath hybrid geometric buffer, multipath secondary frequency features for characterizing high-frequency reflection or refraction effects are obtained in the neural feature field.

4. The neural rendering method for multi-path full-frequency global illumination of complex materials according to claim 1 or 2, characterized in that, The directed graph structure modeling based on multi-path secondary frequency characteristics is used to characterize the propagation relationship of multiple ejection optical paths, resulting in propagation features characterizing optical path propagation information and global features characterizing overall optical path information, including: The initial intersection point of the initial geometry buffer in the scene is taken as the root node, and the ejection intersection points of the multipath hybrid geometry buffer in the scene are taken as child nodes. Based on the reflection or transmission events in the light transmission process, a directed graph structure is constructed to characterize the propagation relationship of multiple ejection light paths. Based on the directed graph structure, the features of child nodes are aggregated to the parent node layer by layer to obtain the propagation features; Based on the directed graph structure, global features are obtained by aggregating the features of all nodes globally.

5. The neural rendering method for multi-path full-frequency global illumination of complex materials according to claim 4, characterized in that, The directed graph structure is specifically constructed as a path tree structure, where each node represents an interaction between light and the surface of an object in the scene, each directed edge represents the process of light propagating from the parent node to the child node, and all paths together constitute a hierarchical branch sequence starting from the root node.

6. The neural rendering method for multi-path full-frequency global illumination of complex materials according to claim 1, characterized in that, The method of fusing multi-path frequency joint features based on dominant frequency features, propagation features, and global features to generate fused features with full-frequency characteristics includes: The main frequency features, propagation features and global features are input into the fusion network to generate filter convolution kernels and weighted interpolation weights at multiple resolutions. The propagation features are filtered using a multi-resolution convolution kernel and then weighted and mixed to obtain the filtered multipath secondary frequency features. The primary frequency feature and the filtered multipath secondary frequency feature are weighted and interpolated according to the weighted interpolation weights to generate the fused feature.

7. The neural rendering method for multi-path full-frequency global illumination of complex materials according to claim 6, characterized in that, The fusion network includes: The encoder is used to encode the main frequency features, propagation features and global features of the input at multiple scales. The decoder, which skips connections with the encoder, is used to fuse multi-scale features and dynamically generate filter convolution kernels and weighted interpolation weights at multiple resolutions. The weighted interpolation weights are three-channel weights, which correspond to the filtered multipath secondary frequency features, primary frequency features, and zero vector, respectively. After normalization, they are used to perform weighted interpolation on the filtered multipath secondary frequency features and primary frequency features.

8. A neural rendering system for multi-path full-frequency global illumination of complex materials, implemented using the neural rendering method for multi-path full-frequency global illumination of complex materials as described in any one of claims 1 to 7, characterized in that, include: The module includes a geometric buffer query and feature extraction module, a multi-path feature structured modeling module, a multi-path frequency joint feature fusion module, and a radiance prediction and neural rendering module. The geometric buffer query and feature extraction module is used to perform scene query for the emitted light from the pixel to obtain the initial geometric buffer and the multi-path hybrid geometric buffer of the light ricocheting multiple times, and extract the main frequency feature and the multi-path secondary frequency feature in the neural feature field based on the two respectively. The multi-path feature structured modeling module is used to perform directed graph structured modeling based on multi-path secondary frequency features to characterize the propagation relationship of multiple ejection optical paths, and obtain the propagation features characterizing the propagation information of the optical path and the global features characterizing the overall optical path information. The multi-path frequency joint feature fusion module is used to perform multi-path frequency joint feature fusion based on the main frequency feature, propagation feature and global feature to generate fused features with full frequency characteristics; The radiance prediction and neural rendering module is used to estimate the final radiance value of the pixel based on the fusion features, and to complete the neural rendering of global illumination.

9. An electronic device comprising a memory and one or more processors, the memory for storing a computer program, characterized in that, The processor is used to implement the neural rendering method for multipath full-frequency global illumination for complex materials as described in any one of claims 1 to 7 when executing a computer program.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the neural rendering method for multipath full-frequency global illumination for complex materials as described in any one of claims 1 to 7.

Images