A general neural rendering method and system based on illumination function generation model

Abstract

The application discloses a general neural rendering method and system based on an illumination function generation model, and belongs to the technical field of computer graphics. The method comprises the following steps: collecting light source information containing multi-view observation data and collecting scene information; constructing an illumination function generation model containing a light source encoding module and a light source decoding module, which is used for converting the light source information into a neural illumination representation, and jointly inferring based on the neural illumination representation and scene features of a rendering point; and generating a final rendering image conforming to a real illumination distribution based on the inference result. The application can realize general neural rendering across light sources and scenes, can generate images with offline rendering quality at a near real-time computing cost, and has good expansibility and wide application value by taking into account the sense of reality, stability and rendering efficiency.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics technology, specifically relating to a general neural rendering method and system based on a lighting function generation model. Background Technology

[0002] In recent years, neural rendering, as an important research direction at the intersection of computer graphics and computer vision, has gradually become a core technology driving the integration of intelligent graphics generation and realistic rendering. Neural rendering introduces deep learning models, enabling the system to automatically learn the complex relationships between geometry, materials, and lighting, thereby achieving efficient and realistic image generation. Traditional rendering pipelines rely on explicit physical modeling methods, which have good controllability and versatility, but suffer from high computational costs and limited rendering efficiency when dealing with highly complex scenes containing subtle geometry, complex reflections, or global illumination. Neural rendering, leveraging the advantages of deep neural networks in feature extraction and nonlinear fitting, transforms the image generation process into an end-to-end learning task, significantly improving image realism and cross-viewpoint consistency, allowing rendering systems to obtain near-physically realistic results at a lower computational cost.

[0003] In modern graphics engines, the application of deep learning technology is no longer limited to image recognition and generation, but is gradually integrated into various key stages of the traditional rendering process to enhance image performance and operational efficiency. For example, super-resolution technologies (such as DLSS and FSR) use deep learning models to reconstruct high-resolution images from low-resolution rendering results. By combining auxiliary information such as depth, normals, and motion vectors, they can restore rich high-frequency details and temporal consistency while reducing rendering overhead. Neural network denoising technology is widely used in low-sampling rendering such as path tracing and global illumination. By learning the statistical laws of illumination distribution, it effectively removes Monte Carlo noise and significantly shortens the convergence time of high-quality images. Neural material generation (Material Synthesis) uses neural networks to infer surface reflection properties, such as albedo, roughness, and normal maps, from real-world images to achieve automated material reconstruction and style-consistent content synthesis. View interpolation technology generates intermediate viewpoints or interpolated frame results based on neural reprojection and completion methods, significantly improving the smoothness and dynamic performance of camera motion. These methods are typically embedded as modular components in the rendering pipeline, achieving a dual improvement in performance and visual quality without altering the core rendering architecture.

[0004] Unlike the aforementioned "enhanced" deep learning techniques, neural rendering methods aim to directly learn rendering equations, that is, to predict the final radiance of pixels from scene input under specific conditions using neural networks. This type of method abandons the explicit modeling of complex BRDFs and lighting equations in traditional rendering, instead utilizing the network's nonlinear approximation capabilities to directly fit the light transfer function. Compared to physically analytical models, neural rendering equations can naturally express high-dimensional, nonlinear lighting effects and adapt to complex material variations and indirect lighting phenomena, thus exhibiting stronger representational capabilities in specific scenes. As a learnable alternative module to traditional rendering pipelines, neural rendering methods can efficiently approximate computationally expensive rendering effects such as global illumination and soft shadows, providing a new acceleration pathway for real-time rendering.

[0005] However, while existing neural rendering methods demonstrate good performance in specific tasks and scenes, they still have significant limitations. On the one hand, they typically rely on specific datasets for training, making it difficult to maintain stable output under unseen lighting, material, or viewpoint conditions. On the other hand, their network architecture designs are mostly optimized for single tasks or fixed input distributions, lacking the ability to generalize to different scenes and lighting conditions. Therefore, to address these issues, it is necessary to propose a novel neural rendering framework with generalization and reusability. This framework should retain the advantages of nonlinear fitting in neural networks while introducing parameterizable rendering function expressions, incorporating lighting and scene modeling into a unified generation mechanism. Summary of the Invention

[0006] In view of the above, the purpose of this invention is to provide a general neural rendering method and system based on a lighting function generation model. The model can not only learn complex lighting distribution and reflection behavior, but also adaptively adjust under different scenes and light source configurations, realizing generalized neural rendering across light sources and scenes. This provides a key path for the evolution of neural rendering technology from a task-driven method to a universal rendering system, and lays the foundation for the future development of intelligent graphics rendering.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides a general neural rendering method based on a lighting function generation model, comprising the following steps:

[0009] Collect light source information and scene information that include multi-view observation data;

[0010] A lighting function generation model containing a light source encoding module and a light source decoding module is constructed to convert light source information into a neural lighting representation and to perform joint inference based on the neural lighting representation and scene features of the drawing points;

[0011] The final rendered image is generated based on the inference results, conforming to the actual lighting distribution.

[0012] Preferably, the acquisition of light source information including multi-view observation data includes:

[0013] The target light source is observed and sampled from multiple perspectives using an offline rendering engine. The collected data includes the radiance, position information, and direction information of the light source from each perspective, in order to describe the shape, material, and spatial distribution characteristics of the light source.

[0014] Preferably, the collected scene information includes:

[0015] The real-time rendering engine acquires geometric buffer data of the scene from the camera's perspective, including normal maps, depth maps, reflectivity maps, roughness maps, and diffuse color maps, to characterize the geometric structure and surface optical properties of objects in the scene.

[0016] Simultaneously, geometric buffer data from the perspective of the light source in the target scene are acquired, including normal maps, depth maps, and diffuse color maps, to characterize the spatial occlusion relationship and energy interaction characteristics between the light source and the scene.

[0017] Preferably, the light source information is converted into a neural illumination representation through a light source encoding module, including:

[0018] The light source information observed from multiple perspectives is normalized by taking the maximum value in the R, G, and B channels to obtain the multi-view observation image, and the maximum brightness value of each channel is recorded as the illumination intensity factor.

[0019] The normalized multi-view observation image is divided into several multi-view observation slices. Each slice is expanded into a high-dimensional vector and then mapped to the neural space through a multilayer perceptron to obtain multiple illumination morphemes.

[0020] Lighting terms are input into a 4D Transformer for joint modeling in view space and image space to restore the global consistency and spatial correlation of lighting, resulting in modeled lighting terms.

[0021] The modeled illumination lexical units are projected onto the neural position field by a light field generator based on a cross-attention mechanism, and reconstructed into a three-plane neural illumination representation of the neural illumination field.

[0022] Preferably, the light source decoding module performs joint inference based on the neural lighting representation and scene information of the drawing points, including:

[0023] Based on the scene location, the corresponding light source features are sampled from the neural illumination representation of the neural illumination field output by the light source encoding module, and the direct light component, indirect light component, and shadow component are inferred by the direct light decoder, indirect light decoder, and shadow decoder respectively based on the light source features and scene information.

[0024] Preferably, in the direct light decoder, a fully connected network is used to predict pixel-level direct light illumination results, i.e., direct light components, based on the sampled light source features and the geometric buffer data of the scene from the camera's perspective.

[0025] Preferably, in the indirect light decoder, a fully connected network is used to infer the global indirect lighting result of the scene, i.e., the indirect light component, based on the sampled light source features, the geometric buffer data of the scene from the camera's perspective, and the geometric buffer data of the scene from the light source's perspective.

[0026] Preferably, in the shadow decoder, a multi-scale kernel prediction network is used to jointly infer the soft shadow result, i.e. the shadow component, based on the sampled light source features and occlusion cues including the distance from the scene to the light source, the distance from the occluder to the light source, the shadow of the point light source, the dot product of the scene surface normal and the camera normal, and the scene depth.

[0027] Preferably, generating the final rendered image that conforms to the true illumination distribution based on the inference result includes:

[0028] The inferred results are synthesized according to the physical lighting model, and the light intensity factor recorded in the light source encoding module is used to restore the light intensity of the synthesized results in order to generate a final drawn image that conforms to the real lighting distribution.

[0029] Secondly, embodiments of the present invention also provide a general neural rendering device based on a lighting function generation model, which is implemented using the above-mentioned general neural rendering method based on a lighting function generation model, including: a light source information extraction unit, a scene information extraction unit, a light source encoding unit, a light source decoding unit, and a result merging unit;

[0030] The light source information extraction unit is used to collect light source information containing multi-view observation data;

[0031] The scene information extraction unit is used to collect scene information;

[0032] The light source encoding unit is used to convert light source information into neural illumination representation;

[0033] The light source decoding unit is used to perform joint inference based on the neural lighting representation and scene features of the drawing points;

[0034] The result merging unit is used to generate a final drawn image that conforms to the true illumination distribution based on the inference results.

[0035] Compared with the prior art, the beneficial effects of the present invention include:

[0036] This invention achieves unified neural rendering modeling for complex light sources and diverse scenes through a collaborative design of light source information extraction, scene information extraction, light source encoding, light source decoding, and lighting synthesis. It can fully extract and encode lighting and scene feature information without imposing any constraints on the light source form, and accurately reconstruct direct light, indirect light, and shadows through layered light source decoding. Compared with existing technologies, this invention significantly improves the generalization ability of neural rendering across light sources and scenes, generating images with offline rendering quality at near real-time computational cost, balancing realism, stability, and rendering efficiency, and possessing good scalability and broad application value. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the general neural rendering method based on a lighting function generation model provided in this embodiment of the invention.

[0039] Figure 2 This is a schematic diagram of the structure and process of the light source encoding module provided in an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the structure and process of the light source decoding module provided in an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of the structure and process of the shadow decoder provided in an embodiment of the present invention;

[0042] Figure 5 This is the rendering result of the present invention under a large number of new scenes and new light sources;

[0043] Figure 6 This is a comparison of the present invention with existing real-time ray tracing rendering methods and neural rendering methods;

[0044] Figure 7 This is a schematic diagram of the structure and process of a general neural rendering system based on a lighting function generation model provided in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0046] The inventive concept of this invention is as follows: Existing neural rendering methods typically require retraining models for specific scenes or tasks, and cannot maintain stable performance under different geometric structures or lighting distributions, making them difficult to extend and reuse in engineering applications. Therefore, in order to overcome the bottlenecks of existing neural rendering methods in terms of generalization and transferability, this invention provides a general neural rendering method and system based on a lighting function generation model. By constructing a lighting function generation model, the rendering equation is regarded as a high-dimensional mapping jointly determined by light source parameters and scene parameters. The neural network extracts information from the light source and scene, and jointly infers the final rendering result, realizing a neural rendering framework that can uniformly adapt to different light sources, different materials, and even different scene conditions.

[0047] like Figure 1 As shown in the example, this embodiment provides a general neural rendering method based on a lighting function generation model, including the following steps:

[0048] S1 collects light source information containing multi-view observation data.

[0049] After acquiring the light source input, traditional real-time rendering and existing neural rendering typically constrain the light source. These constraints include, but are not limited to, simplifying the light source distribution into non-real light sources such as points and surfaces, constraining the placement, brightness, and number of light sources, and binding the light source to the scene. This allows the lighting information under the constraints to be described using simple positional information, but it also makes it impossible to generalize across light sources.

[0050] In this embodiment, to achieve generalized neural rendering across light sources, sufficiently rich multi-view light source information is extracted to describe the differences in geometry and illumination distribution between different light sources. The target light source is observed and sampled from multiple perspectives using an offline rendering engine. The collected data includes the radiance, position, and orientation of the light source from each perspective. Through discrete sampling from multiple perspectives, the shape, material, and spatial distribution characteristics of the light source can be fully described, providing rich input data for the illumination encoding module and bridging the gap between real light source observations and neural illumination representations.

[0051] S2, collect scene information.

[0052] In this embodiment, to obtain geometric and material description information of the scene from the current rendering input, and to provide multi-channel structured input for neural rendering, the Unreal Engine (UE) is used as the underlying implementation. Through the engine's real-time rendering pipeline, the geometric buffer data (i.e., G-Buffer) of the scene from the camera's perspective is extracted in real time in each frame. This includes, but is not limited to, normal maps, depth maps, reflectivity maps, roughness maps, diffuse color maps, and other material property maps. This information can fully characterize the geometric structure and surface optical properties of objects in the scene.

[0053] In addition, to achieve cross-scene lighting modeling, the scene is observed from the perspective of the light source to obtain geometric buffer data from the perspective of the light source, including but not limited to normal maps, depth maps and diffuse color maps, which are used to characterize the spatial occlusion relationship and energy interaction characteristics between the light source and the scene.

[0054] By fusing information from both the camera's perspective and the light source's perspective, a mapping relationship between illumination transmission and scene geometry can be established, forming complete scene description features. This provides comprehensive input support for subsequent neural illumination coding and joint decoding of the light source and scene, serving both as a basis for subsequent direct and indirect light calculations and as a means to provide accurate geometric occlusion cues for shadow decoding. This allows the invention to maintain real-time performance and consistency even in complex dynamic scenes.

[0055] S3 converts light source information into neural lighting representation through the light source encoding module in the lighting function generation model.

[0056] In this embodiment, the light source encoding module is used to enable the neural rendering system to generalize across different light source scenarios. This module can receive and process input information from various forms of light sources, including but not limited to point lights, area lights, volumetric lights, and complex real-world light sources. The module first performs structured completion and feature enhancement on the input light source information; then, through feature extraction and encoding networks, it maps the enhanced light source information into a compact, efficient, and scene-generalizable neural lighting representation. Compared to the parameterized assumptions about light sources in traditional real-time rendering technologies (such as only supporting point lights, area lights, or lights in specific directions), this module does not impose any constraints or pre-set assumptions on the form of light sources, and can directly model and represent any complexly distributed real light sources. This design significantly improves the rendering system's ability to depict real lighting phenomena, enabling it to exhibit higher-order lighting effects that are difficult to achieve with traditional parameterized light sources, such as multiple scattering, non-homogeneous reflection, and lighting under complex geometric occlusion. In terms of rendering performance, although traditional offline rendering can achieve accurate simulation of unconstrained light sources through ray tracing, the complex geometric shapes and material characteristics of real light sources often result in extremely low effective light path sampling efficiency, which cannot meet the requirements of real-time computation. In contrast, the light source encoding module, by introducing the high-dimensional representation capability of neural networks, encodes complex light source distributions into compact latent representations, enabling realistic reconstruction and rendering of complex light sources at near real-time computational cost.

[0057] like Figure 2 As shown, in the light source encoding module, firstly, the maximum values ​​of the input Radiance data in the R, G, and B channels are normalized, and the maximum values ​​are recorded as the intensity factor for subsequent result recovery. Then, the normalized Radiance image is concatenated with the position and orientation images and divided into several multi-view observation slices. Each slice is unfolded into a high-dimensional vector and mapped to the neural space through a multilayer perceptron (MLP) to obtain multiple illumination terms. These illumination terms are subsequently input into a 4D Transformer for joint modeling in the view space and image space to restore the global consistency and spatial correlation of illumination, resulting in modeled illumination terms. Finally, a light field generator based on a cross-attention mechanism projects the modeled illumination terms onto the neural position field, reconstructing a three-plane (XY, YZ, XZ) neural illumination representation, achieving an efficient conversion from discrete multi-view observations to a continuous illumination field representation.

[0058] S4 uses the light source decoding module in the lighting function generation model to perform joint inference based on the neural lighting representation and the scene features of the drawing points.

[0059] In this embodiment, the light source decoding module is used to jointly decode the neural lighting representation obtained by the light source encoding module and multi-channel scene information to generate the final rendering result. As the core synthesis unit of the neural rendering system, this module is responsible for high-dimensional fusion of light source features with scene geometry, materials, and viewing angle information, thereby achieving unified lighting reconstruction across light sources and scenes. Functionally, the light source decoding module can be further subdivided into multiple sub-tasks to perform layered decoding for different lighting components. These sub-tasks include, but are not limited to, generating direct light components, indirect light components, and shadow components respectively through a direct light decoder, an indirect light decoder, and a shadow decoder.

[0060] like Figure 3 As shown, in the light source decoding module, three-plane sampling is first used to sample the corresponding light source features from the neural illumination representation of the neural illumination field output by the light source encoding module according to the scene position. Then, based on the light source features and scene information, the direct light component, indirect light component and shadow component are generated by the direct light decoder, indirect light decoder and shadow decoder respectively.

[0061] The direct light decoder is used to reconstruct the direct radiative transmission relationship between the light source and the scene surface. It mainly extracts and matches features based on the directionality of the light source and the geometric features of the scene, and achieves realistic lighting performance by predicting the irradiance distribution under direct illumination. Specifically, in the direct light decoder, a fully connected network is used to predict pixel-level direct light illumination results, i.e., direct light components, based on the sampled light source features and the geometric buffer data of the scene from the camera's perspective.

[0062] The indirect light decoder is used to model multiple reflections, refractions, and global illumination effects within a scene. It extracts deep features from the scene's reflectivity, roughness, and global structure information, and combines this with the latent feature vectors of the light source to estimate the propagation and concentration of indirect light energy within the scene, achieving global illumination effects that are difficult to represent in traditional real-time rendering. Specifically, the indirect light decoder uses a fully connected network to infer the global indirect illumination result of the scene, i.e., the indirect light component, based on sampled light source features, geometric buffer data of the scene from the camera's perspective, and geometric buffer data of the scene from the light source's perspective.

[0063] The shadow decoder is used to characterize shadow regions caused by scene geometric occlusion. Based on the view frustum relationship between the light source and the camera, it analyzes the differences in lighting occlusion paths and depth, and generates a spatially continuous and smoothly edged shadow distribution map through a lighting consistency learning mechanism, thereby improving the geometric consistency and visual realism of the final rendering result. Specifically, in the shadow decoder, a multi-scale kernel prediction network is used to jointly infer soft shadow results, i.e., shadow components, based on sampled light source features and occlusion cues calculated from the light source's viewpoint. Occlusion cues include the distance from the scene to the light source, the distance from occluders in the scene to the light source, the point light source shadow obtained from the distance between the two, the dot product of the scene surface normal and the camera normal, and the scene depth. For example... Figure 4 As shown, the point light source shadows are first downsampled at different scales to obtain multi-scale point light source shadows. Then, the neural network infers the filtering kernels and upsampling kernels for point light source shadows at different scales based on light source features and shadow cues. In the filtering and upsampling block, the small-scale point light source shadows are filtered and upsampled to the original resolution, resulting in multiple soft shadow maps with the original resolution but filtered at different scales. For the original-scale point light source shadows, the network only predicts the filtering kernel. Subsequently, at the original scale, the fusion weights of multiple soft shadow maps and the overall shadow correction coefficients are additionally predicted. The filtered shadows at different scales are then mixed and corrected to obtain the final shadow result. Through the lighting decoding module, unified lighting decoding across light sources and scenes is achieved.

[0064] S5 generates the final rendered image that conforms to the true illumination distribution based on the inference results.

[0065] In this embodiment, the direct light component, indirect light component, and shadow component are synthesized according to the physical lighting model, and the light intensity factor recorded in the light source encoding module is used to restore the light intensity of the synthesized result to generate a final rendered image that conforms to the real lighting distribution. The synthesis method may include a combination of direct light, indirect light, and shadow, or the decoupling and merging of direct light component and reflectivity, or the supplementation of light source energy. The specific strategy is configured by the user or controlled by the network.

[0066] like Figure 3 As shown, in one embodiment, after obtaining the direct light component, shadow component, and indirect light component respectively, the direct light component is multiplied by the shadow component and the diffuse color to obtain a direct light with shadow and affected by diffuse reflection. This direct light and indirect light are then multiplied by the illuminance factor and added to the radiance of the light source to obtain the final rendering result.

[0067] Through the above steps, the system achieves both high fidelity and real-time performance of rendering results while maintaining physical consistency, significantly improving the cross-light source generalization ability and real-time rendering performance of the neural rendering system.

[0068] To further verify the effectiveness of the present invention, such as Figure 5 As shown, the rendering results are displayed under a large number of new scenes and new light sources. It can be seen that the present invention has a strong generalization ability across scenes and across light sources.

[0069] like Figure 6 The diagram shows a comparison between the present invention and existing real-time ray tracing rendering methods and neural rendering methods. Real-time ray tracing rendering methods utilize a combination of low-sample ray tracing and the latest real-time noise reduction OIDN, while the latest neural rendering method uses LightFormer. It can be seen that real-time ray tracing rendering methods struggle to accurately handle the complex light transmission caused by lighting fixtures, and the high sampling complexity of the fixtures leads to significant differences between the OIDN and the reference. LightFormer also shows significant differences from the results of the present invention in the highlight and shadow portions. The method of the present invention exhibits a more realistic rendering effect.

[0070] Table 1 shows a comparison of the numerical metrics of the present invention with those of the existing real-time ray tracing rendering methods OIDN and LightFormer, including time metrics, PSNR metrics, LPIPS metrics, SSIM metrics, and sMAPE metrics. It can be seen that, under the same time conditions, the present method far surpasses the real-time rendering method and the neural rendering method in various image quality metrics.

[0071] Table 1. Comparison of index values ​​between this method and existing methods

[0072]

[0073] In summary, the general neural rendering method based on a lighting function generation model provided by this invention achieves a complete workflow through the coordinated operation of various steps, from real light source data acquisition and physically consistent HDR light source reconstruction to geometry and material preprocessing compatible with traditional pipelines, and finally to efficient real-time decoding based on neural representation. This not only solves the problem of accurately modeling and efficiently rendering real light sources in traditional real-time rendering, but also possesses cross-scene generalization capabilities. It can be quickly integrated into industrial-grade rendering systems without retraining for each new scene or light source, thus providing high-quality, low-cost, and physically realistic light source rendering solutions for fields such as games, film and television, VR / AR, and digital twins.

[0074] Based on the same inventive concept, such as Figure 7 As shown, this embodiment of the invention also provides a general neural rendering system 700 based on a lighting function generation model, including: a light source information extraction unit 710, a scene information extraction unit 720, a light source encoding unit 730, a light source decoding unit 740, and a result merging unit 750.

[0075] The light source information extraction unit 710 is used to collect light source information containing multi-view observation data.

[0076] The scene information extraction unit 720 is used to collect scene information.

[0077] The light source encoding unit 730 is used to convert light source information into neural illumination representation.

[0078] The light source decoding unit 740 is used for joint inference based on neural lighting representation and scene features of drawing points.

[0079] The result merging unit 750 is used to generate a final drawn image that conforms to the true illumination distribution based on the inference results.

[0080] It should be noted that the general neural rendering system based on the illumination function generation model provided in the above embodiments belongs to the same inventive concept as the general neural rendering method based on the illumination function generation model. For details of its specific implementation process, please refer to the embodiments of the general neural rendering method based on the illumination function generation model, which will not be repeated here.

[0081] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A general neural rendering method based on a lighting function generation model, characterized in that, Includes the following steps: Collect light source information and scene information that include multi-view observation data; Construct a lighting function generation model that includes a light source encoding module and a light source decoding module; wherein, The light source encoding module is used to convert light source information into a neural illumination representation, including: normalizing the light source information observed from multiple perspectives by taking the maximum values ​​in the R, G, and B channels to obtain a multi-view observation image, and recording the maximum brightness of each channel as an illumination intensity factor; dividing the normalized multi-view observation image into several multi-view observation slices, unfolding each slice into a high-dimensional vector, and mapping it to the neural space through a multilayer perceptron to obtain multiple illumination terms; inputting the illumination terms into a 4DTransformer to perform joint modeling in the view space and image space to restore the global consistency and spatial correlation of illumination, and obtaining the modeled illumination terms; projecting the modeled illumination terms onto the neural position field through a light field generator based on a cross-attention mechanism to reconstruct a three-plane neural illumination representation of the neural illumination field; The light source decoding module is used to perform joint inference based on the neural illumination representation and the scene features of the drawing points, including: sampling the corresponding light source features from the neural illumination representation of the neural illumination field output by the light source encoding module according to the scene position, and inferring the direct light component, indirect light component and shadow component based on the light source features and scene information through the direct light decoder, indirect light decoder and shadow decoder respectively. The final rendered image is generated based on the inference results, conforming to the actual lighting distribution.

2. The general neural rendering method based on the illumination function generation model according to claim 1, characterized in that, The acquisition of light source information, which includes multi-view observation data, includes: The target light source is observed and sampled from multiple perspectives using an offline rendering engine. The collected data includes the radiance, position information, and direction information of the light source from each perspective, in order to describe the shape, material, and spatial distribution characteristics of the light source.

3. The general neural rendering method based on the illumination function generation model according to claim 1, characterized in that, The collected scene information includes: The real-time rendering engine acquires geometric buffer data of the scene from the camera's perspective, including normal maps, depth maps, reflectivity maps, roughness maps, and diffuse color maps, to characterize the geometric structure and surface optical properties of objects in the scene. Simultaneously, geometric buffer data from the perspective of the light source in the target scene are acquired, including normal maps, depth maps, and diffuse color maps, to characterize the spatial occlusion relationship and energy interaction characteristics between the light source and the scene.

4. The general neural rendering method based on the illumination function generation model according to claim 1, characterized in that, In the direct light decoder, a fully connected network is used to predict pixel-level direct light illumination results, i.e., direct light components, based on the sampled light source features and the geometric buffer data of the scene from the camera's perspective.

5. The general neural rendering method based on the illumination function generation model according to claim 1, characterized in that, In the indirect light decoder, a fully connected network is used to infer the global indirect lighting result of the scene, i.e., the indirect light component, based on the sampled light source features, the geometric buffer data of the scene from the camera's perspective, and the geometric buffer data of the scene from the light source's perspective.

6. The general neural rendering method based on the illumination function generation model according to claim 1, characterized in that, In the shadow decoder, a multi-scale kernel prediction network is used to jointly infer the soft shadow result, i.e. the shadow component, based on the sampled light source features and occlusion cues including the distance from the scene to the light source, the distance from the occluder to the light source, the point light source shadow, the dot product of the scene surface normal and the camera normal, and the scene depth.

7. The general neural rendering method based on the illumination function generation model according to claim 1, characterized in that, The process of generating a final rendered image that conforms to the true illumination distribution based on the inference results includes: The inferred results are synthesized according to the physical lighting model, and the light intensity factor recorded in the light source encoding module is used to restore the light intensity of the synthesized results in order to generate a final drawn image that conforms to the real lighting distribution.

8. A general neural rendering system based on a lighting function generation model, implemented using the general neural rendering method based on a lighting function generation model as described in any one of claims 1 to 7, characterized in that, include: The system includes a light source information extraction unit, a scene information extraction unit, a light source encoding unit, a light source decoding unit, and a result merging unit. The light source information extraction unit is used to collect light source information containing multi-view observation data; The scene information extraction unit is used to collect scene information; The light source encoding unit is used to convert light source information into neural illumination representation; The light source decoding unit is used to perform joint inference based on the neural lighting representation and scene features of the drawing points; The result merging unit is used to generate a final drawn image that conforms to the true illumination distribution based on the inference results.

Images