A rendering method and device based on implicit light transfer function merging

Abstract

The present invention discloses a rendering method and device based on implicit light transfer function merging, which belongs to the field of image rendering. Specifically, the method comprises: pre-dividing all objects except light sources in a three-dimensional scene into multiple object groups and defining the insertion order of the object groups, then sequentially inserting the divided object groups into the scene, and combining light source sampling data, target perspective-related data and object group-related data through implicit light transfer function calculation to predict the light transmission change field of the object group in the current three-dimensional scene, and finally merging the light transmission change field and the mask map of the object group into the rendering result before the object group is inserted through matrix calculation. The splitting and merging method can be applied to various types of dynamic scenes and can also improve the rendering speed of dynamic scenes.

Description

Technical Field

[0001] The present invention belongs to the field of image rendering, and in particular relates to a rendering method and device based on implicit light transfer function merging. Background Art

[0002] Rendering technology has always been one of the most important research topics in computer graphics and is widely used in fields such as computer-aided design, electronic entertainment, and film and television special effects. Its main goal is to quickly and efficiently visualize virtual scenes based on given descriptions and requirements. Depending on the rendering effect, rendering technology can be divided into realistic rendering technology and non-realistic rendering technology. The main goal of realistic rendering technology is to realistically simulate and restore various complex light and shadow effects in the real world in virtual scenes, generating images with high realism. Non-realistic rendering technology is mainly dedicated to presenting rendering results in specific artistic styles, such as cartoons, sketches, and oil paintings.

[0003] Rendering techniques, especially photorealistic rendering techniques, require extensive computation to achieve high-quality results. For example, Monte Carlo path tracing, based on simulating the physical behavior of light propagation in a scene, requires casting tens of thousands of rays per pixel and performing complex interactive calculations with the scene to produce a converging rendering result. This can result in rendering times that can take tens of minutes or even days. Therefore, minimizing rendering time while ensuring quality is a key challenge for most rendering techniques.

[0004] The coordinated development of software and hardware is an effective means of improving rendering efficiency. The advancement of rasterization algorithms and related graphics rendering pipeline hardware has ushered in the era of real-time rendering. The emergence of real-time denoising algorithms and dedicated ray tracing hardware (RTCore) has further enabled real-time dynamic global illumination. More efficient denoising algorithms reduce the computational cost of achieving the same rendering quality, while faster dedicated ray tracing hardware increases the amount of computation per unit time. The combination of these two factors can further improve rendering efficiency.

[0005] In recent years, neural networks have demonstrated tremendous potential in many fields, including rendering. Simultaneously, the rapid development of specialized neural network hardware, such as Tensor Processing Units (TPUs) and National Processing Units (NPUs), has significantly increased the computational speed of neural network algorithms. Neural network-based denoising techniques rapidly achieve high-quality results by reducing noise in images computed using Monte Carlo path tracing algorithms. These methods are currently the most popular real-time global illumination techniques, but they still rely on Monte Carlo methods or rendering results from the graphics rendering pipeline, making them primarily dependent on specialized hardware such as those supporting real-time ray tracing or graphics shaders. This limits their versatility in many application scenarios and hinders their ability to benefit from the development of specialized neural network hardware. Rendering methods based on neural scene representations utilize neural networks to extract the mapping between global illumination results and implicit scene representations from large amounts of data. Rendering results are directly obtained by inputting the corresponding scene representation and related features into a trained network. However, these methods currently only support the rendering of dynamic scenes that appear in the scene layout of the training sample set. Therefore, a fast rendering method using neural network chips that supports flexible and dynamic scenes is still lacking. Summary of the Invention

[0006] In view of the above, the purpose of the present invention is to provide a rendering method and device based on implicit light transfer function merging, which uses a neural network to predict the light transfer change field of each object group in the scene, and merges the light transfer change field through matrix calculation to quickly obtain dynamic rendering results.

[0007] To achieve the above-mentioned object of the invention, an embodiment provides a rendering method based on implicit light transfer function merging, comprising:

[0008] Step 1: Obtain a three-dimensional scene, and divide all objects in the three-dimensional scene except the light source into at least one object group and define the insertion order of the object group;

[0009] Step 2: sampling the light source in the three-dimensional scene to obtain light source sampling data;

[0010] Step 3: Generate geometric features and material features of the visible parts of the 3D scene at the target perspective to form target perspective related data, and also generate a mask map of each object group at the target perspective;

[0011] Step 4: For each object group, perform the following process: first, insert the object group into the 3D scene before insertion in the insertion order, and generate object group-related data; then, based on at least one of the light source sampling data, the target view-related data, and the object group-related data, use the implicit light transfer function embodied by the neural network to calculate and predict the light transfer change field of the object group in the 3D scene after insertion; finally, combine the light transfer change field with the mask map of the object group through matrix calculation to the rendering result before the object group is inserted, so as to obtain the rendering result after the object group is inserted;

[0012] Step 5: After all object groups have been drawn in step 4, the dynamic drawing result of the three-dimensional scene is obtained and output.

[0013] In one embodiment, when sampling light sources in a three-dimensional scene, different light source sampling data is collected based on different types of light sources, including:

[0014] When the light source is a surface light source, importance sampling is performed on the light source surface according to the light intensity, and the geometric features, light intensity, and sampling probability at the sampling point are recorded to form the light source sampling data. The geometric features at the sampling point include the sampling point position and the surface normal vector at the sampling point.

[0015] When the light source is an environment map, the importance sampling of the light direction is performed according to the light intensity. The sampling direction, light intensity and sampling probability are recorded to form the light source sampling data, and the sampling direction is used as a geometric feature.

[0016] When the light source is a point light source, the point light source position and light intensity are recorded to form the light source sampling data, and the point light source position is used as a geometric feature;

[0017] When the light source is directional light, the lighting direction and light intensity corresponding to the directional light are recorded to form light source sampling data, and the lighting direction corresponding to the directional light is used as a geometric feature.

[0018] In one embodiment, the object group related data includes self-sampled data, scene observation data, and occlusion observation data of the object group;

[0019] The process of generating the self-sampling data of the object group includes: uniformly and randomly sampling the surfaces of the objects in the object group, and recording the geometric features and material features at the sampling points to form the self-sampling data;

[0020] The process of generating scene observation data of the object group includes: obtaining a panoramic observation of the three-dimensional scene during the insertion process at the geometric center of the object group, and recording the geometric features and material features of the observed scene surface to form the scene observation data;

[0021] The process of generating occlusion observation data of the object group includes: using geometric features in the scene observation data as the occlusion observation data.

[0022] In one embodiment, light source sampling data, target view-related data, and object group-related data are required to undergo data space transformation and encoding before being input into the neural network to calculate the light transmission change field, including:

[0023] For the light source sampling data, after transforming the geometric features therein from the world space to the local space of the object group, a first encoder is used to encode the transformed light source sampling data to obtain a light source code;

[0024] For the object group related data, geometric features of the self-sampled data, scene observation data, and occlusion observation data included therein are transformed from the world space to the local space of the object group, and a second encoder is used to encode the transformed self-sampled data, scene observation data, and occlusion observation data to obtain a self-code, an environment code, and an occlusion code respectively;

[0025] For the target perspective related data, the geometric features therein are transformed from the world space to the local space of the object group to obtain the transformed target perspective related data;

[0026] The light source code, self-code, environment code, occlusion code and transformed target perspective related data are input into the neural network, and the implicit light transfer function embodied by the neural network is used for transmission calculation to output the light transfer change field of the object group in the three-dimensional scene after insertion.

[0027] In one embodiment, the light transmission variation field is combined with the mask map of the object group and merged into the rendering result before the object group is inserted through matrix calculation using the following formula:

[0028]

[0029] Among them, M m (o,ω) represents observing the light source from the camera position o to the direction ω The illuminated object group m is inserted into the 3D scene Regarding the mask map of object group m, Indicates that the object group m is under the light source Lighting 3D scene The light transmission variation field in Indicates the drawing result before the object group m is inserted. Represents the drawing result after the object group m is inserted;

[0030] Before being applied, the neural network needs to undergo parameter optimization. The loss function used in parameter optimization is the difference between the predicted value of the light transmission change field and the true value label of the light transmission change field. The true value label of the light transmission change field is obtained by the following method:

[0031]

[0032] in, represents the true value label of the light transmission variation field, Indicates the true value of the drawing result before the object group m is inserted, Represents the true value of the drawing result after the object group m is inserted.

[0033] In one embodiment, when a global illumination rendering result is required, first, the light source code, self-coder, occlusion code, and transformed target view-related data are input into a first neural network, and a first light transmission change field of the object group under direct illumination in the three-dimensional scene after insertion is predicted and calculated. Then, the light source code, self-coder, environment code, and transformed target view-related data are input into a second neural network, and a second light transmission change field of the object group under indirect illumination in the three-dimensional scene after insertion is predicted and calculated.

[0034] Then, the first light transmission variation field is combined with the mask map of the object group and merged into the direct light rendering result before the object group is inserted through matrix calculation to obtain the direct light rendering result after the object group is inserted; the second light transmission variation field is combined with the mask map of the object group and merged into the indirect light rendering result before the object group is inserted through matrix calculation to obtain the indirect light rendering result after the object group is inserted;

[0035] Finally, the direct light rendering result and the indirect light rendering result after the object group is inserted are summed to obtain the global illumination rendering result after the object group is inserted.

[0036] In one embodiment, the first neural network needs to undergo parameter optimization before being applied. The loss function used in the parameter optimization is the difference between the predicted value of the first light transmission variation field and the true value label of the first light transmission variation field, wherein the true value label of the first light transmission variation field is obtained by:

[0037]

[0038] Among them, M m (o,ω) represents observing the light source from the camera position o to the direction ω The lighting object group m is inserted into the current 3D scene Regarding the mask map of object group m, Indicates that the object group m is under the light source The current 3D scene being illuminated The ground truth label of the first light transport variation field in , Indicates the true value of the direct light rendering result before the object group m is inserted. Represents the true value of the direct light rendering result after the object group m is inserted;

[0039] The second neural network needs to undergo parameter optimization before being applied. The loss function used in the parameter optimization is the difference between the predicted value of the second light transmission variation field and the true value label of the second light transmission variation field. The true value label of the second light transmission variation field is obtained by the following method:

[0040]

[0041] in, Indicates that the object group m is under the light source The current 3D scene being illuminated The true value label of the second light transmission change field in, Indicates the true value of the indirect light rendering result before the object group m is inserted. Indicates the true value of the indirect light rendering result after the object group m is inserted.

[0042] To achieve the above-mentioned object of the invention, an embodiment further provides a rendering device based on implicit light transfer function merging, comprising:

[0043] An object division module is used to obtain a three-dimensional scene, and divide all objects in the three-dimensional scene except the light source into at least one object group and define the insertion order of the object group;

[0044] A light source sampling module is used to sample light sources in a three-dimensional scene to obtain light source sampling data;

[0045] A target perspective related data generation module is used to generate geometric features and material features of the visible parts of the three-dimensional scene under the target perspective to form target perspective related data, and is also used to generate a mask map of each object group under the target perspective;

[0046] A dynamic rendering module is configured to execute the following process for each object group: first, inserting the object group into the three-dimensional scene before insertion in an insertion order and generating object group-related data; then, using an implicit light transfer function embodied by a neural network to calculate a light transfer change field of the object group in the three-dimensional scene after insertion based on at least one of light source sampling data, target view-related data, and object group-related data; finally, merging the light transfer change field and the object group's mask map into a rendering result before the object group is inserted through a matrix calculation method to obtain a rendering result after the object group is inserted;

[0047] The output module is used to obtain and output the dynamic rendering results of the three-dimensional scene after all object groups have been rendered by the dynamic rendering module.

[0048] To achieve the above-mentioned object of the invention, an embodiment further provides a rendering device based on implicit light transfer function merging, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0049] Step 1: Obtain a three-dimensional scene, and divide all objects in the three-dimensional scene except the light source into at least one object group and define the insertion order of the object group;

[0050] Step 2: sampling the light source in the three-dimensional scene to obtain light source sampling data;

[0051] Step 3: Generate geometric features and material features of the visible parts of the three-dimensional scene at the target perspective to form target perspective related data, and also generate a mask map of each object group at the target perspective;

[0052] Step 4: For each object group, perform the following process: first, insert the object group into the 3D scene before insertion in the insertion order, and generate object group-related data; then, based on at least one of the light source sampling data, the target view-related data, and the object group-related data, use the implicit light transfer function embodied by the neural network to calculate and predict the light transfer change field of the object group in the 3D scene after insertion; finally, combine the light transfer change field with the mask map of the object group through matrix calculation to the rendering result before the object group is inserted, so as to obtain the rendering result after the object group is inserted;

[0053] Step 5: After all object groups have been drawn in step 4, the dynamic drawing result of the three-dimensional scene is obtained and output.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] By pre-dividing all objects except the light source in the three-dimensional scene into at least one object group and defining the insertion order of the object groups, the divided object groups are then inserted into the scene in order, and the light source sampling data, target perspective related data and object group related data are combined through implicit light transfer function calculation to predict the light transmission change field of the object group in the current three-dimensional scene. Finally, the light transmission change field is combined with the mask map of the object group through matrix calculation and merged into the drawing result before the object group is inserted. The split and then merge method can be applied to various types of dynamic scenes and can also improve the drawing speed of dynamic scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 The embodiment provides a flowchart of a rendering method based on implicit light transfer function merging;

[0058] Figure 2 This is a schematic diagram of a drawing result obtained by inserting an object group into a three-dimensional scene provided by an embodiment;

[0059] Figure 3 3 is a schematic structural diagram of a rendering device based on implicit light transfer function merging provided in an embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0061] Figure 1 The embodiment provides a flow chart of a rendering method based on implicit light transfer function merging. Figure 1 As shown, the rendering method based on implicit light transfer function merging provided by the embodiment includes the following steps:

[0062] Step 1: Obtain a three-dimensional scene, and divide all objects in the three-dimensional scene except the light source into at least one object group and define the insertion order of the object group.

[0063] In this embodiment, the acquired 3D scene contains multiple objects to be rendered. To implement split rendering to accommodate various types of 3D scenes, all objects in the 3D scene, excluding light sources, are divided into any number of object groups. A single object group can consist of a single object or multiple objects, and all objects in the 3D scene belong to and only belong to a specific object group. That is, a single object cannot be repeatedly assigned to two or more object groups.

[0064] In the embodiment, the insertion order of the object groups will not affect the quality of the final rendering result to a certain extent. Therefore, the insertion order of the object groups is not restricted and the insertion order of each object group can be arbitrarily specified.

[0065] Step 2: Sample the light source in the three-dimensional scene to obtain light source sampling data.

[0066] In this embodiment, a three-dimensional scene may include multiple light sources of different types. To improve the convenience and accuracy of subsequent calculations of the light transmission change field of the object group, all light sources in the scene are pre-sampled. Specifically, different light source sampling data is collected based on the different light source types, including:

[0067] When the light source is a surface light source, importance sampling is performed on the light source surface according to the light intensity, and the geometric features, light intensity, and sampling probability at the sampling points are recorded to form light source sampling data, where the geometric features at the sampling points include but are not limited to the sampling point position and the surface normal vector at the sampling point;

[0068] When the light source is an environment map, the importance of the light direction is sampled according to the light intensity. The sampling direction, light intensity and sampling probability are recorded to form the light source sampling data. The sampling direction is used as a geometric feature for coordinate transformation and encoding.

[0069] When the light source is a point light source, the point light source position and light intensity are recorded to form the light source sampling data, and the point light source position is used as a geometric feature for coordinate transformation and encoding;

[0070] When the light source is directional light, the lighting direction and light intensity corresponding to the directional light are recorded to form light source sampling data, and the lighting direction corresponding to the directional light is used as a geometric feature for coordinate transformation and encoding.

[0071] Step 3: Generate geometric features and material features of the visible parts of the three-dimensional scene under the target perspective to form target perspective related data, and also generate a mask map of each object group under the target perspective.

[0072] In this embodiment, to improve the convenience and accuracy of the subsequent calculation of the light transmission change field for the object group, target perspective-related data and a mask map of the object group at the target perspective are also pre-generated. Specifically, based on the target camera parameters, rasterization or ray tracing techniques are used to calculate and record the geometric and material features of the visible portion of the original 3D scene at the target perspective. Geometric features include, but are not limited to, the position and normal vector of the surface observed at the target perspective, and material features include, but are not limited to, the reflectivity and roughness of the surface observed at the target perspective.

[0073] In this embodiment, to improve the convenience and accuracy of the subsequent calculation of the light transmission change field of the object group, a mask map of the observable object group from the target perspective is also required to be pre-generated. Specifically, the mask map corresponding to each object group observed from the target perspective is calculated based on the object position and camera position.

[0074] Step 4: For each object group, the implicit light transfer function embodied by the neural network is used to calculate the light transfer change field corresponding to the object group and merge them to obtain the rendering result.

[0075] In this embodiment, a 3D scene formed by removing all objects except the light source from the 3D scene is used as the initial 3D scene. Then, each object group is traversed in the insertion order, and the following process is performed for each traversed object group:

[0076] (a) First, the object group is inserted into the pre-insertion 3D scene in the insertion order, and the self-sampling data, scene observation data, and occlusion observation data of the object group are generated to form the object group related data.

[0077] In an embodiment, when traversing the first object group, the 3D scene before insertion is the initial 3D scene; when traversing other object groups except the first object group, the 3D scene before insertion is the 3D scene updated after the previous object group is inserted.

[0078] In an embodiment, the self-sampling data generation process of the object group includes: uniformly and randomly sampling the surfaces of all objects in the object group, recording the geometric features and material features at the sampling points to form self-sampling data, wherein the geometric features include but are not limited to the sampling point position and the surface normal vector of the sampling point, and the material features include but are not limited to the reflectivity and roughness of the surface material at the sampling point.

[0079] In an embodiment, the scene observation data generation process of the object group includes: obtaining a panoramic observation of the three-dimensional scene during the insertion process at the geometric center of the object group, and recording the geometric features and material features of the observed scene surface to form scene observation data, wherein the geometric features include but are not limited to the position and normal vector of the observed scene surface, and the material features include but are not limited to the reflectivity and roughness of the observed scene surface.

[0080] In this embodiment, a camera is positioned at the geometric center of the object group, performing a panoramic observation of the 3D scene during insertion. The observed surface positions, surface normals, and the reflectivity and roughness of the surface materials are recorded. The panoramic observation data is specifically in the form of recorded images in six directions (+x, -x, +y, -y, +z, and -z) with a 90° field of view. These images include both geometric and material feature images.

[0081] In an embodiment, the process of generating occlusion observation data of an object group includes: using geometric features in the scene observation data as the occlusion observation data, that is, using the position and normal vector of the scene surface observed in the scene observation data as the occlusion observation data.

[0082] (b) Then, based on the light source sampling data, target view-related data, and object group-related data, the implicit light transfer function embodied by the neural network is calculated to predict the light transfer change field of the object group in the three-dimensional scene after insertion.

[0083] As an algorithmic mathematical model for distributed parallel information processing, neural networks mainly rely on the complexity of the system and achieve the purpose of processing information by adjusting the interconnected relationships between a large number of internal nodes, that is, the implicit light transfer function.

[0084] In order to realize the calculation of the light transmission change field corresponding to various object groups, the light transmission change field calculated based on the neural network Defined as:

[0085]

[0086] in, Indicates observing the light source from the camera position o to the direction ω Lighting scene The drawing result, M m (o,ω) represents observing the light source from the camera position o to the direction ω The illuminated object group m is inserted into the 3D scene Regarding the mask image of object group m, that is, emitting light from camera position o to direction ω, if the first intersecting object belongs to object group m, then M m (o,ω) the result is 1, otherwise 0, Indicates observing the light source from the camera position o to the direction ω The illuminated object group m is inserted into the 3D scene The rendering results, light transmission change field Describes inserting the object group m into the light source Lighting scene Specifically, if the observed surface belongs to the newly inserted object group m, the light transmission change field The result is the rendering result after the object group m is inserted; otherwise, the light transmission change field The result is the change in the rendering result of the observed surface before and after the insertion of object group m.

[0087] In this embodiment, the neural network needs to be parameter optimized before being applied to calculate the light transmission variation field. The loss function used in the parameter optimization is the difference between the predicted value of the light transmission variation field and the true value label of the light transmission variation field. Based on the above definition of the light transmission variation field, the true value label of the light transmission variation field is obtained by the following method:

[0088]

[0089] in, represents the true value label of the light transmission variation field, Indicates the true value of the drawing result before the object group m is inserted, Represents the true value of the drawing result after the object group m is inserted.

[0090] In this embodiment, light source sampling data, target view-related data, and object group-related data are required to undergo data space transformation and encoding before being input into the neural network to calculate the light transmission change field, including:

[0091] For the light source sampling data, the geometric features are transformed from world space to the local space of the object group. A first encoder is then used to encode the transformed light source sampling data to obtain a light source code. The first encoder can employ a neural network. The geometric features contained in the light source sampling data are coordinate-transformed using a transformation matrix from world space to local space. The transformed geometric features, along with the light intensity and sampling probability, form the transformed light source sampling data.

[0092] For the object group related data, the geometric features of the self-sampling data, scene observation data and occlusion observation data included therein are transformed from the world space to the local space of the object group, and then the transformed self-sampling data, scene observation data and occlusion observation data are respectively encoded by a second encoder to obtain self-encoding, environmental encoding and occlusion encoding. Among them, the second encoder can adopt a neural network. Similarly, the geometric features contained in the self-sampling data are coordinate-transformed by the transformation matrix from the world space to the local space, and the geometric features after the coordinate transformation and the material features at the sampling point constitute the transformed self-sampling data. Similarly, the geometric features contained in the scene observation data are coordinate-transformed by the transformation matrix from the world space to the local space, and the geometric features after the coordinate transformation and the material features of the scene surface constitute the transformed scene observation data. Similarly, the geometric features contained in the occlusion observation data are coordinate-transformed by the transformation matrix from the world space to the local space, and the geometric features after the coordinate transformation are used as the transformed occlusion observation data.

[0093] For the target view-related data, the geometric features are transformed from world space to the local space of the object group to obtain the transformed target view-related data. Similarly, the geometric features contained in the target view-related data are transformed using the transformation matrix from world space to local space. The transformed geometric features and the material features of the visible part form the transformed target view-related data.

[0094] In an embodiment, after the spatial transformation and encoding are completed, the light source code, self-code, environment code, occlusion code and the transformed target perspective related data are input into the neural network, and the implicit light transfer function embodied by the neural network is used for transmission calculation to output the light transfer change field of the object group in the three-dimensional scene after insertion.

[0095] (c) Finally, the light transmission change field is combined with the mask map of the object group and merged into the drawing result before the object group is inserted through matrix calculation to obtain the drawing result after the object group is inserted.

[0096] In this embodiment, after obtaining the light transmission change field of the object group in the three-dimensional scene after insertion, the light transmission change field is combined with the mask map of the object group through matrix calculation to merge into the rendering result before the object group is inserted:

[0097]

[0098] Among them, M m (o,ω) represents observing the light source from the camera position o to the direction ω The illuminated object group m is inserted into the 3D scene Regarding the mask map of object group m, Indicates that the object group m is under the light source Lighting 3D scene The light transmission variation field in Indicates the drawing result before the object group m is inserted. Indicates the drawing result after object group m is inserted.

[0099] Step 5: After all object groups have been rendered, the dynamic rendering results of the three-dimensional scene are obtained and output.

[0100] In the embodiment, after all object groups are traversed, the rendering result after the last object group is inserted is output as the dynamic rendering result of the three-dimensional scene.

[0101] In one embodiment, when a non-global illumination drawing result is required, the light source code, self-code, environment code and transformed target perspective related data are input into the neural network for prediction to obtain the corresponding light transmission change field, which is then merged to obtain the drawing result after the object group is inserted.

[0102] In one embodiment, when global illumination rendering results are required, the light transmission variation field of the global illumination can be decomposed. Specifically, the light transmission variation field can be further decomposed into a first light transmission variation field under direct illumination and a second light transmission variation field under indirect illumination. Based on this, the global illumination rendering results obtained include:

[0103] (a) First, the light source code, self-coder, occlusion code and the transformed target perspective related data are input into the first neural network, and the first light transmission change field of the object group under direct lighting in the three-dimensional scene after insertion is predicted and calculated; the light source code, self-coder, environment code and the transformed target perspective related data are input into the second neural network, and the second light transmission change field of the object group under indirect lighting in the three-dimensional scene after insertion is predicted and calculated.

[0104] In this embodiment, the first neural network needs to undergo parameter optimization before being applied. The loss function used in the parameter optimization is the difference between the predicted value of the first light transmission variation field and the true value label of the first light transmission variation field. Also based on the definition of the light transmission variation field of the neural network, the true value label of the first light transmission variation field is obtained through the following process:

[0105]

[0106] in, Indicates that the object group m is under the light source The current 3D scene being illuminated The ground truth label of the first light transport variation field in , Indicates the true value of the direct light rendering result before the object group m is inserted. Indicates the true value of the direct light rendering result after the object group m is inserted.

[0107] In this embodiment, the second neural network needs to undergo parameter optimization before being applied. The loss function used in the parameter optimization is the difference between the predicted value of the second light transmission change field and the true value label of the second light transmission change field. Also based on the definition of the light transmission change field of the neural network, the true value label of the second light transmission change field is obtained through the following process:

[0108]

[0109] in, Indicates that the object group m is under the light source The current 3D scene being illuminated The true value label of the second light transmission change field in, Indicates the true value of the indirect light rendering result before the object group m is inserted. Indicates the true value of the indirect light rendering result after the object group m is inserted.

[0110] (b) Then, the first light transmission variation field is combined with the mask map of the object group and merged into the direct light rendering result before the object group is inserted through matrix calculation to obtain the direct light rendering result after the object group is inserted; the second light transmission variation field is combined with the mask map of the object group and merged into the indirect light rendering result before the object group is inserted through matrix calculation to obtain the indirect light rendering result after the object group is inserted. This can be expressed as:

[0111]

[0112] in, represents the first light transmission change field of object group m under direct illumination, Represents the direct light rendering result before object group m is inserted. Indicates the direct light rendering result after the object group is inserted, Mm (o,ω) represents the mask image of the object group m;

[0113]

[0114] in, represents the second light transmission change field of object group m under indirect lighting, Indicates the indirect light drawing result before the object group m is inserted. Indicates the indirect light rendering result after the object group is inserted, M m (o,ω) represents the mask image of the object group m;

[0115] (c) Finally, the direct light rendering result and the indirect light rendering result after the object group is inserted are summed to obtain the global illumination rendering result after the object group is inserted. This can be expressed as:

[0116]

[0117] in, Indicates the drawing result of global illumination after the object group is inserted.

[0118] The rendering method based on implicit light transfer function merging provided in the above embodiment uses a neural network to predict the light transfer change field of each object group in the scene. The light transfer change field is an object (group)-oriented function. These light transfer change fields are merged through matrix calculation to quickly obtain dynamic rendering results. It is suitable for dynamic rendering of various three-dimensional scenes and improves the versatility of rendering. Figure 2 The following is an example of the global illumination rendering result obtained by inserting the object group into the three-dimensional scene using the above-mentioned rendering method based on implicit light transfer function merging. Figure 2 The rendering method based on implicit light transfer function merging provided in the above embodiment supports flexible scene editing by inserting object groups and sampling materials of scene observation data.

[0119] Figure 3 : is a structural diagram of a drawing device based on implicit light transfer function merging provided in an embodiment. Figure 3 As shown, the embodiment also provides a rendering device based on implicit light transfer function merging, including:

[0120] An object division module is used to obtain a three-dimensional scene, divide all objects in the three-dimensional scene except the light source into multiple object groups, and define the insertion order of the object groups;

[0121] A light source sampling module is used to sample all light sources in a three-dimensional scene to obtain light source sampling data;

[0122] A target perspective related data generation module is used to generate geometric features and material features of the visible parts of the three-dimensional scene under the target perspective to form target perspective related data, and is also used to generate a mask map of each object group under the target perspective;

[0123] The dynamic rendering module is used to predict the light transmission change field corresponding to each object group by using the implicit light transmission function calculation embodied by the neural network and merge them to obtain the rendering results.

[0124] Specifically, in the dynamic rendering module, for each object group, the following process is performed: first, the object group is inserted into the three-dimensional scene before insertion in the insertion order, and self-sampling data, scene observation data, and occlusion observation data of the object group are generated to form the object group related data; then, based on the light source sampling data, target perspective related data, and object group related data, the implicit light transfer function embodied by the neural network is used to calculate and predict the light transfer change field of the object group in the three-dimensional scene after insertion; finally, the light transfer change field is combined with the mask map of the object group through matrix calculation and merged into the rendering result before the object group is inserted, so as to obtain the rendering result after the object group is inserted;

[0125] The output module is used to obtain and output the dynamic rendering results of the three-dimensional scene after all object groups have been rendered by the dynamic rendering module.

[0126] It should be noted that the aforementioned embodiments of the rendering device based on ILF merging are illustrated using the aforementioned functional modules for dynamic rendering. The aforementioned functions can be assigned to different functional modules as needed, i.e., the internal structure of the terminal or server can be divided into different functional modules to perform all or part of the functions described above. Furthermore, the aforementioned embodiments of the rendering device based on ILF merging and the embodiments of the rendering method based on ILF merging share the same concept. The specific implementation process is detailed in the embodiments of the rendering method based on ILF merging and will not be further elaborated here.

[0127] An embodiment further provides a rendering device based on implicit light transfer function merging, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, rendering steps based on implicit light transfer function merging are implemented, specifically including the following steps:

[0128] Step 1: Obtain a 3D scene, divide all objects in the 3D scene except the light source into multiple object groups, and define the insertion order of the object groups;

[0129] Step 2: Sampling all light sources in the three-dimensional scene to obtain light source sampling data;

[0130] Step 3: Generate geometric features and material features of the visible parts of the three-dimensional scene at the target perspective to form target perspective related data, and also generate a mask map of each object group at the target perspective;

[0131] Step 4: For each object group, the implicit light transfer function embodied by the neural network is used to calculate the light transfer change field corresponding to the object group and merge them to obtain the rendering result.

[0132] In step 4, specifically, for each object group, the following process is performed: first, the object group is inserted into the three-dimensional scene before insertion in the insertion order, and self-sampling data, scene observation data, and occlusion observation data of the object group are generated to form object group related data; then, based on the light source sampling data, target perspective related data, and object group related data, an implicit light transfer function embodied by a neural network is used to calculate and predict the light transfer change field of the object group in the three-dimensional scene after insertion; finally, the light transfer change field is combined with the mask map of the object group through matrix calculation and merged into the rendering result before the object group is inserted, so as to obtain the rendering result after the object group is inserted;

[0133] Step 5: After all object groups have been drawn in step 4, the dynamic drawing result of the three-dimensional scene is obtained and output.

[0134] In practical applications, the memory can be a local volatile memory such as RAM, a non-volatile memory such as ROM, FLASH, a floppy disk, a mechanical hard disk, etc., or a remote storage cloud. The processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a graphics processing unit (GPU), or a neural network processor (NPU). In other words, the rendering method steps based on implicit light transfer function merging can be implemented using these processors.

[0135] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A rendering method based on implicit light transfer function merging, characterized in that: The following steps are involved: Step 1: Obtain a three-dimensional scene, and divide all objects in the three-dimensional scene except the light source into at least one object group and define the insertion order of the object group; Step 2: sampling the light source in the three-dimensional scene to obtain light source sampling data; Step 3: Generate geometric features and material features of the visible parts of the 3D scene at the target perspective to form target perspective related data, and also generate a mask map of each object group at the target perspective; Step 4: For each object group, perform the following process: first, insert the object group into the 3D scene before insertion in the insertion order, and generate object group-related data; then, based on at least one of the light source sampling data, the target view-related data, and the object group-related data, use the implicit light transfer function embodied by the neural network to calculate and predict the light transfer change field of the object group in the 3D scene after insertion; finally, combine the light transfer change field with the mask map of the object group through matrix calculation to the rendering result before the object group is inserted, so as to obtain the rendering result after the object group is inserted; Step 5: After all object groups have been drawn in step 4, the dynamic drawing result of the three-dimensional scene is obtained and output.

2. The rendering method based on implicit light transfer function merging according to claim 1, characterized in that: When sampling light sources in a 3D scene, different light source sampling data is collected based on different light source types, including: When the light source is a surface light source, importance sampling is performed on the light source surface according to the light intensity, and the geometric features, light intensity, and sampling probability at the sampling point are recorded to form the light source sampling data. The geometric features at the sampling point include the sampling point position and the surface normal vector at the sampling point. When the light source is an environment map, the importance sampling of the light direction is performed according to the light intensity. The sampling direction, light intensity and sampling probability are recorded to form the light source sampling data, and the sampling direction is used as a geometric feature. When the light source is a point light source, the point light source position and light intensity are recorded to form the light source sampling data, and the point light source position is used as a geometric feature; When the light source is directional light, the lighting direction and light intensity corresponding to the directional light are recorded to form light source sampling data, and the lighting direction corresponding to the directional light is used as a geometric feature.

3. The rendering method based on implicit light transfer function merging according to claim 1, characterized in that: The object group related data includes self-sampling data, scene observation data and occlusion observation data of the object group; The self-sampling data generation process of the object group includes: uniformly and randomly sampling the surfaces of the objects in the object group, recording the geometric features and material features at the sampling points to form the self-sampling data; The process of generating scene observation data of the object group includes: obtaining a panoramic observation of the three-dimensional scene during the insertion process at the geometric center of the object group, and recording the geometric features and material features of the observed scene surface to form the scene observation data; The process of generating occlusion observation data of the object group includes: using geometric features in the scene observation data as the occlusion observation data.

4. The rendering method based on implicit light transfer function merging according to claim 1, characterized in that: Before light source sampling data, target view-related data, and object group-related data are input into the neural network to calculate the light transmission change field, they need to undergo data space transformation and encoding, including: For the light source sampling data, after transforming the geometric features therein from the world space to the local space of the object group, a first encoder is used to encode the transformed light source sampling data to obtain a light source code; For the object group related data, geometric features of the self-sampled data, scene observation data, and occlusion observation data included therein are transformed from the world space to the local space of the object group, and a second encoder is used to encode the transformed self-sampled data, scene observation data, and occlusion observation data to obtain a self-code, an environment code, and an occlusion code respectively; For the target perspective related data, the geometric features therein are transformed from the world space to the local space of the object group to obtain the transformed target perspective related data; The light source code, self-code, environment code, occlusion code and transformed target perspective related data are input into the neural network, and the implicit light transfer function embodied by the neural network is used for transmission calculation to output the light transfer change field of the object group in the three-dimensional scene after insertion.

5. The rendering method based on implicit light transfer function merging according to claim 1, characterized in that: The following formula is used to combine the light transmission change field with the object group's mask map through matrix calculation to merge it into the drawing result before the object group is inserted: Among them, M m (o,ω) represents observing the light source from the camera position o to the direction ω The illuminated object group m is inserted into the 3D scene Regarding the mask map of object group m, Indicates that the object group m is under the light source Lighting 3D scene The light transmission variation field in Indicates the drawing result before the object group m is inserted. Represents the drawing result after the object group m is inserted; Before being applied, the neural network needs to undergo parameter optimization. The loss function used in parameter optimization is the difference between the predicted value of the light transmission change field and the true value label of the light transmission change field. The true value label of the light transmission change field is obtained by the following method: in, represents the true value label of the light transmission variation field, Indicates the true value of the drawing result before the object group m is inserted, Represents the true value of the drawing result after the object group m is inserted.

6. The rendering method based on implicit light transfer function merging according to claim 4, characterized in that: When global illumination rendering results are needed, first, the light source code, self-coder, occlusion code, and transformed target perspective related data are input into the first neural network, and the first light transmission change field of the object group under direct illumination in the three-dimensional scene after insertion is predicted and calculated. The light source code, self-coder, environment code, and transformed target perspective related data are input into the second neural network, and the second light transmission change field of the object group under indirect illumination in the three-dimensional scene after insertion is predicted and calculated. Then, the first light transmission variation field is combined with the mask map of the object group and merged into the direct light rendering result before the object group is inserted through matrix calculation to obtain the direct light rendering result after the object group is inserted; the second light transmission variation field is combined with the mask map of the object group and merged into the indirect light rendering result before the object group is inserted through matrix calculation to obtain the indirect light rendering result after the object group is inserted; Finally, the direct light rendering result and the indirect light rendering result after the object group is inserted are summed to obtain the global illumination rendering result after the object group is inserted.

7. The rendering method based on implicit light transfer function merging according to claim 5, characterized in that: The first neural network needs to undergo parameter optimization before being applied. The loss function used in the parameter optimization is the difference between the predicted value of the first light transmission variation field and the true value label of the first light transmission variation field. The true value label of the first light transmission variation field is obtained by the following method: Among them, M m (o,ω) represents observing the light source from the camera position o to the direction ω The lighting object group m is inserted into the current 3D scene Regarding the mask map of object group m, Indicates that the object group m is under the light source The current 3D scene being illuminated The ground truth label of the first light transport variation field in , Indicates the true value of the direct light rendering result before the object group m is inserted. Represents the true value of the direct light rendering result after the object group m is inserted; The second neural network needs to undergo parameter optimization before being applied. The loss function used in the parameter optimization is the difference between the predicted value of the second light transmission variation field and the true value label of the second light transmission variation field. The true value label of the second light transmission variation field is obtained by the following method: in, Indicates that the object group m is under the light source The current 3D scene being illuminated The true value label of the second light transmission change field in, Indicates the true value of the indirect light rendering result before the object group m is inserted. Indicates the true value of the indirect light rendering result after the object group m is inserted.

8. A rendering device based on implicit light transfer function merging, characterized in that: include: An object division module is used to obtain a three-dimensional scene, and divide all objects in the three-dimensional scene except the light source into at least one object group and define the insertion order of the object group; A light source sampling module is used to sample light sources in a three-dimensional scene to obtain light source sampling data; A target perspective related data generation module is used to generate geometric features and material features of the visible parts of the three-dimensional scene under the target perspective to form target perspective related data, and is also used to generate a mask map of each object group under the target perspective; A dynamic rendering module is configured to execute the following process for each object group: first, inserting the object group into the three-dimensional scene before insertion in an insertion order and generating object group-related data; then, using an implicit light transfer function embodied by a neural network to calculate a light transfer change field of the object group in the three-dimensional scene after insertion based on at least one of light source sampling data, target view-related data, and object group-related data; finally, merging the light transfer change field and the object group's mask map into a rendering result before the object group is inserted through a matrix calculation method to obtain a rendering result after the object group is inserted; The output module is used to obtain and output the dynamic rendering results of the three-dimensional scene after all object groups have been rendered by the dynamic rendering module.

9. A rendering device based on implicit light transfer function merging, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: Step 1: Obtain a three-dimensional scene, and divide all objects in the three-dimensional scene except the light source into at least one object group and define the insertion order of the object group; Step 2: sampling the light source in the three-dimensional scene to obtain light source sampling data; Step 3: Generate geometric features and material features of the visible parts of the three-dimensional scene at the target perspective to form target perspective related data, and also generate a mask map of each object group at the target perspective; Step 4: For each object group, perform the following process: first, insert the object group into the 3D scene before insertion in the insertion order, and generate object group-related data; then, based on at least one of the light source sampling data, the target view-related data, and the object group-related data, use the implicit light transfer function embodied by the neural network to calculate and predict the light transfer change field of the object group in the 3D scene after insertion; finally, combine the light transfer change field with the mask map of the object group through matrix calculation to the rendering result before the object group is inserted, so as to obtain the rendering result after the object group is inserted; Step 5: After all object groups have been drawn in step 4, the dynamic drawing result of the three-dimensional scene is obtained and output.

Images