Dynamic scene global illumination rendering method, device, system and storage medium based on neural network

Abstract

The present invention discloses a method, device, system and storage medium for global illumination rendering of dynamic scenes based on neural networks, including: generating scene representations for three-dimensional scenes from two aspects, namely, camera perspective and light source; generating global illumination shading results according to the scene representation corresponding to the light source and the scene representation corresponding to the camera perspective as inputs of the neural network. In this way, illumination coding, illumination mixing and illumination decoding are used to realize real-time global illumination prediction of dynamic scenes, making complex illumination, a large number of dynamic objects, animations, etc. possible, and being able to perform real-time global illumination simulation of full dynamic scenes under complex light sources.

Description

Technical Field

[0001] The present invention belongs to the technical field of design graphics drawing, and in particular relates to a method, device, system and storage medium for dynamic scene global illumination drawing based on a neural network. Background Art

[0002] Three-dimensional graphics rendering is a process of generating images from three-dimensional models. The characteristics of a graphics rendering method are mainly reflected in two aspects: one is the time taken to generate the image, and the other is the quality of the generated image. According to the characteristics of different rendering methods, rendering methods are divided into offline rendering and real-time rendering. Offline rendering technology is usually used for high-quality simulation in three-dimensional scenes such as film and television special effects, and real-time rendering is mostly used in three-dimensional games, large-scale scene interactive display, virtual reality and other fields. At present, with the demands brought by complex application scenarios, people have put forward higher requirements on the generation time and generation quality of rendering methods, which makes traditional rendering methods insufficient.

[0003] The simulation of global illumination is a decisive factor affecting the realism of the generated image. Traditional real-time rendering methods often greatly simplify the calculation of global illumination, making the realism of the generated images insufficient to meet the requirements. On the other hand, simulation based on the Monte Carlo method can obtain unbiased estimation results, but its calculation time is not enough to meet the requirements of real-time rendering, and the generated images often contain noise. At the same time, it is difficult to obtain smooth results that ensure time series continuity.

[0004] In recent years, neural rendering methods have been proposed to try to solve the above problems, hoping to obtain real-time high-realistic global illumination simulation results with good temporal continuity, such as a neural network-based augmented reality rendering method disclosed in a patent application with publication number CN111652963A, and a real-time global illumination rendering method based on radial basis function neural network fitting disclosed in a patent application with publication number CN105389843A. However, these methods often have certain constraints on the scene, which makes them only support static scenes, or rigid body transformations of some objects, and cannot meet the real-time global illumination rendering requirements of fully dynamic scenes under complex light sources (dynamic light sources, materials, camera parameters, objects with animation, etc.). Summary of the invention

[0005] In view of the above, the purpose of the present invention is to provide a method, device, system and storage medium for global illumination rendering of dynamic scenes based on neural networks, which can perform real-time global illumination simulation of fully dynamic scenes under complex light sources.

[0006] In order to achieve the above-mentioned invention object, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention further provides a method for rendering global illumination of a dynamic scene based on a neural network, characterized in that it comprises the following steps:

[0008] Generate scene representation for the 3D scene from the camera perspective and light source;

[0009] The global illumination shading result is generated based on the scene representation corresponding to the light source and the scene representation corresponding to the camera perspective as the input of the neural network.

[0010] Preferably, the method further comprises: using the scene representation corresponding to the light source and the scene representation corresponding to the camera perspective as inputs of the neural network to generate a global illumination shading result.

[0011] According to the light source information in the three-dimensional scene and the scene representation corresponding to each light source, the neural network is used to perform illumination encoding to obtain the neural representation of each light source;

[0012] The neural network based on the attention mechanism mixes the neural representation of all light sources in the three-dimensional scene with the scene representation corresponding to the camera perspective to obtain a neural representation containing all light source information;

[0013] The neural representation containing all light source information is mixed with the scene representation corresponding to the camera perspective and decoded by the neural network to obtain the global illumination shading result.

[0014] Preferably, the scene representation comprises geometry buffer information and depth information.

[0015] Preferably, according to the light source information in the three-dimensional scene and the scene representation corresponding to each light source, a neural network is used to perform illumination encoding to obtain a neural representation of each light source, including:

[0016] Generate at least one corresponding virtual point light source and screen space shading clues of shadows and highlights for each light source according to the light source information in the three-dimensional scene and the scene representation corresponding to each light source, and obtain the neural representation of each light source through neural network encoding;

[0017] Among them, virtual point light sources are divided into two categories: direct light sources and indirect light sources. The direct light sources are obtained by uniform random sampling on the surface of the light source, and the indirect light sources are each texel stored in the geometry buffer information.

[0018] Preferably, for the screen space shading clue of the shadow, the distance from the occluded object to the light source corresponding to each shading point is obtained by sampling the shadow map, and the distance from the shading point to the light source is combined to obtain the shadow representation as the screen space shading clue of the shadow by performing difference and division operations;

[0019] For the screen space shading clue of the highlight, the direction from the shading point to the light source and the half-angle vector obtained by considering the camera position constitute the screen highlight representation, which serves as the screen space shading clue of the highlight.

[0020] Preferably, the virtual point light source, and the screen space shading cues of shadows and highlights are encoded by a neural network to obtain a neural representation of each light source, including:

[0021] For all virtual point light sources corresponding to each light source, a virtual point light source encoding network is used to encode each virtual point light source to obtain a neural representation of the virtual point light source, and a pooling operation is performed on the neural representations of each virtual point light source to obtain a neural representation of the virtual point light source of each light source;

[0022] For the screen space shading clues of shadows and highlights, a shading clue encoding network is used to encode the screen space shading clues of shadows and the screen space shading clues of highlights respectively to obtain the neural representation of the screen space shading clues;

[0023] The neural representation of each light source is obtained by combining the neural representation of the virtual point light source and the neural representation of the screen-space shading cues.

[0024] Preferably, the neural network based on the attention mechanism mixes the neural representation of all light sources of the three-dimensional scene with the scene representation to obtain a neural representation containing all light source information, including:

[0025] The scene representation is used as the query information of the attention mechanism, and the neural representation of each light source is used as the key information. The mixing coefficient of each light source is calculated, and the neural representation of each light source is used as the value information. The value information of all light sources is weighted and summed in combination with the mixing coefficient to obtain the neural representation containing all light source information.

[0026] Preferably, the neural representation containing all light source information is mixed with the scene representation and decoded by a neural network to obtain a global illumination result, including:

[0027] Neural network decoding includes direct lighting decoding operations, indirect lighting decoding operations, and shadow decoding operations;

[0028] The neural representation containing all light source information is combined with the scene representation corresponding to the camera view, and then subjected to direct lighting decoding operation, indirect lighting decoding operation and shadow decoding operation respectively, to obtain the direct lighting, indirect lighting and shadow under the current camera view;

[0029] The global illumination shading result is obtained by mixing direct lighting with shadows and adding indirect lighting.

[0030] In a second aspect, an embodiment of the present invention further provides a dynamic scene global illumination rendering device based on a neural network, comprising:

[0031] A scene representation module is used to generate scene representations of a three-dimensional scene from two aspects: camera perspective and light source.

[0032] The rendering module is used to generate a global illumination shading result based on the scene representation corresponding to the light source and the scene representation corresponding to the camera perspective as the input of the neural network.

[0033] In the third aspect, an embodiment of the present invention also provides a dynamic scene global illumination rendering system based on a neural network, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned dynamic scene global illumination rendering method based on a neural network when executing the computer program.

[0034] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium on which a computer program is stored, and when the computer program is processed and executed, the above-mentioned dynamic scene global illumination rendering method based on neural network is implemented.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] Based on the scene representations obtained from the camera and light source respectively during scene rasterization, real-time global illumination prediction of dynamic scenes is achieved using illumination encoding, illumination mixing, and illumination decoding. This makes complex lighting, a large number of dynamic objects, animation, etc. possible, and enables real-time global illumination simulation of fully dynamic scenes under complex light sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 is a flow chart of a method for rendering global illumination of a dynamic scene based on a neural network provided by an embodiment of the present invention;

[0039] Figure 2 is a flow chart of generating global illumination shading results provided by an embodiment of the present invention;

[0040] Figure 3 is a flowchart of generating global illumination shading results provided by an embodiment of the present invention;

[0041] Figure 4 is a flowchart of illumination coding provided by an embodiment of the present invention;

[0042] Figure 5 is a structural schematic diagram of a dynamic scene global illumination rendering device based on a neural network provided by an embodiment of the present invention;

[0043] Figure 6 It is a structural schematic diagram of a drawing module provided in an embodiment. DETAILED DESCRIPTION

[0044] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0045] In order to realize real-time global illumination calculation for dynamic scenes, the embodiments of the present invention provide a method, device, system and storage medium for global illumination rendering of dynamic scenes based on neural networks. Based on the multi-light rendering framework, multiple neural network modules are used to effectively learn the shading results in the three-dimensional scene, and obtain high-quality global illumination results in real time in the dynamic scene.

[0046] like Figure 1 The real-time global illumination method for dynamic scenes provided in the embodiment includes the following steps:

[0047] S110, generating a scene representation for the three-dimensional scene from two aspects, namely, the camera perspective and the light source.

[0048] In the embodiment, a rasterization technique is used to obtain a scene representation of a three-dimensional scene with the camera and the center of each light source as the observation point. Specifically, for the camera, a rasterization technique, a ray casting technique, etc. are used to obtain a scene representation of the three-dimensional scene from the camera's perspective, and the scene representation includes geometry buffer (G-Buffers) information, wherein the geometry buffer contains scene and material related information such as position, normal, reflectivity, and roughness.

[0049] For each light source, rasterization technology, ray casting technology, etc. are also used to obtain the scene representation of the three-dimensional scene from the perspective of each light source. In addition to the geometric buffer (G-Buffers) information, the scene representation also includes depth information, which is similar to the shadow map. Considering compatibility with different light source types, when rasterizing the scene from the light source's perspective, the center point of the light source is used as the observation point. The light source's observation results of the scene are stored in a hexahedral map, thereby ensuring that the geometric buffer (G-Buffers) information in all directions is stored.

[0050] S120, generating a global illumination shading result according to the scene representation corresponding to the light source and the scene representation corresponding to the camera perspective as inputs of the neural network.

[0051] In an embodiment, the scene representation corresponding to the light source and the scene representation corresponding to the camera perspective can be directly input into a neural network, and a global illumination shading result is generated through calculation by the neural network.

[0052] In another embodiment, a scene representation corresponding to the light source and a scene representation corresponding to the camera view are used as inputs of a neural network to generate a global illumination shading result, such as Figure 2 and Figure 3 As shown, it also includes:

[0053] S121, according to the light source information in the three-dimensional scene and the scene representation corresponding to each light source, a neural network is used to perform illumination encoding to obtain a neural representation of each light source. Specifically, according to the light source information in the three-dimensional scene and the scene representation corresponding to each light source, at least one corresponding virtual point light source and screen space shading cues of shadows and highlights are generated for each light source, and the neural network encoding is performed to obtain a neural representation of each light source.

[0054] For each light source, at least one virtual point light source is first generated for each light source. Figure 4 As shown, virtual point light sources are divided into two categories: direct light sources and indirect light sources. The direct light sources are obtained by uniform random sampling on the light source surface, and the indirect light sources are each texel stored in the geometry buffer information.

[0055] For these two types of light sources, a virtual point light source encoding network is used to encode each virtual point light source to obtain the neural representation of the virtual point light source, and a pooling operation is performed on the neural representations of each virtual point light source to obtain the virtual point light source neural representation of each light source.

[0056] For each light source in the scene, screen space shading clues for shadows and highlights corresponding to each light source are also generated. For the screen space shading clues for shadows, the distance from the occluded object to the light source corresponding to each shading point is obtained by sampling the shadow map, and the distance from the shading point to the light source is combined to obtain the shadow representation as the screen space shading clue for the shadow through difference and division operations; for the screen space shading clues for highlights, the direction from the shading point to the light source and the half-angle vector obtained by considering the camera position constitute the screen highlight representation, which is used as the screen space shading clue for the highlight.

[0057] After obtaining the screen space shading cues of shadows and highlights, a shading cue encoding network is used to encode the screen space shading cues of shadows and the screen space shading cues of highlights respectively to obtain a neural representation of the screen space shading cues.

[0058] Finally, the neural representation of each light source is combined with the neural representation of the virtual point light source and the neural representation of the screen space shading cue to obtain the neural representation of each light source. Figure 3 As shown, for each light source, the above encoding process is performed through light source encoding. After all light source encoding is completed, step S122 is continued.

[0059] S122, a neural network based on an attention mechanism mixes the neural representation of all light sources in the three-dimensional scene with the scene representation corresponding to the camera perspective to obtain a neural representation containing all light source information.

[0060] A neural network based on an attention mechanism is used to mix the neural representations of all light sources in the scene to obtain a neural representation that can represent all light source information in the scene. Considering that the light source information required for each shading point during the shading process is different, in an embodiment, the scene representation is used as the query information of the attention mechanism, and the neural representation of each light source is used as the key information, the mixing coefficient of each light source is calculated, and the neural representation of each light source is used as the value information, and the value information of all light sources is weighted and summed in combination with the mixing coefficient to obtain a neural representation containing all light source information.

[0061] S123, mixing the neural representation containing all light source information with the scene representation corresponding to the camera perspective and decoding them through a neural network to obtain a global illumination shading result.

[0062] After obtaining the neural representation of all light sources in the scene, the final global illumination shading result with global illumination effect can be obtained through illumination decoding combined with the information of the current shading point.

[0063] The input of the illumination decoding process is the combination of the neural representation containing all light source information and the scene representation corresponding to the camera perspective. Neural network decoding includes direct illumination decoding operations, indirect illumination decoding operations, and shadow decoding operations. Each type of decoding operation has a similar network structure, using a neural network based on a multi-layer perceptron. Such a network structure makes each pixel relatively independent and can obtain complete illumination information in the scene, which can be lightweight.

[0064] During specific decoding, the neural representation containing all light source information is combined with the scene representation corresponding to the camera perspective and then undergoes direct lighting decoding operations, indirect lighting decoding operations, and shadow decoding operations to obtain the direct lighting, indirect lighting, and shadows under the current camera perspective. The direct lighting and shadows are mixed and the indirect lighting is added to obtain the global illumination shading result.

[0065] The neural network-based global illumination rendering method for dynamic scenes provided in the embodiment can effectively learn the illumination distribution in the dynamic scene through the above steps, and predict the global illumination results under variable light sources, materials, and object animations in real time based on the scene representation from the camera and light sources in the scene.

[0066] like Figure 5 As shown, the embodiment also provides a dynamic scene global illumination drawing device 500 based on a neural network, a scene representation module 510 and a drawing module 520, wherein the scene representation module 510 is used to generate a scene representation for a three-dimensional scene from two aspects, namely, a camera perspective and a light source; the drawing module 520 is used to generate a global illumination shading result based on the scene representation corresponding to the light source and the scene representation corresponding to the camera perspective as inputs of the neural network.

[0067] In one embodiment, Figure 6 As shown, the drawing module 520 includes an illumination encoding module 521, an illumination mixing module 522, and an illumination decoding module 523, wherein the illumination encoding module 521 is used to obtain a neural representation of each light source by performing illumination encoding through a neural network according to the light source information in the three-dimensional scene and the scene representation corresponding to each light source; the illumination mixing module 522 is used to mix the neural representation of all light sources in the three-dimensional scene with the scene representation corresponding to the camera perspective based on a neural network of an attention mechanism to obtain a neural representation containing all light source information; the illumination decoding module 523 is used to mix the neural representation containing all light source information with the scene representation corresponding to the camera perspective and obtain a global illumination shading result after decoding through a neural network.

[0068] It should be noted that when the global illumination drawing device for dynamic scenes based on a neural network provided in the embodiment performs global illumination drawing for dynamic scenes, the division of the above-mentioned functional modules shall be used as an example for illustration, and the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the global illumination drawing device for dynamic scenes based on a neural network provided in the embodiment belongs to the same concept as the embodiment of the global illumination drawing method for dynamic scenes based on a neural network, and its specific implementation process is detailed in the embodiment of the global illumination drawing method for dynamic scenes based on a neural network, which will not be repeated here.

[0069] The embodiment further provides a dynamic scene global illumination rendering system based on a neural network, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the dynamic scene global illumination rendering method based on a neural network is implemented, comprising the following steps:

[0070] S110, generating scene representations for the three-dimensional scene from two aspects, namely, camera perspective and light source;

[0071] S120, generating a global illumination shading result according to the scene representation corresponding to the light source and the scene representation corresponding to the camera perspective as inputs of the neural network.

[0072] The embodiment further provides a computer storage medium on which a computer program is stored. When the computer program is processed and executed, the method for rendering global illumination of dynamic scenes based on a neural network is implemented, comprising the following steps:

[0073] S110, generating scene representations for the three-dimensional scene from two aspects, namely, camera perspective and light source;

[0074] S120, generating a global illumination shading result according to the scene representation corresponding to the light source and the scene representation corresponding to the camera perspective as inputs of the neural network.

[0075] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0076] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for rendering global illumination of dynamic scenes based on a neural network, characterized in that: The following steps are involved: Generate scene representation for the 3D scene from the camera perspective and light source; A global illumination shading result is generated based on the scene representation corresponding to the light source and the scene representation corresponding to the camera perspective as inputs of the neural network, including: according to the light source information in the three-dimensional scene and the scene representation corresponding to each light source, illumination encoding is performed through a neural network to obtain a neural representation of each light source; a neural network based on an attention mechanism mixes the neural representations of all light sources in the three-dimensional scene with the scene representation corresponding to the camera perspective to obtain a neural representation containing all light source information, specifically including: using the scene representation as query information of the attention mechanism, and using the neural representation of each light source as key information, calculating a mixing coefficient for each light source, and using the neural representation of each light source as value information, and performing weighted summation of the value information of all light sources in combination with the mixing coefficient to obtain a neural representation containing all light source information; the neural representation containing all light source information is mixed with the scene representation corresponding to the camera perspective and decoded through a neural network to obtain a global illumination shading result.

2. The method for rendering global illumination of dynamic scenes based on a neural network according to claim 1, characterized in that: According to the light source information in the 3D scene and the scene representation corresponding to each light source, the neural network is used to encode the illumination to obtain the neural representation of each light source, including: Generate at least one corresponding virtual point light source and screen space shading clues of shadows and highlights for each light source according to the light source information in the three-dimensional scene and the scene representation corresponding to each light source, and obtain the neural representation of each light source through neural network encoding; Among them, virtual point light sources are divided into two categories: direct light sources and indirect light sources. The direct light sources are obtained by uniform random sampling on the surface of the light source, and the indirect light sources are each texel stored in the geometry buffer information.

3. The method for rendering global illumination of dynamic scenes based on a neural network according to claim 2, characterized in that: For the screen space shading clue of the shadow, the distance from the occluded object to the light source corresponding to each shading point is obtained by sampling the shadow map. At the same time, the distance from the shading point to the light source is combined, and the shadow representation is obtained by subtraction and division operations as the screen space shading clue of the shadow; For the screen space shading clue of the highlight, the direction from the shading point to the light source and the half-angle vector obtained by considering the camera position constitute the screen highlight representation, which serves as the screen space shading clue of the highlight.

4. The method for rendering global illumination of dynamic scenes based on a neural network according to claim 2, characterized in that: The virtual point light sources, as well as the screen space shading cues for shadows and highlights, are encoded with a neural network to obtain a neural representation of each light source, including: For all virtual point light sources corresponding to each light source, a virtual point light source encoding network is used to encode each virtual point light source to obtain a neural representation of the virtual point light source, and a pooling operation is performed on the neural representations of each virtual point light source to obtain a neural representation of the virtual point light source of each light source; For the screen space shading clues of shadows and highlights, a shading clue encoding network is used to encode the screen space shading clues of shadows and the screen space shading clues of highlights respectively to obtain the neural representation of the screen space shading clues; The neural representation of each light source is obtained by combining the neural representation of the virtual point light source and the neural representation of the screen-space shading cues.

5. The method for rendering global illumination of dynamic scenes based on a neural network according to claim 1, characterized in that: The neural representation containing all light source information is mixed with the scene representation and decoded by the neural network to obtain the global illumination result, including: Neural network decoding includes direct lighting decoding operations, indirect lighting decoding operations, and shadow decoding operations; The neural representation containing all light source information is combined with the scene representation corresponding to the camera view, and then subjected to direct lighting decoding operation, indirect lighting decoding operation and shadow decoding operation respectively, to obtain the direct lighting, indirect lighting and shadow under the current camera view; The global illumination shading result is obtained by mixing direct lighting with shadows and adding indirect lighting.

6. A dynamic scene global illumination rendering device based on a neural network, characterized in that: include: A scene representation module is used to generate scene representations of a three-dimensional scene from two aspects: camera perspective and light source. A drawing module is used to generate a global illumination shading result based on the scene representation corresponding to the light source and the scene representation corresponding to the camera perspective as inputs of the neural network, including: according to the light source information in the three-dimensional scene and the scene representation corresponding to each light source, the neural network performs illumination encoding to obtain a neural representation of each light source; the neural network based on the attention mechanism mixes the neural representations of all light sources in the three-dimensional scene with the scene representation corresponding to the camera perspective to obtain a neural representation containing all light source information, specifically including: using the scene representation as the query information of the attention mechanism, and the neural representation of each light source as key information, calculating the mixing coefficient of each light source, and using the neural representation of each light source as value information, and combining the mixing coefficient to perform weighted summation of the value information of all light sources to obtain a neural representation containing all light source information; the neural representation containing all light source information is mixed with the scene representation corresponding to the camera perspective and decoded through the neural network to obtain a global illumination shading result.

7. A dynamic scene global illumination rendering system based on a neural network, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the neural network-based dynamic scene global illumination rendering method described in any one of claims 1 to 5 is implemented.

8. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is processed and executed, the neural network-based dynamic scene global illumination rendering method described in any one of claims 1 to 5 is implemented.

Images