Construction method of ambient light shielding model and ambient light shielding method

Abstract

The invention belongs to the related technical field of real-time rendering, and particularly relates to a construction method of an ambient light shielding model and an ambient light shielding method, the model is constructed based on a convolutional neural network in deep learning, and the proposed convolutional neural network comprises a patch module, an encoder, a decoder and a reverse patch module; the patch module is used for segmenting an input feature map with the size of h * w * c into feature maps with the size of # imgabs0 #; the encoder is used for encoding the characteristic pattern with the size of # imgabs1 #; the decoder is used for decoding the output of the encoder; and the reverse patch module is used for obtaining an image with the size of h * w * c based on the decoding features. And after the patch module divides the image into patches, each patch can be independently processed in parallel, so that the calculation efficiency is improved. According to the method, only a small amount of extra space is needed to store network parameters, meanwhile, a high-quality ambient light shielding effect can be generated in a short time, and application requirements can be met in a realistic manner.

Description

Technical Field

[0001] The present invention belongs to the technical field related to real-time rendering, and more specifically, relates to a method for constructing an ambient light occlusion model and an ambient light occlusion method. Background Art

[0002] Ambient occlusion technology is a common global illumination technology. It simulates the phenomenon of complex objects blocking each other's ambient light at a low cost, greatly improving the information richness of the rendered image. Ambient light refers to a uniform light field formed by multiple diffuse reflections of light in a scene under the illumination of a strong light source (such as sunlight).

[0003] Currently, rendering is divided into offline rendering and real-time rendering. Real-time physically based ambient occlusion is not possible on current hardware platforms, so in offline renderers, it is often assumed that ambient light is a smooth and uniform field function that changes slowly with position and approximates these functions. Due to the uniformity of the light field function, general ambient light shading results are relatively smooth and lack details. This problem can be solved by multiplying an ambient occlusion attenuation term during the shading process. The definition of the ambient occlusion attenuation term is in is the visibility term, which describes the neighborhood direction at the x position ω Whether there is an object blocking the image, if blocked, it is 0, if not blocked, it is 1. For the sunken parts in the scene (such as the corners of the wall, the small holes on the ground), the attenuation coefficient well describes the phenomenon that part of the ambient light cannot reach the surface due to the occlusion of the surrounding geometry, which greatly enhances the three-dimensional sense of the whole scene.

[0004] There are two types of ambient occlusion (AO) commonly used in traditional real-time rendering, AO based on pre-calculation and AO based on real-time calculation in screen space. Pre-calculated AO writes the accurate numerical solution of the ambient occlusion term into the map and samples the map to obtain the corresponding value in real-time rendering. The screen space method occurs after the normal rendering process, generates a series of sample points in the neighborhood of the shaded pixel in the screen space, obtains scene information by sampling the depth map obtained by rendering, and estimates the value of the ambient occlusion attenuation term.

[0005] Traditional ambient occlusion rendering often faces various challenges. For offline methods based on pre-calculation, additional storage space is required to store pre-calculated AO maps. The storage space required for this method increases linearly with the number of pre-calculated AO maps. For real-time methods based on screen space, although the performance requirements of real-time rendering can be met, since they only rely on visible information on the screen, errors such as light leakage or inaccurate occlusion may occur when processing occlusion and edge effects. At the same time, these algorithms based on screen space often use the Monte Carlo method to estimate the value of the visible function integral. In order to eliminate the noise caused by Monte Carlo, these algorithms often require a denoising algorithm to filter the calculation results, which further brings additional computational overhead. Summary of the invention

[0006] In view of the above defects or improvement needs of the prior art, the present invention provides a method for constructing an ambient light occlusion model and an ambient light occlusion method, the purpose of which is to achieve ambient light occlusion efficiently.

[0007] To achieve the above object, according to one aspect of the present invention, a method for constructing an ambient occlusion model is provided, comprising:

[0008] Construct a convolutional neural network, which includes a patch module, an encoder, a decoder and a reverse patch module; the patch module is used to convert the size of h ×w×c The input feature map is split into The encoder is used to encode a feature map of size The decoder is used to decode the encoded features output by the encoder to obtain decoded features; the reverse patch module splices the decoded features into a size of h ×w×c images;

[0009] The convolutional neural network is trained with the normal map and the depth map as input and the corresponding ambient occlusion map as output to obtain an ambient occlusion model and complete the construction, wherein the normal map and the depth map are stacked and fused as the input feature map.

[0010] Furthermore, the convolutional neural network is a U-Net structure.

[0011] Furthermore, the convolution module in the encoder is a depth-separable convolution module, and / or the convolution module in the decoder is a depth-separable convolution module.

[0012] Furthermore, the depthwise separable convolution module is used as a n×n×m The equivalent replacement of the convolution module consists of two convolution blocks with shapes n×n×1The spatial convolution of and shape is 1×1×m The depth of convolution.

[0013] Furthermore, each training sample is constructed in the following way:

[0014] Through the ray tracing renderer, the normal, depth and AO value of each pixel in the scene at a certain perspective are calculated to obtain the normal map, depth map and ambient occlusion map at that perspective, forming a training sample.

[0015] According to another aspect of the present invention, there is provided an ambient light shielding method, comprising:

[0016] Using a rasterizer to render the target scene at the perspective to be rendered, and obtaining a depth map and a normal map at the perspective to be rendered;

[0017] The depth map and the normal map are fused and input into an ambient occlusion model to obtain an ambient occlusion map under the perspective to be rendered, thereby completing ambient occlusion, wherein the ambient occlusion model is constructed by the ambient occlusion model construction method as described above.

[0018] A rendering method with ambient occlusion, characterized by comprising:

[0019] Using an ambient light occlusion method as described above, an occlusion map of the target scene to be rendered is obtained;

[0020] Rendering the target scene without opening ambient occlusion at the perspective to be rendered to obtain a rendering result;

[0021] The occlusion map and the rendering result are merged to achieve rendering with ambient occlusion.

[0022] According to another aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the method steps as described above, the method steps as described above, and / or the method steps as described above when executing the computer program.

[0023] According to another aspect of the present invention, a computer-readable storage medium is provided, which includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the method steps described above, the method steps described above, and / or the method steps described above.

[0024] According to another aspect of the present invention, a computer program product is provided, comprising a computer program or instructions, characterized in that when the computer program or instructions are executed by a processor, the method steps as described above, the method steps as described above and / or the method steps as described above are implemented.

[0025] In general, compared with the prior art, the technical solution conceived by the present invention has the following beneficial effects:

[0026] 1. The present invention proposes a method for constructing an ambient light occlusion model. The proposed convolutional neural network includes a patch module, an encoder, a decoder and a reverse patch module; the patch module is used to split the input feature map of size h×w×c into The encoder is used to map the feature map of size The feature map is encoded to obtain the encoded features; the decoder is used to decode the encoded features output by the encoder to obtain the decoded features; the inverse patch module splices the decoded features into an image of size h×w×c. After the image is divided into patches using the patch module, each patch can be processed independently and in parallel, which greatly improves the computational efficiency. Therefore, the method of the present invention proposes a method based on a convolutional neural network to simulate the AO effect, which only requires a small amount of additional space to store the parameters of the neural network, and can also generate high-quality AO effects in less time, and its realism can meet the application requirements.

[0027] 2. The present invention further preferably uses depthwise separable convolution to replace standard convolution, which has fewer parameters than the existing standard convolution module, can speed up the calculation time of the convolution operation, and can also ensure the simulated AO effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the structure of a convolutional neural network provided by an embodiment of the present invention;

[0029] Figure 2 Schematic diagram of the convolutional neural network structure of the U-Net structure provided by an embodiment of the present invention;

[0030] Figure 3 is a schematic diagram of a normal estimation algorithm provided by an embodiment of the present invention;

[0031] Figure 4 It is an ambient light occlusion effect diagram obtained based on the ambient light occlusion model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0033] Embodiment 1

[0034] A method for constructing an ambient occlusion model, comprising:

[0035] Construct a convolutional neural network, such as Figure 1 As shown in Figure 1, it includes a patch module, an encoder, a decoder, and a reverse patch module; the patch module is used to split the input feature map of size h×w×c into The encoder is used to map the feature map of size The decoder is used to decode the encoded features output by the encoder to obtain decoded features; the reverse patch module splices the decoded features into an image of size h×w×c;

[0036] The convolutional neural network is trained with the normal map and the depth map as input and the corresponding ambient occlusion map as output to obtain the ambient occlusion model and complete the construction, wherein the normal map and the depth map are stacked and fused as the input feature map.

[0037] Regarding the patch module, specifically, the patch module is used to split c input feature maps of size h×w into n×n×c (i.e., n 2 c) size is The square feature subgraph of h×w×c, that is, c in h×w×c is the number of channels of the entire feature graph, n 2 c is the number of channels. The encoder is used to encode n×n×c channels of size The square feature sub-graphs are encoded synchronously, and each size is extracted The features of the square feature subgraph are n×n×c in size The features of the square feature sub-graphs are fused into an overall feature, namely the above-mentioned encoding feature. The reverse patch module will restore the feature map to its original size, i.e., h×w×c, at the end of the method. The method of this embodiment introduces the patch module and the reverse patch module. After the image is divided into patches using the patch module, each patch can be processed independently and in parallel, which greatly improves the computational efficiency.

[0038] As a preferred implementation, the convolutional neural network is a U-Net structure.

[0039] The structure of convolutional neural network is as follows Figure 2 As shown, it includes a patch feature extraction module (i.e., a patch module), a feature encoder, a feature decoder, and a patch feature restoration module (i.e., a reverse patch module). The working principle of the U-Net structure is:

[0040] The input enters the patch feature extraction module (i.e., the patch module) to preliminarily extract low-level features in the image;

[0041] Input low-level features into the feature encoder to extract features from low to high levels. The feature encoder consists of a series of cascaded feature encoder units, each of which consists of a convolution module (to further extract features), an activation module, and a maximum pooling module (to further extract features);

[0042] The output features of the encoding units at each level are input into the feature decoder, and the high-level features are gradually restored to low-level decoding features. The feature decoder is composed of a series of cascaded feature decoder units, each of which is composed of a maximum pooling module, a convolution module, and an activation module. The input of each level of the decoder unit is: the decoding features output by the previous level decoder unit and the fusion features of the encoding features output by the encoder unit at the corresponding level, and the decoding features are obtained;

[0043] The decoded features are input into the inverse patch module to obtain the final ambient occlusion image.

[0044] Regarding the maximum pooling layer, the maximum pooling layer is to h×w×c The input feature map is divided into For each feature subgraph, the maximum pooling layer selects n×n The largest element among the elements is output. These largest elements will eventually form a feature map.

[0045] Regarding the activation module (LeakyReLU layer), the formula of the LeakyReLU activation function is: It solves the problem that the ReLU function has no learnable gradient when the input value is less than 0.

[0046] As a preferred implementation, the convolution module in the encoder is a depth-wise separable convolution module, and / or the convolution module in the decoder is a depth-wise separable convolution module.

[0047] As a further preferred embodiment, the depthwise separable convolution module is used as a n×n×m The equivalent replacement of the convolution module consists of two convolution blocks with shapes n×n×1 The spatial convolution of and shape is 1×1×mThe depth of the convolution, the number of parameters is relatively large n×n×m The convolution module is smaller, which can speed up the computation time of the convolution operation.

[0048] As a preferred implementation, each training sample is constructed in the following way:

[0049] Through the ray tracing renderer, the normal, depth and AO value of each pixel in the scene at a certain perspective are calculated to obtain the normal map, depth map and occlusion map at that perspective, forming a training sample.

[0050] Finally, the trained ambient occlusion model is deployed to the target renderer, including:

[0051] Select a series of target scenes. Use an offline renderer with ray tracing capabilities (e.g. Blender) to render the AO maps of the scenes at different viewpoints. Output the AO maps and the camera space normal maps and camera space depth maps at the corresponding viewpoints into the dataset.

[0052] The normal map and depth map in the dataset are concatenated in the depth dimension and input into the model, and the AO map in the dataset is used as the output of the model to supervise the model training.

[0053] Export the parameters of the trained model and deploy them to the target renderer. The deployment method varies depending on the deployment platform. For example, for the Unity platform, export the trained model to the onnx format and deploy it using the built-in barracuda library.

[0054] Embodiment 2

[0055] An ambient occlusion method, comprising:

[0056] Using a rasterizer to render the target scene at the perspective to be rendered, and obtaining a depth map and a normal map at the perspective to be rendered;

[0057] The depth map and the normal map are fused and input into an ambient occlusion model to obtain an ambient occlusion map under the perspective to be rendered, thereby completing ambient occlusion, wherein the ambient occlusion model is constructed by the ambient occlusion model construction method as described above.

[0058] Render the scene in the target renderer (rasterizer) to obtain information such as the depth map, normal map, and rendering results under the perspective to be rendered.

[0059] As for the depth map, since almost all mainstream real-time rendering methods currently require depth buffer technology to determine the visibility of geometric bodies, after S3 rendering is completed, the depth buffer used in the rendering process can be obtained and used as the depth map input of the model.

[0060] For normal maps, different processing methods are used according to the rendering pipeline. Currently, the mainstream real-time rendering pipeline can be divided into two categories: deferred rendering and forward rendering. These two categories can be discussed separately:

[0061] For the deferred rendering pipeline, since one of the steps in its process is to write the geometric information such as the normals in the scene into the G-Buffer, the normals can be directly extracted from the G-Buffer to obtain the normal map.

[0062] For the forward rendering pipeline, the normal information can be indirectly inferred from the depth map. The algorithm is shown as follows Figure 3 As shown, specifically including:

[0063] Traverse each pixel in the depth map and convert the depth value z of the sampled pixel into a linear camera depth according to the camera parameters near plane distance n and far plane distance f. And output to the linear depth map, the conversion formula is:

[0064]

[0065] The Sobel operator is used to calculate the gradient of the linear depth map on the horizontal and vertical axes of the image space. Specifically, it includes: traversing each pixel in the linear depth map, performing convolution on its 3x3 neighborhood one by one, and the convolution kernel is in

[0066] Traverse each pixel in the screen space, and calculate the gradient of the world space coordinates X and Y in the horizontal and vertical axes of the screen space according to the camera parameter fov angle a and the screen aspect ratio aspect. The formula is:

[0067]

[0068] Where x and y are the coordinates of each pixel in the screen space.

[0069] Traverse each pixel in the screen space and calculate the normal N based on the two estimated gradient directions. The formula is:

[0070]

[0071] For the extracted normal map and depth map, the two maps are merged pixel by pixel, and the merged result is input into the network to obtain the output result of AO.

[0072] The relevant technical solution is the same as that in Embodiment 1 and will not be described in detail here.

[0073] Embodiment 3

[0074] A rendering method with ambient occlusion, comprising:

[0075] Using an ambient light occlusion method as described above, an occlusion map of the target scene to be rendered is obtained;

[0076] Rendering the target scene without opening ambient occlusion at the perspective to be rendered to obtain a rendering result;

[0077] The occlusion map and the rendering result are merged to achieve rendering with ambient occlusion.

[0078] The rendering method with ambient light occlusion provided in this embodiment can render with ambient light occlusion effect in real time, and its realism meets the application requirements, such as Figure 4 As shown, it can also be deployed on mobile platforms.

[0079] Optionally, the rendering method may include two stages, a pre-training stage and a real-time rendering stage. The pre-training stage is similar to other neural network methods, using a pre-prepared data set to train the network. In the real-time rendering stage, the user needs to infer the network in real time during the rendering of each frame and process the picture according to the network inference results, including the following steps:

[0080] Build a convolutional neural network.

[0081] Construct training data sets and test data sets, input the training data sets into the network for iterative training, obtain the trained network, test the network with the test data sets to evaluate the network performance. Finally, deploy the network to the corresponding renderer.

[0082] Get the scene's depth map and other geometric information when rendering the scene.

[0083] Normal information is extracted from the rendering result. For a deferred rendering method, the normal information can be directly extracted from the rendered G-Buffer. For a non-deferred rendering method (eg, forward rendering) in which normals cannot be extracted, the normal value can be estimated by depth.

[0084] The normal and depth data are input into the network for inference to obtain the inferred ambient occlusion result (AO map); finally, the AO map and the rendering result can be fused to achieve rendering with ambient occlusion.

[0085] Compared with the traditional method based on screen space, this method has a more realistic screen performance to meet the needs of users. And compared with the traditional method based on pre-calculation, it greatly reduces the storage space required by the algorithm without significantly increasing the required calculation time.

[0086] The relevant technical solution is the same as that in Embodiment 1 and will not be described in detail here.

[0087] Embodiment 4

[0088] The present application also relates to an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0089] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The memory may be used to store computer programs and / or modules, and the processor may perform various functions of the electronic device by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory.

[0090] The relevant technical solutions are the same as above and will not be described in detail here.

[0091] Embodiment 5

[0092] The present application also relates to a computer-readable storage medium having a computer program stored thereon, and the steps of the above method are implemented when the computer program is executed by a processor.

[0093] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0094] The relevant technical solutions are the same as above and will not be described in detail here.

[0095] Embodiment 6

[0096] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the method of the above embodiment of the present application.

[0097] The relevant technical solutions are the same as above and will not be described in detail here.

[0098] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing an ambient occlusion model, characterized in that: include: Construct a convolutional neural network, which includes a patch module, an encoder, a decoder and a reverse patch module; the patch module is used to convert the size of h ×w×c The input feature map is split into The encoder is used to encode a feature map of size The decoder is used to decode the encoded features output by the encoder to obtain decoded features; The reverse patch module concatenates the decoded features into a size of h ×w×c images; The convolutional neural network is trained with the normal map and the depth map as input and the corresponding ambient occlusion map as output to obtain an ambient occlusion model and complete the construction, wherein the normal map and the depth map are stacked and fused as the input feature map.

2. The construction method according to claim 1, characterized in that: The convolutional neural network is a U-Net structure.

3. The construction method according to claim 1, characterized in that: The convolution module in the encoder is a depth-separable convolution module, and / or the convolution module in the decoder is a depth-separable convolution module.

4. The construction method according to claim 3, characterized in that: The depthwise separable convolution module is used as a n×n×m The equivalent replacement of the convolution module consists of two convolution blocks with shapes n×n×1 The spatial convolution of and shape is 1×1×m The depth of convolution.

5. The construction method according to any one of claims 1 to 4, characterized in that: Each training sample is constructed in the following way: Through the ray tracing renderer, the normal, depth and AO value of each pixel in the scene at a certain perspective are calculated to obtain the normal map, depth map and ambient occlusion map at that perspective, forming a training sample.

6. An ambient light shielding method, characterized in that: include: Using a rasterizer to render the target scene at the perspective to be rendered, and obtaining a depth map and a normal map at the perspective to be rendered; The depth map and the normal map are fused and input into an ambient occlusion model to obtain an ambient occlusion map under the perspective to be rendered, thereby completing ambient occlusion, wherein the ambient occlusion model is constructed by a method for constructing an ambient occlusion model as described in any one of claims 1 to 5.

7. A rendering method with ambient occlusion, characterized in that: include: Using an ambient light occlusion method as claimed in claim 6, obtaining an ambient light occlusion map of the target scene to be rendered from a viewing angle; Rendering the target scene without opening ambient occlusion at the perspective to be rendered to obtain a rendering result; The ambient occlusion map and the rendering result are merged to achieve rendering with ambient occlusion.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method steps according to any one of claims 1 to 5, the method steps according to claim 6 and / or the method steps according to claim 7 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed by a processor, the device where the storage medium is located is controlled to perform the method steps as described in any one of claims 1 to 5, the method steps as described in claim 6 and / or the method steps as described in claim 7.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method steps according to any one of claims 1 to 5, the method steps according to claim 6 and / or the method steps according to claim 7 are implemented.

Images