Target model rendering method, device and storage medium

Abstract

Embodiments of the present application provide a target model rendering method and device and a storage medium. In the process of model rendering based on a random sampling method, the terminal device renders multiple noise layers and an auxiliary layer through multiple random samplings, and then sends the multiple noise layers and the auxiliary layer to a server. The server calculates a noise distribution map according to the multiple noise layers, and performs noise reduction processing on the multiple noise layers under the guidance of the noise distribution map and the auxiliary layer through a neural network model to obtain a noise reduction result image. The terminal device can display the noise reduction result image. On the one hand, the random sampling method for the target model achieves fast rendering and reduces rendering time consumption. On the other hand, the neural network model on the server side achieves noise reduction on the rendered image, which is conducive to providing faster rendering speed and higher image quality without user awareness, so as to balance the rendering speed and image quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a target model rendering method, device and storage medium. BACKGROUND

[0002] In some methods of image rendering based on random sampling, for example, rendering techniques based on Monte-Carlo simulation method, random sampling is usually performed on a model to be rendered, and the rendering result is rendered. The random sampling operation causes a certain noise in the image obtained by rendering. The higher the sampling rate, the smaller the noise, but the longer the rendering time. In order to obtain a high-quality rendered image in a relatively short rendering time, a lower sampling rate can be used for image rendering to obtain a rendered image with noise, and then the image with noise is denoised. However, the traditional denoising method for model rendering result has poor denoising effect. Therefore, a new solution is needed. SUMMARY

[0003] Aspects of the present application provide a target model rendering method, device and storage medium to reduce the noise generated by the method based on random sampling for rendering the model and improve the rendering quality.

[0004] The embodiment of the present application provides a target model rendering method, comprising: obtaining a plurality of noise layers and an auxiliary layer not containing illumination information; the plurality of noise layers and the auxiliary layer are obtained by rendering the results of a plurality of random samplings of a target model; calculating a noise distribution map of the plurality of random sampling operations according to the plurality of noise layers; inputting the plurality of noise layers, the noise distribution map and the auxiliary layer into a neural network model, so that the neural network model performs denoising processing on the plurality of noise layers according to the noise distribution information in the noise distribution map and the auxiliary layer, and obtains a denoised result image of the plurality of noise layers.

[0005] The embodiment of the present application also provides a target model rendering method, comprising: in response to a rendering instruction, performing a plurality of random samplings on a target model to be rendered; rendering the results of the plurality of random samplings to obtain a plurality of noise layers and an auxiliary layer not containing illumination information; sending the plurality of noise layers and the auxiliary layer to a server, so that the server generates a noise distribution map of the plurality of random sampling operations according to the plurality of noise layers and performs denoising processing on the plurality of noise layers according to the noise distribution map and the auxiliary layer; and displaying the denoised result image returned by the server as a rendering result image.

[0006] The embodiment of the present application further provides a rendering method of a target model, comprising: in response to a rendering instruction, performing multiple random samplings on a target model to be rendered; rendering results of the multiple random samplings to obtain multiple noise layers and an auxiliary layer not containing light information; calculating a noise distribution map of the multiple random sampling operations according to the multiple noise layers; performing noise reduction processing on the multiple noise layers according to the noise distribution map and noise distribution information in the auxiliary layer by using a neural network model to obtain a noise reduction result image; and displaying the noise reduction result image as a rendering result image.

[0007] The embodiment of the present application further provides a server, comprising: a memory and a processor; the memory is used to store one or more computer instructions; and the processor is used to execute the one or more computer instructions to perform steps in the method provided by the embodiment of the present application.

[0008] The embodiment of the present application further provides a terminal device, comprising: a display component, a memory and a processor.

[0009] The memory is used to store one or more computer instructions; and the processor is used to execute the one or more computer instructions to perform steps in the method provided by the embodiment of the present application through the display component.

[0010] The embodiment of the present application further provides a computer readable storage medium storing a computer program, which can implement steps in the method provided by the embodiment of the present application when executed by a processor.

[0011] In the embodiment of the present application, the terminal device sends multiple noise layers and an auxiliary layer to a server after obtaining the multiple noise layers and the auxiliary layer by performing multiple random samplings during rendering based on the method of random sampling. The server calculates a noise distribution map according to the multiple noise layers, and performs noise reduction processing on the multiple noise layers under the guidance of the noise distribution map and the auxiliary layer by using a neural network model to obtain a noise reduction result image. The noise reduction result image can be displayed as a rendering result by the terminal device. Based on this implementation, on the one hand, the fast rendering is realized based on the random sampling of the target model, and the rendering time is reduced; on the other hand, the noise reduction of the rendering image is realized based on the neural network model on the server side, which is conducive to providing faster rendering speed and higher image quality without user awareness, so as to balance the rendering speed and image quality. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0013] Figure 1 A structural schematic diagram of a model rendering system provided for an exemplary embodiment of the present application is shown in FIG. 1.

[0014] Figure 2a A schematic diagram of a neural network model training process provided for an exemplary embodiment of the present application is shown in FIG. 2.

[0015] Figure 2b A comparison schematic diagram of image denoising before and after is shown in FIG. 3.

[0016] Figure 3 A flowchart of a rendering method of a target model provided for an exemplary embodiment of the present application is shown in FIG. 4.

[0017] Figure 4 A flowchart of a rendering method of a target model provided for an exemplary embodiment of the present application is shown in FIG. 4.

[0018] Figure 5 A flowchart of a rendering method of a target model provided for an exemplary embodiment of the present application is shown in FIG. 4.

[0019] Figure 6 A structural schematic diagram of a terminal device provided for an exemplary embodiment of the present application is shown in FIG. 5.

[0020] Figure 7 A structural schematic diagram of a server provided for an exemplary embodiment of the present application is shown in FIG. 6. DETAILED DESCRIPTION

[0021] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in detail with the embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0022] In the rendering technology based on Monte-Carlo Simulation method, the model to be rendered needs to be sampled. In order to reduce the rendering time, a lower sampling rate can be used to sample the model and render a rendering image containing noise, and then a denoising method is used to denoise the rendering image. However, the traditional denoising method for the rendering result of the model has poor denoising effect.

[0023] To solve the above technical problems, in some embodiments of the present application, a solution is provided, which includes a model rendering system composed of a terminal device and a server. In the model rendering system, the terminal device can provide a rendering function based on random sampling, and the server can provide a function of denoising the rendered image to the terminal device based on a neural network, thereby reducing the noise of the model rendering result. The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0024] Figure 1 An exemplary embodiment of the model rendering system provided by the present application is shown in the structural schematic diagram as Figure 1 The model rendering system 100 includes a terminal device 10 and a server 20.

[0025] In the present embodiment, the terminal device 10 refers to a device capable of providing image rendering operation to a user based on a random sampling manner, and having a communication function. In different application scenarios, the implementation form of the terminal device 10 will also be different. For example, in some scenarios, the terminal device 10 can be manifested as a user-side mobile phone, tablet computer, computer device, etc., and the user can initiate a rendering operation on some three-dimensional model through an application program or a browser provided by the terminal device 10. The three-dimensional model can include but is not limited to a model obtained by modeling a specific scene, a physical object or a virtual object. When rendering the three-dimensional model, the material and light information of the model can also be obtained.

[0026] In the model rendering system 100, the server 20 is a device capable of providing a denoising processing function and a communication function. In some embodiments, the server 20 can be implemented as a conventional server, a cloud server, a cloud host, a virtual center, etc. The present embodiment does not limit the server. The composition of the server device mainly includes a processor, a hard disk, a memory, a system bus, etc., which is similar to the general computer architecture and will not be described here.

[0027] In the present embodiment, the terminal device 10 is mainly used to respond to the rendering instruction of the user and perform multiple random samplings on the target model to be rendered. The target model can be provided by the user or obtained by the terminal device 10 from the network, and the present embodiment does not limit it. After multiple random samplings on the target model, the results of the multiple random samplings can be rendered to obtain multiple noise layers and auxiliary layers without light information. The noise layer refers to a rendered layer containing noise, and the result of one random sampling on the target model can be rendered to obtain a layer containing noise. The auxiliary layer does not contain light information and has a small amount of noise, which can be used to guide denoising.

[0028] The sampling rate of the random sampling can be set to a small value, so as to sample the target model at a low frequency and improve the rendering speed. The sampling rate can be less than a preset threshold, which can be set by the user. For example, in some scenarios, when rendering a certain model, if the sampling rate is 1000, the model needs to be sampled and rendered 1000 times, which consumes a lot of time. If the sampling rate is set to 500, the model needs to be sampled and rendered 500 times, the rendering time is halved, and the rendering efficiency is higher.

[0029] In some optional embodiments, the terminal device 10 can include an electronic display screen through which the user can initiate the rendering operation. The electronic display screen can include a liquid crystal display (LCD) and a touch panel (TP). If the electronic display screen includes a touch panel, the electronic display screen can be implemented as a touch screen that can receive an input signal from the user to obtain the rendering request of the user. Of course, in other optional embodiments, the terminal device 10 can include physical buttons or voice input devices for providing the rendering request to the user, which will not be described here.

[0030] In some embodiments, the terminal device 10 locally deploys a neural network model for performing the noise reduction operation. The terminal device 10 can calculate the noise distribution map of the multiple random sampling operations according to the multiple noise map layers. The noise distribution map is used as prior knowledge required for noise reduction to guide the subsequent noise reduction process.

[0031] After obtaining the noise distribution map and the auxiliary layer, the terminal device 10 can use the neural network model to perform noise reduction processing on the multiple noise map layers according to the noise distribution information in the noise distribution map and the auxiliary layer to obtain a noise reduction result image. The terminal device 10 can input the noise distribution map, the auxiliary layer, and the multiple noise map layers into the local neural network model, so that the neural network model performs noise reduction processing on the multiple noise map layers according to the noise distribution information in the noise distribution map and the auxiliary layer to obtain a noise reduction result image. This implementation does not depend on the network and reduces the time required for rendering.

[0032] In other embodiments, the neural network model is deployed on the server 20. The terminal device 10 can send a noise reduction request to the server 20 to implement noise reduction processing through the server 20. The terminal device 10 can send the multiple noise map layers and the auxiliary layer to the server 20. Thus, the server 20 calculates the noise distribution map of the multiple random sampling operations according to the multiple noise map layers, and can perform noise reduction processing on the multiple noise map layers according to the noise distribution information in the noise distribution map and the auxiliary layer through the neural network model, and returns the noise reduction result image. In this implementation, the neural network model is deployed on the server 20, which is beneficial to reduce the operating pressure of the terminal device 10.

[0033] In the implementation, the neural network model can respectively denoise each of the plurality of noise layers to obtain a plurality of denoised layers, and then combine the plurality of denoised layers to obtain the denoised result image. Alternatively, the neural network model can first combine the plurality of noise layers to obtain a combined layer, and then denoise the combined layer to obtain the denoised result image. The implementation is not limited in this regard.

[0034] In the implementation, the neural network model (NN) can be implemented as one or more of a convolutional neural network (CNN), a deep neural network (DNN), a graph convolutional network (GCN), a recurrent neural network (RNN), and a long short-term memory (LSTM), or can be derived from one or more of the above neural networks. The implementation is not limited in this regard.

[0035] After the terminal device 10 obtains the denoised result image, the terminal device 10 can display the denoised result image as a rendering result image.

[0036] In this implementation, the rendering speed can be improved by randomly sampling the target model multiple times, and the quality of the rendering result image can be improved by denoising, so as to balance the rendering speed and the image quality.

[0037] In the model rendering system 100, to implement the above data interaction process between the terminal device 10 and the server 20, the terminal device 10 and the server 20 can establish a communication connection. The specific communication connection mode can be determined according to the actual application scenario.

[0038] In some example implementations, the terminal device 10 and the server 20 can communicate with each other in a wired or wireless manner. The wireless communication manner includes short-distance communication manners such as Bluetooth, ZigBee, infrared, and WiFi (WIreless-Fidelity), and long-distance wireless communication manners such as LORA. The wireless communication manner can also include a wireless communication manner based on a mobile network. When the mobile network is used for communication, the network standard of the mobile network can be any one of 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), 5G, WiMax, and the like.

[0039] In the embodiment, the terminal device sends the plurality of noise layers and the auxiliary layer to the server after rendering the plurality of noise layers and the auxiliary layer through multiple random sampling. The server calculates a noise distribution map according to the plurality of noise layers, and performs noise reduction processing on the plurality of noise layers under the guidance of the noise distribution map and the auxiliary layer through the neural network model to obtain a denoising result image. The denoising result image can be used as a rendering result and displayed by the terminal device. Based on this implementation, on the one hand, the random sampling of the target model is used to achieve fast rendering and reduce rendering time. On the other hand, the neural network model on the server side is used to reduce the noise of the rendering image, which is conducive to providing faster and higher-quality rendering effects without user awareness, thereby achieving a balance between rendering speed and image quality.

[0040] In the above and the following embodiments of the present application, when the terminal device 10 obtains the auxiliary layer without illumination information, at least one of the normal layer, the depth layer and the albedo layer rendered according to the random results of the target model can be obtained. The normal layer is obtained by sampling and rendering the normal direction of a point on the model, for example, the normal direction of a point is 45°, and the normal direction of another point is 50°. The normal information can be used to calculate the illumination information of the pixel when shading. The depth layer is obtained by sampling and rendering the depth value of a point on the model, for example, the depth of a point is 65, and the depth of another point is 70. The depth information can be used to reflect the concave and convex details and shadows of the object surface. The albedo layer is obtained by sampling the albedo value of the material corresponding to a point on the model, for example, if the material of a point is glass, the albedo value of the point is 0. The normal information can also be used to calculate the illumination information of the pixel when shading.

[0041] The auxiliary layer does not contain illumination information, and the amount of noise introduced by random sampling operation is small. Therefore, in the embodiment, the neural network model can use the noise distribution map as prior knowledge of noise distribution, and use the auxiliary layer as guidance knowledge for noise reduction, to perform noise reduction processing on the plurality of noise layers.

[0042] In some optional embodiments, the terminal device 10 or the server 20 can calculate the noise distribution feature generated by the multiple random sampling operations according to the color information of each of the multiple noise layers and the sampling rate corresponding to the multiple random sampling operations, to obtain the noise distribution map. The color information of any noise layer refers to the values of the pixels in the noise layer in the R, G, and B (red, green, and blue) color channels. The sampling rate refers to the number of random sampling operations on the target model in the rendering process. The noise distribution feature is used to describe the distribution rule of the noise introduced by the random sampling rendering process.

[0043] The noise distribution feature can be calculated by the mean absolute error or the mean square error of the multiple noise layers, which is not limited in the embodiments. The following will be described by examples.

[0044] Optionally, the mean value of the multiple noise layers obtained by the multiple random sampling operations can be calculated to obtain a mean noise layer, and the residual sum of squares of the multiple noise layers and the mean noise layer can be calculated respectively. The residual sum of squares is averaged according to the sampling number corresponding to the multiple random sampling operations to obtain the noise distribution feature generated by the multiple random sampling operations. The above calculation process can be described by the following formula:

[0045]

[0046]

[0047] wherein NoiseMap represents the noise distribution map, N represents the sampling rate, ColorSpp i represents the noise layer obtained by the i-th random sampling, ColorSpp mean represents the mean value of the noise layers obtained by the N random sampling operations.

[0048] The above formula describes the process of calculating the noise distribution feature by the mean square error. In some other embodiments, the average absolute error can be used to calculate the noise distribution feature, as shown in the following formula:

[0049]

[0050] Based on the above formula, the noise distribution feature corresponding to each pixel position can be calculated to determine the noise distribution map.

[0051] After obtaining the noise distribution map, the server 20 can input the noise distribution map, the multiple noise layers obtained by the multiple random sampling operations, and the auxiliary layer into the neural network model.

[0052] The model parameters in the neural network model can be obtained by training the training data. In the training process, the neural network model can learn the ability to denoise the noise layer according to the noise distribution map and the auxiliary layer.

[0053] The following will be described in combination with Figure 2a The training process of the neural network model will be described exemplarily.

[0054] In some embodiments, for the server 20, in addition to performing the denoising function described in the foregoing embodiments, the server 20 can also perform the training function of the neural network model.

[0055] In this implementation, the plurality of noise layers obtained by the plurality of random sampling and rendering described in the foregoing embodiments, the noise distribution map and the auxiliary layer can be used as training data. After the server 20 obtains the denoising result image of the plurality of noise layers based on the foregoing embodiments, the server 20 can calculate the denoising loss of the neural network model by using the frequency domain information of the denoising result image. After obtaining the denoising loss, the server 20 can optimize the neural network model by converging the denoising loss of the neural network model to a specified range as an optimization target. The frequency domain information is used to describe the frequency characteristics of the denoising result image. Generally, noise is high-frequency information in an image. In this embodiment, the frequency domain information of the denoising result image can be obtained by using Fourier transform, or the frequency domain information of the denoising result image can be obtained by using wavelet transform, which is not limited in this embodiment. The following will be described exemplarily by taking wavelet transform as an example.

[0056] Optionally, when the server 20 calculates the denoising loss of the neural network model by using the frequency domain information of the denoising result image, the server 20 can perform wavelet transform on the denoising result image to obtain a plurality of wavelet transform images at different scales. For the denoising result image, four images can be obtained after wavelet transform, i.e., an approximation image, a horizontal detail image, a vertical detail image and a diagonal detail image. The server 20 can use any one of the four images to calculate the denoising loss, which will be described exemplarily by taking the diagonal detail image in the subsequent embodiments.

[0057] After wavelet transform, the energy of the image is mainly concentrated in the large-scale subband, while the coefficients of the small-scale high-frequency subband have smaller amplitudes and lower energy. When the noise is high, the coefficients of the small-scale high-frequency subband can be regarded as noise. Based on this, after obtaining multiple wavelet transform images at different scales, the server 20 can extract diagonal detail images with scales smaller than a set scale threshold from these multiple wavelet transform images; based on the cumulative value of the absolute values ​​of the pixels in the diagonal detail images, a portion of the noise reduction loss of the neural network model is obtained. Optionally, the set scale threshold can be 2, 3, etc., and this embodiment does not impose any restrictions. If the set scale threshold is 2, a diagonal detail image with a scale of 1 can be selected for noise reduction loss calculation; if the set scale threshold is 3, a diagonal detail image with a scale of 1 or 2 can be selected for noise reduction loss calculation, and this embodiment does not impose any restrictions.

[0058] In some embodiments, the denoising loss of the neural network model includes not only a portion of the loss calculated based on the frequency domain, but also another portion of the loss calculated based on the target image to be denoised.

[0059] Optionally, when the server 20 calculates the denoising loss of the neural network model using the frequency domain information of the denoised result image, it can calculate the first denoising loss based on the error between the denoised result image and the denoised target image. Optionally, the first denoising loss can be calculated by... Figure 2a The L1 (norm) loss function is used for calculation, but other loss functions, such as L2 (regular) loss or cross-entropy loss, can also be used. This embodiment is not limited to these. Optionally, after performing wavelet transform on the denoised image to obtain multiple wavelet transformed images at different scales, diagonal detail images with scales smaller than a set scale threshold can be obtained from these multiple wavelet transformed images. The second denoising loss (i.e., the frequency domain noise loss shown in Figure 2) is calculated based on the cumulative absolute values ​​of the pixels in the diagonal detail images. The joint loss of the first denoising loss and the second denoising loss is calculated as the denoising loss of the neural network model.

[0060] After determining the denoising loss of the neural network model, server 20 can optimize the neural network model by focusing on converging the denoising loss to a specified range. This optimization operation can be performed iteratively. After each iteration, the denoising loss of the neural network model can be calculated using the method described above. If the denoising loss converges to the specified range, the iteration can be stopped, which will not be elaborated further.

[0061] Based on the above implementation methods, a neural network model with high noise reduction capability can be trained. For example... Figure 2bAs shown, the noise image before noise reduction contains more noise, and the image is blurred and unclear. After noise reduction by the neural network model, the image quality is greatly improved, and the details are more abundant. The neural network model trained by the server 20 can be run on the server 20 side for use by terminal devices or other third-party users. The neural network model can also be deployed on other devices for local use by other devices, which is not limited by the present embodiment.

[0062] In addition to the model rendering system provided in the foregoing embodiments, the present embodiment also provides a rendering method of a target model, which will be exemplarily described below.

[0063] Figure 3 FIG. 1 is a flow diagram of a rendering method of a target model provided by an exemplary embodiment of the present application. When the method is executed on the server side, it can include the steps shown in FIG. 1. Figure 3

[0064] Step 301: Obtain a plurality of noise image layers and an auxiliary layer not containing illumination information; the plurality of noise image layers and the auxiliary layer are rendered according to the results of multiple random sampling of a target model.

[0065] Step 302: Calculate a noise distribution map of the multiple random sampling operations according to the plurality of noise image layers.

[0066] Step 303: Input the plurality of noise image layers, the noise distribution map, and the auxiliary layer into a neural network model, so that the neural network model performs noise reduction processing on the plurality of noise image layers according to the noise distribution information in the noise distribution map and the auxiliary layer, to obtain a noise-reduced result image of the plurality of noise image layers.

[0067] In some exemplary embodiments, one way of calculating the noise distribution map of the multiple random sampling operations according to the plurality of noise image layers includes: calculating noise distribution characteristics generated by the multiple random sampling according to the color information of each of the plurality of noise image layers and the sampling rate corresponding to the multiple random sampling operations, to obtain the noise distribution map.

[0068] In some exemplary embodiments, one way of calculating the noise distribution characteristics generated by the multiple random sampling according to the color information of each of the plurality of noise image layers and the sampling rate corresponding to the multiple random sampling operations includes: calculating the color mean of the pixels at the same position in the plurality of noise image layers to obtain a mean noise image; calculating the color value residual of each of the plurality of noise image layers and the mean noise image; and calculating the mean square error of the plurality of noise image layers as the noise distribution characteristics generated by the multiple random sampling according to the color value residual corresponding to each of the plurality of noise image layers and the sampling rate.

[0069] ​In some example embodiments, a manner of obtaining the auxiliary layer without illumination information obtained through the multiple random samplings includes: obtaining at least one of a normal layer, a depth layer, and a diffuse reflection layer rendered according to results of the multiple random samplings of the target model.

[0070] In some example embodiments, after the denoising processing is performed on the multiple noise layers to obtain a denoising result image of the multiple noise layers, the method further includes: calculating a denoising loss of the neural network model by using frequency domain information of the denoising result image; and optimizing the neural network model by taking convergence of the denoising loss of the neural network model to a specified range as an optimization target.

[0071] In some example embodiments, a manner of calculating the denoising loss of the neural network model by using the frequency domain information of the denoising result image includes: calculating a first denoising loss according to an error between the denoising result image and a denoising target image; performing wavelet transform on the denoising result image to obtain multiple wavelet transform images at different scales; obtaining a diagonal line detail image with a scale less than a set scale threshold from the multiple wavelet transform images; calculating a second denoising loss according to a cumulative value of absolute values of pixel points in the diagonal line detail image; and calculating a joint loss of the first denoising loss and the second denoising loss as the denoising loss of the neural network model.

[0072] In this embodiment, after the server obtains the multiple noise layers and the auxiliary layer rendered through the multiple random samplings, the server can calculate a noise distribution map according to the multiple noise layers, and perform denoising processing on the multiple noise layers by using the neural network model under the guidance of the noise distribution map and the auxiliary layer to obtain a denoising result image. Based on this implementation, on the one hand, the fast rendering is realized based on the manner of random sampling on the target model, and the rendering time consumption is reduced; on the other hand, the denoising is realized on the rendered image based on the neural network model on the server side, which is conducive to providing a rendering effect with faster speed and higher image quality without user awareness, so as to realize the balance between rendering speed and image quality.

[0073] Figure 4 is a flowchart of a rendering method of a target model provided by another example embodiment of the present application. When the method is executed on a terminal device side, the method can include the steps as shown in Figure 4

[0074] Step 401: In response to a rendering instruction, performing multiple random samplings on a target model to be rendered.

[0075] Step 402: Rendering results of the multiple random samplings to obtain multiple noise layers and an auxiliary layer without illumination information.

[0076] ​Step 403, send the plurality of noise layers and the auxiliary layer to the server, so that the server generates a noise distribution map of the plurality of random sampling operations according to the plurality of noise layers and performs noise reduction processing on the plurality of noise layers according to the noise distribution map and the auxiliary layer.

[0077] Step 404, return the noise reduction result image returned by the server according to the noise reduction processing as a rendering result image for display.

[0078] Optionally, in some embodiments, after the terminal device obtains the noise reduction result image through the neural network model, the noise reduction result image can be sent to the server for optimizing the neural network model, so that the server calculates the noise reduction loss of the neural network model according to the frequency domain information of the noise reduction result image; the noise reduction loss of the neural network model is converged to a specified range as an optimization target, and the neural network model is optimized. After the server obtains the optimized neural network model, the optimized neural network model can be distributed to the terminal device to update the neural network model running on the terminal device, so as to continuously optimize the rendering effect on the terminal device side.

[0079] In this embodiment, in the process of rendering based on the random sampling method, after the terminal device renders the noise layer, the noise layer and the auxiliary layer are sent to the server, and the noise reduction result image is returned to the terminal device after the server performs noise reduction processing based on the neural network model. Therefore, the terminal device can display the rendering image after noise reduction to the user, which on the one hand realizes fast rendering based on the random sampling method, and on the other hand realizes noise reduction of the rendering image based on the noise reduction algorithm of the server, which is beneficial to providing faster and higher quality rendering effect without user awareness.

[0080] Figure 5 is a flowchart of a rendering method of a target model provided by another exemplary embodiment of the present application. When executed on the terminal device side, the method can include the steps as shown in Figure 5

[0081] Step 501, in response to a rendering instruction, performing multiple random sampling on a target model to be rendered.

[0082] Step 502, rendering the results of the multiple random sampling to obtain a plurality of noise layers and an auxiliary layer not containing illumination information.

[0083] Step 503, calculating a noise distribution map of the plurality of random sampling operations according to the plurality of noise layers.

[0084] Step 504, using a neural network model to perform noise reduction processing on the plurality of noise layers according to the noise distribution map and noise distribution information in the auxiliary layer, to obtain a noise reduction result image.​

[0085] Step 505: display the denoised result image as a rendering result image.

[0086] In this embodiment, the terminal device side runs a neural network model that can perform image denoising operations. The neural network model can be trained on the other device (e.g., server) side and deployed on the terminal device. The terminal device can perform denoising processing on the image rendered based on the random sampling-based rendering method during the execution of the rendering operation, thereby outputting a denoised result image that meets specific image quality requirements.

[0087] In some example embodiments, a way to calculate the noise distribution map of the multiple random sampling operations according to the multiple noise layers includes: calculating noise distribution characteristics generated by the multiple random sampling operations according to the color information of each of the multiple noise layers and the sampling rate corresponding to the multiple random sampling operations, to obtain the noise distribution map.

[0088] In some example embodiments, a way to calculate the noise distribution characteristics generated by the multiple random sampling operations according to the color information of each of the multiple noise layers and the sampling rate corresponding to the multiple random sampling operations includes: calculating the color mean of the pixels at the same position in the multiple noise layers to obtain a mean noise map; calculating the color value residual of each of the multiple noise layers and the mean noise map; and calculating the mean square error of the multiple noise layers as the noise distribution characteristics generated by the multiple random sampling operations according to the color value residual of each of the multiple noise layers and the sampling rate.

[0089] In some example embodiments, a way to obtain the auxiliary layer that does not contain illumination information obtained by the multiple random sampling includes: obtaining at least one of the normal map layer, the depth map layer, and the diffuse reflection layer rendered according to the results of the multiple random sampling of the target model.

[0090] Optionally, in some embodiments, after the terminal device obtains the denoised result image through the neural network model, the terminal device can send the denoised result image to a server for optimizing the neural network model, so that the server calculates the denoising loss of the neural network model according to the frequency domain information of the denoised result image; and the server takes the convergence of the denoising loss of the neural network model to a specified range as an optimization goal, and optimizes the neural network model. After the server obtains the optimized neural network model, the server can distribute the optimized neural network model to the terminal device to update the neural network model running on the terminal device, thereby continuously optimizing the rendering effect on the terminal device side.

[0091] In this embodiment, the terminal device locally runs a neural network model for denoising. In the process of rendering based on the random sampling method, after obtaining the noise layer and the auxiliary layer, the terminal device can calculate a noise distribution map according to the noise layer, and input the noise distribution map, the noise layer and the auxiliary layer into the neural network model. The auxiliary layer does not contain illumination information, and therefore has less noise. Thus, the neural network model can take the noise distribution map as prior knowledge, and perform denoising processing on the noise layer according to the prior knowledge and the auxiliary layer, to improve the image quality of the image rendered based on the random algorithm.

[0092] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 501 to 504 can be device A; for another example, the execution subject of steps 501 and 502 can be device A, and the execution subject of step 503 can be device B; and so on.

[0093] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appearing in a specific order are included, but it should be clear that these operations can be executed or in parallel without following the order in which they appear in this text. The serial numbers of the operations, such as 501, 502, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel.

[0094] It should be noted that the "first", "second" and the like in this text are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do "first" and "second" represent different types.

[0095] Figure 6 is a structural schematic diagram of a terminal device provided by an exemplary embodiment of the present application, which is suitable for the model rendering system provided by the foregoing embodiments. As shown in Figure 6 the terminal device includes a memory 601, a processor 602 and a display component 603.

[0096] The memory 601 is used to store computer programs and can be configured to store other various data to support operations on the terminal device. Examples of these data include instructions of any application program or method for operating on the terminal device, contact data, phonebook data, messages, pictures, videos, etc.

[0097] In some embodiments, the processor 602 is coupled with the memory 601 to execute a computer program in the memory 601 to, in response to a rendering instruction, perform multiple random samplings on a target model to be rendered, render results of the multiple random samplings to obtain multiple noise layers and an auxiliary layer not containing illumination information, and send the multiple noise layers and the auxiliary layer to a server to enable the server to generate a noise distribution map of the multiple random sampling operations according to the multiple noise layers and perform noise reduction processing on the multiple noise layers according to the noise distribution map and the auxiliary layer to obtain a noise reduction result image, and display the noise reduction result image as a rendering result image by the display component 603.

[0098] In the present embodiment, in the process of rendering based on the random sampling method, after obtaining the noise layer by rendering, the terminal device sends the noise layer and the auxiliary layer to the server, and returns the noise reduction result image to the terminal device after the server performs noise reduction processing based on the neural network model. Thus, the terminal device can display the rendering image after noise reduction to the user, on the one hand, the random sampling method is used to realize fast rendering, and on the other hand, the noise reduction algorithm based on the server is used to realize noise reduction on the rendering image, which is conducive to providing faster speed and higher quality of rendering effect without user awareness.

[0099] In some other embodiments, the processor 602 is coupled with the memory 601 to execute a computer program in the memory 601 to, in response to a rendering instruction, perform multiple random samplings on a target model to be rendered, render results of the multiple random samplings to obtain multiple noise layers and an auxiliary layer not containing illumination information, calculate a noise distribution map of the multiple random sampling operations according to the multiple noise layers, perform noise reduction processing on the multiple noise layers according to the noise distribution map and noise distribution information in the auxiliary layer by using a neural network model to obtain a noise reduction result image, and display the noise reduction result image as a rendering result image by the display component 603.

[0100] In the present embodiment, the terminal device locally runs the neural network model for noise reduction. In the process of rendering based on the random sampling method, after obtaining the noise layer and the auxiliary layer by rendering, the terminal device can calculate the noise distribution map according to the noise layer, and input the noise distribution map, the noise layer and the auxiliary layer into the neural network model. The auxiliary layer does not contain illumination information, and thus has less noise. Thus, the neural network model can take the noise distribution map as prior knowledge, and perform noise reduction processing on the noise layer according to the prior knowledge and the auxiliary layer to improve the quality of the image rendered based on the random algorithm.

[0101] Further, as shown in FIG. 6, the terminal device can further include a communication component 604, which is coupled with the processor 602 and configured to send the multiple noise layers and the auxiliary layer to the server and receive the noise reduction result image from the server. Figure 6As shown, the terminal device further includes a communication component 604, a power supply component 605, an audio component 606, and other components. Figure 6 Only some components are shown schematically, and it does not mean that the terminal device only includes Figure 6 the components shown.

[0102] Correspondingly, the embodiment of the present application also provides a computer readable storage medium storing a computer program, which can implement each step that can be executed by the terminal device in the method embodiment when the computer program is executed.

[0103] Figure 7 The structure schematic diagram of the server provided by an example embodiment of the present application is shown, and the server is suitable for the model rendering system provided by the foregoing embodiments. As Figure 7 As shown, the server includes a memory 701, a processor 702, and a communication component 703.

[0104] The memory 701 is used to store computer programs, and can be configured to store other various data to support operations on the server. Examples of these data include instructions for any application or method operating on the server, contact data, phonebook data, messages, pictures, videos, and the like.

[0105] The processor 702 is coupled with the memory 701, and is used to execute the computer program in the memory 701, so as to: acquire, through the communication component 703, a plurality of noise image layers rendered according to results of a plurality of random samplings of a target model and an auxiliary image layer not containing illumination information obtained by the plurality of random samplings; calculate a noise distribution map of the plurality of random sampling operations according to the plurality of noise image layers; and input the plurality of noise image layers, the noise distribution map, and the auxiliary image layer into a neural network model, so that the neural network model performs noise reduction processing on the plurality of noise image layers according to noise distribution information in the noise distribution map and the auxiliary image layer, to obtain a denoising result image of the plurality of noise image layers.

[0106] Further optionally, when calculating the noise distribution map of the plurality of random sampling operations according to the plurality of noise image layers, the processor 702 is specifically used to: calculate noise distribution characteristics generated by the plurality of random samplings according to color information of each of the plurality of noise image layers and a sampling rate corresponding to the plurality of random sampling operations, to obtain the noise distribution map.

[0107] Further, the processor 702 is further configured to: calculate a color mean value of pixels at the same position in the plurality of noise layers to obtain a mean noise layer; calculate a color value residual of each of the plurality of noise layers and the mean noise layer; and calculate a mean square error of the plurality of noise layers as the noise distribution feature generated by the plurality of random samplings according to the color value residual of each of the plurality of noise layers and the sampling rate.

[0108] Further, the processor 702 is further configured to: obtain at least one of a normal layer, a depth layer, and a diffuse reflection layer rendered according to a result of the plurality of random samplings of the target model.

[0109] Further, the processor 702 is further configured to: calculate a denoising loss of the neural network model by using frequency domain information of the denoising result image; and optimize the neural network model by taking the denoising loss of the neural network model converging to a specified range as an optimization target.

[0110] Further, the processor 702 is further configured to: calculate a first denoising loss according to an error between the denoising result image and a denoising target image; perform wavelet transform on the denoising result image to obtain a plurality of wavelet transform images at different scales; obtain a diagonal line detail image with a scale less than a set scale threshold from the plurality of wavelet transform images; calculate a second denoising loss according to a cumulative value of absolute values of pixel points in the diagonal line detail image; and calculate a joint loss of the first denoising loss and the second denoising loss as the denoising loss of the neural network model.

[0111] Further, as shown in Figure 7 the server further includes a power supply component 704 and other components. Figure 7 Some components are only schematically shown in the server, and it does not mean that the server only includes Figure 7 the components shown.

[0112] In this embodiment, after the server obtains multiple noise layers and an auxiliary layer through multiple random sampling renderings, the server can calculate a noise distribution map according to the multiple noise layers, and perform noise reduction processing on the multiple noise layers under the guidance of the noise distribution map and the auxiliary layer through a neural network model to obtain a denoising result image. Based on this implementation, on the one hand, the random sampling manner for the target model is used to realize fast rendering and reduce the rendering time, and on the other hand, the neural network model on the server side is used to realize noise reduction on the rendered image, which is beneficial to providing a faster and higher-quality rendering effect without user awareness, so as to balance the rendering speed and image quality.

[0113] Correspondingly, the embodiment of the application further provides a computer readable storage medium storing a computer program, which can implement each step that can be executed by the server in the method embodiment when the computer program is executed.

[0114] The memory in the above Figure 6 and Figure 7 may be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0115] The communication component in the above Figure 6 and Figure 7 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G or 5G, or a combination thereof. In an example embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component can be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0116] The display component in the above Figure 6 includes a screen, which can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide and a gesture on the touch panel. The touch sensor can not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0117] superior Figure 6 The audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0118] The above Figure 6 and Figure 7 The power supply component provides power to the various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0119] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0121] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1

[0123] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0124] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the processor can execute instructions. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, or other memory technologies, CD-ROM, digital versatile disc (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information for access by a computing device. In no case does the medium include a transitory signal.

[0125] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EEPROM), flash memory or other memory technologies, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible by a computing device. According to the definition herein, computer readable media does not include transitory media such as modulated data signals and carrier waves.

[0126] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0127] ​​The above merely provides an example of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the scope of claims of the present application.

Claims

1. A rendering method of a target model, characterized by, The method comprises the following steps: obtaining a plurality of noise layers and an auxiliary layer not containing illumination information; the plurality of noise layers and the auxiliary layer are rendered according to the results of a plurality of random samplings of a target model, and the sampling rate of the random sampling is less than a preset threshold, so as to perform low-frequency sampling on the target model; according to the plurality of noise layers, a noise distribution map of the plurality of random sampling operations is calculated, including: calculating the color mean value of the pixels at the same position in the plurality of noise layers to obtain a mean noise map; respectively calculating the color value residuals of the plurality of noise layers and the mean noise map; according to the color value residuals corresponding to the plurality of noise layers respectively and the sampling rate, calculating the mean square error of the plurality of noise layers as the noise distribution characteristics generated by the plurality of random samplings, so as to obtain the noise distribution map; inputting the plurality of noise layers, the noise distribution map and the auxiliary layer not containing illumination information into a neural network model, so that the neural network model performs noise reduction processing on the plurality of noise layers according to the noise distribution information in the noise distribution map and the auxiliary layer not containing illumination information, to obtain a denoised result image of the plurality of noise layers.

2. The method of claim 1, wherein, obtaining an auxiliary layer not containing illumination information, comprising: obtaining at least one of a normal map layer, a depth map layer and a diffuse reflection layer rendered according to the results of a plurality of random samplings of the target model.

3. The method according to claim 1 or 2, characterized in that, After the noise reduction processing on the plurality of noise layers to obtain the denoised result image of the plurality of noise layers, further comprising: calculating the noise reduction loss of the neural network model by using the frequency domain information of the denoised result image; optimizing the neural network model with the optimization goal of converging the noise reduction loss of the neural network model to a specified range.

4. The method of claim 3, wherein, calculating the noise reduction loss of the neural network model by using the frequency domain information of the denoised result image, comprising: calculating a first noise reduction loss according to the error between the denoised result image and a denoised target image; performing wavelet transform on the denoised result image to obtain a plurality of wavelet transform images at different scales; from the plurality of wavelet transform images, obtaining a diagonal line detail image with a scale less than a set scale threshold; calculating a second noise reduction loss according to the cumulative value of the absolute values of the pixel points in the diagonal line detail image; calculating the joint loss of the first noise reduction loss and the second noise reduction loss as the noise reduction loss of the neural network model.

5. A rendering method of a target model, characterized by, The method comprises the following steps: in response to a rendering instruction, performing a plurality of random samplings on a target model to be rendered; rendering the results of the plurality of random samplings to obtain a plurality of noise layers and an auxiliary layer not containing illumination information, and the sampling rate of the random sampling is less than a preset threshold, so as to perform low-frequency sampling on the target model; The plurality of noise layers and the auxiliary layer are sent to a server, so that the server generates a noise distribution map of the plurality of random sampling operations according to the plurality of noise layers and performs noise reduction processing on the plurality of noise layers according to the noise distribution map and the auxiliary layer; wherein the calculation method of the noise distribution map comprises: calculating the color mean value of the pixels at the same position in the plurality of noise layers to obtain a mean noise map; calculating the color value residual of each of the plurality of noise layers and the mean noise map; calculating the mean square error of the plurality of noise layers according to the color value residual corresponding to each of the plurality of noise layers and the sampling rate, as the noise distribution feature generated by the plurality of random sampling operations, to obtain the noise distribution map; The noise reduction result image returned by the server according to the noise reduction processing is taken as a rendering result image for display.

6. A rendering method of a target model, characterized by, Comprising: In response to a rendering instruction, performing multiple random sampling on a target model to be rendered; Rendering the results of the multiple random sampling to obtain a plurality of noise layers and an auxiliary layer that does not contain lighting information, wherein the sampling rate of the random sampling is less than a preset threshold, so as to perform low-frequency sampling on the target model; According to the plurality of noise layers, a noise distribution map of the plurality of random sampling operations is calculated, including: calculating the color mean value of the pixels at the same position in the plurality of noise layers to obtain a mean noise map; calculating the color value residual of each of the plurality of noise layers and the mean noise map; calculating the mean square error of the plurality of noise layers according to the color value residual corresponding to each of the plurality of noise layers and the sampling rate, as the noise distribution feature generated by the plurality of random sampling operations, to obtain the noise distribution map; Using a neural network model, noise reduction processing is performed on the plurality of noise layers according to the noise distribution information in the noise distribution map and the auxiliary layer, to obtain a noise reduction result image; The noise reduction result image is taken as a rendering result image for display.

7. A server, characterized by Comprising: A memory and a processor; The memory is used to store one or more computer instructions; The processor is used to execute the one or more computer instructions for: performing the steps in the method of any one of claims 1-4.

8. A terminal device, comprising: Comprising: A display component, a memory and a processor; The memory is used to store one or more computer instructions; The processor is used to execute the one or more computer instructions for: performing the steps in the method of claim 5 or 6 through the display component.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the steps in the method of any one of claims 1-6.

Images