Three-dimensional model rendering method, image denoising model training method and related device

Abstract

The present disclosure provides a three-dimensional model rendering method, a training method of an image denoising model and related devices. It relates to the technical field of artificial intelligence. The method comprises: performing ray tracing rendering on a three-dimensional model based on a target sampling number to obtain a noisy image with noise; wherein the target sampling number is less than a preset threshold; performing denoising processing on the noisy image using a first neural network module to obtain a preliminary denoised image with part of the noise removed; aligning the preliminary denoised image and auxiliary rendering features in the frequency domain to obtain intermediate features; the auxiliary rendering features at least include normal features and lighting features used to optimize the preliminary denoised image; and processing the intermediate features based on an image generator to generate a denoised image of the three-dimensional model. The present disclosure can quickly render an image with high quality at low cost.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, three-dimensional model rendering, image rendering and the like. BACKGROUND

[0002] In the field of home decoration design, a home decoration design effect picture is generally completed by a home decoration design software before formal construction, so as to determine the final decoration scheme. Many home decoration design software supports constructing a three-dimensional effect picture based on a three-dimensional model. In the related art, a three-dimensional model designed is rendered to generate a three-dimensional effect picture approximating a real lighting condition.

[0003] However, a rendering engine needs to sample a three-dimensional model a high number of times to construct a realistic three-dimensional effect picture. A high sampling rate can render a high-quality picture, but the cost is high and the time is long. When the sampling number is low, the obtained three-dimensional effect picture has a lot of noise. SUMMARY

[0004] The present disclosure provides a home decoration model control method and device, electronic equipment and storage medium, to solve or alleviate one or more technical problems in the related art.

[0005] In a first aspect, the present disclosure provides a three-dimensional model rendering method, comprising:

[0006] performing ray tracing rendering on the three-dimensional model based on a target sampling number to obtain a noisy image with noise; wherein the target sampling number is less than a preset threshold;

[0007] performing noise reduction processing on the noisy image by using a first neural network module to obtain a preliminary denoised image in which part of the noise is removed;

[0008] aligning the preliminary denoised image and auxiliary rendering features in the frequency domain to obtain intermediate features; the auxiliary rendering features at least include normal features and lighting features used to optimize the preliminary denoised image;

[0009] processing the intermediate features by using an image generator to generate a denoised image of the three-dimensional model.

[0010] In a second aspect, the present disclosure provides a training method of an image denoising model, wherein the image denoising model includes a pre-trained first neural network module, an intermediate feature acquisition module and an image generator, and the method comprises:

[0011] Inputting a sample image into the image denoising model to obtain a denoised image of the sample image; the sample image is obtained by performing ray tracing rendering on the sample three-dimensional model based on a target sampling number, and performing ray tracing rendering on the sample three-dimensional model based on a preset sampling number higher than the target sampling number to obtain a high-definition image of the sample image;

[0012] determining a loss value based on the denoised image and the high-definition image;

[0013] Adjusting model parameters of the image denoising model based on the loss value;

[0014] The first neural network module is used to perform preliminary noise reduction on the sample image to obtain a preliminary noise reduction sample;

[0015] The intermediate feature extraction module is used to process the preliminary denoised samples and the sample auxiliary rendering features of the sample image to align the preliminary denoised samples and the sample auxiliary rendering features in the frequency domain to obtain intermediate features of the sample image; the sample auxiliary rendering features at least include normal features and lighting features required for rendering the sample image and used to optimize the preliminary denoised image;

[0016] The image generator is used to process the intermediate features of the sample image to obtain the denoised image.

[0017] In a third aspect, the present disclosure provides a three-dimensional model rendering device, comprising:

[0018] a rendering module configured to perform ray tracing rendering on the three-dimensional model based on a target sampling number to obtain a noisy image having noise; wherein the target sampling number is less than a preset threshold, and the auxiliary rendering features are used to describe at least normal features and lighting features of the three-dimensional model;

[0019] A denoising module, configured to perform denoising on the noisy image using the first neural network module to obtain a preliminary denoised image with some noise removed;

[0020] An alignment module is used to align the preliminary denoised image and the auxiliary rendering features in the frequency domain to obtain intermediate features; the auxiliary rendering features include at least normal features and illumination features used to optimize the preliminary denoised image;

[0021] The generation module is used to process the intermediate features based on the image generator to generate a denoised image of the three-dimensional model.

[0022] In a fourth aspect, the present disclosure provides a training device for an image denoising model, wherein the image denoising model includes a pre-trained first neural network module, an intermediate feature acquisition module, and an image generator, and the device includes:

[0023] The input module is configured to input a sample image into the image denoising model to obtain a denoised image of the sample image; the sample image is obtained by performing ray tracing rendering on a sample three-dimensional model based on a target sampling number, and the sample three-dimensional model is rendered based on a preset sampling number higher than the target sampling number to obtain a high-definition image of the sample image;

[0024] The determination module is configured to determine a loss value based on the denoised image and the high-definition image.

[0025] The adjustment module is configured to adjust a model parameter of the image denoising model based on the loss value.

[0026] The first neural network module is configured to perform preliminary denoising on the sample image to obtain a preliminary denoised sample.

[0027] The intermediate feature extraction module is configured to process the preliminary denoised sample and a sample auxiliary rendering feature of the sample image to align the preliminary denoised sample and the sample auxiliary rendering feature in a frequency domain to obtain an intermediate feature of the sample image; the sample auxiliary rendering feature at least includes a normal feature and an illumination feature required for rendering the sample image and used for optimizing the preliminary denoised image.

[0028] The image generator is configured to process the intermediate feature of the sample image to obtain the denoised image.

[0029] In a fifth aspect, an electronic device is provided, including:

[0030] at least one processor; and

[0031] a memory communicatively connected to the at least one processor; wherein

[0032] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any of the embodiments of the present disclosure.

[0033] In a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the method according to any of the embodiments of the present disclosure.

[0034] In a seventh aspect, a computer program product is provided, including a computer program which, when executed by a processor, implements the method according to any of the embodiments of the present disclosure.

[0035] The embodiments of the present disclosure can quickly and low-cost render a three-dimensional model into a high-quality image.

[0036] It is to be understood that the details set forth herein do not limit the scope of the disclosure to the one embodiment described. Rather, the scope encompasses numerous alternatives, modifications, and equivalents. Various features, which are believed to be novel, are described herein. It will be apparent, however, to one skilled in the art that many modifications, combinations, subcombinations, and equivalents can be made and still be within the scope of the disclosure. Accordingly, the disclosure is not to be limited by what has been particularly shown and described. BRIEF DESCRIPTION OF DRAWINGS

[0037] In the drawings, like reference numerals refer to same or similar functionalities throughout the several views. The drawings are not necessarily to scale. It should be understood that the drawings only depict some embodiments in accordance with the disclosure and should not be considered to be limiting of its scope.

[0038] Figure 1 is a flowchart of a method for rendering a three-dimensional model according to an embodiment of the disclosure;

[0039] Figure 2 is a flowchart of constructing an intermediate feature according to an embodiment of the disclosure;

[0040] Figure 3 is a structural diagram of an image denoising model according to an embodiment of the disclosure;

[0041] Figure 4 is a flowchart of a training method of an image denoising model according to an embodiment of the disclosure;

[0042] Figure 5 is a structural diagram of an image denoising model according to an embodiment of the disclosure;

[0043] Figure 6 is a flowchart of determining a loss value according to an embodiment of the disclosure;

[0044] Figure 7 is an effect comparison diagram provided according to an embodiment of the disclosure;

[0045] Figure 8 is another effect comparison diagram provided according to an embodiment of the disclosure;

[0046] Figure 9 is another effect comparison diagram provided according to an embodiment of the disclosure;

[0047] Figure 10 is still another effect comparison diagram provided according to an embodiment of the disclosure;

[0048] Figure 11 is a structural diagram of a three-dimensional model rendering device according to an embodiment of the disclosure;

[0049] Figure 12 is a structural diagram of a training device of an image denoising model according to an embodiment of the disclosure;

[0050] Figure 13 is a block diagram of an electronic device for implementing the three-dimensional model rendering method / image denoising model training method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0051] The present disclosure will be described in further detail below with reference to the drawings. The same reference numbers in different drawings represent the same or similar elements / functionally similar elements. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0052] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following detailed description of the embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without some specific details. In some examples, methods, means, elements and circuits that are well known to those skilled in the art are not described in detail in order to highlight the main ideas of the present disclosure.

[0053] In addition, it should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0054] As set forth in the background, rendering high-quality pictures using a high sampling rate is costly and time-consuming. There are also related technologies that first render a noisy picture using a low sampling rate, and then use a denoising algorithm to remove noise in order to obtain a high-quality image. However, the effect is good or bad depends on the algorithm.

[0055] In view of this, the embodiments of the present disclosure provide a three-dimensional model rendering method, as shown in Figure 1 The method is shown in the flowchart, which includes:

[0056] S101, based on a target sampling number, performing ray tracing rendering on a three-dimensional model to obtain a noisy image with noise; wherein the target sampling number is less than a preset threshold.

[0057] The three-dimensional model of the embodiments of the present disclosure can be a three-dimensional scene obtained by home decoration design.

[0058] In implementation, the three-dimensional model is rendered by ray tracing using a rendering engine. In the embodiments of the present disclosure, in order to improve the rendering efficiency, the number of rendering samples can be reduced, and thus the obtained image is a noisy image, which is referred to as a noisy image in the embodiments of the present disclosure. The noisy image allows a large noise and is very blurred. Then, the image quality is optimized through noise reduction processing.

[0059] In S102, a first neural network module is used to perform noise reduction processing on the noisy image, to obtain a preliminary denoised image in which part of the noise is removed.

[0060] In the embodiments of the present disclosure, the first neural network model is a lightweight deep learning denoising network module. The first neural network module can be a neural network model constructed by a convolutional network and / or a fully connected network. For example, in implementation, an oidn (Open Image Denoise) network can be used as the first neural network model.

[0061] Generally, a neural network model can perform noise reduction processing on a noisy image. However, most of the features learned by the neural network model are derived from training samples in the spatial domain. Moreover, it is found through research that the neural network model is more inclined to optimize low-frequency features in the learning process, and it will remove part of the high-frequency features as noise, so that the details of the denoised image are lost. Therefore, in the embodiments of the present disclosure, the optimization of the image is completed in the frequency domain.

[0062] In S103, the preliminary denoised image and the auxiliary rendering features are aligned in the frequency domain to obtain intermediate features; the auxiliary rendering features at least include normal features and lighting features used to optimize the preliminary denoised image.

[0063] The lighting features, as the name implies, can be understood as the lighting conditions of each sampling position point in the three-dimensional model, which can include descriptions of lighting intensity, lighting angle, light source characteristics, refraction, and reflection conditions.

[0064] In addition to the lighting features, the auxiliary rendering features can also include normal features, depth features, reflection features, and other features that can be used to optimize image edges and texture details. The normal features are used to describe the distance of the sampling position point relative to the camera; the depth features, as the name implies, describe the depth of each sampling position point in the three-dimensional model.

[0065] In implementation, while obtaining the noisy image through ray tracing rendering by the rendering engine, the rendering auxiliary data of the three-dimensional model can also be obtained, such as the structure information of the three-dimensional model (i.e., describing the structural features of the model object in the three-dimensional model), the mapping information, the UV coordinate information (wherein the origin of the UV coordinate system can be the origin of the camera coordinate system), and the camera information (such as camera intrinsic parameters, extrinsic parameters, and viewing angle).

[0066] The rendering auxiliary data can be used to obtain intermediate products of the noise image for the rendering engine. Based on the rendering auxiliary data, auxiliary rendering features can be obtained by calculation. For example, reflection features and depth features can be obtained by calculation based on the structure information, map information, UV coordinate information, and camera information of the three-dimensional model. The reflection features can be calculated by a ray tracing algorithm, and the depth features can be obtained by obtaining the Z-axis coordinate value of the three-dimensional model.

[0067] The illumination features and normal features can be calculated based on existing technologies, and the embodiments of the present disclosure are not limited in this regard. The focus of the embodiments of the present disclosure is how to use the auxiliary rendering features to optimize the image quality.

[0068] It can be understood that the image denoised by the first neural network module may, to varying degrees, lose the original details and textures, and these details and textures can be included in the auxiliary rendering features to improve the image rendering quality.

[0069] In S104, the intermediate features are processed based on the image generator to generate a denoised image of the three-dimensional model.

[0070] The network structure of the image generator can be set according to actual requirements, for example, a convolutional network and / or a fully connected network can be used to implement the image generator, and the embodiments of the present disclosure are not limited in this regard.

[0071] In summary, in the embodiments of the present disclosure, the three-dimensional model is rendered at a lower sampling rate, which can reduce the cost and time consumption. Then, the first neural network module is used to preliminarily denoise the image to improve the image quality. On this basis, since the preliminarily denoised image will lose image details and textures, the image quality is further optimized based on the image details and textures included in the auxiliary rendering features in the embodiments of the present disclosure. Moreover, in order to make up for the characteristics of the traditional spatial domain image processing that easily fits low-frequency features and loses details, the embodiments of the present disclosure operate in the frequency domain and optimize the details based on the auxiliary rendering features. Finally, the image is optimized based on the image generator to obtain a high-quality rendering effect. In summary, the embodiments of the present disclosure provide a low-cost three-dimensional model rendering method, and a high-quality denoised image can be rendered.

[0072] In some embodiments, aligning the preliminarily denoised image and the auxiliary rendering features in the frequency domain can be implemented by converting the preliminarily denoised image and the auxiliary rendering features to frequency domain features. Then, the attention mechanism is used to learn the key intermediate features under the guidance of the frequency domain features of the auxiliary rendering features, so as to generate the final high-quality denoised image by the image generator. Specifically, in the embodiments of the present disclosure, the intermediate features can be constructed based on the manner as shown in FIG. 8, including the following steps: Figure 2 ​

[0073] S201, transform the preliminary denoising image to a frequency domain to obtain image frequency domain features.

[0074] S202, transform the auxiliary rendering features to a frequency domain to obtain auxiliary frequency domain features.

[0075] It should be noted that the execution order of S201 and S202 is not limited, that is, they can be executed simultaneously or sequentially.

[0076] It should be understood that S201 and S202 can obtain corresponding frequency domain features based on DCT (Discrete Cosine Transform).

[0077] S203, processing the image frequency domain features and the auxiliary frequency domain features based on an attention mechanism to obtain fusion features.

[0078] Among them, self-attention and cross-attention can be used to extract fusion features.

[0079] In order to improve the denoising effect and restore the details and textures of the three-dimensional model as much as possible, in the embodiments of the present disclosure, as shown in Figure 2 S203 can be further implemented as:

[0080] S2031, performing feature extraction on the image frequency domain features to obtain first features.

[0081] S2032, performing feature extraction on the auxiliary frequency domain features to obtain second features.

[0082] It should be noted that the execution order of S2031 and S2032 is not limited.

[0083] In implementation, at least one convolutional layer can be used to extract features from the image frequency domain features to obtain the first features. The convolutional layer can use a 1*1 convolutional neural network.

[0084] Similarly, at least one convolutional layer can be used to extract features from the auxiliary frequency domain features to obtain the second features. The convolutional layer can use a 1*1 convolutional neural network.

[0085] S2033, constructing input features of an attention module based on the first features and the second features.

[0086] Among them, the attention mechanism often needs query features Q, key features K and value features V to extract features. In order to improve the effect of denoising, in the embodiments of the present disclosure, the first features extracted from the image frequency domain features can be used as the query features of the attention module; and the second features extracted from the auxiliary frequency domain features can be used as the key features and the value features of the attention module.

[0087] It can be understood that, on the spectrum diagram after the preliminary denoising result and the rendering auxiliary feature are subjected to DCT, since there is no actual physical meaning, the frequency domain information can be integrated by feature extraction. Then, the first feature is regarded as a query, a key of the frequency domain information of the rendering auxiliary feature (the information extracted from the rendering auxiliary feature can be regarded as enhanced texture features), and finally multiplied by a value to enhance the extracted texture information.

[0088] Therefore, in the embodiments of the present disclosure, by reasonably constructing the input features, the denoising effect can be further improved, and the edges and textures of the generated image can be enhanced.

[0089] In S2034, the input features are processed by using the attention module to obtain the fusion features.

[0090] In the embodiments of the present disclosure, the attention module directly uses the connections and nonlinear mappings between neurons in the neural network to realize the transmission between features, and the obtained fusion features are closer to the quality of the rendered image of high sampling rate. The frequency domain features of the noise image and the auxiliary rendering features are further captured by feature extraction, and the key features are used to guide the attention module to interactively learn the two features. The high-dimensional core features can be extracted therefrom, so as to facilitate the image generator to generate high-quality images, and finally improve the denoising effect.

[0091] In S204, the fusion features are transformed into the spatial domain to obtain the intermediate features.

[0092] In a possible implementation, the fusion features can be subjected to inverse Fourier transform to restore the extracted key features to the spatial domain.

[0093] In another possible implementation, in order to improve the quality of the final image, the first feature can be transformed into the spatial domain by inverse Fourier transform to obtain first inverse transform features; and the fusion features can be transformed into the spatial domain by inverse Fourier transform to obtain second inverse transform features; and the first inverse transform features and the second inverse transform features are fused to obtain the intermediate features.

[0094] The first inverse transform features and the second inverse transform features can be processed by concatenation (concatenation) to fuse the two features.

[0095] It can be understood that the fusion features themselves introduce more details and textures on the basis of the preliminary denoising image by the auxiliary rendering features. However, the fusion features themselves are important key features, and the first features, i.e., the frequency domain of the preliminary denoising image, still retain a large number of non-important features. By combining the fusion features and the first features, the fusion of global information and important information can be realized, thereby facilitating the construction of high-quality intermediate features to improve the quality of the denoising image.

[0096] In summary, in the embodiments of the present disclosure, the learning in the frequency domain can make up for the loss of detailed information of the preliminary denoised image, and the combination of the two can enhance the rendering effect of the finally generated image and improve the rendering quality.

[0097] In the embodiments of the present disclosure, the image generator can be implemented by using the generator of the GAN network (Generative Adversarial Networks), and the structure of the image generator is not limited in the embodiments of the present disclosure.

[0098] Similarly, in order to improve the rendering quality, the preliminary denoised image and the intermediate feature are input into the image generator to obtain the denoised image of the three-dimensional model.

[0099] Therefore, the image generator can not only construct the image based on the intermediate feature as the key feature, but also optimize the constructed image based on the preliminary denoised image, thereby improving the rendering effect of the three-dimensional model.

[0100] In summary, the embodiments of the present disclosure can provide an image denoising model, which can include a first neural network module, an intermediate feature acquisition module, and an image generator, so as to realize low-cost and efficient three-dimensional model rendering.

[0101] The specific structure of the image denoising model can be as shown in Figure 3 For ease of understanding, the three-dimensional model rendering method of the embodiments of the present disclosure will be described in conjunction with Figure 3

[0102] First, the rendering engine samples the three-dimensional model by a low sampling rate, and the image rendered according to the sampling is a noise image. At the same time, auxiliary rendering features are obtained by rendering. The auxiliary rendering features are obtained by simple calculation from the rendering auxiliary data in the rendering engine, and these auxiliary rendering features provide illumination information, normal information, etc.

[0103] After the noise image is denoised by the lightweight first neural network module, a preliminary denoised image is obtained.

[0104] The intermediate feature acquisition module performs the following operations to obtain the intermediate feature:

[0105] 1) The preliminary denoised image and the auxiliary rendering feature are transformed by DCT to obtain the image frequency domain feature of the preliminary denoised image; the auxiliary rendering feature is transformed by DCT to obtain the auxiliary frequency domain feature of the auxiliary rendering feature.

[0106] ​2), the image frequency domain features of the preliminary denoising image are extracted to obtain first features; the auxiliary frequency domain features of the auxiliary rendering features are extracted to obtain second features. Through the feature extraction in the frequency domain, the feature reconstruction of the illumination and normal information of the auxiliary rendering features in the frequency domain is completed, so as to align the preliminary denoising image in the frequency domain.

[0107] 3), the first features and the second features are interactively learned through the attention module to obtain fused features.

[0108] 4), then the first features and the fused features are inversely Fourier transformed to obtain intermediate features in the spatial domain.

[0109] Therefore, by extracting the frequency domain features (including the image frequency domain features and the auxiliary frequency domain features), the preliminary denoising image and the auxiliary rendering features can be preliminarily aligned in the frequency domain. It can be understood that the center of the spectrum expresses low frequency, and the surrounding expresses high frequency. The high frequency describes the places where the pixel points in the image change greatly, that is, the texture detail characteristics of the image. Aligning the high frequency of the preliminary denoising image with the high frequency in the auxiliary rendering features can enhance the texture details, so as to achieve the effect of denoising and restoring the real details.

[0110] Finally, the preliminary denoising image and the intermediate features are processed by the image generator to generate a high-quality denoising image.

[0111] In summary, since the texture detail part of the preliminary noise image is blurred, and the illumination and normal information of the auxiliary rendering features can exactly contain this part of the blurred information, the attention unit is used to map the detail information of the auxiliary rendering features to the Fourier features of the preliminary denoising result, so as to correct the blurred part in the preliminary denoising result and obtain the intermediate features with texture detail information. Providing the intermediate features to the image generator can facilitate the generation of high-quality images.

[0112] In order to improve the rendering quality, based on the same technical concept, the disclosure embodiment also provides a training method of an image denoising model, wherein, as described above, the image denoising model includes a pre-trained first neural network module, an intermediate feature acquisition module and an image generator, as shown in Figure 4 the training method includes:

[0113] S401, inputting a sample image into the image denoising model to obtain a denoising image of the sample image; the sample image is obtained by ray tracing rendering of a sample three-dimensional model based on a target sampling number, and the sample three-dimensional model is ray traced and rendered based on a preset sampling number higher than the target sampling number to obtain a high-definition image of the sample image.

[0114] S402, determining a loss value based on the denoising image and the high-definition image.

[0115] In implementation, as shown in Figure 5 The discriminant model can be added after the image denoising model to optimize the parameters of the image denoising model based on the adversarial generation mechanism. The generated denoised image can be input into the discriminant model, and the discriminant model can distinguish whether the input denoised image is a generated high-definition denoised image or a real high-definition image obtained based on a high sampling rate by the rendering engine. The discriminant model can be a trained model, which is not limited in the embodiments of the present disclosure.

[0116] S403, adjusting the model parameters of the image denoising model based on the loss value.

[0117] The first neural network module is configured to perform preliminary denoising on the sample image to obtain a preliminary denoised sample.

[0118] The intermediate feature extraction module is configured to process the preliminary denoised sample and a sample auxiliary rendering feature of the sample image to align the preliminary denoised sample and the sample auxiliary rendering feature in the frequency domain to obtain an intermediate feature of the sample image. The sample auxiliary rendering feature at least includes a normal feature and an illumination feature required for rendering the sample image to optimize the preliminary denoised image.

[0119] The image generator is configured to process the intermediate feature of the sample image to obtain a denoised image.

[0120] In the embodiments of the present disclosure, for an image with noise, preliminary denoising can be implemented based on the first neural network module. Then, details are reconstructed based on the features in the frequency domain, and the denoising effect is further improved to improve the rendering quality. Moreover, supervised training based on the high-definition images rendered by multiple sampling can enable the model to restore images close to the rendering engine.

[0121] In a possible implementation, the difference between the denoised image of the sample image and the high-definition image can be determined to obtain a loss value.

[0122] In another possible implementation, to improve the model training effect, the loss value can be determined based on the denoised image, as shown in Figure 6 The loss value can be determined based on the difference between the denoised image and the high-definition image of the sample image.

[0123] S601, determining a denoising loss based on the difference between the denoised image and the high-definition image of the sample image.

[0124] S602, inputting the denoised image into a discriminator model to identify the image type of the denoised image; determining a discriminant loss based on the predicted image type and the real type label; and

[0125] S603, determining a Fourier loss between the denoised image and the high-definition image.

[0126] Wherein, the denoising image can be transformed to the frequency domain, and the high-definition image can be transformed to the frequency domain, and the difference between the two in the frequency domain is determined, and thus the Fourier loss is obtained. In addition, the execution order of S601-S603 is not limited.

[0127] S604, weighted sum is performed based on the denoising loss, the discriminative loss and the Fourier loss, and a loss value is obtained.

[0128] Here, the model parameters are preliminarily adjusted using L1Loss, and then the model parameters are fine-tuned using the following loss function.

[0129] Specifically, the loss function can be as shown in expression (1):

[0130] L=a*L1loss+b*GAN loss+c*fourier loss (1)

[0131] Wherein, L represents the loss value; L1loss represents the denoising loss; GAN loss represents the discriminative loss; fourier loss represents the Fourier loss; a, b, c are all weights. When implemented, a can be required to be greater than b, and b can be required to be greater than c.

[0132] In the embodiments of the present disclosure, the loss is comprehensively measured in multiple dimensions, which is beneficial to adjust the model parameters of different modules in the model, accelerate the convergence speed of the model, and improve the model training effect.

[0133] Figures 7-10 The experimental effect comparison chart of the embodiments of the present disclosure is shown in the following table. Figures 7-10 In the table, N represents a noise image, GT represents a high-definition image obtained by rendering a three-dimensional model with high frequency sampling, and is used as a comparison chart. M represents a rendering image provided by the embodiments of the present disclosure, and 0 represents a rendering image of other deep learning models.

[0134] Figure 7 In the table, it can be known from the image content in the mirror that the embodiments of the present disclosure can accurately express the details of the mirror reflecting light. In 0, there are various noises, and the image details cannot be smoothly expressed.

[0135] Figure 8 In the table, in the high-light environment, the curtain fold part is a part with many details. It can be known through comparison that the scheme provided by the embodiments of the present disclosure can more smoothly display the curtain folds.

[0136] Figure 9 In the table, in the low-brightness environment, the curtain part reflects the details of alternating highlights and dark tones. It can be known through comparison that the image curtain restored by the embodiments of the present disclosure is relatively smooth, and the effects reflected by other models are some chaotic noises.

[0137] Figure 10In the case of a cabinet with some patterns on the surface, the embodiments of the present disclosure can reflect the texture of the patterns and the material, while other models reflect more noisy and disordered effects.

[0138] Based on the same technical concept, the embodiments of the present disclosure also provide a three-dimensional model rendering device 1100, as shown in the accompanying drawings, comprising: Figure 11

[0139] The rendering module 1101 is configured to perform ray tracing rendering on the three-dimensional model based on a target sampling number to obtain a noisy image with noise; wherein the target sampling number is less than a preset threshold.

[0140] The denoising module 1102 is configured to perform denoising processing on the noisy image using a first neural network module to obtain a preliminary denoised image in which part of the noise is removed.

[0141] The alignment module 1103 is configured to align the preliminary denoised image and auxiliary rendering features in the frequency domain to obtain intermediate features; the auxiliary rendering features at least include normal features and lighting features used to optimize the preliminary denoised image.

[0142] The generation module 1104 is configured to process the intermediate features based on an image generator to generate a denoised image of the three-dimensional model.

[0143] In some embodiments, the alignment module comprises:

[0144] The first processing unit is configured to transform the preliminary denoised image to the frequency domain to obtain image frequency domain features; and.

[0145] The second processing unit is configured to transform the auxiliary rendering features to the frequency domain to obtain auxiliary frequency domain features.

[0146] The fusion unit is configured to process the image frequency domain features and the auxiliary frequency domain features based on an attention mechanism to obtain fusion features.

[0147] The third processing unit is configured to transform the fusion features to the spatial domain to obtain the intermediate features.

[0148] In some embodiments, the fusion unit comprises:

[0149] The first extraction unit is configured to perform feature extraction on the image frequency domain features to obtain first features; and,

[0150] The second extraction unit is configured to perform feature extraction on the auxiliary frequency domain features to obtain second features.

[0151] The construction unit is configured to construct input features of the attention module based on the first features and the second features.

[0152] The fourth processing unit is configured to process the input features using the attention module to obtain the fusion features.​

[0153] In some embodiments, the input features of the attention module include query features, key features and value features. The construction unit is configured to:

[0154] the first features extracted from the image frequency domain features are taken as the query features of the attention module; and

[0155] the second features extracted from the auxiliary frequency domain features are taken as the key features and the value features of the attention module.

[0156] In some embodiments, the third processing unit is configured to:

[0157] transform the first features into a spatial domain by inverse Fourier transform to obtain first inverse transform features; and

[0158] transform the fused features into the spatial domain by inverse Fourier transform to obtain second inverse transform features;

[0159] fuse the first inverse transform features and the second inverse transform features to obtain intermediate features.

[0160] In some embodiments, the generation module is configured to:

[0161] input the preliminary denoised image and the intermediate features into an image generator to obtain a denoised image of the three-dimensional model.

[0162] Based on the same technical concept, the embodiments of the present disclosure further provide a training device 1200 of an image denoising model, wherein the image denoising model includes a pre-trained first neural network module, an intermediate feature acquisition module and an image generator, as shown in Figure 12 , which includes:

[0163] The input module 1201 is configured to input a sample image into the image denoising model to obtain a denoised image of the sample image; the sample image is obtained by ray tracing rendering of a sample three-dimensional model based on a target sampling number, and the sample three-dimensional model is ray tracing rendered based on a preset sampling number higher than the target sampling number to obtain a high-definition image of the sample image.

[0164] The determination module 1202 is configured to determine a loss value based on the denoised image and the high-definition image.

[0165] The adjustment module 1203 is configured to adjust model parameters of the image denoising model based on the loss value.

[0166] The first neural network module is configured to preliminarily denoise the sample image to obtain a preliminarily denoised sample.

[0167] The intermediate feature extraction module is configured to process the preliminary denoised sample and sample auxiliary rendering features of the sample image, to align the preliminary denoised sample and the sample auxiliary rendering features in the frequency domain, and to obtain intermediate features of the sample image; the sample auxiliary rendering features at least include normal features and illumination features required for rendering the sample image, and the normal features and the illumination features are used to optimize the preliminary denoised image.

[0168] The image generator is configured to process the intermediate features of the sample image, to obtain a denoised image.

[0169] In some embodiments, the determining module is configured to:

[0170] determine a denoising loss based on a difference between the denoised image of the sample image and the high-definition image; and

[0171] input the denoised image into the discriminator model to identify an image type of the denoised image;

[0172] determine a discrimination loss based on the predicted image type and a true type label; and

[0173] determine a Fourier loss between the denoised image and the high-definition image;

[0174] perform weighted summation based on the denoising loss, the discrimination loss, and the Fourier loss to obtain a loss value.

[0175] The specific functions and examples of the modules and units of the apparatuses of the embodiments of the present disclosure are described in the related description of the corresponding steps in the above method embodiments, which will not be described here.

[0176] In the technical solutions of the present disclosure, the acquisition, storage, and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0177] Figure 13 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. As shown in the figure, the electronic device includes a memory 1310 and a processor 1320, and the memory 1310 stores a computer program that can run on the processor 1320. The number of memories 1310 and processors 1320 can be one or more. The memory 1310 can store one or more computer programs, which, when executed by the electronic device, cause the electronic device to perform the method provided in the above method embodiments. The electronic device can also include a communication interface 1330 for communicating with external devices and transmitting data. Figure 13

[0178] ​If the memory 1310, the processor 1320 and the communication interface 1330 are implemented independently, the memory 1310, the processor 1320 and the communication interface 1330 can be connected with each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 13 Only one thick line is used to represent the bus in the middle, but it does not mean that there is only one bus or only one type of bus.

[0179] Optionally, in a specific implementation, if the memory 1310, the processor 1320 and the communication interface 1330 are integrated on a chip, the memory 1310, the processor 1320 and the communication interface 1330 can complete communication between each other through an internal interface.

[0180] It should be understood that the above processor can be a Central Processing Unit (CPU), and can also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. It is worth noting that the processor can be an Advanced RISC Machines (ARM) architecture processor.

[0181] Furthermore, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0182] In the above embodiments, all or part of the steps can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the steps can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present disclosure are generated. The computer can be a general purpose computer, a special purpose computer, a computer network or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (for example: coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example: infrared, Bluetooth, microwave, etc.) mode. The computer readable storage medium can be any available medium accessible by a computer, or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example: floppy disk, hard disk, magnetic tape), an optical medium (for example: digital versatile disc (DVD)) or a semiconductor medium (for example: solid state disk (SSD)) and the like. It is worth noting that the computer readable storage medium mentioned in the present disclosure can be a non-volatile storage medium, in other words, it can be a non-transitory storage medium.

[0183] A person of ordinary skill in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or the program can instruct the related hardware to complete, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0184] In the description of the embodiments of the present disclosure, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0185] In the description of the embodiments of the present disclosure, unless otherwise specified, " / " means or, for example, A / B can mean A or B. "And / or" in this document only describes the relationship between associated objects, which means that there can be three relationships, for example, A and / or B, which can mean: A exists alone, A and B exist together, and B exists alone.

[0186] In the description of the embodiments of the present disclosure, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more.

[0187] The above only describes exemplary embodiments of the present disclosure, and is not intended to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A three-dimensional model rendering method, comprising: Performing ray tracing rendering on the three-dimensional model based on a target sampling number to obtain a noise image with noise; wherein the target sampling number is less than a preset threshold; Using a first neural network module to perform denoising on the noisy image to obtain a preliminary denoised image with some noise removed; Aligning the preliminary denoised image and auxiliary rendering features in the frequency domain to obtain intermediate features; the auxiliary rendering features at least include normal features and illumination features for optimizing the preliminary denoised image; processing the intermediate features based on an image generator to generate a denoised image of the three-dimensional model; The aligning the preliminary denoised image and the auxiliary rendering features in the frequency domain to obtain intermediate features includes: Transforming the preliminary denoised image into the frequency domain to obtain image frequency domain features; and Transforming the auxiliary rendering features into the frequency domain to obtain auxiliary frequency domain features; Processing the image frequency domain features and the auxiliary frequency domain features based on the attention mechanism to obtain fused features; wherein the image frequency domain features are used to construct the query Q required by the attention mechanism, and the auxiliary frequency domain features are used to construct the key K and value V required by the attention mechanism; The fused features are transformed into a spatial domain to obtain the intermediate features.

2. The method according to claim 1, wherein The processing of the image frequency domain features and the auxiliary frequency domain features based on the attention mechanism to obtain fused features includes: Extracting the image frequency domain features to obtain a first feature; and Performing feature extraction on the auxiliary frequency domain feature to obtain a second feature; constructing input features of an attention module based on the first features and the second features; The attention module is used to process the input features to obtain the fused features.

3. The method according to claim 2, wherein: The input features of the attention module include query features, key features and value features; The constructing the input features of the attention module based on the first feature and the second feature includes: Using the first feature extracted from the image frequency domain feature as the query feature of the attention module; and The second feature extracted from the auxiliary frequency domain feature is used as the key feature and the value feature of the attention module.

4. The method according to claim 2, wherein: The transforming the fused features into a spatial domain to obtain the intermediate features includes: Transforming the first feature into the spatial domain by inverse Fourier transform to obtain a first inverse transformed feature; and Transforming the fused features into the spatial domain by inverse Fourier transform to obtain a second inverse transformed feature; The first inverse transformation feature and the second inverse transformation feature are fused to obtain the intermediate feature.

5. The method according to any one of claims 1 to 4, wherein The processing of the intermediate features by an image generator to generate a denoised image of the three-dimensional model includes: The preliminary denoised image and the intermediate features are input into the image generator to obtain a denoised image of the three-dimensional model.

6. A training method for an image denoising model, wherein: The image denoising model includes a pre-trained first neural network module, an intermediate feature acquisition module, and an image generator. The method includes: Inputting a sample image into the image denoising model to obtain a denoised image of the sample image; the sample image is obtained by performing ray tracing rendering on the sample three-dimensional model based on a target sampling number, and performing ray tracing rendering on the sample three-dimensional model based on a preset sampling number higher than the target sampling number to obtain a high-definition image of the sample image; determining a loss value based on the denoised image and the high-definition image; Adjusting model parameters of the image denoising model based on the loss value; The first neural network module is used to perform preliminary noise reduction on the sample image to obtain a preliminary noise reduction sample; The intermediate feature extraction module is used to process the preliminary denoised samples and the sample auxiliary rendering features of the sample image to align the preliminary denoised samples and the sample auxiliary rendering features in the frequency domain to obtain intermediate features of the sample image; the sample auxiliary rendering features at least include normal features and lighting features required for rendering the sample image and used to optimize the preliminary denoised samples; The image generator is used to process the intermediate features of the sample image to obtain the denoised image; The aligning the preliminary denoised sample and the sample auxiliary rendering feature in the frequency domain to obtain the intermediate feature of the sample image includes: Transforming the preliminary denoised samples into the frequency domain to obtain image frequency domain features; and Transforming the sample auxiliary rendering features into the frequency domain to obtain auxiliary frequency domain features; Processing the image frequency domain features and the auxiliary frequency domain features based on the attention mechanism to obtain fused features; wherein the image frequency domain features are used to construct the query Q required by the attention mechanism, and the auxiliary frequency domain features are used to construct the key K and value V required by the attention mechanism; The fused features are transformed into a spatial domain to obtain the intermediate features.

7. The method according to claim 6, wherein: The determining of the loss value based on the denoised image comprises: determining a denoising loss based on a difference between the denoised image and the high-definition image of the sample image; and, Inputting the denoised image into a discriminator model to identify the image type of the denoised image; Determine the discriminative loss based on the predicted image type and the true type label; and determining a Fourier loss between the denoised image and the high-definition image; The loss value is obtained by performing weighted summation based on the denoising loss, the discrimination loss, and the Fourier loss.

8. A three-dimensional model rendering device, comprising: a rendering module, configured to perform ray tracing rendering on the three-dimensional model based on a target sampling number to obtain a noise image having noise; wherein the target sampling number is less than a preset threshold; a denoising module, configured to perform denoising on the noisy image using a first neural network module to obtain a preliminary denoised image from which part of the noise has been removed; an alignment module, configured to align the preliminary denoised image and auxiliary rendering features in the frequency domain to obtain intermediate features; the auxiliary rendering features at least including normal features and illumination features for optimizing the preliminary denoised image; a generating module for processing the intermediate features based on an image generator to generate a denoised image of the three-dimensional model; The alignment module is specifically used to: Transforming the preliminary denoised image into the frequency domain to obtain image frequency domain features; and Transforming the auxiliary rendering features into the frequency domain to obtain auxiliary frequency domain features; Processing the image frequency domain features and the auxiliary frequency domain features based on the attention mechanism to obtain fused features; wherein the image frequency domain features are used to construct the query Q required by the attention mechanism, and the auxiliary frequency domain features are used to construct the key K and value V required by the attention mechanism; The fused features are transformed into a spatial domain to obtain the intermediate features.

9. A training device for an image denoising model, wherein: The image denoising model includes a pre-trained first neural network module, an intermediate feature acquisition module and an image generator, and the device includes: an input module, configured to input a sample image into the image denoising model to obtain a denoised image of the sample image; the sample image is obtained by performing ray tracing rendering on a sample three-dimensional model based on a target sampling number, and further, performing ray tracing rendering on the sample three-dimensional model based on a preset sampling number higher than the target sampling number to obtain a high-definition image of the sample image; a determination module, configured to determine a loss value based on the denoised image and the high-definition image; An adjustment module, configured to adjust model parameters of the image denoising model based on the loss value; The first neural network module is used to perform preliminary noise reduction on the sample image to obtain a preliminary noise reduction sample; The intermediate feature extraction module is used to process the preliminary denoised samples and the sample auxiliary rendering features of the sample image to align the preliminary denoised samples and the sample auxiliary rendering features in the frequency domain to obtain intermediate features of the sample image; the sample auxiliary rendering features at least include normal features and lighting features required for rendering the sample image and used to optimize the preliminary denoised samples; The image generator is used to process the intermediate features of the sample image to obtain the denoised image; The aligning the preliminary denoised sample and the sample auxiliary rendering feature in the frequency domain to obtain the intermediate feature of the sample image includes: Transforming the preliminary denoised samples into the frequency domain to obtain image frequency domain features; and Transforming the sample auxiliary rendering features into the frequency domain to obtain auxiliary frequency domain features; Processing the image frequency domain features and the auxiliary frequency domain features based on the attention mechanism to obtain fused features; wherein the image frequency domain features are used to construct the query Q required by the attention mechanism, and the auxiliary frequency domain features are used to construct the key K and value V required by the attention mechanism; The fused features are transformed into a spatial domain to obtain the intermediate features.

10. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images