Three-dimensional model rendering method, image denoising model training method and related device

Abstract

The invention provides a three-dimensional model rendering method, an image denoising model training method and related devices. Relates to the technical field of artificial intelligence. The method comprises the following steps: performing ray tracing rendering on a three-dimensional model based on target sampling times to obtain a noise image with noise; wherein the target sampling frequency is smaller than a preset threshold value; carrying out noise reduction processing on the noise image by adopting a first neural network module to obtain a preliminary noise reduction image of which part of noise is removed; aligning the preliminary noise reduction image and the auxiliary rendering feature in a frequency domain to obtain an intermediate feature; the auxiliary rendering features at least comprise normal features and illumination features used for optimizing the preliminary noise reduction image; the intermediate features are processed based on an image generator to generate a noise-reduced image of the three-dimensional model. According to the embodiment of the invention, high-quality images can be rendered quickly at low cost.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as deep learning, three-dimensional model rendering, and image rendering. Background Art

[0002] In the field of home decoration design, it is generally necessary to use home decoration design software to complete the home decoration design renderings before formal construction, so as to determine the final decoration plan. Many home decoration design software support the construction of 3D renderings based on 3D models. In related technologies, it is necessary to render the designed 3D model to generate a 3D rendering that is close to the real lighting conditions.

[0003] However, the rendering engine needs to sample the 3D model a high number of times to build a realistic 3D rendering. A high sampling rate can render high-quality images, but it is costly and time-consuming. When the sampling number is low, the resulting 3D rendering has a lot of noise. Summary of the invention

[0004] The present disclosure provides a control method, device, electronic device and storage medium for a home decoration model to solve or alleviate one or more technical problems in related technologies.

[0005] In a first aspect, the present disclosure provides a three-dimensional model rendering method, comprising:

[0006] Based on the target sampling number, ray tracing rendering is performed on the three-dimensional model to obtain a noisy image with noise; wherein the target sampling number is less than a preset threshold;

[0007] The first neural network module is used to perform denoising on the noisy image to obtain a preliminary denoised image with some noise removed;

[0008] Aligning the preliminary denoised image and the auxiliary rendering features in the frequency domain to obtain intermediate features; the auxiliary rendering features at least include normal features and illumination features for optimizing the preliminary denoised image;

[0009] The intermediate features are processed based on the image generator to generate a denoised image of the three-dimensional model.

[0010] In a second aspect, the present disclosure provides a method for training an image denoising model, wherein the image denoising model includes a pre-trained first neural network module, an intermediate feature acquisition module, and an image generator, and the method includes:

[0011] Inputting a sample image into the image denoising model to obtain a denoised image of the sample image; the sample image is obtained by performing ray tracing rendering on a sample three-dimensional model based on a target sampling number, and performing ray tracing rendering on the sample three-dimensional model based on a preset sampling number higher than the target sampling number to obtain a high-definition image of the sample image;

[0012] determining a loss value based on the denoised image and the high-definition image;

[0013] Adjusting a model parameter of the image denoising model based on the loss value;

[0014] Wherein, the first neural network module is used to perform preliminary denoising on the sample image to obtain a preliminary denoised sample;

[0015] The intermediate feature extraction module is used to process the preliminary denoised samples and the sample auxiliary rendering features of the sample image to align the preliminary denoised samples and the sample auxiliary rendering features in the frequency domain to obtain the intermediate features of the sample image; the sample auxiliary rendering features at least include normal features and illumination features required for rendering the sample image and used to optimize the preliminary denoised image;

[0016] The image generator is used to process the intermediate features of the sample image to obtain the denoised image.

[0017] In a third aspect, the present disclosure provides a three-dimensional model rendering device, comprising:

[0018] A rendering module, used for performing ray tracing rendering on the three-dimensional model based on a target sampling number to obtain a noisy image with noise; wherein the target sampling number is less than a preset threshold, and the auxiliary rendering feature is used to describe at least a normal feature and a lighting feature of the three-dimensional model;

[0019] A denoising module, configured to perform denoising on the noisy image using the first neural network module to obtain a preliminary denoised image with some noise removed;

[0020] An alignment module, used for aligning the preliminary denoised image and the auxiliary rendering features in the frequency domain to obtain intermediate features; the auxiliary rendering features at least include normal features and illumination features for optimizing the preliminary denoised image;

[0021] The generation module is used for processing the intermediate features based on the image generator to generate a denoised image of the three-dimensional model.

[0022] In a fourth aspect, the present disclosure provides a training device for an image denoising model, wherein the image denoising model includes a pre-trained first neural network module, an intermediate feature acquisition module and an image generator, and the device includes:

[0023] An input module, used for inputting a sample image into the image denoising model to obtain a denoised image of the sample image; the sample image is obtained by performing ray tracing rendering on a sample three-dimensional model based on a target sampling number, and performing ray tracing rendering on the sample three-dimensional model based on a preset sampling number higher than the target sampling number to obtain a high-definition image of the sample image;

[0024] A determination module, configured to determine a loss value based on the denoised image and the high-definition image;

[0025] An adjustment module, used for adjusting model parameters of the image denoising model based on the loss value;

[0026] Wherein, the first neural network module is used to perform preliminary denoising on the sample image to obtain a preliminary denoised sample;

[0027] The intermediate feature extraction module is used to process the preliminary denoised samples and the sample auxiliary rendering features of the sample image to align the preliminary denoised samples and the sample auxiliary rendering features in the frequency domain to obtain the intermediate features of the sample image; the sample auxiliary rendering features at least include normal features and illumination features required for rendering the sample image and used to optimize the preliminary denoised image;

[0028] The image generator is used to process the intermediate features of the sample image to obtain the denoised image.

[0029] In a fifth aspect, an electronic device is provided, including:

[0030] at least one processor; and

[0031] a memory communicatively connected to the at least one processor; wherein,

[0032] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0033] In a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any method according to the embodiments of the present disclosure.

[0034] In a seventh aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements any method according to the embodiments of the present disclosure.

[0035] The disclosed embodiments can render high-quality images for three-dimensional models quickly and at low cost.

[0036] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments provided according to the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0038] Figure 1 is a flowchart of a three-dimensional model rendering method according to an embodiment of the present disclosure;

[0039] Figure 2 is a schematic diagram of a process for constructing an intermediate feature according to an embodiment of the present disclosure;

[0040] Figure 3 is a structural schematic diagram of an image denoising model according to an embodiment of the present disclosure;

[0041] Figure 4 is a flowchart of a training method for an image denoising model according to an embodiment of the present disclosure;

[0042] Figure 5 is a structural schematic diagram of an image denoising model according to an embodiment of the present disclosure;

[0043] Figure 6 is a schematic diagram of a process for determining a loss value according to an embodiment of the present disclosure;

[0044] Figure 7 is a comparison diagram of effects provided according to an embodiment of the present disclosure;

[0045] Figure 8 is another effect comparison diagram provided according to an embodiment of the present disclosure;

[0046] Fig. 9 is another effect comparison diagram provided according to an embodiment of the present disclosure;

[0047] Fig.10 is another effect comparison diagram provided according to an embodiment of the present disclosure;

[0048] Fig.11 is a structural schematic diagram of a three-dimensional model rendering device according to an embodiment of the present disclosure;

[0049] Fig.12 is a structural schematic diagram of a training device for an image denoising model according to an embodiment of the present disclosure;

[0050] Fig.13 It is a block diagram of an electronic device used to implement the 3D model rendering method / image denoising model training method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] The present disclosure will be further described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0052] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0053] In addition, it should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0054] As described in the background technology, using a high sampling rate to render high-quality images is costly and time-consuming. There are also related technologies that first use a low sampling rate to render a noisy image, and then use a denoising algorithm to remove the noise in the hope of obtaining a high-quality image. However, the effect depends on the quality of the algorithm.

[0055] In view of this, a three-dimensional model rendering method is provided in an embodiment of the present disclosure. Figure 1 As shown, it is a schematic diagram of the process of the method, comprising:

[0056] S101, based on a target sampling number, performing ray tracing rendering on a three-dimensional model to obtain a noisy image with noise; wherein the target sampling number is less than a preset threshold.

[0057] The three-dimensional model of the embodiment of the present disclosure may be a three-dimensional scene obtained through home decoration design.

[0058] During implementation, a rendering engine is used to perform ray tracing rendering on the three-dimensional model. In the embodiment of the present disclosure, in order to improve rendering efficiency, the number of rendering sampling times can be reduced, and the image obtained is noisy, which is referred to as a noisy image in the embodiment of the present disclosure. The noisy image allows for large noise and is very blurred. Then, the image quality is optimized through noise reduction processing.

[0059] S102, using a first neural network module to perform denoising on the noisy image to obtain a preliminary denoised image with part of the noise removed.

[0060] In the disclosed embodiment, the first neural network model is a lightweight deep learning denoising network module. The first neural network module can be a neural network model constructed by a convolutional network and / or a fully connected network. For example, during implementation, an oidn (Open Image Denoise) network can be used as the first neural network model.

[0061] In general, although the neural network model can perform noise reduction on noisy images, most of the features learned by the neural network model come from training samples in the spatial domain. Moreover, it has been found through research that during the learning process, the neural network model tends to optimize low-frequency features, and it will treat some high-frequency features as noise and remove them, which will cause the denoised image to lose details. In view of this, in the embodiments of the present disclosure, it is proposed to optimize the image in the frequency domain.

[0062] In S103, the preliminary denoised image and the auxiliary rendering features are aligned in the frequency domain to obtain intermediate features; the auxiliary rendering features at least include normal features and illumination features for optimizing the preliminary denoised image.

[0063] Among them, the lighting characteristics, as the name suggests, can be understood as the lighting conditions of each sampling position point in the three-dimensional model, which may include lighting intensity, lighting angle, light source characteristics, description of refraction and reflection conditions, etc.

[0064] In addition to illumination features, auxiliary rendering features can also include normal features, depth features, reflection features, etc., which can be used to optimize image edge and texture details. Among them, the normal feature is used to describe the distance of the sampling position point relative to the camera; the depth feature, as the name implies, describes the depth of each sampling position point in the 3D model.

[0065] During implementation, while a noise image is obtained by performing ray tracing rendering through a rendering engine, rendering auxiliary data of the three-dimensional model can be obtained, such as structural information of the three-dimensional model (i.e., describing the structural features of the model object in the three-dimensional model), mapping information, UV coordinate information (wherein the origin of the UV coordinate system can be the origin of the camera coordinate system), and camera information (such as camera intrinsic parameters, extrinsic parameters, viewing angle) and other rendering auxiliary data.

[0066] Among them, the rendering auxiliary data can be an intermediate product of the noise image obtained by the rendering engine. Based on these rendering auxiliary data, auxiliary rendering features can be obtained by calculation. For example, the reflection feature and depth feature are obtained by calculation through the structural information, mapping information, UV coordinate information and camera information of the three-dimensional model. Among them, the reflection feature can be calculated by the ray tracing algorithm, and the Z-axis coordinate value of the three-dimensional model can be obtained to obtain the depth feature.

[0067] The illumination features and normal features can be calculated based on the existing technology, and the embodiments of the present disclosure do not limit this. The embodiment of the present disclosure focuses on how to use auxiliary rendering features to optimize image quality.

[0068] It can be understood that the image after denoising using the first neural network module may lose the original detail texture to varying degrees, and these details can be included in the auxiliary rendering features to improve the image rendering quality.

[0069] S104, processing the intermediate features based on the image generator to generate a denoised image of the three-dimensional model.

[0070] Among them, the network structure of the image generator can be set according to actual needs. For example, it can be implemented by using a convolutional network and / or a fully connected network, which is not limited in the embodiments of the present disclosure.

[0071] In summary, in the examples disclosed herein, a lower sampling rate is used to render the three-dimensional model, which can reduce costs and time. Then, the first neural network module is used to perform preliminary denoising on the image to improve the image quality. On this basis, since the preliminary denoised image will lose image details and texture, the image quality is further optimized based on the image detail texture contained in the auxiliary rendering features in the embodiments disclosed herein. Moreover, in order to make up for the fact that traditional spatial domain image processing easily fits low-frequency features and loses details, the embodiments disclosed herein operate in the frequency domain and optimize details based on auxiliary rendering features. Finally, the image is optimized based on the image generator to obtain a high-quality rendering effect. In summary, the embodiments disclosed herein provide a low-cost three-dimensional model rendering method, and can render a high-quality denoised image.

[0072] In some embodiments, aligning the preliminary denoised image and the auxiliary rendering features in the frequency domain can be implemented by converting both the preliminary denoised image and the auxiliary rendering features into the frequency domain to obtain their respective frequency domain features. Then, an attention mechanism is used to learn key intermediate features under the guidance of the frequency domain features of the auxiliary rendering features, so that the image generator can generate the final high-quality denoised image. Specifically, in the embodiments of the present disclosure, based on the following example, Figure 2 The method shown in the figure constructs the intermediate features, including the following steps:

[0073] S201, transforming the preliminary denoised image into the frequency domain to obtain image frequency domain features.

[0074] S202, transforming the auxiliary rendering features into the frequency domain to obtain auxiliary frequency domain features.

[0075] It should be noted that the execution order of S201 and S202 is not limited, that is, the two can be executed simultaneously or in sequence.

[0076] It should be understood that S201 and S202 can obtain corresponding frequency domain features based on DCT (Discrete Cosine Transform).

[0077] S203, processing the image frequency domain features and the auxiliary frequency domain features based on the attention mechanism to obtain fusion features.

[0078] Among them, self-attention and cross-attention can be used to extract fusion features.

[0079] In order to improve the noise reduction effect and restore the details and texture of the three-dimensional model as much as possible, in the embodiment of the present disclosure, Figure 2 As shown, step S203 can be further implemented as follows:

[0080] S2031, extracting the frequency domain features of the image to obtain a first feature.

[0081] S2032, extract the auxiliary frequency domain feature to obtain a second feature.

[0082] It should be noted that the execution order of S2031 and S2032 is not limited.

[0083] During implementation, at least one convolution layer may be used to extract the image frequency domain features to obtain the first feature. The convolution layer may use a 1*1 convolutional neural network.

[0084] Similarly, at least one convolution layer may be used to extract the auxiliary frequency domain feature to obtain the second feature. The convolution layer may use a 1*1 convolutional neural network.

[0085] S2033, constructing input features of the attention module based on the first feature and the second feature.

[0086] Among them, the attention mechanism often requires query feature Q, key feature K and value feature V for feature extraction. In order to improve the effect of noise reduction, in the embodiment of the present disclosure, the first feature extracted from the image frequency domain feature can be used as the query feature of the attention module; and the second feature extracted from the auxiliary frequency domain feature can be used as the key feature and value feature of the attention module.

[0087] It can be understood that since there is no actual physical meaning in the spectrum diagram of the preliminary denoising result and the rendering auxiliary feature after DCT, the frequency domain information can be integrated through feature extraction. After that, the first feature is regarded as a query, and the key of the frequency domain information of the rendering auxiliary feature (the information extracted by the rendering auxiliary feature can be regarded as an enhanced texture feature) is queried, and finally multiplied by the value to enhance the extracted texture information.

[0088] Therefore, in the embodiments of the present disclosure, by reasonably constructing input features, the effect of noise reduction can be further improved and the edges and textures of the generated images can be enhanced.

[0089] S2034, using the attention module to process the input features to obtain fused features.

[0090] In the disclosed embodiment, the attention module directly utilizes the connection and nonlinear mapping between neurons in the neural network to realize the transfer between features, and the obtained fusion features are closer to the quality of the rendering image rendered at a high sampling rate. The frequency domain features of the noise image and the frequency domain features of the auxiliary rendering features are further captured through feature extraction, and the key features are used to guide the attention module to interactively learn the two features, from which high-dimensional core features can be extracted, so as to facilitate the image generator to generate high-quality images, and ultimately improve the noise reduction effect.

[0091] S204, transforming the fused features into the spatial domain to obtain intermediate features.

[0092] In a possible implementation, an inverse Fourier transform may be performed on the fused features to restore the extracted key features to the spatial domain.

[0093] In another possible implementation, in order to improve the final image quality, the first feature can be transformed into the spatial domain through an inverse Fourier transform to obtain a first inverse transform feature; and the fused feature can be transformed into the spatial domain through an inverse Fourier transform to obtain a second inverse transform feature; the first inverse transform feature and the second inverse transform feature can be fused to obtain an intermediate feature.

[0094] The first inverse transformation feature and the second inverse transformation feature may be processed by concat (splicing) to merge the two.

[0095] It can be understood that the fusion feature itself introduces more details and textures on the basis of the preliminary denoised image through the auxiliary rendering feature. However, the fusion feature itself is an important key feature, and the first feature, that is, the frequency domain of the preliminary denoised image still retains a large number of non-important features. By combining the fusion feature with the first feature, the fusion of global information and important information can be achieved, which is conducive to constructing high-quality intermediate features to improve the quality of the denoised image.

[0096] In summary, in the embodiments of the present disclosure, learning in the frequency domain can make up for the detail information lost in the preliminary denoised image. The combination of the two can enhance the rendering effect of the final generated image and improve the rendering quality.

[0097] In the embodiment of the present disclosure, the image generator can be implemented using a generator of a GAN network (Generative Adversarial Networks), and the structure of the image generator is not limited in the embodiment of the present disclosure.

[0098] Similarly, in order to improve the rendering quality, in the embodiment of the present disclosure, the preliminary denoised image and the intermediate features are input into the image generator to obtain the denoised image of the three-dimensional model.

[0099] Therefore, the image generator can not only construct an image based on the key feature of the intermediate feature, but also optimize the constructed image based on the preliminary denoised image, thereby improving the rendering effect of the three-dimensional model.

[0100] In summary, the embodiments of the present disclosure may provide an image denoising model, which may include a first neural network module, an intermediate feature acquisition module, and an image generator, so as to achieve low-cost and efficient three-dimensional model rendering.

[0101] The specific structure of the image denoising model can be shown as follows Figure 3 For ease of understanding, combined with Figure 3 To illustrate the three-dimensional model rendering method of the embodiment of the present disclosure.

[0102] First, the rendering engine samples the 3D model at a low sampling rate, and the rendered image is a noise image. At the same time, auxiliary rendering features are obtained through rendering. Auxiliary rendering features are obtained from the rendering auxiliary data in the rendering engine through simple calculations. These rendering auxiliary features provide lighting information, normal information, etc.

[0103] After the noisy image is denoised by the lightweight first neural network module, a preliminary denoised image is obtained.

[0104] The intermediate feature acquisition module performs the following operations to obtain the intermediate features:

[0105] 1) The preliminary denoised image and the auxiliary rendering features are transformed by DCT to obtain the image frequency domain features of the preliminary denoised image; the auxiliary rendering features are transformed by DCT to obtain the auxiliary frequency domain features of the auxiliary rendering features.

[0106] 2) The image frequency domain features of the preliminary denoised image are subjected to feature extraction to obtain the first feature; the auxiliary frequency domain features of the auxiliary rendering features are subjected to feature extraction to obtain the second feature. Through feature extraction in the frequency domain, the feature reconstruction of the illumination and normal information of the auxiliary rendering features in the frequency domain is completed, so as to align the preliminary denoised image in the frequency domain.

[0107] 3) The first feature and the second feature are interactively learned by the attention module to obtain a fusion feature.

[0108] 4) Then the first feature and the fusion feature are transformed by inverse Fourier transform to obtain the intermediate feature in the spatial domain.

[0109] Therefore, by extracting the frequency domain features (including image frequency domain features and auxiliary frequency domain features), the preliminary denoised image and the auxiliary rendering features can be preliminarily aligned in the frequency domain. It can be understood that the center of the spectrum expresses low frequencies and the surrounding expresses high frequencies. High frequencies describe places where the pixel points in the image vary greatly, that is, the image texture detail characteristics. Aligning the high frequencies of the preliminary denoised image with the high frequencies in the auxiliary rendering features can enhance the texture details, thereby achieving the effect of denoising and restoring the real details.

[0110] Finally, an image generator is used to process the preliminary denoised image and intermediate features to generate a high-quality denoised image.

[0111] In summary, since the texture details in the preliminary noise image are partially blurred, and the illumination and normal information of the auxiliary rendering features can just contain this part of the blurred information, the attention unit is used to map the detail information of the auxiliary rendering features to the Fourier features of the preliminary denoising results to correct the blurred parts in the preliminary denoising results and obtain intermediate features with texture detail information. Providing this intermediate feature to the image generator can facilitate the generation of high-quality images.

[0112] In order to improve the rendering quality, the present disclosure also provides a method for training an image denoising model based on the same technical concept, wherein, as described above, the image denoising model includes a pre-trained first neural network module, an intermediate feature acquisition module and an image generator, such as Figure 4 As shown, the training method includes:

[0113] S401, inputting the sample image into the image denoising model to obtain a denoised image of the sample image; the sample image is obtained by ray tracing and rendering the sample three-dimensional model based on a target sampling number, and ray tracing and rendering the sample three-dimensional model based on a preset sampling number higher than the target sampling number to obtain a high-definition image of the sample image.

[0114] S402, determining a loss value based on the denoised image and the high-definition image.

[0115] When implementing, Figure 5 As shown, a discriminant model can be added after the image denoising model to optimize the parameters of the image denoising model based on the adversarial generation mechanism. The generated denoised image can be input into the discriminant model, and the discriminant model determines whether the input denoised image is a high-definition denoised image generated by denoising or a real rendered high-definition image obtained by the rendering engine based on a high sampling rate. The discriminant model can be a trained model, which is not limited in the embodiments of the present disclosure.

[0116] S403: Adjust model parameters of the image denoising model based on the loss value.

[0117] The first neural network module is used to perform preliminary noise reduction on the sample image to obtain a preliminary noise reduction sample;

[0118] The intermediate feature extraction module is used to process the preliminary denoising samples and the sample auxiliary rendering features of the sample image, so as to align the preliminary denoising samples and the sample auxiliary rendering features in the frequency domain to obtain the intermediate features of the sample image; the sample auxiliary rendering features at least include the normal features and illumination features required for rendering the sample image and used to optimize the preliminary denoising image;

[0119] The image generator is used to process the intermediate features of the sample image to obtain a denoised image.

[0120] In the disclosed embodiment, for noisy images, preliminary denoising can be achieved based on the first neural network module. Then, details are reconstructed based on the characteristics of the frequency domain, thereby improving the denoising effect and rendering quality. Moreover, supervised training of high-definition images rendered by multiple samplings can enable the model to restore images close to the rendering engine.

[0121] In a possible implementation, the difference between a denoised image and a high-definition image of the sample image may be determined to obtain a loss value.

[0122] In another possible implementation, in order to improve the model training effect, the loss value is determined based on the denoised image, such as Figure 6 As shown, it can also be implemented as:

[0123] S601, determining a denoising loss based on a difference between a denoised image of a sample image and a high-definition image.

[0124] S602, inputting the denoised image into a discriminator model to identify the image type of the denoised image; determining a discrimination loss based on the predicted image type and the true type label; and,

[0125] S603, determining the Fourier loss between the denoised image and the high-definition image.

[0126] The denoised image may be transformed into the frequency domain, and the high-definition image may be transformed into the frequency domain, and the difference between the two in the frequency domain may be determined, thereby obtaining the Fourier loss. In addition, the execution order of S601-S603 is not limited.

[0127] S604, performing weighted summation based on the denoising loss, the discrimination loss, and the Fourier loss to obtain a loss value.

[0128] Here, L1Loss is used to make a preliminary adjustment to the model parameters, and then the following loss function is used to fine-tune the model parameters.

[0129] Specifically, the loss function can be expressed as expression (1):

[0130] L=a*L1loss+b*GAN loss+c*fourier loss (1)

[0131] Among them, L represents the loss value; L1loss represents the denoising loss; GAN loss represents the discrimination loss; fourier loss represents the Fourier loss; a, b, and c are all weights. During implementation, a can be required to be greater than b, and b can be greater than c.

[0132] In the disclosed embodiment, comprehensively measuring the loss in multiple dimensions is beneficial to adjusting the model parameters of different modules in the model, accelerating the convergence speed of the model, and improving the model training effect.

[0133] Figure 7-10 This is a comparison chart of the experimental results of the embodiments of the present disclosure. Figure 7-10 In the figure, N represents a noise image, GT represents a high-definition image obtained by high-frequency sampling and rendering of a three-dimensional model, which is used as a comparison image. M represents a rendering image provided by an embodiment of the present disclosure, and 0 represents a rendering image of other deep learning models.

[0134] Figure 7 It can be seen from the image content in the mirror that the disclosed embodiment can more accurately express the details of the light reflected by the mirror. However, in 0, there are multiple noises and the image details cannot be smoothly expressed.

[0135] Figure 8 In the high-light environment, the curtain folds are the parts with more details. By comparison, it can be seen that the solution provided by the embodiment of the present disclosure can display the curtain folds more smoothly.

[0136] Fig. 9 In the example, in a low-brightness environment, the curtain part shows the details of alternating highlights and dark tones. By comparison, it can be seen that the curtain image restored by the embodiment of the present disclosure has a smoother score, while the effect of other models is some messy noise.

[0137] Fig.10In the embodiment, there are some patterns on the cabinet surface. The embodiment of the present disclosure can reflect the texture of the patterns and materials, while the effect reflected by other models is more messy noise.

[0138] Based on the same technical concept, the embodiment of the present disclosure also provides a 3D model rendering device 1100, such as Fig.11 As shown, including:

[0139] The rendering module 1101 is used to perform ray tracing rendering on the three-dimensional model based on the target sampling number to obtain a noise image with noise; wherein the target sampling number is less than a preset threshold;

[0140] A denoising module 1102 is used to perform denoising on the noisy image using a first neural network module to obtain a preliminary denoised image with some noise removed;

[0141] An alignment module 1103 is used to align the preliminary denoised image and the auxiliary rendering features in the frequency domain to obtain intermediate features; the auxiliary rendering features at least include normal features and illumination features for optimizing the preliminary denoised image;

[0142] The generating module 1104 is used to process the intermediate features based on the image generator to generate a denoised image of the three-dimensional model.

[0143] In some embodiments, the alignment module comprises:

[0144] A first processing unit is used to transform the preliminary denoised image into the frequency domain to obtain the image frequency domain features; and

[0145] A second processing unit is used to transform the auxiliary rendering features into the frequency domain to obtain auxiliary frequency domain features;

[0146] A fusion unit is used to process the image frequency domain features and auxiliary frequency domain features based on the attention mechanism to obtain fusion features;

[0147] The third processing unit is used to transform the fused features into the spatial domain to obtain intermediate features.

[0148] In some embodiments, the fusion unit comprises:

[0149] A first extraction unit is used to extract the image frequency domain feature to obtain a first feature; and

[0150] A second extraction unit is used to extract the auxiliary frequency domain feature to obtain a second feature;

[0151] A construction unit, configured to construct an input feature of an attention module based on the first feature and the second feature;

[0152] The fourth processing unit is used to process the input features using the attention module to obtain fused features.

[0153] In some embodiments, the input features of the attention module include query features, key features, and value features. A construction unit is used to:

[0154] The first feature extracted from the image frequency domain feature is used as the query feature of the attention module; and,

[0155] The second features extracted from the auxiliary frequency domain features are used as the key features and value features of the attention module.

[0156] In some embodiments, the third processing unit is configured to:

[0157] Transforming the first feature into the spatial domain by inverse Fourier transform to obtain a first inverse transform feature; and,

[0158] Transforming the fused features into the spatial domain by inverse Fourier transform to obtain a second inverse transform feature;

[0159] The first inverse transformation feature and the second inverse transformation feature are fused to obtain an intermediate feature.

[0160] In some embodiments, a generating module is used to:

[0161] The preliminary denoised image and the intermediate features are input into the image generator to obtain the denoised image of the 3D model.

[0162] Based on the same technical concept, the embodiment of the present disclosure also provides a training device 1200 for an image denoising model, wherein the image denoising model includes a pre-trained first neural network module, an intermediate feature acquisition module and an image generator, such as Fig.12 As shown, including:

[0163] An input module 1201 is used to input a sample image into an image denoising model to obtain a denoised image of the sample image; the sample image is obtained by performing ray tracing rendering on the sample three-dimensional model based on a target sampling number, and performing ray tracing rendering on the sample three-dimensional model based on a preset sampling number higher than the target sampling number to obtain a high-definition image of the sample image;

[0164] A determination module 1202, configured to determine a loss value based on the denoised image and the high-definition image;

[0165] An adjustment module 1203, used to adjust the model parameters of the image denoising model based on the loss value;

[0166] The first neural network module is used to perform preliminary noise reduction on the sample image to obtain a preliminary noise reduction sample;

[0167] The intermediate feature extraction module is used to process the preliminary denoising samples and the sample auxiliary rendering features of the sample image, so as to align the preliminary denoising samples and the sample auxiliary rendering features in the frequency domain to obtain the intermediate features of the sample image; the sample auxiliary rendering features at least include the normal features and illumination features required for rendering the sample image and used to optimize the preliminary denoising image;

[0168] The image generator is used to process the intermediate features of the sample image to obtain a denoised image.

[0169] In some embodiments, the determining module is configured to:

[0170] determining a denoising loss based on a difference between a denoised image and a high definition image of the sample image; and,

[0171] Input the denoised image into the discriminator model to identify the image type of the denoised image;

[0172] Determine the discriminative loss based on the predicted image type and the true type label; and,

[0173] Determine the Fourier loss between the denoised image and the high-definition image;

[0174] The loss value is obtained by weighted summation based on denoising loss, discriminant loss and Fourier loss.

[0175] For the description of specific functions and examples of each module and unit of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0176] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0177] Fig.13 FIG. 1 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Fig.13 As shown, the electronic device includes: a memory 1310 and a processor 1320, and the memory 1310 stores a computer program that can be run on the processor 1320. The number of memories 1310 and processors 1320 can be one or more. The memory 1310 can store one or more computer programs. When the one or more computer programs are executed by the electronic device, the electronic device executes the method provided by the above method embodiment. The electronic device may also include: a communication interface 1330, which is used to communicate with external devices and perform data exchange transmission.

[0178] If the memory 1310, the processor 1320 and the communication interface 1330 are implemented independently, the memory 1310, the processor 1320 and the communication interface 1330 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0179] Optionally, in a specific implementation, if the memory 1310, the processor 1320 and the communication interface 1330 are integrated on a chip, the memory 1310, the processor 1320 and the communication interface 1330 can communicate with each other through an internal interface.

[0180] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0181] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0182] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (for example: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (for example: infrared, Bluetooth, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc. It is worth noting that the computer-readable storage medium mentioned in the present disclosure may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0183] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0184] In the description of the embodiments of the present disclosure, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0185] In the description of the embodiments of the present disclosure, unless otherwise specified, " / " means or, for example, A / B can mean A or B. "And / or" in this article is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0186] In the description of the embodiments of the present disclosure, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0187] The above description is only an exemplary embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A three-dimensional model rendering method, comprising: Based on the target sampling number, ray tracing rendering is performed on the three-dimensional model to obtain a noise image with noise; wherein the target sampling number is less than a preset threshold; Using a first neural network module to perform denoising on the noisy image to obtain a preliminary denoised image with some noise removed; Aligning the preliminary denoised image and the auxiliary rendering features in the frequency domain to obtain intermediate features; the auxiliary rendering features at least include normal features and illumination features for optimizing the preliminary denoised image; The intermediate features are processed based on an image generator to generate a denoised image of the three-dimensional model.

2. The method according to claim 1, wherein: The step of aligning the preliminary denoised image and the auxiliary rendering features in the frequency domain to obtain intermediate features includes: Transforming the preliminary denoised image into the frequency domain to obtain image frequency domain features; and, Transforming the auxiliary rendering features into the frequency domain to obtain auxiliary frequency domain features; Processing the image frequency domain features and the auxiliary frequency domain features based on an attention mechanism to obtain fused features; The fused features are transformed into a spatial domain to obtain the intermediate features.

3. The method according to claim 2, wherein: The processing of the image frequency domain features and the auxiliary frequency domain features based on the attention mechanism to obtain fusion features includes: Extracting the image frequency domain features to obtain a first feature; and Performing feature extraction on the auxiliary frequency domain feature to obtain a second feature; Constructing an input feature of an attention module based on the first feature and the second feature; The attention module is used to process the input features to obtain the fused features.

4. The method according to claim 3, wherein: The input features of the attention module include query features, key features and value features; The constructing the input feature of the attention module based on the first feature and the second feature includes: using the first feature extracted from the image frequency domain feature as the query feature of the attention module; and, The second feature extracted from the auxiliary frequency domain feature is used as the key feature and the value feature of the attention module.

5. The method according to claim 3, wherein: The step of transforming the fused features into a spatial domain to obtain the intermediate features comprises: Transforming the first feature into the spatial domain by inverse Fourier transform to obtain a first inverse transformed feature; and, Transforming the fused features into the spatial domain by inverse Fourier transform to obtain a second inverse transform feature; The first inverse transformation feature and the second inverse transformation feature are fused to obtain the intermediate feature.

6. The method according to any one of claims 1 to 5, wherein: The processing of the intermediate features by an image generator to generate a denoised image of the three-dimensional model comprises: The preliminary denoised image and the intermediate features are input into the image generator to obtain a denoised image of the three-dimensional model.

7. A training method for an image denoising model, wherein: The image denoising model includes a pre-trained first neural network module, an intermediate feature acquisition module and an image generator, and the method includes: Inputting a sample image into the image denoising model to obtain a denoised image of the sample image; the sample image is obtained by performing ray tracing rendering on a sample three-dimensional model based on a target sampling number, and performing ray tracing rendering on the sample three-dimensional model based on a preset sampling number higher than the target sampling number to obtain a high-definition image of the sample image; Determining a loss value based on the denoised image and the high-definition image; Adjusting a model parameter of the image denoising model based on the loss value; Wherein, the first neural network module is used to perform preliminary denoising on the sample image to obtain a preliminary denoised sample; The intermediate feature extraction module is used to process the preliminary denoised samples and the sample auxiliary rendering features of the sample image to align the preliminary denoised samples and the sample auxiliary rendering features in the frequency domain to obtain the intermediate features of the sample image; the sample auxiliary rendering features at least include normal features and illumination features required for rendering the sample image and used to optimize the preliminary denoised image; The image generator is used to process the intermediate features of the sample image to obtain the denoised image.

8. The method according to claim 7, wherein: The determining of the loss value based on the denoised image comprises: determining a denoising loss based on a difference between the denoised image and the high definition image of the sample image; and, Inputting the denoised image into a discriminator model to identify the image type of the denoised image; Determining the discrimination loss based on the predicted image type and the true type label; and, determining a Fourier loss between the denoised image and the high definition image; The loss value is obtained by performing weighted summation based on the denoising loss, the discrimination loss and the Fourier loss.

9. A three-dimensional model rendering device, comprising: A rendering module, used for performing ray tracing rendering on the three-dimensional model based on a target sampling number to obtain a noise image with noise; wherein the target sampling number is less than a preset threshold; A denoising module, configured to perform denoising processing on the noisy image using a first neural network module to obtain a preliminary denoised image with some noise removed; An alignment module, used for aligning the preliminary denoised image and the auxiliary rendering features in the frequency domain to obtain intermediate features; the auxiliary rendering features at least include normal features and illumination features for optimizing the preliminary denoised image; A generating module is used to process the intermediate features based on an image generator to generate a denoised image of the three-dimensional model.

10. A training device for an image denoising model, wherein: The image denoising model includes a pre-trained first neural network module, an intermediate feature acquisition module and an image generator, and the device includes: An input module, used for inputting a sample image into the image denoising model to obtain a denoised image of the sample image; the sample image is obtained by performing ray tracing rendering on a sample three-dimensional model based on a target sampling number, and performing ray tracing rendering on the sample three-dimensional model based on a preset sampling number higher than the target sampling number to obtain a high-definition image of the sample image; A determination module, configured to determine a loss value based on the denoised image and the high-definition image; An adjustment module, used for adjusting the model parameters of the image denoising model based on the loss value; Wherein, the first neural network module is used to perform preliminary denoising on the sample image to obtain a preliminary denoised sample; The intermediate feature extraction module is used to process the preliminary denoised samples and the sample auxiliary rendering features of the sample image to align the preliminary denoised samples and the sample auxiliary rendering features in the frequency domain to obtain the intermediate features of the sample image; the sample auxiliary rendering features at least include normal features and illumination features required for rendering the sample image and used to optimize the preliminary denoised image; The image generator is used to process the intermediate features of the sample image to obtain the denoised image.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

Images