Detection of detail loss in denoised images

Abstract

To provide a method, system, and program which form a dataset configured for learning a convolutional neural network (CNN) architecture including an image feature extractor.SOLUTION: The method comprises: providing pairs of images each of which includes a reference image and a denoised image; for each pair of images, providing the pair of images to a pre-trained CNN architecture similar to a CNN architecture for which a formed dataset is configured; and computing an error map representing a difference between a first normalized feature of the denoised image and a second normalized feature of the reference image. The first and second normalized features are output of the same layer of the pre-trained CNN architecture. The method also comprises adding each denoised image and the error map to the dataset.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to the field of computer programs and systems, and more specifically, to a method, system, and program for detecting detail loss of a noise-reduced image.

Background Art

[0002] The market offers numerous systems and programs for the design, engineering, and manufacturing of objects. CAD stands for Computer-Aided Design, and refers to software solutions for designing objects, for example. CAE stands for Computer-Aided Engineering, and refers to software solutions for simulating the physical behavior of future products, for example. CAM stands for Computer-Aided Manufacturing, and refers to software solutions for defining manufacturing processes and operations, for example. In such computer-aided design systems, graphical user interfaces play a crucial role in terms of technical efficiency. These technologies can be integrated into Product Lifecycle Management (PLM) systems. PLM is a business strategy that helps companies share product data, apply common processing, and leverage enterprise knowledge to help develop products from concept to lifecycle, across the concept of an extended enterprise. Dassault Systèmes' PLM solutions (under the trademarks CATIA, ENOVIA, and DELMIA) provide an Engineering Hub for organizing product engineering knowledge, a Manufacturing Hub for managing manufacturing engineering knowledge, and an Enterprise Hub that enables enterprise integration and connectivity to the Engineering and Manufacturing Hubs. Together, these systems provide an open object model that links products, processes, and resources to enable dynamic, knowledge-based product creation and decision support, facilitating product definition, manufacturing preparation, production, and service optimization.

[0003] Against this backdrop, the field of computer graphics offers increasingly useful technologies. In this field, ray-traced images are used to generate images with extremely high visual realism, and the importance of techniques for rendering ray-traced images is growing.

[0004] Monte Carlo rendering is a known technique used to render high-quality ray-traced images. It relies on numerical integration using random samples to compute higher-order integrals of the underlying rendering equations that represent light reflection. Because sampling is random, Monte Carlo rendering has a probabilistic nature, and therefore, the resulting image is noisy. Increasing the number of samples increases the accuracy of Monte Carlo rendering, and iteratively, it converges to the correct result. "Pharr, M., Jakob, W., & Humphreys, G. (2018), Physically Based Rendering: From Theory To Implementation" provides a detailed examination of the key mathematical concepts underlying Monte Carlo rendering. Since Monte Carlo rendering is a computationally intensive process requiring significant time and resources, minimizing the required number of samples while reducing noise-induced variance is a crucial task.

[0005] Some algorithms utilize sampling strategies that sample as close as possible to the underlying distribution in order to reduce variance. Several sampling strategies exist that can achieve this, such as uniform sampling, stratified sampling, and importance sampling. These sampling methods are known as fixed-scheme sampling strategies. Adaptive sampling is a sampling strategy that defines the sampling density based on the aforementioned samples, rather than being a fixed scheme. Essentially, adaptive sampling makes it possible to distribute samples in the image space where samples are needed to minimize output variance. Many adaptive sampling methods require a reconstruction process. The background techniques for adaptive sampling are investigated in detail in "Zwicker, M., Jarosz, W., Lehtinen, J., Moon, B., Ramamoorthi, R., Rousselle, F., Sen, P., Soler, C., Yoon, S.-E. (2015), Recent Advances in Adaptive Sampling and Reconstruction for Monte Carlo Rendering".

[0006] Sampling algorithms enable high-quality image rendering by utilizing appropriate sampling strategies, but they have the drawback that the number of samples required to achieve low-noise images can be computationally enormous. Monte Carlo rendering denoising refers to techniques aimed at reducing the variance of sampled ray-traced images. Monte Carlo rendering denoising can be described as both a priori and ex post facto techniques. "Zwicker, M., Jarosz, W., Lehtinen, J., Moon, B., Ramamoorthi, R., Rousselle, F., Sen, P., Soler, C., Yoon, S.-E. (2015), Recent Advances in Adaptive Sampling and Reconstruction for Monte Carlo Rendering" investigates ex post facto techniques for Monte Carlo rendering denoising in detail. Ex post facto methods act on the renderer's output with little to no additional information (such as normal buffers or albedo buffers) and no knowledge of internal calculations. The key is to use a family of reconstruction filters and create an error estimate of the filter output. Many current systems are based on the concept of nonlinear image space filters, as described in "Rushmeier, HE, & Ward, GJ (1994), Energy preserving non-linear filters," and the use of auxiliary features such as normal buffers and albedo buffers, as introduced in "McCool, MD (1999), Anisotropic Diffusion for Monte Carlo Noise Reduction."

[0007] Neural networks, particularly convolutional neural networks, are being implemented in computer vision systems, including denoising systems. Examples of the use of convolutional neural networks in computer vision systems such as image classification, and in generative image systems such as image synthesis, restoration, and denoising are presented in "Li, Z., Yang, W., Peng, S., & Liu, F. (2020), A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects," "Liu, G., Reda, FA, Shih, KJ, Wang, T.-C., Tao, A., & Catanzaro, B. (2018), Image Inpainting for Irregular Holes Using Partial Convolutions," and "Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008), Extracting and composing robust features with denoising autoencoders." In "Gatys, LA, Ecker, AS, & Bethge, M. (2015), A Neural Algorithm of Artistic Style," a method is described for transferring style from one image to another using the weights of a pre-trained VGG19 convolutional neural network and an image classification network as the loss function.In "Zhang, R., Isola, P., Efros, AA, Shechtman, E., & Wang, O. (2018), The Unreasonable Effectiveness of Deep Features as a Perceptual Metric," it is described how to evaluate perceptual similarity between two images using the weights of a pre-trained VGG19 convolutional neural network, which is achieved by training a network that acts on a filter map generated by a pre-trained classification network to measure perceptual similarity. In "Bako, S., Vogels, T., Mcwilliams, B., Meyer, M., Novak, J., Harvill, A., Sen, P., DeRose, T., Rousselle, F. (2017), Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings," a denoiser is described that uses a neural network to estimate a local weighting kernel used to compute each denoising pixel from adjacent pixels. In "Chaitanya, CR, Kaplanyan, AS, Schied, C., Salvi, M., Lefohn, A., Nowrouzezahrai, D., & Aila, T. (2017), Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder," denoising that acts on image sequences and directly predicts denoising pixels from the input is described.Commercial denoiser solutions such as Nvidia OptiX Denoiser or Intel Open Image Denoise also exist, and these are partially based on the technology described in "Chaitanya, CR, Kaplanyan, AS, Schied, C., Salvi, M., Lefohn, A., Nowrouzezahrai, D., & Aila, T. (2017), Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder".

[0008] While denoising reduces dispersion caused by noise, in the case of a laser-like structure background in the final frame rendering, for example, the denoising process can cause a loss of fine detail and blur the image. "Kuznetsov, A., Kalantari, NK, & Ramamoorthi, R. (2018), Deep Adaptive Sampling for Low Sample Count Rendering" describes a method that combines adaptive sampling and denoising using two neural networks. One of the neural networks operates in front of the renderer, focusing on a low sample count and generating an adaptive sampling map from a single sample image. The other neural network denoises the adaptively sampled rendering. However, because the neural network used for sampling has a low sample count, it does not properly sample thin structures in noisy areas, resulting in a loss of detail after denoising. "Hasselgren, J., Munkberg, J., Salvi, M., Patney, A., & Lefohn, A. (2020), Neural Temporal Adaptive Sampling and Denoising" extends the system that acts on image sequences in "Kuznetsov, A., Kalantari, NK, & Ramamoorthi, R. (2018)" to the time domain, creating repeating loops by temporally stretching previous frames and inputting them into an adaptive sample map network, but this also suffers from the same problem of loss of detail after denoising.

[0009] Figures 6 and 7 show examples of noise reduction errors detected by prior art methods.

[0010] Figure 6 shows a ray-traced image of furniture with a leather texture. A large number of Monte Carlo samples were used to render this image, meaning the image is quite close to the reference image. For images rendered with a large number of Monte Carlo samples, denoising can degrade quality compared to an undenoised input. In fact, denoising can blur the leather structure. The top left image of Figure 6 shows the denoised image, and the upper left section shows the respective reference images. A visual inspection of the denoised and reference images clearly reveals the degradation in quality. While the degradation can be partially detected with standard metrics, in images with a large number of samples, it is not possible to systematically distinguish the areas degraded by the denoiser. The lower left section of Figure 6 shows a standard metric representing the mean absolute error between the denoised and reference images. However, in terms of standard metrics, the actual pixel error is very low. In fact, the lower left section of Figure 6 shows an error map representing the mean absolute error between the denoised and reference images. Darker areas in the error map correspond to areas with little or no error. Brighter areas correspond to areas with larger errors. The error map shown in the lower left of Figure 6 does not identify the loss of detail in the leather texture. Another metric is the measurement of the structural similarity index (SSIM). The lower right of Figure 6 shows an error map showing the SSIM between the denoised image and the reference image. The SSIM shows a slight improvement in the display of areas degraded by denoising, and assigns a slightly higher error to these areas. However, the SSIM is still not sufficient as a detection metric. In fact, the SSIM error map shows a blurry error, making it difficult to clearly distinguish areas where small structural details are distorted.

[0011] Figure 7 shows a similar example using a rendering of a plastic bottle. Visual inspection reveals that the denoised image of the upper left portion shows blurred lettering on the bottle compared to the reference image. However, neither the mean absolute error nor the SSIM captures the blurred area of ​​the lettering.

[0012] Against this backdrop, there is still a need for improved methods to detect the loss of detail in denoised images. [Overview of the project]

[0013] Therefore, a computer implementation method is provided for forming a dataset configured to train a convolutional neural network (CNN) architecture. The CNN architecture includes an image feature extractor. This method is - A step of providing a plurality of image pairs, each pair having a reference image and a denoising image thereof, -For each of the above image pairs, - A step of providing the image pair to a pre-trained CNN architecture similar to the CNN architecture to which the resulting dataset is constructed. The step of calculating the difference between a first normalized feature of a denoised image and a second normalized feature of a reference image, wherein the first and second normalized features are the outputs of the same layer of a pre-trained CNN architecture. • A step of calculating an error map representing the calculated difference, and • Steps to add each denoised image and error map to the dataset. Includes.

[0014] This method may include one or more of the following: The calculation of the error map includes the following: - A step of downsampling the resolution of the calculated difference, and - A step of obtaining a coarse error map by calculating an error map at the resolution of the downsampled difference, wherein each pixel of the error map has a color that conforms to a color scale that penalizes the calculated difference. The pre-trained CNN architecture is a Visual Geometric Group (VGG) architecture. The Visual Geometric Group (VGG) architecture is the VGG-19 architecture, where the first normalization feature of the denoised image and the second normalization feature of the reference image are extracted from layers 7 and / or 9 and / or 12 of the VGG-19 architecture. The additional steps include: - Add the noise image for each denoised image, error map, and reference image to the dataset, where the noise image is the original image from which the denoised image was obtained. Each denoised image is the result of inputting an image with fewer samples than the reference image, after denoising it using a denoiser, into the denoiser.

[0015] Furthermore, a data structure is provided that represents a dataset that can be formed according to this method.

[0016] Furthermore, a computer implementation method for using the dataset represented by the data structure is provided. This method includes training a convolutional neural network (CNN) architecture on the dataset.

[0017] This computer implementation method may include one or more of the following: Learning is supervised learning; The CNN architecture to be trained is identical to the pre-trained CNN architecture and has the same number of layers. All layers of the CNN architecture to be trained that are after the layers of the pre-trained CNN architecture that generated the first and second normalization features are removed.

[0018] A computer program is further provided that includes instructions for performing any of the methods herein.

[0019] A computer-readable data storage medium recording the computer program and / or data structure is further provided.

[0020] Furthermore, a data set that can be formed according to the method herein is provided.

[0021] A computer-implemented method of machine learning is also provided for training a convolutional neural network (CNN) architecture to estimate degradation generated by a denoiser on a ray-traced image. The method includes: - Providing a data set, wherein the data set: · Providing a plurality of image pairs, each pair having a reference image and a respective denoised image, the step; · For each image pair: ·· Providing the image pair to a pre-trained CNN architecture similar to the CNN architecture to be trained to estimate degradation generated by a denoiser on a ray-traced image; ·· Calculating a difference between a first normalized feature of the denoised image and a second normalized feature of the reference image, wherein the first and second normalized features are outputs of the same layer of the pre-trained CNN architecture, the calculating step; ·· Calculating an error map representing the calculated difference, and; ·· Adding each denoised image, error map, and respective noisy image of the reference image to the data set, wherein the noisy image is the image from which the denoised image was obtained, the adding step; Providing the data set obtained by. - A step of training a CNN architecture based on a provided dataset, wherein the training step includes receiving, as inputs, an image generated by a denoiser and its corresponding noisy image in the dataset, and outputting an error map.

[0022] This method may include one or more of the following: · The CNN architecture to be trained further receives, as inputs, an albedo map and a normal buffer for each denoised image in the dataset. · The training is supervised learning. · The CNN architecture to be trained has the same number of layers as and is identical to a pre-trained CNN architecture, and all layers of the CNN architecture to be trained that are after the layers of the pre-trained CNN architecture that generated the first and second normalized features are removed. · Calculating the error map includes the following: - A step of downsampling the calculated difference resolution, and - A step of obtaining a rough error map by calculating an error map at the downsampled difference resolution, where each pixel of the error map has a color along a color scale that penalizes the calculated difference. · The pre-trained CNN architecture and the CNN architecture to be trained are Visual Geometric Group (VGG) architectures. · The Visual Geometric Group (VGG) architecture is the VGG-19 architecture, and the first normalized feature of the denoised image and the second normalized feature of the reference image are extracted from the 7th and / or 9th and / or 12th layers of the VGG-19 architecture.

[0023] Furthermore, a trainable CNN architecture is provided, which can be trained by machine learning methods to train a convolutional neural network (CNN) architecture to estimate the degradation generated by a denoiser on ray-traced images.

[0024] Furthermore, a method is provided for computer implementation using a CNN architecture. This method includes: - A step of providing first and second Monte Carlo ray tracing images that are rendered from the same shot but are different from each other, - A step of providing a first denoised image of the first image, - A step of applying the first image and the first denoising image to the CNN architecture and obtaining an error map, - A step of calculating a blend map from an acquired error map, wherein the blend map represents the signal-to-noise decibel level of each region of the error map, - A step of estimating the variance from the first and second images, - A step of calculating a sample map showing the number of Monte Carlo samples required to reach a certain signal-to-noise decibel level from the estimated variance between the first and second images. - A step of rendering a third Monte Carlo ray tracing image of the shot using the number of Monte Carlo samples identified from the sample map, - The steps of inputting the rendered third Monte Carlo rendered ray tracing image into a denoiser and obtaining a corresponding denoised image, and - The method comprising the step of blending the third Monte Carlo rendered ray tracing image with the acquired corresponding denoising image, wherein the blend is weighted for each region of the blend map having a signal-to-noise decibel level higher than a threshold.

[0025] This method of use may include one or more of the following: The step of calculating a blend map from the acquired error map may include the following: - A step of applying a sigmoid function to the error map to normalize the error map, - A step to calculate the decibel value for each region of the error map from the target root mean square error between the normalized error map and the target error map. The first Monte Carlo ray tracing image provided is rendered with a first number of Monte Carlo samples, and the second Monte Carlo ray tracing image rendered is rendered with a second number of Monte Carlo samples, where the first number of Monte Carlo samples is greater than or equal to the second number of Monte Carlo samples. The second Monte Carlo sample is half the number of the first Monte Carlo sample. The first Monte Carlo ray tracing image provided was rendered according to a first random seed, and the second Monte Carlo ray tracing image provided was rendered according to a second random seed, where the first number of Monte Carlo samples is equal to the second number of Monte Carlo samples.

[0026] Further computer programs are provided that include instructions for performing any of these methods.

[0027] A computer-readable data storage medium on which the computer program and / or the neural network is recorded is further provided. [Brief explanation of the drawing]

[0028] Next, the present invention will be described with reference to the attached drawings, based on examples that are not limiting. [Figure 1] A flowchart of this method is shown below. [Figure 2] A flowchart of this method is shown below. [Figure 3] A flowchart of this method is shown below. [Figure 4] An example of the system's graphical user interface is shown. [Figure 5] An example of the system is shown. [Figure 6] This demonstrates the detection of errors in noise reduction using prior art methods. [Figure 7] This demonstrates the detection of errors in noise reduction using prior art methods. [Figure 8] Here is an example of a CNN architecture. [Figure 9] An example of a pre-trained CNN architecture is shown. [Figure 10] An example of a coarse error map is shown. [Figure 11] An example of an error map calculated according to this method is shown. [Figure 12] An example of an error map calculated according to this method is shown. [Figure 13] An example of an error map calculated according to this method is shown. [Figure 14] An example of a CNN architecture used for training is shown. [Figure 15] This example shows a CNN architecture that learns using supervised learning. [Figure 16] This example shows an error map estimated by a CNN architecture trained on a dataset that is formed. [Figure 17] This example shows an error map estimated by a CNN architecture trained on a dataset that is formed. [Figure 18] This example shows an error map estimated by a CNN architecture trained on a dataset that is formed. [Figure 19] A noise image with the decibel values ​​of each signal-to-noise pair is shown as an example, along with the corresponding reference image. [Figure 20] Examples of denoised images are shown according to different target signal-to-noise decibel values. [Figure 21] Examples of denoised images are shown according to different target signal-to-noise decibel values. [Modes for carrying out the invention]

[0029] Referring to Figure 1, a method is provided for forming a dataset that is configured to train a convolutional neural network (CNN) architecture that includes an image feature extractor.

[0030] As is well known in the field of machine learning, processing input by a neural network involves applying an operation to that input, where the operation is defined by data containing weight values. Therefore, training a neural network involves the step of determining weight values ​​based on a dataset configured for such training. Thus, a dataset contains data pieces, each forming a training sample. The training samples represent a variety of situations in which the neural network will be used after training. Any dataset referenced herein may contain more than 1,000, 10,000, 1,000,000, or 1,000,000 training samples.

[0031] Therefore, computer implementation methods for forming such datasets have been specifically proposed, and these methods are sometimes referred to as "dataset formation methods." Furthermore, computer implementation methods for performing such learning have also been proposed, and these methods will be explained in more detail later and are sometimes referred to as "learning methods."

[0032] The machine learning process may include a method for creating a dataset and / or a method for learning.

[0033] The machine learning process may include both a dataset formation method and a training method. This training method may, for example, be performed on, at least partially, the dataset formed by the dataset formation method, and may, for example, be performed after the dataset formation method. Such a machine learning process is particularly efficient and improves accuracy.

[0034] Alternatively, the machine learning process may include steps of a method for forming a dataset and performing any other computer implementation (other than the proposed training method) for training a neural network on that dataset. Alternatively, the machine learning process may include steps of performing the training method on a dataset provided by any other computer implementation (other than the proposed dataset forming method), such as another method for forming or searching for such a dataset.

[0035] A "dataset" refers to a data structure containing training samples, configured for machine learning, which is stored in the memory of a computer system and accessible by the processor. A method for creating a dataset may include the step of storing the dataset in non-volatile memory.

[0036] Referring again to Figure 1, in step S10, a pair of images is provided. Each pair has a reference image and a corresponding denoising image. The reference image is a highly detailed and low-noise image that corresponds to an image substantially close to the exact solution of the rendering equations that determine the image. "Denoiser" is an arbitrary algorithm representing a set of calculations, at least part of the algorithm that takes a ray-traced image as input, where the ray-traced image is rendered with any number of samples or any sampling strategy. The denoiser applies a set of signal processing calculations to the input ray-traced image. For example, the denoiser may apply a filter to the ray-traced image. In the field of signal processing, particularly in applications to denoising ray-traced images, this set of calculations follows post-processing techniques. The specific calculations that the denoiser performs on the ray-traced image are not the subject of this invention, and therefore details of its application are omitted. Standard references for post-processing and other denoising methods include "Zwicker, M., Jarosz, W., Lehtinen, J., Moon, B., Ramamoorthi, R., Rousselle, F., Sen, P., Soler, C., Yoon, S.-E, (2015), Recent Advances in Adaptive Sampling and Reconstruction for Monte Carlo Rendering". Each denoised image corresponds to the output of the denoiser and the result of applying the denoiser calculation to an input image sampled from the same rendering equation as the reference image.

[0037] The method further includes step S20, for each image pair, including a reference image and its respective denoised image, providing the image pair to a pre-trained CNN architecture. Convolutional neural network (CNN) architectures are widely known in the art and have a wide range of applications. A standard reference for convolutional neural networks is "Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998), Gradient-based learning applied to document recognition." As is well known in the field of neural networks, a CNN architecture includes layers of image feature extractors and / or fully connected image classifiers, each of which includes multiple layers, and features may be output from one or more layers of the image feature extractors. A pre-trained CNN architecture is analogous to the CNN architecture to which the resulting dataset is constructed. "Alike" means that the pre-trained CNN architecture includes at least one common layer with an image feature extractor of a CNN architecture that learns on the resulting dataset. A common layer refers to a layer that provides the same output for the same input. The common layer is the layer used in calculation step S30. The pre-trained CNN architecture may be trained on any image dataset known in the art, and further details are omitted. As an example, the pre-trained CNN architecture may be pre-trained on the dataset ImageNet.

[0038] The method further includes step S30, which calculates the difference between a first normalized feature of the denoised image and a second normalized feature of the reference image. The first and second normalized features are output from the same layers of a pre-trained CNN architecture. "Normalization" means that the output values ​​of the layers corresponding to the features are adjusted (e.g., rescaled) to the same numerical magnitude. In other words, normalization can be performed by adding or subtracting features before calculating the difference, thereby giving the same weight to the numerical values ​​of the features extracted from the denoised image and the reference image. Thus, the denoised image and the reference image are input to a pre-trained CNN architecture, and the features are extracted from each layer of the image feature extractor of the pre-trained CNN architecture. In this way, the difference is not the direct difference between the denoised image and the reference image, but rather is calculated from the features output by one or more layers of the image feature extractor.

[0039] The method further includes step S40 of calculating an error map representing the calculated difference. The error map is a collection of values, for example, two-dimensional values, where each element is a numerical value corresponding to the difference between a first normalized feature of the denoised image and a second normalized feature of the reference image. Traditionally, the error map may establish the calculated difference as the difference between the first normalized feature and the second normalized feature of the reference image. Thus, each value in the error map represents the distance that the first normalized feature of the denoised image does not respect compared to the second normalized feature of the reference image. Therefore, the error map can be interpreted as a measure of how much detail the denoiser retains, depending on each feature of the reference image.

[0040] The method further includes step S50, which adds each denoised image and error map to the dataset. Thus, the method adds each denoised image and error map to a data structure containing the dataset. For each provided image pair, training examples containing each denoised image and error map are successively added to the dataset. The method may form training examples from at least each denoised image and error map. For example, the method may further add annotations or additional data to the training examples. Thus, the dataset is formed of pairs of denoised images annotated with corresponding error maps, where each pair indicates how well detail is preserved in the denoised image and how well the denoiser preserves detail.

[0041] Therefore, this method utilizes the capabilities of a convolutional neural network to form a dataset that identifies degraded areas generated by the denoiser. This is made possible thanks to error maps that define metrics based on normalization features output from a pre-trained CNN architecture. The normalization features allow for particularly accurate identification of small areas degraded by the denoiser. For example, the normalization features can detect small structures blurred by the denoiser, such as the texture of leather. The normalization features can also particularly accurately detect ray-traced image objects that are furthest from the camera viewpoint in the ray-traced image scene and are blurred by the denoiser.

[0042] This method is implemented on a computer. That is, the steps (or substantially all steps) of the method are performed by at least one computer or any similar system. Thus, the steps of this method are performed by a computer, possibly fully automatically or semi-automatically. For example, the initiation of at least some steps of this method may be performed through user-computer interaction. The required level of user-computer interaction may depend on the expected level of automation and be balanced with the need to implement the user's wishes. For example, this level may be user-defined and / or predefined.

[0043] A typical example of a computer implementing this method is running it on a system adapted for this purpose. The system may include a processor coupled with memory and a graphical user interface (GUI), where memory stores a computer program containing instructions for performing this method. Memory may also store datasets. Memory is any hardware adapted for such storage and may include several physically distinct components (e.g., one for the program and possibly one for the datasets).

[0044] Ray-traced images may include scenes containing one or more modeled objects that may have been designed in a CAD system, or that may result from a design created in a CAD system. The term CAD system refers to a system, such as CATIA, that is adapted to design modeled objects based on their graphical representation. In this case, the data defining the modeled objects includes data that enables the representation of the modeled objects. A CAD system may provide a representation of a CAD modeled object, for example, using edges or lines, and in certain cases faces or surfaces. Lines, edges, or surfaces may be represented in various ways, with non-uniform rational B-splines (NURBS) being an example. Specifically, a CAD file may contain specifications from which geometry may be generated, enabling the generation of the representation. The specifications for a modeled object may be stored in a single CAD file or multiple CAD files. The typical size of a file representing a modeled object in a CAD system is in the range of 1 megabyte per part. A modeled object can also typically be an assembly of thousands of parts.

[0045] In CAD, modeled objects can typically be 3D modeled objects, which represent products such as parts or assemblies of parts, or in some cases, product assemblies. A "3D modeled object" means an object modeled with data that enables 3D representation. 3D representation allows parts to be viewed from all angles. For example, a 3D modeled object, when represented in 3D, may be processed and rotated based on any of its axes, or any axis on the displayed screen. This excludes 2D icons, in particular, that are not modeled in 3D. 3D representation makes design easier (i.e., it statistically speeds up the process for designers to perform tasks). Since product design is part of the manufacturing process, this speeds up the manufacturing process in industry.

[0046] The 3D modeled object may represent the geometry of a product after its virtual design has been completed, for example, by a CAD software solution or CAD system. The product may be a (e.g., mechanical) part or assembly of parts (an assembly of parts may be considered a part itself from the perspective of this method, or parts and assemblies of parts are equivalent, as this method can be applied independently to each part of an assembly), or more generally, an assembly of any rigid body (e.g., a movable mechanism). CAD software solutions enable the design of products in a wide range of industries without limitation, including aerospace, architecture, construction, consumer goods, high-tech equipment, industrial equipment, transportation, marine, and / or offshore oil / gas production or transportation. The 3D modeled objects designed by this method may be industrial products that are any mechanical parts, such as parts for land vehicles (e.g., automobile and light truck equipment, racing cars, motorcycles, truck and motor equipment, trucks and buses, trains, etc.), parts for aircraft vehicles (e.g., airframe equipment, aerospace equipment, propulsion equipment, defense products, aircraft equipment, space equipment, etc.), parts for marine vehicles (e.g., naval equipment, commercial ships, offshore equipment, yachts and workboats, marine equipment, etc.), general mechanical parts (e.g., industrial manufacturing machinery, heavy machinery or equipment, installation equipment, industrial equipment products, metalwork products, tire manufacturing products, etc.), electrical machinery or electronic components (e.g., home appliances, security and / or control and / or measurement products, computing and communication equipment, semiconductors, medical devices and equipment, etc.), consumer goods (e.g., furniture, home and garden products, leisure goods, fashion products, products of durable goods retailers, products of textile retailers, etc.), and packaging (e.g., food and beverages and tobacco, beauty and personal care, household goods packaging, etc.).

[0047] Figure 2 shows an example of a GUI for a system where the system is a CAD system.

[0048] GUI2100 may be a typical CAD-like interface having standard menu bars 2110, 2120 and bottom and side toolbars 2140, 2150. Such menus and toolbars include a set of user-selectable icons, each associated with one or more actions or functions known in the art. Some of these icons are associated with software tools suitable for editing and working with 3D modeled objects 2000 displayed in GUI2100. The software tools may be grouped into workbenches. Each workbench has a subset of software tools. In particular, one of the workbenches is an editing workbench suitable for editing the geometric features of the modeled product 2000. During operation, the designer may, for example, pre-select a portion of the object 2000, select the appropriate icon, and then begin operations (e.g., changing dimensions, color, etc.), or edit geometric constraints. For example, a common CAD operation is modeling the extrusion and folding of 3D modeled objects displayed on the screen. The GUI may, for example, display data 2500 related to the displayed product 2000. As an example in the diagram, the data 2500 and its 3D representation 2000, displayed as a "feature tree," relate to a brake assembly including a brake caliper and disc. The GUI may further display various types of graphic tools 2130, 2070, 2080 for, for example, facilitating 3D orientation of objects, initiating simulations of the operation of the product being edited, or rendering various attributes of the displayed product 2000. The cursor 2060 may be controlled by a haptic device to allow the user to interact with the graphic tools.

[0049] Figure 3 shows an example of the system, which is a client computer system, such as a user's workstation.

[0050] The client computer in this embodiment includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000, and random access memory (RAM) 1070 also connected to the bus. The client computer further includes a graphics processing unit (GPU) 1110 associated with video random access memory 1100, also connected to the bus. The video RAM 1100 is also known in the art as a frame buffer. A mass storage device controller 1020 manages access to mass storage devices such as a hard drive 1030. Mass storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, such as semiconductor memory devices like EPROMs, EEPROMs, and flash memory devices, magnetic disks like internal hard disks and removable disks, magneto-optical disks, and CD-ROM disks 1040. Any of the above may be complemented by or incorporated into a specially designed application-specific integrated circuit (ASIC). A network adapter 1050 manages access to the network 1060. The client computer may also include haptic devices 1090, such as a cursor control device and a keyboard. A cursor control device is used in a client computer to allow the user to selectively position the cursor at any desired location on the display 1080. Furthermore, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes several signal generating devices for inputting control signals to the system. Typically, the cursor control device may be a mouse, and the mouse buttons are used to generate signals. Alternatively or additionally, the client computer system may include a pressure-sensitive pad and / or a pressure-sensitive screen.

[0051] The computer program may include instructions that can be executed by a computer, and the instructions include means for causing the system to perform the Method. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuits, or in computer hardware, firmware, software, or a combination thereof. The program may be implemented, for example, as a device such as a product tangibly embodied in a machine-readable storage device for execution by a programmable processor. The steps of the Method may be performed by a programmable processor that executes a program of instructions to perform the functions of the Method by acting on input data and producing an output. Thus, the processor may be programmable or coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to transmit data and instructions to them. The application program may be implemented in a high-level procedural programming language or an object-oriented programming language, or in assembly language or machine language, as necessary. In any case, the language may be a compiled or interpreted language. The program may be a full installation program or an update program. In any case, the application of the program on the system results in instructions for performing the Method.

[0052] Refer again to Figure 1, and we will further illustrate with a flowchart an example of a computer implementation method for forming a dataset.

[0053] The images of the image pair provided in step S10 may be ray tracing images. The ray tracing images may include a scene containing one or more objects. The ray tracing images may include different objects or phenomena. For example, how and from where the scene is viewed is determined by the camera model. The ray tracing images are rendered using the Monte Carlo method. As is widely known in the art, Monte Carlo rendering is based on numerical integration using random samples of the corresponding rendering equation. For example, the rendering equation describes the amount of light reaching the camera (in the case of the camera model included in the embodiment) from a certain point on an object in the scene as the sum of the light emitted from the object (if it is a light source itself) and the amount of light reflected by the object. The random sample may be set from a predetermined number, for example, a positive integer n. The Monte Carlo renderer, for example, executes n sampling steps as an iterative process, and a sample is obtained at any step s where 0 < s < n. As is known in the field of Monte Carlo rendering, as the number of samples increases, Monte Carlo rendering becomes more accurate, and as the number of steps increases, Monte Carlo rendering converges to an ideal image corresponding to the exact solution of the corresponding rendering equation. That is, the Monte Carlo samples converge to the distribution underlying the corresponding rendering equation. For example, if s and s’ are two numerical values such that 0 < s < s’ < n, the ray tracing image obtained with s Monte Carlo samples has a lower level of detail (i.e., a lower degree of convergence) than the ray tracing image obtained with s’ Monte Carlo samples. The specific method of obtaining samples is not the subject of the present invention and is merely described for illustrative purposes. For example, in the ray tracing image obtained with s Monte Carlo samples, there may be more noise in the area corresponding to an object far from the camera, or the area of an image with fine structures such as a leather texture or text may be distorted. Since small details and / or distant objects have a large variance, the denoiser may degrade the corresponding area.For example, such degradation can manifest as blurring of fine details in an image, or loss of detail in objects that are far from the camera.

[0054] A “reference image” is a Monte Carlo rendering ray tracing image obtained by sampling substantially close to the underlying distribution to reduce variance. “Substantially close” means that, according to a given criterion, the system can determine that the underlying distribution of the corresponding rendering equation for that image is substantially approximated by a discrete distribution obtained through Monte Carlo sampling. The variance of the discrete reference image distribution is substantially small. “Each denoised image” is another Monte Carlo ray tracing image resulting from input to the denoiser, and this other image is sampled from the same rendering equation as the corresponding rendering equation for the reference image. Specifically, the number of samples in this other image is less than or equal to the number of samples in the reference image. This other image is also called a noisy image of the reference image. For example, a noisy image may be obtained by sampling s=512 samples, and the corresponding reference image may be obtained by sampling s=131,072 or more samples.

[0055] Given a pair of images, the method calculates a dataset of denoised images. The next step of the method is performed for each pair of images provided. The method provides the pair of images to a pre-trained CNN architecture similar to the CNN architecture to which the resulting dataset is constructed (S20). "Similar" means that the pre-trained CNN architecture has at least one common layer with image feature extractors for CNN architectures that learn on the resulting dataset. A common layer means a layer that provides the same output for the same input. The pre-trained CNN architecture may have all common layers with image feature extractors for CNN architectures that learn on the resulting dataset. As is well known from the field of neural networks, an "image feature extractor" comprises a set of layers, each layer of which is configured to output features of a corresponding input image. Features of a corresponding input image (also known as a feature map) are well known from the field of neural networks, particularly convolutional neural networks, and correspond to a two-dimensional arrangement of values ​​corresponding to the output of the corresponding layer. Features may be represented by an image according to the values ​​of that two-dimensional arrangement, i.e., a matrix. Each value corresponds to a feature value extracted by the corresponding layer. For example, a layer may apply a convolution operation to an input image or the output of another layer. The output of the convolution operation (for example, a convolution using a filter matrix) corresponds to a feature map. A "fully connected image classifier" is a layer in a CNN architecture that takes a feature map output by an image feature extractor as input and outputs a label for the corresponding image.

[0056] In this method, the difference between the first normalized feature of the denoised image and the second normalized feature of the reference image is calculated (S30). The first and second normalized features are the outputs of the same layer of the pre-trained CNN architecture. The difference can be any numerical difference between the first normalized feature map and the second normalized feature map corresponding to the same layer. The difference may be calculated using, for example, the Euclidean distance between the feature matrix values, or any other arbitrary distance metric such as the absolute norm. Figure 9 shows an example of the difference between the first normalized feature of the denoised image and the second normalized feature of the reference image, obtained by obtaining the layer outputs of the pre-trained VGG-19CNN architecture. The denoised image is input to the pre-trained CNN architecture. Each layer of the pre-trained CNN architecture extracts features. The seventh layer from the right of the input image outputs a feature map of the reference image consisting of 256 channels, where the x-direction is 1 / 4 of the input image resolution and the y-direction is 1 / 4 of the input image resolution. The seventh layer from the right of the input image outputs a second feature of the reference image. The two outputs are normalized before the difference is calculated, as shown in the green section in the center of Figure 9. For example, the difference of normalized features output from the seventh layer may further include the differences of other normalized features output by other layers, such as the ninth layer which has 256 channels with x-direction 1 / 4 of the input image resolution and y-direction 1 / 4 of the input image resolution and outputs a feature map of the reference image, and / or the twelfth layer which has 512 channels with x-direction 1 / 8 of the input image resolution and y-direction 1 / 8 of the input image resolution and outputs a feature map of the reference image. Optionally, if the layers have different resolutions, average pooling may be applied to set the layer outputs to the same resolution. Figure 10 shows an example of an error map (calculated in step S40) obtained by applying average pooling after calculating the difference and before calculating the error map.

[0057] Next, the method calculates an error map representing the calculated difference (S40). The error map may be an image, where each pixel of the image has a color value, and each color value corresponds to the calculated difference between the values ​​of the first normalization feature and the second normalization feature corresponding to the output of the same layer. Therefore, the error map can be considered as a map that shows how well the denoised image respects the reference image with respect to the first and second normalization features, respectively. Thus, the smaller the error, the smaller the difference, meaning that the denoised image and the reference match better, that is, that the denoiser retains sufficient detail after the denoising process. Conventionally, the error map consists of non-negative values, with large positive values ​​corresponding to large differences and small values ​​having small non-negative values. The values ​​of the error map with zero values ​​consist of zero values ​​or the minimum value conventionally selected. For example, the pixels of the error map may have grayscale values, where 0 corresponds to no error and 255 corresponds to the maximum difference. Other arbitrary color scales may be defined conventionally. There may be no limit to the difference, but this method may always map values ​​exceeding a threshold to the maximum value allowed on the color scale, which is simply a matter of implementation.

[0058] Next, the method adds each denoised image and error map to the dataset (S50). The dataset is configured to train the CNN architecture to estimate the error due to denoising, and training is performed based on the error map and the corresponding denoised image. The additional training samples in the dataset represent diverse situations after the CNN architecture has been trained, i.e., diverse situations in the corresponding online application after offline training. Thus, the error maps in the dataset train the CNN architecture to identify regions degraded by the denoiser in the denoised images, where regions with a higher level of degradation correspond to a set of pixel map values ​​with substantially high error. Substantially high error means that the pixel map values ​​exceed a certain threshold, which may be a conventionally established threshold. These regions may contain fine structures or objects located far from the camera viewpoint and may be better identified depending on the diversity of the training samples. As is known from the field of machine learning, the dataset formed affects the training speed of the CNN architecture and the quality of training, i.e., the accuracy of estimating the error map of the trained CNN architecture. The dataset may be formed with a total number of data that takes into account the intended quality of training. This number can be 1,000, 10,000, or even more than 100,000 elements. The amount of data in the dataset follows a trade-off between the accuracy achieved by the CNN architecture and the speed of training. For example, 500 data points are suitable for fast training, while 100,000 provide high-quality training. As another example, approximately 2,500 data points are needed to train a CNN architecture using supervised learning in 24 hours.

[0059] For example, the spatial resolution of the error map may be adjusted to improve the detection of areas where the denoiser degrades the image. Step S40, which calculates an error map representing the calculated difference, may include the step of first downsampling the resolution of the calculated difference, and then calculating the error map at the resolution of the downsampled difference, where each pixel of the error map has a color according to a color scale that penalizes the calculated difference. Thus, the spatial resolution of the error map is reduced, and the resulting error map is a coarser error map. The error map may be downsampled by any known means, for example, using mean pooling. Thus, a CNN architecture trained on a dataset formed by this method is well-suited to finding areas of detail loss. In fact, a map with high spatial resolution may contain very small spatial variations between areas of high error and areas of low error, i.e., a few pixels in the error map indicating high error colors according to a conventional color scale are surrounded by several pixels indicating low or no error. If the image resolution is too high, the CNN architecture may be more accurate, but it will require more samples to identify very small spatial variations of error. In contrast, a coarse error map shows coarse spatial variation; that is, pixels with high errors are averaged out to constitute a larger spatial area of ​​the image, making it easier to train the CNN architecture. Downsampling can be implemented in any way that strikes a suitable trade-off between the accuracy and training time of the CNN architecture.

[0060] For example, a pre-trained CNN architecture may use a state-of-the-art CNN architecture such as the Visual Geometric Group (VGG) architecture. The CNN architecture may have various layers, including convolutional layers, pooling layers, fully connected input layers, and fully connected output layers. The details of the configuration of such CNN architectures are widely known and will not be explained in further detail. The various layers of the VGG architecture can extract information (i.e., features or deep features) from the corresponding input image.

[0061] As an example, good results were obtained when the pre-trained CNN architecture was a pre-trained VGG-19 architecture. The main architecture behind the VGG-19 neural network is described in the paper "Simonyan, K., & Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition". The VGG-19 architecture may be pre-trained on a commonly available dataset such as the image dataset ImageNet. Figure 8 shows the main components of the VGG-19 architecture, a state-of-the-art architecture that leverages improvements over convolutional neural networks, and VGG19 includes 16 convolutional layers (denoted as conv), 3 fully connected layers (denoted as FC), 5 max pooling layers (denoted as pool), and 1 softmax layer (denoted as Softmax). Traditionally, layers are numbered from the leftmost layer to the rightmost layer as shown in Figure 8. Because VGG-19 is pre-trained on the image dataset ImageNet, its feature acquisition is particularly accurate. ImageNet is an image dataset organized according to the hierarchical nouns of WordNet, where each node in the hierarchy represents hundreds or thousands of images. ImageNet contains an average of over 500 images per node. Thus, the error map is calculated from the difference between the first normalized feature of the denoised image and the second normalized feature of the reference image, where the first and second normalized features are the outputs of the same layers of a pre-trained VGG-19 architecture. Referring again to Figure 9, as an example, the first normalized feature of the denoised image and the second normalized feature of the reference image are extracted from layers 7 and / or 9 and / or 12 of the image feature extractor, where layers 7 and / or 9 and / or 12 are counted from the leftmost to the rightmost layers shown in Figure 9, as before. All layers shown in gray have been removed. As an example, the first normalized feature of the denoised image and the second normalized feature of the reference image are extracted from layers 7, 9 and 12 of the image feature extractor. The characteristic feature is that the output sizes of different layers are normalized to be equal.For example, the layers of the image feature extractor in the VGG-19 architecture include a downsampling step, and the spatial resolutions of layers 7 and / or 9 and / or 12 are different. In fact, both layers 7 and 9 output a feature map of the reference image consisting of 256 channels where the x-direction is 1 / 4 of the input image resolution and the y-direction is 1 / 4 of the input image resolution, while layer 12 outputs a feature map of the reference image consisting of 512 channels where the x-direction is 1 / 8 of the input image resolution and the y-direction is 1 / 8 of the input image resolution. Therefore, for each channel, the error map may be downsampled to the same resolution to obtain a coarser error map. Figure 10 shows a coarser error map obtained by normalizing the features and performing downsampling by averaging before calculating the error map. The coarser error map shows the same weights for the outputs of layers 7 and / or 9 and / or 12, and these layers have been found to best distinguish the degradation between the reference image and the denoised image. The error map may be further downsampled to provide a coarser error map, which may better identify areas of degradation caused by the denoiser.

[0062] For example, this method may further add each denoised image, error map, and each noisy image of the reference image to the dataset (S50). The noisy image is the original image from which the denoised image was obtained. Typically, noisy images are those with fewer Monte Carlo samples than the corresponding reference image, thus generating noise. The noisy image is provided to the denoiser to reduce noise, and the denoised image is the result of the denoising process performed by the denoiser. The dataset formed by this method allows the CNN architecture to learn to estimate the error of the denoised image output by the denoiser without accessing the reference image. This is particularly important in real-world situations where the reference image is inaccessible because obtaining it would be computationally intensive and difficult.

[0063] For example, the method may further form a dataset suitable for supervised training. The method may configure at least some of the data within the dataset's data structure to be used as ground truth data for supervised training. Referring to Figure 15, a machine learning process for supervised learning of a function is shown. An example of ground truth data is shown at the top of Figure 15. Noise images and function buffers obtained by Monte Carlo rendering are input to a denoiser to generate a denoised image. The denoised image and reference image are provided to the dataset formation method. The dataset formation method outputs an error map, which is configured as ground truth data. This method may generate any amount of ground truth data required for supervised learning. The method may store the dataset in memory for later use on the same system or for transmission to another system. In offline training, which can be run on the same system or another system, the CNN architecture can be trained in a supervised manner by comparing the output of the CNN architecture with the generated ground truth data.

[0064] For example, each denoised image is the result of inputting another image sampled from the same rendering equation as the reference image into a denoiser, where the number of samples is strictly less than the number of samples in the reference image. The user may input any number of samples as long as it is strictly less than the number of samples in the reference image. Thus, the resulting dataset may contain data fragments that exhibit relatively large variations due to noise, thereby providing a CNN architecture that learns to distinguish the degradation by the denoiser at multiple noise variation values. This other image is sometimes further called the “noise image”. Figures 11–13 show an example of a set of images including the reference image, the corresponding noise image, the denoised image resulting from inputting the noise image into the denoiser, and the corresponding error map. For example, Figure 11 shows a noise image of a scene containing multiple elements, such as a curtain on the side and a sculpture at the far end of the camera's viewpoint. Because this specific example is an example of a large number of samples, it is difficult to see the degradation of the noise image due to denoising. The corresponding error map shows the degradation. Figure 12 is a zoomed version of the same scene including the area of ​​the curtain. Figure 13 is a zoomed version of the same scene including the sculpture at the far end of the camera's viewpoint.

[0065] For example, a dataset formation method may form a data structure that represents the dataset. Thus, the formed dataset may be represented by a data structure. This data structure may be stored in non-volatile memory or transmitted directly to another computer system over a network, thereby enabling subsequent use. For example, a dataset may be organized as an EXR image file so that high dynamic range (HDR) content can be stored in non-volatile memory (such as a disk), where the data on the disk is in a folder, one shot per folder, and the file names include descriptive postfixes that describe the content.

[0066] Referring to the flowchart in Figure 2, further computer implementation methods for performing learning are proposed and provided, which are also called “Learning Methods.” These Learning Methods use the dataset obtained in the “Dataset Formation Method.” These Methods include the step of training a convolutional neural network (CNN) architecture based on the said dataset. These Methods include step Q10, which provides the formed dataset. The provided dataset may be formed at different times, in different places, using different systems, and / or by different individuals or entities. These Methods further include step Q20, which trains a CNN architecture based on the formed dataset. Step Q20, which trains a state-of-the-art CNN architecture that outputs an error map that estimates the loss of detail in the image generated by a denoiser compared to a corresponding noise image input to the denoiser, can be performed by any known means.

[0067] For example, the training step Q20 may include a step that takes images generated by a denoiser and their corresponding noise images from a dataset as input and outputs an error map. Training can be performed by any known means using a state-of-the-art CNN architecture that outputs an error map that estimates the loss of detail in the image generated by the denoiser compared to the corresponding noise image input to the denoiser. Thus, the CNN architecture learns for estimating denoising degradation thanks to a dataset provided which is configured for the CNN architecture to learn from training samples for the purpose of estimating denoising degradation.

[0068] Therefore, a CNN architecture trained according to this method estimates the degradation produced by the denoiser with the planned estimation accuracy intended by the sample size. Furthermore, thanks to the various training examples provided by the resulting dataset, the CNN architecture can identify fine structural details or zone regions, including objects far from the camera's viewpoint.

[0069] For example, step Q20, which involves training, may be performed as supervised learning. Therefore, training further includes a step of using error maps as ground truth data for supervised learning. In particular, training may use error maps of the dataset provided in step Q10 as ground truth data. Since the dataset provided in step Q10 is formed from pairs of denoised images and corresponding reference images, the error maps are appropriate ground truth data that reflect the loss of detail compared to the reference. The CNN architecture can be trained according to any known means in supervised learning, and the specific training process is a detail of implementation. The ray-traced images may include scenes, such as indoor shots, product shots, or close-up shots. Therefore, a CNN architecture trained according to this method can detect problematic areas where detail is lost in various scenes.

[0070] For example, the CNN architecture trained in training step Q20 is identical to the pre-trained CNN architecture and has the same number of layers. Since it is not necessary to extract other information such as deep features, and only the information that outputs the first and second normalization features needs to be extracted, all layers of the CNN architecture that come after the layers of the pre-trained CNN architecture that generated the first and second normalization features are removed. The VGG-19 architecture enables an efficient machine learning process, which is illustrated in detail in the following example.

[0071] Figures 9, 14, and 15 illustrate this example, showing a pre-trained CNN architecture, a CNN architecture trained according to this learning method, and an example of the machine learning process, respectively. The pre-trained CNN architecture used to form the dataset according to the dataset formation method and the CNN architecture trained to estimate the degradation due to denoising are based on the VGG-19 architecture. Figure 15 shows the machine learning process, including the learning method process (prediction section) based on the dataset (ground truth section) formed by the dataset formation method. Both the pre-trained CNN architecture used to form the dataset and the CNN architecture trained by this learning method are identical VGG-19 architectures with the same number of layers. In this example, the pre-trained VGG-19 architecture is set up as described above, and further details are omitted.

[0072] Referring to Figure 14, the CNN architecture trained to estimate the degradation produced by the denoiser uses the same layers as the pre-trained VGG-19 architecture, without adding a normalization step, and is trained in a supervised manner on the ground truth data set up in the provided dataset. In this specific example, the CNN architecture further accepts albedo and normal buffers as input. This should be understood as one option for the embodiment.

[0073] The following describes an example of a learning method using the dataset obtained in the "Dataset Formation Method". A CNN architecture may be trained to estimate the degradation generated by a denoiser on ray-traced images. Therefore, the trained CNN architecture may also be called an "estimator architecture". Learning may be based on the formed dataset. The degradation estimation may be represented by an error map that identifies areas where the denoiser image has lost detail (for example, fine details are blurred). The training step Q20 includes receiving the dataset, the image generated by the denoiser and its corresponding noise image, as input. The dataset is obtained according to the example described above. The training step further includes outputting an error map.

[0074] Therefore, the estimator architecture trained according to this example is particularly fast at estimating the denoiser-generated degradation on ray-traced images compared to other online optimization procedures. This is because the estimator architecture is first trained in the offline phase and then applied in the online phase without further optimization. Furthermore, the error map output by the estimator architecture accurately estimates the denoiser-generated degradation without using a reference image. In fact, since the provided dataset includes denoised images and corresponding error maps, the estimator architecture trained according to this learning method can estimate small and / or thin structures lost by the denoiser and detect objects far from the camera's viewpoint in the scene.

[0075] For example, an estimator architecture trained according to this learning method further accepts an albedo map and a normal buffer as input for each denoised image in the dataset. The number of input channels is the same as that used by state-of-the-art denoiser systems. Therefore, this method can detect the loss of fine structural detail. In fact, as is well known in the art, albedo and normal buffers are typically used by denoisers to better preserve detail, and therefore, the trained CNN architecture can estimate the further loss of detail caused by the denoiser.

[0076] For example, the estimator architecture may be trained using supervised learning. Therefore, the training further includes the step of using error maps as ground truth data for supervised learning, as described above. The estimator architecture can be trained using any known means in supervised learning, and the specific training process is a detail of implementation. The ray-traced images may include scenes, such as indoor shots, product shots, or close-up shots. Thus, an estimator architecture trained according to this method can detect problematic areas where detail is lost in various scenes.

[0077] For example, the estimator architecture is identical to the pre-trained CNN architecture and has the same number of layers. All layers of the estimator architecture that come after the layers that generated the first and second normalization features of the pre-trained CNN architecture are removed. Therefore, the estimator architecture has the same number of layers as the pre-trained CNN architecture, but it can save computation time compared to training with more layers.

[0078] As an example, good results were obtained when the estimator architecture was a VGG-19 feature extractor with layers 0 through 12. The dataset formed by this dataset formation method may contain thousands of shots (i.e., training samples), each shot containing various objects in a scene under various lighting conditions with all kinds of structural details. For example, objects in the scene may include objects with fine details, such as leather, carpet, stonework, wallpaper, rough metal, metal flooring, or plastic. Each shot may include a reference image, a denoising image, and an error map. As an example, the VGG-19 feature extractor is trained in a supervised manner, as is well known in the art. The best results are obtained when the VGG-19 feature extractor further includes albedo and normal buffers as inputs.

[0079] Next, we will explain how to use the estimator architecture, referring to the flowchart in Figure 3. This method uses the trained estimator architecture to dynamically denoise ray-traced images. The basic idea is to sample the ray-traced image with a first number of samples, estimate the variance, and calculate a sample map that shows the number of samples required to reach a specific level of detail. The application of the sample map is performed based on a blend map that shows areas of the image that have not converged sufficiently in the context of Monte Carlo sampling. This is done by calculating a blend map that identifies areas of the image that have not converged significantly, according to the target signal-to-noise decibel level. The sample map then establishes, for the blend map, the number of samples required to obtain better detail in areas with high signal-to-noise ratio decibel levels. This method is further explained below.

[0080] The method includes step P10, which provides first and second Monte Carlo ray-traced images rendered from the same shot. A shot means a ray-traced image that includes a scene containing one or more objects, rendered using the Monte Carlo method. The first Monte Carlo ray-traced image is different from the second image. "Different" means that the first and second Monte Carlo ray-traced images are rendered from the same shot but are rendered using different sampling processes, e.g., different initial conditions (such as a random seed), different number of samples, or different sampling processes. As is known by itself from the field of Monte Carlo methods, the sampling process of the image is random; that is, the samples of the shot's rendering equation are taken according to a probabilistic (i.e., non-deterministic) criterion. Thus, because the sampling process is performed differently, these images have different regions that represent noise. Monte Carlo samples may be taken by any known sampling process, e.g., uniform sampling from a random seed. The system may obtain two images by any method known in the art. The system may perform Monte Carlo sampling from the corresponding shots based on the shot's rendering equation stored in non-volatile memory. For example, the system may be configured to perform Monte Carlo rendering for the first image and Monte Carlo sampling for the second image in a separate process within the same system. This should be understood as merely one option for implementation. Alternatively, the first and second images may be acquired directly from non-volatile memory or received from another computer system, for example, via a network. Because the sampling is different, the first and second images will show noise in different areas, and the method uses the first and second images to estimate the number of samples required to achieve a satisfactory quality of shot, which will be further explained in the steps of this method.

[0081] This method further includes step P20 of providing a first denoising image of the first image. Any known denoising method known in the art may be used. The first image and the first denoising image are then applied to a trained estimator architecture so that an error map P30 is obtained. Each pixel of the error map may represent a color indicating the error due to denoising, where the error is the error estimated by the estimator architecture. The error map may contain regions of the same error, i.e., sets of consecutive pixels in the error map having the same (or substantially the same) values. For example, the error map may be a coarse map obtained by averaging error maps, and thus a wider region is provided than a fine map.

[0082] This method of use further includes step P40 of calculating a blend map from the obtained error map. The blend map represents the signal-to-noise decibel level of each region of the error map. As is well known in the art, loss of detail is directly related to the signal-to-noise ratio of the image. Larger errors due to denoising correspond to a lower signal-to-noise ratio, and smaller errors correspond to a higher signal-to-noise ratio. Therefore, the blend map shows the decibel level of the region corresponding to the error map resulting from denoising. For example, the blend map may represent the decibel value of the peak signal-to-noise ratio, calculated from the target root mean square error using, for example, the formula: -20*log10(rmse), where rmse represents the target root mean square error. The system may predefine the target root mean square error. Therefore, the blend map represents multiple regions within the blend map that have decibel values ​​lower than the desired signal-to-noise decibel value required to achieve an accurate image. Therefore, the system can define a weighted blend according to the signal-to-noise decibel values ​​of the blend map.

[0083] Next, the variances of the first and second images are estimated (P50). The variances may be estimated from the first and second images by any means. Because the first and second images are generated from different sampling processes, noise occurs in different regions due to the random nature of Monte Carlo sampling. The regions exhibiting noise in the first and second images may have some degree of cross-correlation; for example, the regions may be independent (in a probabilistic sense) or substantially correlated. The variance is estimated from the combination of the first and second images. For example, the variance may be calculated from the image resulting from (first image + second image) / 2.

[0084] From the variances estimated from the first and second images, this method may further calculate a sample map (P60). The sample map shows the number of Monte Carlo samples required to reach the signal-to-noise decibel level necessary to achieve a suitable quality. The signal-to-noise decibel level may be set by the user or the system. The sample map may be calculated using the estimated variances and Monte Carlo convergence characteristics. The system may determine the number of samples to reduce the variance between the first and second images by any means known in the art that utilizes Monte Carlo convergence characteristics. The number of samples required to reach a certain signal-to-noise decibel level may be greater than, for example substantially greater than, the number of samples performed to render the first and second images.

[0085] Next, a third Monte Carlo ray tracing image of the shot is rendered using the sample map (P70). The third image is rendered using the number of Monte Carlo samples identified from the sample map. The third image is sampled according to the number of samples required to reach a certain signal-to-noise decibel level. Therefore, the sampled third image is more detailed than the first and second images, and that detail is added in a particularly efficient way. In fact, since sampling is performed according to the number of samples required to reach a certain signal-to-noise decibel level, this method focuses only on adding samples to the areas indicated by the sample map that are required to reach the signal-to-noise decibel level. This is particularly efficient compared to other sampling methods, such as uniform sampling, where sampling is performed uniformly across areas of the image and sampling may be performed even where there is no need to add further detail.

[0086] Next, the third image is provided as input to the denoiser (P80), thereby obtaining the corresponding denoised image.

[0087] The resulting denoised image corresponds to the third image and is blended with the third image (P90). The third image has better detail than the first image, but still requires denoising. Blending is performed by blending areas with less noise with the denoised image, depending on the signal-to-noise decibel level. This is done according to a blend map. Blending weights each region of the blend map that has a signal-to-noise decibel level higher than the threshold. For example, in this method, regions with decibel values ​​above 40 decibels may be blended, but those below that threshold may not be blended. This is because, in this case, the first denoised image would be too noisy to produce an acceptable result. The threshold may be set according to the desired quality and is only one option for implementation.

[0088] Therefore, this method improves the denoising process and enhances the quality of ray-traced images without the high computational cost of rendering high-resolution or reference images, as the denoising process is improved to remove noise only from areas that do not undergo quality degradation during denoising. In fact, this method focuses on adaptively distributing a large number of samples to areas that are not denoised, by detecting problematic areas in the image via an error map. As it is a fully automated process, no user intervention is required, and time-consuming manual inspection is eliminated. Furthermore, rendering costs are also reduced due to the more efficient sample distribution calculated from the sample map.

[0089] Due to noise variability, the error map obtained from a trained estimator architecture may show regions with large or even unrestricted errors. For example, this method may calculate a blend map from the obtained error map by normalizing the error map. Normalization may be performed by applying a sigmoid function to the error map. As is widely known in engineering, the sigmoid function is a bounded, smooth real function defined for all real input values. The sigmoid function takes, for example, values ​​corresponding to the color scale of the error map as input, representing the error value at each pixel of the error map. The sigmoid function has non-negative derivatives at each point and at one inflection point. The properties of the sigmoid function are used to map unrestricted errors as follows: below the inflection point, the value mapped by the sigmoid function is small, and below the inflection point, the mapped value decreases significantly. Thus, the sigmoid function can suppress substantially small noise. The extent to which small noise is suppressed depends on the definition of the tunable inflection point, as is known in the art. Large noise, such as unrestricted noise, is noise that maps over inflection points. In this case, as is widely known, the sigmoid function can be considered a smooth saturation function that maps large errors up to the maximum value allowed by the sigmoid function. For example, the sigmoid function may be chosen to normalize the error values ​​over a bounded interval, for example, the interval [0,1]. For example, the sigmoid function is the hyperbolic tangent function or the logistic function. Thus, the sigmoid function may map unrestricted values ​​over a restricted interval [0,1]. Since unrestricted values ​​are mapped to a blend map, the blend map is clearly defined in all its regions. This method calculates the blend map by calculating the decibel value for each region of the error map from the target root mean square error between the normalized error map and the target error map. The blend threshold can be set by the user from the calculated decibel levels. For example, regions with low signal-to-noise decibel levels are ignored because there is too much noise. Therefore, regions with high signal-to-noise decibel levels can be identified and further improved.Therefore, this method of use can focus more on improving areas with low error.

[0090] For example, the first image may be rendered with a first number of Monte Carlo samples, and the second image may be rendered with a second number of Monte Carlo samples. “Rendered with” means that the rendering is based on either the first or second number, which may further involve adjustments depending on the choice of sampling implementation. The first number of Monte Carlo samples may be greater than or equal to the second number of Monte Carlo samples. For example, the first and second numbers may be equal, but because it is a probabilistic sampling process, the samples will provide different renderings. The first number of Monte Carlo samples may be greater than the second number of Monte Carlo samples. Thus, the variance can be considered an indicator of how noisy the first image is, and the method may be considered a way to improve detail with respect to the first image.

[0091] This method of use may also involve obtaining a second image by using the same sampling process used to obtain the first image. For example, a first number of Monte Carlo samples is set, and the sampling process starts with the first number. The sampling process then stores a second image corresponding to half of the first number of samples. Thus, the second image is rendered with a second number of Monte Carlo samples, which is half of the first number of Monte Carlo samples. The rendering of the first image may correspond to rendering with the first number of samples, or to rendering that is a further refinement of rendering with the first number of samples. This method of rendering the first and second images is particularly efficient as a sampling process for obtaining the first and second images because it is optimized so that both renderings are performed in a single run within the system. For example, the system may set the first number to, for example, n=128 samples and start the sampling process. The image corresponding to the execution of an intermediate step in the sampling process, i.e., s=64 samples, is then stored in memory as the second image, and the system then continues sampling until it reaches n samples. Next, the image obtained with the first number of n samples is further adapted by removing 64 samples s corresponding to the second image. For example, this adaptation may be performed by multiplying the image corresponding to n=128 samples by 2 and subtracting the second image corresponding to 64 samples s. The arithmetic operation is understood as an operation performed using the image matrix. The result is an image with sample count s'=ns, which corresponds to rendering the first image after adapting the first number of samples. The resulting first image (adapted) is independent of the samples s of the second image (corresponding to s' samples), meaning that the noise between the two images is not structurally correlated. This yields the same result as rendering two images independently with the same number of samples, but without restarting the sampling process.Therefore, this method allows rendering the first and second images without restarting the sampling process, and rendering is performed only when it is necessary to further save the intermediate results of the sampling process (i.e., the second image), thereby integrating better with existing architectural rendering. For example, uniform rendering of a Monte Carlo image is an iterative process, with one additional sample accumulating for each pixel of the image in each iteration. This means that at each point in sampling time, according to uniform sampling, only the cumulative version of the image that exists exists. Thus, by obtaining the second image from the same process of the same image, the second image can be obtained by simply removing the samples from the first image. If the first image is fitted by removing the samples from the second image, the first and second images are independent, making it easier to calculate the variance and improving the quality of the rendering.

[0092] For example, a first Monte Carlo ray-traced image provided may be rendered according to a first random seed, and a second Monte Carlo ray-traced image provided may be rendered according to a second random seed. A random seed is an initial condition for initiating any sampling strategy chosen to perform sampling. As is known by itself from the field of Monte Carlo rendering, a first random seed different from a second random seed will generate different samples of the same shot, thereby obtaining different renderings of the same shot. The first number of Monte Carlo samples may be the same as the second number of Monte Carlo samples. Because the first and second random seeds are different, it is guaranteed that the noise present in the first image after the sampling process will be independent of the noise present in the second image, yet the required number of samples remains the same. Example of creating a dataset The combinations of examples described below will be explained with reference to Figures 8 to 13.

[0093] First, we will describe the metric used to calculate the error map. This metric compares images using features from a VGG19 network pre-trained on ImageNet. The VGG19 network includes a convolutional feature extractor and a fully connected classifier. Figure 8 shows a flowchart of the layers of the VGG19 architecture. The metric calculates features from a dataset that includes images with various types of detail, their respective denoised images, and a reference image. In the test implementation, we found that layers 7, 9, and 12 of the VGG19 architecture were the layers that best extracted structural detail features.

[0094] Each feature is normalized so that the magnitude of the output from different layers is equal before calculating the squared difference between the denoised image and the reference image. Figure 9 shows the calculation of the difference of normalized features. All layers from layer 12 onwards in the VGG-19 architecture, i.e., the layers shown in gray in Figure 9, are discarded from the metric and are not used any further.

[0095] Because the network includes a downsampling step, the spatial resolution of different layers will be different. The spatial resolution of different layers is converted to an average of 1 / 32 of the input resolution. After averaging the output, it is averaged channel by channel to obtain a coarse error map. Figure 10 shows a flowchart of the implementation of downsampling across layers and the resulting coarse error map.

[0096] Figures 11-13 show error maps calculated according to this metric. Figure 11 shows a coarse error map calculated from the reference image, a noisy image with fewer samples than the reference image, a denoised image which is the output of the denoiser, and the normalized difference of features in the VGG19 map. In the error map, dark areas represent areas with low error, and bright areas correspond to areas with high error. This metric identifies the blurring caused by denoising the blue curtain. Figure 12 shows the corresponding zoomed portion of the same image, showing the loss of detail in the curtain area and fine structural features of the floor, such as cracks in the floor. Figure 13 is a zoomed version of an object (image) far from the camera viewpoint. The error map shows the loss of detail in such an object.

[0097] Error map estimation The combinations of examples discussed below will be discussed with reference to Figures 14 to 18.

[0098] Since reference images typically do not exist in real-world situations, it is necessary to generate error maps without reference images. A convolutional neural network is trained to estimate error maps. To train the network, a dataset is created with error maps calculated from the above metrics. The dataset contains 2437 shots of various objects with all kinds of structural details (leather, carpet, stonework, wallpaper, rough metal, metal flooring, fine details of plastic, etc.) under various lighting conditions. Each shot is rendered with a different number of samples (1 to 512), and the reference image is rendered with 131072 samples per pixel (rendering iterations). The aforementioned metrics are used to generate the ground truth output. The network is then trained in a supervised manner for approximately 24 hours. In the test implementation, the training setup provides a good trade-off between the time it takes to train the dataset and the accuracy of the trained network architecture. Figure 14 shows the network architecture. The network architecture is similar to the VGG19 feature extractor, and the aforementioned metrics use layers 0 to 12 of the feature extractor. The main difference is the number of input channels, as the denoised image and the same inputs used by the denoiser (noise image, albedo, and normal buffer) are input to the network. Figure 15 shows supervised learning. The trained network directly outputs error maps corresponding to the input images. Figures 16-18 show estimated error maps obtained by test implementations. Figure 16 corresponds to shots of an indoor scene. The left figure corresponds to a noisy image with few samples. The second image clockwise corresponds to the denoised image. For comparison, the error map corresponding to a comparison of the denoised image and the reference error map is also shown in the lower right, and the estimated error map (also called the predicted error map) is shown in the lower left. Figures 16-18 show good agreement between both error maps.

[0099] Adaptive noise reduction system The combinations of examples discussed below will be discussed with reference to Figures 3 and 19-21.

[0100] The following example concerns an adaptive denoiser for adaptively denoising an image. In this example, a low sample count is used for the variance estimator, but it is necessary to render a sample map with enough samples to achieve, for example, about 50+ dB in the relevant region. Referring to Figure 3, in step P10, the system uniformly renders the scene with a small sample count (e.g., 64 SPP). During rendering, intermediate images of half the sample count (32 in this case) are also saved, which are necessary for estimating the variance. The 32 and 64 SPP images are sent to the variance estimator (P50) to estimate how noisy the image is. The 64 SPP images and the corresponding normal and albedo feature buffers are input to the denoiser (P20) to generate a denoised input for the error estimation network. The resulting error map and the variance estimate output at P50 are then used to compute the adaptive sample map and the final blend map. The tanh function is applied to the squared error map to normalize the error in the error map to the real range [0,1]. Next, the normalized error map is transformed into a blend map containing the decibel levels required for blending. The decibel value of the peak signal-to-noise ratio is calculated from the target root mean square error (RMSE) value by the following formula: -20*log10(RMSE). Depending on the rendering calculation budget, these decibel levels may be below the blend threshold that would prevent blending in areas that are not sufficiently converged. Various decibel levels are used to define the blend boundary. In this example, linear blending is performed from 40 dB (0 blend) to 50 dB (full blend). In the test implementation, the blend threshold was defined by comparing the decibel levels of multiple noises. Figure 19 shows an example of a noisy image along with the decibel value of the signal-to-noise ratio relative to the reference image. Noisy images below 40 dB are noisy, and detail increases as the decibel level increases.

[0101] A sample map P60, which shows the samples needed to reach the desired decibel level, can be calculated using a variance estimator and Monte Carlo convergence properties. The sample map P60 is then sent to the renderer, which adaptively continues rendering according to the map. The renderer's output is denoised, and the denoised image, along with the rendered output, is provided to the blending step P90. The blending step P90 blends the rendered output and the denoised output according to the blend map. Compared to denoising alone, this system achieves significantly better image quality with nearly the same computational effort. In the shot shown in Figure 20, the error for the aforementioned metric is reduced by approximately 40%. In Figure 21, the reduction is approximately 27%.

Claims

1. A computer implementation method for forming a dataset configured to train a convolutional neural network (CNN) architecture including an image feature extractor, - A step (S10) of providing a plurality of image pairs, wherein each pair has a reference image and a corresponding denoising image, - For each of the above image pairs, - Step (S20) of providing the image pair to a pre-trained CNN architecture similar to the CNN architecture to be trained using the formed dataset, wherein the similar pre-trained CNN architecture has at least one common layer with the CNN architecture to be trained, the pre-trained CNN architecture is a Visual Geometric Group (VGG) architecture, the Visual Geometric Group (VGG) architecture is a VGG-19 architecture, and the first normalized feature of the denoised image and the second normalized feature of the reference image are extracted from the 7th and / or 9th and / or 12th layers of the VGG-19 architecture, - A step (S30) of calculating the difference between a first normalization feature of the denoised image and a second normalization feature of the reference image, wherein the first and second normalization features are the outputs of the same layer of the pre-trained CNN architecture. - A step (S40) to calculate an error map representing the calculated difference, and A method comprising the step (S50) of adding the respective denoising images and the error map to the dataset.

2. The step of calculating the error map is, - A step of downsampling the resolution of the calculated difference, and The method according to claim 1, comprising the step of obtaining a coarse error map by calculating an error map at the resolution of the downsampled difference, wherein each pixel of the error map has a color along a color scale that penalizes the calculated difference.

3. The aforementioned additional step is, The method according to claim 1 or 2, comprising the step of adding noise images of each of the denoising images, the error map, and the reference image to the dataset, wherein the noise image is the original image from which the denoising image was obtained.

4. The method according to any one of claims 1 to 3, wherein each of the denoised images is the result of inputting an image with fewer Monte Carlo samples than the reference image, which has been denoised by the denoiser, into the denoiser.

5. A computer implementation method using a dataset that can be formed according to any one of claims 1 to 4, the method comprising the step of training a convolutional neural network (CNN) architecture based on the dataset.

6. The method according to claim 5, wherein the learning step is supervised learning.

7. The method according to claim 5 or 6, wherein the CNN architecture to be trained is identical to and has the same number of layers as the pre-trained CNN architecture, but all layers of the CNN architecture to be trained that are after the layers of the pre-trained CNN architecture that generated the first and second normalization features have been removed.

8. A computer program comprising instructions for performing the method according to any one of claims 1 to 4 and / or the method according to any one of claims 5 to 7.

Images