Intelligent noise reduction

Abstract

The invention relates in particular to a computer-implemented method for learning a machine learning of a convolutional neural network (CNN) architecture for estimating a degradation of ray-traced image generation by a denoiser. The method comprises providing a dataset and learning the CNN architecture based on the provided dataset. The learning comprises inputting an image generated by the denoiser and its corresponding noisy image of the provided dataset and outputting an error map. This forms an improved solution with respect to estimating a degradation of ray-traced image generation by a denoiser.

Description

Technical Field

[0001] This invention relates to the field of computer programs and systems, and more particularly, to methods, systems, and programs for rendering ray-traced images. Background Technology

[0002] A variety of systems and programs are available on the market for the design, engineering, and manufacturing of objects. CAD is an acronym for Computer-Aided Design, which, for example, refers to software solutions for designing objects. CAE is an acronym for Computer-Aided Engineering, which, for example, refers to software solutions for simulating the physical behavior of future products. CAM is an acronym for Computer-Aided Manufacturing, which, for example, refers to software solutions for defining manufacturing processes and operations. In such computer-aided design systems, graphical user interfaces play a crucial role in the efficiency of the technology. These technologies can be embedded in Product Lifecycle Management (PLM) systems. PLM refers to a business strategy that helps companies share product data, apply common processes, and leverage enterprise knowledge to develop products from concept to the end of their lifecycle within the concept of an extended enterprise. PLM solutions offered by Dassault Systèmes (under the trademarks CATIA, ENOVIA, and DELMIA) provide an engineering center for organizing product engineering knowledge, a manufacturing center for managing manufacturing engineering knowledge, and an enterprise center that integrates and connects the enterprise with both the engineering and manufacturing centers. The system collectively delivers an open object model that links products, processes, and resources to enable dynamic, knowledge-based product creation and decision support that drive optimization in product definition, manufacturing readiness, production, and service.

[0003] Against this backdrop, the field of computer graphics has provided increasingly useful techniques. In this field, ray tracing is used to generate images with high visual realism, and the techniques used to render ray-traced images are becoming increasingly important.

[0004] Monte Carlo rendering is a known technique for rendering high-quality ray-traced images. It is based on computing higher-order integrals of the fundamental rendering equations describing light reflection using numerical integration with random samples. Due to random sampling, Monte Carlo rendering has a random nature, resulting in noisy images. Increasing the number of samples can make Monte Carlo rendering more accurate, and iteratively converges to the correct result as the number of samples increases. Pharr, M., Jakob, W., and Humphreys, G. (2018), Physically Based Rendering: From Theory to Implementation, provides a detailed introduction to the main mathematical concepts of basic Monte Carlo rendering. Monte Carlo rendering is a computationally intensive and resource-intensive process, so minimizing the number of samples that must be collected while reducing variance due to noise is a crucial task.

[0005] Some algorithms use sampling strategies to sample as close as possible to the basic distribution, thereby reducing variance. To achieve this, various sampling strategies can be employed, such as uniform sampling, hierarchical sampling, or importance sampling. These are called fixed-scheme sampling strategies. Adaptive sampling is a sampling strategy that defines the sampling density based on previous sampling rather than a fixed scheme. Essentially, adaptive sampling allows for the distribution of samples in the image space where these samples need to minimize the output variance. Most adaptive sampling techniques require a reconstruction process. Zwicker, M., Jarosz, W., Lehtinen, J., Moon, B., Ramamorthi, R., Rousselle, F., Sen, P., Soler, C., Yoon, S.-E. (2015), Recent Advances in Adaptive Sampling and Reconstruction in Monte Carlo Rendering, provides a detailed investigation of background techniques on adaptive sampling.

[0006] While sampling algorithms allow for the rendering of high-quality images using appropriate sampling strategies, a drawback is that the number of samples required to achieve low-noise images may be computationally infeasible. Denoising Monte Carlo rendering refers to techniques aimed at reducing the variance of sampled ray-traced images. It can be described as a priori and posterior techniques. Zwicker, M., Jarosz, W., Lehtinen, J., Moon, B., Ramamoorthi, R., Rousselle, F., Sen, P., Soler, C., Yoon, S.-E. (2015), Recent Advances in Adaptive Sampling and Reconstruction in Monte Carlo Rendering, provides a detailed investigation of background techniques regarding posterior techniques for denoising Monte Carlo rendering. Posterior methods operate on the renderer's output with only a small amount of additional information (such as normal and albedo buffers) and without knowledge of the internal computations. The key idea is to use a series of reconstruction filters and develop an error estimate of the filter output. Most current systems are based on the idea of ​​nonlinear image spatial filters, as described in Rushmeier, HE and Ward, GJ (1994), Energy Preservation Nonlinear Filters, and use auxiliary features such as normal and albedo buffers, as introduced in McCool, MD (1999), Anisotropic Diffusion for Monte Carlo Noise Reduction.

[0007] Neural networks, particularly convolutional neural networks (CNNs), have been deployed in computer vision systems, including denoising systems. Li, Z., Yang, W., Peng, S., and Liu, F. (2020), Research on Convolutional Neural Networks: Analysis, Applications, and Prospects; Liu, G., Reda, FA, Shih, KJ, Wang, T.-C., Tao, A., and Catanzaro, B. (2018), Image Inpainting for Irregular Holes Using Partial Convolution; and Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, PA. (2008), Extracting and Synthesizing Robust Features Using a Denoising Autoencoder, provide examples of the use of CNNs in computer vision systems (such as image classification) and generative image systems (such as image synthesis, inpainting, or denoising). Gatys, LA, Ecker, AS, and Bethge, M. (2015), Neural Algorithms for Artistic Styles, describes the transfer of styles from one image to another using an image classification network as a loss function and by employing the weights of a pre-trained VGG19 convolutional neural network. Zhang, R., Isola, P., Efros, AA, Shechtman, E., and Wang, O. (2018), Unreasonable Validity of Deep Features as Perceptual Metrics, describes the evaluation of perceptual similarity between two images by training a network to operate on filter maps generated by a pre-trained classification network to measure their perceptual similarity, using the weights of a pre-trained VGG19 convolutional neural network. Bako, S., Vogels, T., Mcwilliams, B., Meyer, M., Novák, J., Harvill, A., Sen, P., DeRose, T., Rousselle, F. (2017), Kernel Prediction Convolutional Network for Denoising Monte Carlo Rendering, describes a denoiser in which a neural network is used to estimate a local weighted kernel, which is used to compute each denoised pixel from its neighbors. Chaitanya, CR, Kaplanyan, AS, Schied, C., Salvi, M., Lefohn, A., Nowrouzezahrai, D., and Aila, T. (2017), Interactive Reconstruction of Monte Carlo Image Sequences Using Recurrent Denoising Autoencoders, describes a denoising method that operates on image sequences and predicts denoised pixels directly from the input.Commercial denoising solutions exist, such as NVIDIA OptiX Denoising or Intel Open Image Denoising, which are partly based on the interactive reconstruction technique of Monte Carlo image sequences using a cyclic denoising autoencoder, as described by Chaitanya, CR, Kaplanyan, AS, Schied, C., Salvi, M., Lefohn, A., Nowrouzezahrai, D., and Aila, T. (2017).

[0008] While denoising reduces the variance caused by noise, it can lead to the loss of fine details (e.g., the leather-like background in the final frame render) during the denoising process, resulting in image blurring. Kuznetsov, A., Kalantari, NK, and Ramamoorthi, R. (2018), Depth Adaptive Sampling for Low Sample Count Rendering, describes combining adaptive sampling with denoising using two neural networks. One neural network operates in front of the renderer, generating an adaptively sampled map from a single sample image, focusing on low sample counts. The other neural network denoises the adaptively sampled render. However, because the neural network used for sampling operates under low sample counts, it cannot properly sample thin structures in noisy regions, resulting in loss of detail after denoising. Hasselgren, J., Munkberg, J., Salvi, M., Patney, A., and Lefohn, A. (2020), Neural Temporal Adaptive Sampling and Denoising, extended the system of Kuznetsov, A., Kalantari, NK, and Ramamoorthi, R. (2018) to operate on image sequences by extending the system to the temporal domain and creating a recursive loop by temporally wrapping the previous frame and feeding it into an adaptive sample graph network, but suffers from the same loss of detail problem after denoising.

[0009] In the case of images rendered with high sample counts, including a denoising process degrades quality compared to not denoising the input at all. Therefore, a systematic denoising process is required to avoid degrading highly convergent regions that present small structural details.

[0010] Figure 6 and 7 An example image with a high sample count is shown, where the quality has been reduced to that required for the noise reduction process. Noise reduction errors are detected using existing techniques.

[0011] Figure 6 The upper left portion shows a denoised image with a large number of samples. Figure 6The upper right portion shows a reference image of the denoised image. This image shows furniture containing a leather texture. A visual inspection of the denoised and reference images clearly shows areas of image quality degradation; for example, the leather texture in the reference image is lost after denoising. Standard metrics can be used to partially detect degradation, but the high sampling rate of the images does not allow for systematic differentiation of areas degraded due to the denoiser. The lower left portion shows the standard metrics, which show the mean absolute error between the denoised and reference images. Darker areas in the error map correspond to areas with lower or no error. Brighter areas correspond to areas with larger errors. Figure 6 The lower part of the error diagram shown on the left does not distinguish the loss of details in the leather texture. Figure 6 The lower right portion shows the error map based on the Structural Similarity Index (SSIM) metric between the denoised image and the reference image. SSIM slightly improves the regions shown as degraded by denoising, assigning them slightly higher errors. However, SSIM is still insufficient as a detection metric. In fact, the SSIM error map shows a blurred error, making it difficult to clearly distinguish regions with distorted small structural details.

[0012] Figure 7 A similar example is shown using a rendering of a plastic bottle, where visual inspection confirms that the denoised image in the upper left portion exhibits blurred letters engraved in the plastic bottle compared to the reference image. However, neither mean absolute error nor SSIM (Simplified Mean Absolute Error) can reveal the area of ​​blurred letters.

[0013] In this context, improvements are still needed in rendering ray-traced images. Summary of the Invention

[0014] Therefore, a computer-implemented method is provided for learning a machine learning approach to estimate a degraded convolutional neural network (CNN) architecture generated by a denoiser on a ray-traced image. The method includes:

[0015] - Provide the dataset obtained through the following operations:

[0016] ● Provides image pairs, each pair including a reference image and a corresponding denoised image;

[0017] ●For each image pair:

[0018] ○ The image pairs are fed into a pre-trained CNN architecture, which is similar to the learned CNN architecture used to estimate the degradation generated by the denoiser on the ray-traced images;

[0019] ○ Calculate the difference between the first normalized feature of the denoised image and the second normalized feature of the reference image, wherein the first normalized feature and the second normalized feature are the outputs of the same layer of the pre-trained CNN architecture;

[0020] ○ Calculate an error graph representing the calculated difference; and

[0021] ○ The corresponding denoised image and the error map are added to the corresponding noise image of the dataset and the reference image, wherein the noise image is the image from which the denoised image is obtained;

[0022] - The CNN architecture is learned based on the provided dataset, the learning comprising: taking an image generated by the denoiser and a corresponding noisy image of the dataset as input, and outputting an error map.

[0023] The method may also include one or more of the following:

[0024] ■ The learned CNN architecture also takes an albedo map and a normal buffer as input for each denoised image in the dataset.

[0025] ■The learning described is supervised learning.

[0026] ■ The CNN architecture being learned is the same as the pre-trained CNN architecture and has the same number of layers. All layers of the CNN architecture that were learned after the layers of the pre-trained CNN architecture that generated the first normalized feature and the second normalized feature are removed.

[0027] ■ The calculation error graph includes:

[0028] - Downsample the resolution of the calculated difference; and

[0029] - The error map is calculated using the downsampled resolution of the difference, and each pixel of the error map has a color that follows the color scale of the difference calculated by the penalty, thereby obtaining a coarse error map.

[0030] ■ The pre-trained CNN architecture and the CNN architecture being learned are both Visual Geometry Group (VGG) architectures.

[0031] ■ The Visual Geometry Group (VGG) architecture is the VGG-19 architecture, and wherein the first normalized feature of the denoised image and the second normalized feature of the reference image are extracted from the 7th and / or 9th and / or 12th layers of the VGG-19 architecture.

[0032] It also provides a CNN architecture that can be learned according to the method.

[0033] A method for implementing the CNN architecture using a computer is also provided. The method includes:

[0034] - Provide a first Monte Carlo ray-traced image and a second Monte Carlo ray-traced image rendered from the same lens, wherein the first Monte Carlo ray-traced image and the second Monte Carlo ray-traced image are different;

[0035] - Provide a first denoised image of the first image;

[0036] - An error map is obtained by applying the first image and the first denoised image to the CNN architecture;

[0037] - Calculate a blending map based on the obtained error map, the blending map representing the signal-to-noise ratio (SNR) level in dB for each region of the error map;

[0038] - Estimate the variance from the first image and the second image;

[0039] - Calculate the number of Monte Carlo samples required to indicate the signal-to-noise ratio level based on the estimated variance between the first and second images;

[0040] - Render a third Monte Carlo ray-traced image of the lens using the number of Monte Carlo samples identified from the sample image;

[0041] - Provide a rendered third Monte Carlo-rendered ray-traced image as input to the denoiser to obtain the corresponding denoised image; and

[0042] - The third Monte Carlo rendered ray-traced image is blended with the corresponding denoised image obtained, the blending being weighted for each region in the blended image having a signal-to-noise ratio level above a threshold in Bass.

[0043] The use of the method may include one or more of the following:

[0044] ■ Calculating the mixture plot based on the obtained error plot includes:

[0045] - Normalize the error map by applying the sigmoid function to the error map; and

[0046] - The blending map is calculated by calculating the decibel value of each region of the error map based on the target root mean square error between the normalized error map and the target error map.

[0047] ■ The first Monte Carlo ray-traced image provided has been rendered using a first number of Monte Carlo samples, and the second rendered Monte Carlo ray-traced image has been rendered using a second number of Monte Carlo samples, wherein the first number of Monte Carlo samples is greater than or equal to the second number of Monte Carlo samples.

[0048] ■ The number of the second Monte Carlo sample is half the number of the first Monte Carlo sample.

[0049] ■ The first Monte Carlo ray tracing image provided has been rendered according to the first random seed, and the second Monte Carlo ray tracing image provided has been rendered according to the second random seed, wherein the number of the first Monte Carlo samples is equal to the number of the second Monte Carlo samples.

[0050] A computer program is also provided, which includes instructions for performing any of the methods described.

[0051] A computer-readable data storage medium is also provided, having a computer program and / or neural network recorded thereon.

[0052] A computer-implemented method is also provided for forming a dataset configured to learn a convolutional neural network (CNN) architecture. The CNN architecture includes an image feature extractor. The method includes:

[0053] - Provides image pairs, each pair including a reference image and a corresponding denoised image;

[0054] -For each image pair:

[0055] ● The image pairs are provided to a pre-trained CNN architecture, which is similar to the CNN architecture that the resulting dataset will be configured for.

[0056] ● Calculate the difference between the first normalized feature of the denoised image and the second normalized feature of the reference image, wherein the first normalized feature and the second normalized feature are the outputs of the same layer of the pre-trained CNN architecture;

[0057] ● Calculate an error graph representing the difference being calculated; and

[0058] ● Add the corresponding denoised image and the error map to the dataset.

[0059] The method may also include one or more of the following:

[0060] ■ The calculation of the error graph includes:

[0061] - Downsample the resolution of the calculated difference; and

[0062] - The error map is calculated using the downsampled resolution of the difference, and each pixel of the error map has a color that follows the color scale of the difference calculated by the penalty, thereby obtaining a coarse error map.

[0063] ■ The pre-trained CNN architecture is the Visual Geometry Group (VGG) architecture.

[0064] ■ The Visual Geometry Group (VGG) architecture is the VGG-19 architecture, and wherein the first normalized feature of the denoised image and the second normalized feature of the reference image are extracted from the 7th and / or 9th and / or 12th layers of the VGG-19 architecture.

[0065] ■The additions include:

[0066] - Add the corresponding denoised image of the reference image, the error map, and the corresponding noise image to the dataset, wherein the noise image is an image from which the denoised image has been obtained;

[0067] ■The corresponding denoised image is the result of a lower sample count image of the reference image input to the denoiser and denoised by the denoiser;

[0068] A data structure for representing a dataset that can be formed according to the method is also provided.

[0069] A computer-implemented method is also provided using the dataset represented in the data structure. The method includes: learning a convolutional neural network (CNN) structure based on the dataset.

[0070] The use of the computer-implemented method may include one or more of the following:

[0071] ■The learning described is supervised learning;

[0072] ■ The CNN architecture being learned is the same as the pre-trained CNN architecture and has the same number of layers. All layers of the CNN architecture are learned after the layers of the pre-trained CNN architecture that generated the first normalized feature and the second normalized feature are removed.

[0073] A computer program is also provided, which includes instructions for performing any of the methods described.

[0074] A computer-readable data storage medium is also provided, having computer programs and / or data results recorded thereon.

[0075] A dataset that can be formed according to the method is also provided. Attached Figure Description

[0076] Embodiments of this disclosure will now be described by way of non-limiting example and with reference to the accompanying drawings, in which:

[0077] - Figures 1 to 3 A flowchart of the method is shown;

[0078] - Figure 4 An example of the system's graphical user interface is shown;

[0079] - Figure 5 An example of the system is shown;

[0080] - Figure 6 and 7 This demonstrates the detection of noise reduction error using existing techniques.

[0081] - Figure 8 An example of a CNN architecture is shown;

[0082] - Figure 9 An example of a pre-trained CNN architecture is shown;

[0083] - Figure 10 An example of a coarse error plot is shown;

[0084] - Figures 11 to 13 An example of an error plot calculated according to this method is shown;

[0085] - Figure 14 An example of a CNN architecture used for learning is shown;

[0086] - Figure 15 An example of a CNN architecture that utilizes supervised learning is shown;

[0087] - Figures 16 to 18 An example of an error map estimated by a CNN architecture learned based on the formed dataset is shown;

[0088] - Figure 19 Examples of noisy images with corresponding signal-to-noise ratios in dB and corresponding reference images are shown;

[0089] - Figure 20-21 Examples of images denoised according to different target signal-to-noise ratios in Babel are shown. Detailed Implementation

[0090] The provided method relates to rendering ray-traced images. It is well known that the rendering process typically involves a denoising process using a denoiser. A “denoiser” is any algorithm that represents a series of computations, wherein at least a portion of the algorithm takes a ray-traced image as input, which is rendered using an arbitrary number of samples or a sampling strategy. The denoiser applies a series of signal processing computations on the input ray-traced image. For example, the denoiser may apply filters to the ray-traced image. In the field of signal processing, particularly in applications of denoising ray-traced images, the computation set follows the techniques of posterior methods. The specific computations performed on the ray-traced image by the denoiser are not the subject of this invention, and therefore details of their application are omitted. A standard reference for posterior methods and other methods for denoising is Zwicker, M., Jarosz, W., Lehtinen, J., Moon, B., Ramamoorthi, R., Rousselle, F., Sen, P., Soler, C., Yoon, S.-E., (2015), Recent Advances in Adaptive Sampling and Reconstruction for Monte Carlo Rendering. The corresponding denoised image corresponds to the output of the denoiser, whose input corresponds to the result of the calculation applied by the denoiser, and whose input is an image sampled from the same rendering equation as the reference image.

[0091] Reference Figure 1 The flowchart below describes a computer-implemented method for learning a convolutional neural network (CNN) architecture to estimate the degradation of an image generated by a denoiser from a ray-traced image. CNN architectures are well-known in the field and have been applied in a wide range of applications. A standard reference for CNNs is Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998), applied to gradient-based learning for document recognition. This learning method involves providing a Q10 dataset. The method also includes making Q20 learn the CNN architecture based on the formed dataset. The learning involves taking an image generated by the denoiser and a noisy image of its corresponding dataset as input and outputting an error map. The learning can be performed in any way known from state-of-the-art CNN architectures to output an error map that estimates the loss of detail in the image generated by the denoiser relative to its input to the corresponding noisy image of the denoiser.

[0092] The estimate of the degradation can be represented by an error map that distinguishes regions where the image rendered by the denoiser has lost detail, such as fine elements being blurred. As is known in the field of machine learning, this learning is performed by learning a CNN architecture based on a provided dataset. The provided dataset is now configured to implement this learning. As is known in the field of machine learning, the processing of the input (i.e., the image generated by the denoiser) by a CNN architecture involves applying operations to the input, defined by data including weight values. Therefore, the learning of the neural network involves determining the values ​​of the weights based on a dataset configured for this learning. For this purpose, the provided dataset includes data fragments that form each corresponding training sample. The training samples represent the diversity of situations in which the neural network is used after being learned. Any dataset mentioned herein may include a number of training samples greater than 1000, 10000, 100000, or 1000000.

[0093] Therefore, the dataset that provides Q10 to the learning method is obtained in a way that achieves the expected quality of the estimate (i.e., the data fragments that form the dataset are obtained). The dataset that provides Q10 to the learning method can be formed according to the "dataset formation method" and will be discussed later.

[0094] At least a portion or all of the provided dataset is obtained by providing image pairs, each pair including a reference image and a corresponding denoised image. The provided dataset includes data segments (i.e., training samples) computed based on the image pairs provided to obtain (i.e., form) the dataset. Data segments are obtained by feeding the image pairs to a pre-trained CNN architecture, which is similar to a learned CNN architecture used to estimate the degradation generated by the denoiser on ray-traced images. The system then uses the pre-trained CNN architecture to perform a series of computations on the provided image pairs to obtain the data segments. Obtaining the provided dataset also includes: computing the difference between a first normalized feature of the denoised image and a second normalized feature of the reference image, where the first and second normalized features are outputs of the same layer of the pre-trained CNN architecture. Next, an error map is computed. The error map represents the computed difference. Data segments are obtained by adding the corresponding denoised image, the error map, and a corresponding noisy image of the reference image to the dataset, from which the denoised image has been obtained.

[0095] Based on the dataset that provides Q10 to the learning method, learning Q20 involves taking the image generated by the denoiser and the noisy image of its corresponding dataset as input. Learning Q20 also includes outputting an error map and can be performed using any means known in the field of machine learning (e.g., supervised learning).

[0096] Therefore, the CNN architecture learned according to the learning method is particularly efficient and provides improved accuracy for estimating the degradation generated by the denoiser on ray-traced images. This is because the CNN architecture is first learned offline based on the provided dataset and then simply applied in the online stage without further optimization. Furthermore, due to the diversity of data in the provided dataset configured for learning, the error map output by the CNN architecture accurately estimates the degradation generated by the denoiser without requiring visual comparison or verification of the error with respect to a reference image.

[0097] We will now discuss a computer-implemented method for forming a dataset configured to learn a CNN architecture, which may be referred to as a "dataset forming method." The learning method and / or the dataset forming method can be part of a machine learning process. A machine learning process can significantly include both a dataset forming method and a learning method. For example, in examples following the dataset forming method, the learning method can be performed at least in part based on the dataset formed by the dataset forming method. This machine learning process is particularly efficient and provides improved accuracy.

[0098] The method is implemented by a computer. This means that the steps (or substantially all steps) of the method are executed by at least one computer or any similar system. Therefore, the steps of the method are executed by a computer (potentially fully or semi-automatically). In the example, the triggering of at least some steps of the method can be performed through user-computer interaction. The required level of user-computer interaction may depend on the anticipated level of automation and be balanced with the need to fulfill the user's wishes. In the example, this level may be user-defined and / or predefined.

[0099] A typical example of a computer implementation of a method is to utilize a system suitable for this purpose to execute the method. This system may include a processor coupled to memory and a graphical user interface (GUI), the memory having a computer program recorded thereon containing instructions for executing the method. The memory may also store a dataset. The memory is any hardware suitable for such storage and may comprise several physically distinct parts (e.g., one for the program and one possibly for the dataset).

[0100] Ray-traced images can consist of a scene comprising one or more modeling objects, which may be designed using a CAD system or generated from designs created using a CAD system. A CAD system is defined as any system suitable for designing modeling objects based on their graphical representations, such as CATIA. In this case, the data defining the modeling objects includes data that allows for the representation of the modeling objects. For example, a CAD system can provide a representation of a CAD modeling object using edges or lines (in some cases, faces or surfaces). Lines, edges, or surfaces can be represented in various ways, such as non-uniform rational B-splines (NURBS). Specifically, a CAD file contains specifications from which geometry can be generated, thereby allowing for the generation of representations. The specifications of the modeling objects can be stored in a single CAD file or multiple CAD files. The typical size of a file representing a modeling object in a CAD system is in the range of 1 megabyte per part. And a modeling object can often be a component composed of thousands of parts.

[0101] In the context of CAD, modeling objects can typically be 3D modeling objects, such as representing products, like parts or part assemblies, or possibly product components. The term "3D modeling object" refers to any object that is modeled through data, thus allowing for its 3D representation. 3D representation allows parts to be viewed from all angles. For example, when performing a 3D representation, a 3D modeling object can be manipulated and rotated around any of its axes or around any axis on the screen displaying the representation. This specifically excludes 2D icons that are not 3D modeled. The display of 3D representations aids in design (i.e., increases the speed at which designers can statistically complete their tasks). This accelerates the manufacturing process in the industry, as product design is part of the manufacturing process.

[0102] 3D modeling objects can represent the geometry of a product that will be manufactured in the real world after its virtual design (e.g., using CAD software solutions or CAD systems) is completed. This could include (e.g., mechanical) parts or part assemblies (or assemblies equivalent to parts, since from a methodological perspective, a part assembly might be considered as the part itself, or the method could be applied independently to each part within the assembly), or more generally, any rigid body assembly (e.g., mobile machinery). CAD software solutions allow for product design across a wide and virtually limitless range of industrial sectors, including: aerospace, architecture, construction, consumer goods, high-tech equipment, industrial equipment, transportation, and marine and / or offshore oil / gas production or transportation. Therefore, the 3D modeling objects designed using this method can represent industrial products, which can be any mechanical parts, such as parts for ground transportation vehicles (e.g., including automobiles and light truck equipment, racing cars, motorcycles, trucks and motorized equipment, trucks and buses, trains), parts for aircraft (e.g., including fuselage equipment, aerospace equipment, propulsion equipment, defense products, aviation equipment, space equipment), parts for naval transportation vehicles (e.g., including naval equipment, merchant ships, marine equipment, yachts and workboats, marine equipment), general mechanical parts (e.g., including industrial manufacturing machinery, heavy mobile machinery or equipment, installation equipment, industrial equipment products, metal manufacturing products, tire manufacturing products), electromechanical or electronic parts (e.g., including consumer electronics products, safety and / or control and / or instrumentation products, computing and communication equipment, semiconductors, medical devices and equipment), consumer goods (e.g., including furniture, home and garden products, leisure goods, fashion products, products of hard goods retailers, products of soft goods retailers), and packaging (e.g., including food, beverage and tobacco, beauty and personal care, household product packaging).

[0103] Figure 2 An example of the system's GUI is shown, where the system is a CAD system.

[0104] The GUI 2100 may be a typical CAD-like interface with standard menu bars 2110, 2120 and bottom and side toolbars 2140, 2150. These menus and toolbars contain a set of user-selectable icons, each associated with one or more operations or functions known in the art. Some of these icons are associated with software tools suitable for editing and / or manipulating the 3D modeled object 2000 displayed in the GUI 2100. These software tools may be grouped into workbenches. Each workbench consists of a subset of the software tools. In particular, one of the workbenches is an editing workbench, suitable for editing the geometry of the modeled product 2000. In operation, the designer may, for example, pre-select a portion of the object 2000 and then initiate an operation (e.g., change dimensions, color, etc.) or edit geometric constraints by selecting an appropriate icon. For example, a typical CAD operation is modeling a 3D modeled object displayed on the screen by stamping or folding. The GUI may, for example, display data 2500 related to the displayed product 2000. In this example diagram, the data 2500, displayed as a "feature tree," and its 3D representation 2000 relate to a brake assembly including brake calipers and brake discs. The GUI can further illustrate various types of graphical tools 2130, 2070, and 2080, such as those for facilitating 3D orientation of objects, triggering operational simulations for editing the product, or rendering various properties of the displayed product 2000. A cursor 2060 can be controlled by a haptic device to allow the user to interact with the graphical tools.

[0105] Figure 3 An example of the system is shown, where the system is a client computer system, such as a user's workstation.

[0106] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000, and random access memory (RAM) 1070 also connected to the bus. The client computer is also equipped with a graphics processing unit (GPU) 1110, which is associated with video random access memory 1100 connected to the bus. The video RAM 1100 is also referred to in the art as a frame buffer. A mass storage device controller 1020 manages access to mass storage devices such as a hard disk drive 1030. Mass storage devices suitable for tangibly representing computer program instructions and data include all forms of non-volatile memory, including, for example: semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROMs 1040. Any of the above may be supplemented or incorporated by a specially designed ASIC (Application-Specific Integrated Circuit). A network adapter 1050 manages access to a network 1060. The client computer may also include a haptic device 1090, such as a cursor control device, a keyboard, etc. A cursor control device is used on the client computer to allow the user to selectively position the cursor at any desired location on the monitor 1080. Furthermore, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes multiple signal generating devices for inputting control signals to the system. Typically, the cursor control device can be a mouse, with mouse buttons used to generate signals. Alternatively or additionally, the client computer system may include a sensitive pad and / or a sensitive screen.

[0107] A computer program may include computer-executable instructions, including units for causing the system to perform the method. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, in digital circuitry or in computer hardware, firmware, software, or a combination thereof. The program may be implemented as an apparatus, such as a product tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps may be executed by a programmable processor executing the instruction program to perform the function of the method by manipulating input data and generating output. Therefore, the processor may be programmable and coupled to receive data and instructions from the data storage system, at least one input device, and at least one output device, and to send data and instructions to the data storage system, at least one input device, and at least one output device. The application program may be implemented using a high-level program or an object-oriented programming language, or, if desired, assembly language or machine language. In any case, the language may be a compiled or interpreted language. The program may be a complete installation program or update program. In any case, the application of the program on the system results in instructions for performing the method.

[0108] Return reference Figure 1 , now discuss a computer-implemented method for performing learning (also referred to as a "learning method") for learning a convolutional neural network architecture for estimating degradation generated by a denoiser for a ray-traced image. The learning method includes providing a dataset Q10.

[0109] The provided dataset Q10 can be obtained according to the computer-implemented method. Referring Figure 2 to the flowchart, an example of a method for obtaining a dataset configured to be provided, which is referred to as a "dataset formation method", is further discussed. The provided dataset may have been formed at different times, different locations, using different systems, and / or by different people or entities using the dataset formation method.

[0110] The dataset formation method includes providing an image pair S10. The images of the provided image pair can be ray-traced images. The ray-traced images can include a scene with one or more objects. The ray-traced images can include different objects or phenomena, such as a camera model that determines how and where to view the scene. The ray-traced images are rendered using Monte Carlo techniques. As is well known in the art, Monte Carlo rendering is based on numerical integration of random samples using the corresponding rendering equation. For example, the rendering equation describes the amount of light reaching the camera (in the camera model included in the example) from a point on a scene object as the sum of the light emitted by the object (if the object itself is a light source) and the amount of light reflected by the object. The random samples can be set from a predetermined number (e.g., a positive integer n). The Monte Carlo renderer will perform n sampling steps, for example, as an iterative process, where at any step s (where 0 < s < n), a sample is obtained. As is known in the field of Monte Carlo rendering, increasing the number of samples can make the Monte Carlo rendering more accurate; as the number of steps increases, the Monte Carlo rendering converges to an ideal image corresponding to the exact solution of the corresponding rendering equation, that is, the Monte Carlo samples converge to the underlying distribution of the corresponding rendering equation. For example, let s and s' be two numbers such that 0 < s < s' < n, then the ray-traced image obtained using s Monte Carlo samples is less detailed (i.e., less convergent) than the ray-traced image obtained using s' Monte Carlo samples. The specific way of obtaining the samples is not the subject of this disclosure and is only discussed for illustration. For example, the ray-traced image obtained using s Monte Carlo samples will have more noise in regions corresponding to objects far from the camera, or distort image regions with fine structures, such as leather structures or text in the image. Since small details and / or distant objects have high variance, the denoiser will degrade the corresponding regions. For example, such degradation will be the blurring of the fine structure of the image or the loss of details of objects far from the camera.

[0111] Each image pair includes a reference image and a corresponding denoised image. The "reference image" is a Monte Carlo-rendered ray-traced image whose sampling is substantially close to the fundamental distribution to reduce variance. "Substantially close" means that the fundamental distribution of the corresponding rendering equation of the image, which the system can determine via predetermined criteria, is substantially approximated by a discrete distribution obtained through Monte Carlo sampling. Consequently, the variance of the discrete reference image relative to the distribution of the corresponding rendering equation is substantially small. The "corresponding denoised image" is another Monte Carlo ray-traced image, which is the result of inputting to the denoiser; this other image is sampled from the same corresponding rendering equation of the reference image. Specifically, the number of samples in the other image is less than or equal to the number of samples in the reference image. This other image is also referred to as the noisy image of the reference image. For example, the noisy image can be obtained by sampling s = 512 samples, and its corresponding reference image can be obtained by sampling s = 131072 samples or more.

[0112] In the example, the corresponding denoised image is the result input to the denoiser, and another image is sampled according to the same corresponding rendering equation of the reference image, where the number of samples is strictly less than the number of samples in the reference image. The user can input any number of samples, strictly less than the number of samples in the reference image. Therefore, the resulting dataset can include data segments showing relatively high variance variations due to noise, thus providing a CNN architecture to learn to distinguish the degradation performed by the denoiser among several values ​​of noise variation. The other image can also be referred to as the "noisy image." Figure 11-13 An example of an image set including a reference image, a corresponding noisy image, and a denoised image is shown, wherein the denoised image is the result of inputting the noisy image into a denoiser and a corresponding error map. For example, Figure 11 A noisy image of a scene is shown, which includes several elements such as curtains on the side and a sculpture at the far end of the camera's viewpoint. This particular example is a high-count sample, making it difficult to verify that the noisy image is degraded by denoising. The corresponding error plot illustrates the degradation. Figure 12 It is a scaled-down version of the same scene that includes the curtain area. Figure 13 It is a scaled-down version of the same scene, including the sculpture at the far end of the camera's viewpoint.

[0113] Once image pairs are provided, the dataset forming method computes a dataset of denoised images. For each image pair provided, the next step of the dataset forming method is performed. The dataset forming method provides the image pairs to a pre-trained CNN architecture (S20), similar to the architecture to which the formed dataset will be configured. Similarly, this means that the pre-trained CNN architecture includes at least one common layer with an image feature extractor that will be learned based on the formed dataset. A common layer is defined as a layer that provides the same output for the same input. The common layer is used in computation (S30). The pre-trained CNN architecture may include all common layers with image feature extractors that will be learned based on the formed dataset. As is known in the neural network field, an "image feature extractor" consists of a set of layers, each configured to output features corresponding to the input image. The features corresponding to the input image (also called feature maps) are known in the neural network field (especially convolutional neural networks) and correspond to a two-dimensional permutation of values ​​corresponding to the outputs of the corresponding layers. Features can be represented by the values ​​of the two-dimensional permutation (i.e., a matrix). Each value corresponds to a feature value extracted by the corresponding layer. For example, a layer can apply a convolution operation to an input image or the output of another layer. The output of a convolution operation (typically a convolution with a filter matrix in this example) corresponds to a feature map. A "fully connected image classifier" refers to a layer in a CNN architecture that takes the feature map output by an image feature extractor as input and outputs a label for the corresponding image.

[0114] Next, the dataset formation method calculates the difference between the first normalized feature of the S30 denoised image and the second normalized feature of the reference image, where the first and second normalized features are the outputs of the same layer in a pre-trained CNN architecture. The difference can be any difference between the values ​​of the first and second normalized feature maps corresponding to the same layer. For example, the difference can be calculated using the Euclidean distance between the matrix values ​​of the features or any other distance metric (such as the absolute norm). A "normalized feature" refers to a value that is adjusted (e.g., rescaled) with respect to the same numerical magnitude as the output of the layer corresponding to the feature. In other words, normalization considers the features before calculating the difference, thus assigning equal weight to the numerical values ​​of the features extracted from the denoised image and the reference image. Figure 9An example of the difference between the first normalized feature of the denoised image and the second normalized feature of the reference image is shown by obtaining the output of a layer from a pre-trained VGG-19 architecture. The denoised image is input into a pre-trained CNN architecture. Each layer of the pre-trained CNN architecture extracts features. The seventh layer to the right of the input image outputs a feature map of the reference image consisting of 256 channels, where the x-direction is 1 / 4 of the input image resolution and the y-direction is 1 / 4 of the input image resolution. The second feature of the reference image is output from the same seventh layer to the right. Before calculating the difference, the two outputs are normalized, which is done in... Figure 9 The difference is indicated by the green section in the middle. For example, the difference in normalized features output from the seventh layer may also include the difference in other normalized features output from other layers (such as the ninth layer, whose output is a feature map of a reference image consisting of 256 channels, where the x-direction is 1 / 4 of the input image resolution and the y-direction is 1 / 4 of the input image resolution; and / or the 12th layer, whose output is a feature map of a reference image consisting of 512 channels, where the x-direction is 1 / 8 of the input image resolution and the y-direction is 1 / 8 of the input image resolution). Optionally, if the layers have different resolutions, average pooling can be applied to set the outputs of the layers to the same resolution. Figure 10 An example is shown of the error map obtained by applying average pooling after calculating the difference (calculated in step S40).

[0115] Next, the dataset formation method computes an error map representing the computed difference, S40. The error map is an arrangement of values, such as two-dimensional values, where each element is a numerical value corresponding to the difference between a first normalized feature of the denoised image and a second normalized feature of the reference image. By convention, the error map can be constructed by establishing the computed difference as the difference between the first normalized feature and the second normalized feature of the reference image. Therefore, each value within the error map represents an uncorrelated distance between the second normalized feature of the reference image and the first normalized feature of the denoised image. Thus, the error map can be interpreted as a measure of the degree to which the denoiser preserves detail, based on the corresponding features of the reference image. The error map can be an image where each pixel has a color value, where each color value corresponds to the difference between the computed values ​​of the first and second normalized features, which correspond to the output of the same layer. Therefore, the error map can be viewed as a graph indicating the degree to which the denoised image conforms to the reference image (with respect to its respective first and second normalized features). Therefore, a lower error corresponding to a smaller difference corresponds to better consistency between the denoised image and the reference, thus instructing the denoiser to retain sufficient detail after denoising. By convention, the error map can consist of non-negative values, where large positive values ​​correspond to large differences, and small values ​​will have small non-negative values. Error maps with zero values ​​will consist of zero or any minimum value chosen by convention. For example, pixels in the error map can have grayscale values, where 0 corresponds to no error and 255 corresponds to the maximum difference. Any other color levels can be defined by convention. The difference may be unbounded, but the method will likely always map values ​​above a threshold to the maximum allowed color level; this is simply an implementation issue.

[0116] The dataset formation method then adds the corresponding denoised images and error maps to the dataset S50. Therefore, the dataset formation method configures the dataset for learning a CNN architecture to estimate the error caused by denoising. This CNN architecture uses segments of the dataset as training samples, where learning is performed based on the error maps corresponding to the corresponding denoised images. The training samples in the added dataset represent various applications of the CNN architecture after it has been learned, i.e., its corresponding online application after the offline training phase in the machine learning process.

[0117] In the example, the method also adds the corresponding denoised image, error map, and corresponding noisy image of the reference image to the dataset S50. The noisy image is the image from which the denoised image has already been obtained. Typically, it is an image with a lower Monte Carlo sample value than the corresponding reference image, and therefore exhibits noise. The noisy image is fed to the denoiser to reduce noise, and the denoised image is the result of the denoising process of the denoiser. The dataset formed by this method allows supervised learning of CNN architectures (as known from the field of convolutional neural networks) to learn to estimate the error of the denoised image output by the denoiser without accessing the reference image. This is particularly important in real-world scenarios where access to the reference image is computationally forbidden.

[0118] In the example, this method can also create a dataset suitable for supervised training. The method sets at least a portion of the data in the dataset's data structure to serve as ground truth data for training in supervised mode. (See also...) Figure 15 , Figure 15 It is a machine learning process used for supervised learning of functions. Figure 15 The upper part of the diagram illustrates an example of ground truth data. A noisy image and feature buffer obtained through Monte Carlo rendering are fed into a denoiser to generate a denoised image. The denoised image and a reference image are then provided to a dataset forming method. The dataset forming method outputs an error map, which is then used as ground truth data. This method can generate any amount of ground truth data required for supervised learning. The method can store the dataset in memory for later use within the same system or send it to another system. In offline learning, which can be performed within the same system or another system, the CNN architecture can be learned in a supervised manner by comparing the output of the CNN architecture with the generated ground truth data.

[0119] In the example, the dataset formation method can create a data structure representing the dataset. Therefore, the formed dataset can be represented within this data structure. This data structure can be stored in non-volatile memory or transferred via a network to another computer system, allowing for subsequent use. For example, the dataset can be organized as EXR image files to allow high dynamic range (HDR) content to be stored on non-volatile memory (e.g., a disk), where data on the disk is organized into folders per shot, with filenames containing descriptive suffixes describing the content.

[0120] Return to reference Figure 1The learning method also includes learning the Q20 CNN architecture based on the provided dataset. Learning may involve taking an image generated by the denoiser and a corresponding noisy image from the dataset as input, and outputting an error map. Learning can be performed using any means known for state-of-the-art CNN architectures to output an error map that estimates the loss of detail in the image generated by the denoiser relative to its corresponding noisy image input to the denoiser. Therefore, the CNN architecture is learned to estimate the degradation caused by denoising, as the provided dataset is configured to allow the CNN architecture to learn from training samples for this purpose. In fact, all the added training samples in the dataset represent the diversity of applications of the CNN architecture after it has been learned (i.e., its corresponding online application after offline training), and may include, for example, scenes with one or more objects or different viewpoints of the same scene.

[0121] Learning Q20 involves outputting an error map. As is essentially known from the field of machine learning, the CNN architecture learns to improve its output based on training samples from the dataset provided in step Q10. Therefore, the error map in the dataset provided in step S10 learns the CNN architecture to distinguish regions of a denoised image degraded by a denoiser, where regions with higher degradation correspond to the set of pixel map values ​​with substantially high error. "Substantially high error" means that the pixel map values ​​are above a certain threshold that can be conventionally established. These regions may include fine structures or objects found far from the camera viewpoint, which can be better distinguished based on the diversity of the training samples. As is essentially known from the field of machine learning, the provided dataset affects the learning speed and quality of the CNN architecture, i.e., the accuracy with which the learned CNN architecture estimates the error map. The dataset can be formed using the total number of data points depending on the expected learning quality. This number can be greater than 1,000, 10,000, or 100,000 elements. The amount of data in the dataset follows a trade-off between the accuracy and learning speed achieved by the CNN architecture. For example, 500 data snippets would correspond to fast learning, while 100,000 data snippets would provide high-quality learning. In another example, approximately 2,500 data snippets are needed to learn a CNN architecture within 24 hours, where learning is performed through supervised learning.

[0122] Therefore, the CNN architecture learned according to this method estimates the degradation produced by the denoiser with the expected estimation accuracy (as expected from the sample size). Furthermore, since the resulting dataset provides a wide variety of training examples, the CNN architecture identifies regions with fine structural details or regions including objects found at a distance in the camera view.

[0123] In the example, the learned Q20 CNN architecture can also take an albedo map and a normal buffer as input for each denoised image in the dataset. Therefore, the number of input channels is the same as that used by state-of-the-art denoising systems. Thus, the method is able to detect details lost in fine structures. Indeed, as is known in the art, albedo and normal buffers are typically used by denoisers to better preserve details; therefore, the learned CNN architecture can estimate any further loss of detail caused by the denoiser.

[0124] In the example, learning Q20 can be performed as supervised learning. Therefore, this learning also includes using an error map as ground truth data for supervised learning. It is worth noting that the learning can use an error map from the dataset provided at step Q10 as ground truth data. This learning method can set a portion of the training samples from the provided dataset as ground truth data. Alternatively, the dataset provided by the data structure may have been arranged to provide data for use as ground truth data. Since the dataset provided at step Q10 can be formed from denoised images and corresponding reference image pairs, the error map is a suitable ground truth data reflecting the loss of detail with respect to the reference. A CNN architecture can be learned according to any method known in supervised learning, specifically the learning process of implementing details. Ray-traced images can include scenes, such as interior shots, product shots, or close-up shots. Therefore, a CNN architecture learned according to this method can detect problem areas in various scenes that present lost details.

[0125] In the example, the CNN architecture learned according to this learning method is the same as the pre-trained CNN architecture used to form the provided dataset, and has the same number of layers. All layers after the layers of the pre-trained CNN architecture that generate the first and second normalized features are removed when learning the CNN architecture. Training a CNN architecture can take a very long time (e.g., 24 hours or more), so using the same number of layers as the pre-trained CNN architecture to learn the CNN architecture saves computational time compared to including more layers.

[0126] In the example, the spatial resolution of the error map in the provided dataset can be adjusted to improve the detection of regions where the denoiser degrades the image by learning Q20. For example, the dataset forming method can compute an error map representing the computed difference, which may include first downsampling the resolution of the computed difference and then using the downsampled resolution of the difference to compute the error map, where each pixel of the error map has a color that follows a color scale that penalizes the computed difference. Thus, the spatial resolution of the error map is reduced, resulting in a coarse error map. The error map can be downsampled by any known means (e.g., using average pooling). When learning the Q20 CNN architecture, the CNN architecture learning output is adjusted to an error map with the resolution of the error map in the provided dataset. Therefore, the CNN architecture learned based on the provided dataset is better suited for finding regions of lost detail according to the expected resolution of the error map. In fact, according to the convention of color scale setting, a high spatial resolution map can contain very small spatial variations between regions with high and low errors, i.e., a few pixels in the error map showing a color with high error are surrounded by multiple pixels showing low or no error. If the image resolution is too large, the CNN architecture may be more accurate, but more samples will be needed to identify minute spatial variations in the error. Conversely, a coarse error map shows coarse spatial variations; that is, pixels with high errors are averaged out, making them constitute a larger spatial region of the image, thus facilitating the learning of the CNN architecture. Downsampling can be implemented in any way to achieve a good trade-off between the accuracy and learning time of the CNN architecture.

[0127] In the example, the pre-trained CNN architecture used to form the provided dataset, and the CNN architecture learned according to the learning method, can use any state-of-the-art CNN architecture, such as the Visual Geometric Group (VGG) architecture. The CNN architecture comprises different layers, which may include convolutional layers, pooling layers, fully connected input layers, fully connected output layers, etc. The details of the composition of this CNN architecture are well known and are not further elaborated. The different layers of the VGG architecture allow for the extraction of information (i.e., features or deep features) from the corresponding input images.

[0128] In the examples, good results were observed when the Visual Geometry Group (VGG) architecture was the VGG-19 architecture. The VGG-19 architecture can be used in both the formation and learning methods of the machine learning process.

[0129] The main architecture behind the VGG19 neural network is described in Simonyan, K. & Zisserman, A. (2015), Ultra-deep convolutional networks for large-scale image recognition. In the example, the VGG-19 architecture, pre-trained to form the provided dataset, can be pre-trained based on commonly available datasets such as the ImageNet image dataset. Figure 8 The main components of the VGG-19 architecture are shown. The VGG-19 architecture is a state-of-the-art architecture utilizing improvements to convolutional neural networks; it consists of 16 convolutional layers (denoted as conv), 3 fully connected layers (denoted as FC), 5 max-pooling layers (denoted as pooling), and 1 softmax layer (denoted as softmax). By convention, layers are numbered from left to right, starting with 1. Because it was pre-trained on the ImageNet image dataset, VGG-19 yields particularly accurate features. In fact, ImageNet is an image dataset organized according to the nouns in the WordNet hierarchy, where each node of this hierarchy is depicted by hundreds or thousands of images. On average, each node in ImageNet contains over 500 images.

[0130] Reference Figure 9 , 14 Examples 1 and 15 are provided, which respectively illustrate pre-trained CNN architectures ( Figure 9 CNN architecture that learns according to the learning method Figure 14 ) and example machine learning process ( Figure 15 The pre-trained CNN architecture used to form the dataset according to the dataset formation method and the CNN architecture to be learned to estimate the degradation due to noise reduction are based on the VGG-19 architecture in these figures. Figure 15 The machine learning process is illustrated, including the learning method (prediction part) based on the dataset formed by the dataset formation method (ground truth part). Both the pre-trained CNN architecture used to form the dataset and the CNN architecture learned through the learning method are the same VGG-19 architecture with the same number of layers.

[0131] Figure 9 A diagram of the pre-trained VGG-19 architecture, used as a metric for obtaining the error map, is shown. Therefore, the error map is computed based on the difference between a first normalized feature of the denoised image and a second normalized feature of the reference image; the first and second normalized features are the outputs of the same layer of the pre-trained VGG-19 architecture. In the example, and returning the reference... Figure 9The first normalized features of the denoised image and the second normalized features of the reference image are extracted from layers 7 and / or 9 and / or 12 of the image feature extractor, which are conventionally extracted from layers 7 and / or 9 and / or 12. Figure 9 The layers depicted are enumerated from leftmost to rightmost. All grayed-out layers are removed. Features are normalized to equalize the size of the outputs from different layers. For example, the layers of the image feature extractor in the VGG-19 architecture include downsampling steps, with different spatial resolutions for layers 7 and / or 9 and / or 12. In fact, layers 7 and 9 both output feature maps of a reference image consisting of 256 channels, where the x-direction is 1 / 4 of the input image resolution and the y-direction is 1 / 4 of the input resolution, while layer 12 outputs feature maps of a reference image consisting of 512 channels, where the x-direction is 1 / 8 of the input image resolution and the y-direction is 1 / 8 of the input image resolution. Therefore, the error map can be downsampled for each channel at the same resolution to obtain a coarse error map, for example, 1 / 32 or less of the input resolution. Figure 10 A coarse error map obtained through feature normalization is shown, with downsampling performed via averaging before calculating the error map. The coarse error map shows the same weights for the outputs of layers 7 and / or 9 and / or 12, which was found to best identify degradation between the reference image and the denoised image. The error map can also be downsampled to provide a coarser error map and thus better identify regions of degradation due to the denoiser.

[0132] Reference Figure 14 The CNN architecture being learned to estimate the degradation generated by the denoiser uses the same layers as the pre-trained VGG-19 architecture without adding a normalization stage, and is learned in a supervised manner based on ground truth data set from the provided dataset. In this particular example, the CNN architecture also takes albedo and normal buffer as input. This can be understood as an implementation choice.

[0133] In the example, good results are found when the CNN architecture learned according to the learning method is the VGG-19 architecture (including layers 0 to 12 of the VGG-19 architecture). The CNN architecture can be learned from a provided dataset (e.g., a dataset formed according to a dataset formation method) that includes thousands of shots (i.e., training samples), each shot containing various objects in different lighting scenes with multiple structural details. For example, objects in the scene can include leather, carpet, stonework, wallpaper, rough metal, metal floors, or plastic objects with fine details. Each shot can include a reference image, a denoised image, and an error map. A portion of the error map in the dataset is reserved as ground truth data for supervised learning. In the example, the VGG-19 feature extractor is trained according to a training method known in the art in a supervised manner. Optimal results are found when the VGG-19 feature extractor also includes albedo and normal buffers as input.

[0134] Reference Figure 3 The flowchart above describes the process of using a CNN architecture learned through a learning method. The learned architecture can be referred to as the "estimator architecture." The method used employs the learned estimator structure for adaptive denoising of ray-traced images. The basic idea is to sample the ray-traced image using a first number of samples and estimate the variance to compute a sample map that indicates how many samples are needed to achieve a specific level of detail. The application of the sample map is performed based on a blending map, which indicates regions of the image that are not sufficiently convergent in the Monte Carlo sampling sense. This is done by computing a blending map that distinguishes image regions that have not converged significantly based on the target signal-to-noise ratio (SNR) in dB. Next, the sample map determines the number of samples required to achieve better detail for regions with high SNR levels relative to the blending map. The method used is explained further below.

[0135] The method includes providing a first Monte Carlo ray-traced image and a second Monte Carlo ray-traced image rendered from the same shot by P10. A shot is defined as a ray-traced image comprising a scene having one or more objects to be rendered using Monte Carlo techniques. The first Monte Carlo ray-traced image differs from the second Monte Carlo ray-traced image; the difference indicates that both the first and second Monte Carlo ray-traced images were rendered from the same shot but using different sampling procedures, for example, by setting different initial conditions (e.g., random seed), different sampling numbers, or different sampling strategies. As is inherently known from the field of Monte Carlo methods, the sampling process of an image is random, i.e., the way samples of the rendering equations for the shot are obtained follows a probabilistic (i.e., nondeterministic) criterion. Therefore, due to the different sampling procedures performed, the images exhibit different regions of rendering noise. Monte Carlo samples can be obtained through any known sampling procedure, e.g., uniform sampling from a random seed. The system can obtain both images using any method known in the art. The system can perform Monte Carlo sampling from the corresponding shot based on the rendering equations of the shot stored in non-volatile memory. For example, the system can be configured to perform Monte Carlo rendering on a first image and Monte Carlo sampling on a second image in a separate process within the same system. It should be understood that this is only one implementation option. Alternatively, the first and second images can be obtained from non-volatile memory or received from another computer system, for example, via a network. Due to the different sampling, the first and second images exhibit noise in different regions; the method uses the first and second images to estimate how many samples are needed to achieve appropriate lens quality, which will be further explained in the steps of the method used.

[0136] The method used also includes providing a first denoised image of the first image P20. Any known denoising method known in the art can be used. Next, the first image and the first denoised image are applied to the learned estimator architecture to obtain an error map P30. Each pixel of the error map is rendered with a color indicating the error due to denoising, where the error is estimated by the estimator architecture. The error map can include regions of the same error, i.e., consecutive sets of pixels in the error map with the same (or substantially the same) value. For example, the error map can be a coarse map of the average error map, thus providing a larger area than a fine map.

[0137] The method also includes calculating a P40 blending map based on the obtained error map. The blending map represents the signal-to-noise ratio (SNR) in dB for each region of the error map. As is known in the art, loss of detail is directly related to the SNR of an image. Large errors due to denoising correspond to low SNR, while low errors correspond to high SNR. Therefore, the blending map indicates the dB level of the region corresponding to the error map caused by denoising. For example, the blending map can represent the peak SNR in dB calculated based on the target root mean square error, for example, by using the formula -20*log10(rmse), where rmse represents the target root mean square error. The system can predefine the target root mean square error. Therefore, the blending map represents multiple regions in the blending map that have dB values ​​lower than the desired SNR in dB required to achieve an accurate image. Therefore, the system can define a weighted blend based on the SNR in dB values ​​present in the blending map.

[0138] Next, the variance of P50 from the first and second images is estimated. The variance can be estimated from the first and second images in any way. The first and second images are generated from different sampling processes, and therefore exhibit noise in different regions due to the random nature of Monte Carlo sampling. The regions exhibiting noise in the first and second images can be correlated to any degree; for example, these regions can be independent (in a probabilistic sense) or substantially related. The variance is estimated from the noise in a combination of the first and second images. For example, the variance can be calculated from the image generated from (first image + second image) / 2.

[0139] Based on the estimated variance from the first and second images, the method used can also compute a P60 sample map. The sample map indicates the number of Monte Carlo samples required to achieve the signal-to-noise ratio (SNR) level needed to achieve adequate quality. The SNR level can be set by the user or the system. The sample map can be computed using the estimated variance and Monte Carlo convergence properties. The system can determine the number of remaining samples to reduce the variance between the first and second images using any method known in the art that uses Monte Carlo convergence properties. The number of samples required to achieve the SNR level can be higher (e.g., substantially higher) than the number of samples performed to render the first and second images.

[0140] Next, a third Monte Carlo ray-traced image of the P70 lens is rendered using the sample image. The third image is rendered using the number of Monte Carlo samples identified from the sample image. The third image is sampled according to the number of samples required to reach the signal-to-noise ratio (SNR) level. Therefore, the sampled third image is more detailed than the first and second images, and adds detail in a particularly efficient manner. In fact, since sampling is performed based on the number of samples required to reach the SNR level, the method focuses only on adding samples in the regions where the SNR level indicated by the sample image needs to be reached. This is particularly efficient compared to other sampling methods (e.g., uniform sampling, in which sampling is performed uniformly across the domain of the image, and therefore may be performed where further detail addition is not required).

[0141] Next, the third image is provided as input to P80 to the denoiser, thereby obtaining the corresponding denoised image.

[0142] The resulting denoised image (corresponding to the third image) is blended with the third image (P90). The third image has better detail compared to the first image, but still requires denoising. Blending is now performed so that only the portions of the denoised image with less noise are blended according to the signal-to-noise ratio (SNR) in dB. This is done based on a blending map. In practice, the blending is weighted for each region in the blending map with an SNR level above a threshold in dB. For example, the method might blend regions with SNR values ​​above 40 dB; the method might not attempt to blend below that threshold because the noise in the first denoised image would be too great to provide acceptable results. The threshold can be set based on the desired quality; this is just one implementation option.

[0143] Therefore, the method used improves the denoising process to enhance the quality of ray-traced images without incurring the high computational cost of rendering high-resolution or reference images. This is because the denoising process is optimized to only denoise regions that will not suffer quality loss during denoising. In fact, the method detects problem regions in the image via an error map, thus focusing on adaptively distributing a large number of samples to areas that will not be denoised. User intervention is no longer required, as the process is fully automated, thus avoiding time-consuming manual checks. Furthermore, the rendering cost is reduced because the sample distribution calculated from the sample map is more efficient.

[0144] Due to the variability of noise, the error map obtained from the learned estimator structure may present regions with large or even unbounded errors. In the example, the method used can compute a mixture map based on the obtained error map by normalizing the error map. Normalization can be performed by applying the sigmoid function to the error map. As is well known in engineering, the sigmoid function is a bounded, smoothing real function defined for all real input values. The sigmoid function takes the error value at each pixel of the error map as input, for example, from values ​​corresponding to the color scales of the error map. The sigmoid function has a non-negative derivative at each point and an inflection point. The properties of the sigmoid function for mapping unbounded errors are as follows: any value mapped by the sigmoid function below the inflection point is small, and the value decreases significantly as the mapped value goes further below the inflection point. Therefore, the sigmoid function attenuates very small noise. The degree of attenuation of small noise depends on the definition of the inflection point, and as is known in the art, the inflection point can be adaptively adjusted. Large noise (e.g., unbounded noise) is noise mapped above the inflection point. In this context, it is well known that the sigmoid function can be viewed as a smoothing saturation function that maps large errors up to the maximum allowed by the sigmoid function. In the example, the sigmoid function can be chosen to normalize the error values ​​within a bounded interval (e.g., the interval [0,1]). In the example, the sigmoid function is either a hyperbolic tangent function or a logistic function. Therefore, the sigmoid function can map unbounded values ​​to the bounded interval [0,1]. Since unbounded values ​​are mapped in the blending graph, the blending graph is well defined in all its regions. This method computes the blending graph by calculating the decibel value for each region of the error graph based on the target root mean square error between the normalized error graph and the target error graph. The user can establish a blending threshold based on the calculated decibel level. For example, regions with low signal-to-noise ratio decibel levels can be ignored due to excessive noise. Therefore, regions with high signal-to-noise ratio decibel levels can be distinguished and further improved. Thus, the method can be better focused on improving regions with low errors.

[0145] In the example, a first image can be rendered using a first number of Monte Carlo samples, and a second image can be rendered using a second number of Monte Carlo samples. "Render using..." indicates that the rendering is based on either the first or the second number, which can include further adaptation based on the implementation of the sampling. The first number of Monte Carlo samples can be greater than or equal to the second number of Monte Carlo samples. For example, the first and second numbers can be equal; however, because the sampling process is probabilistic, the samples provide different renderings. The first number of Monte Carlo samples can be greater than the second number of Monte Carlo samples. Therefore, variance can be viewed as an indicator of the noise level of the first image, and the method can be viewed as improving the detail regarding the first image.

[0146] The method used can obtain the second image by using the same sampling process used to obtain the first image. In the example, a first Monte Carlo sample number is set, and the sampling process begins from the first number. The sampling process then saves the second image corresponding to half of the first number of samples. Therefore, the second image is rendered using a second Monte Carlo sample number, where the second number is half of the first Monte Carlo sample number. The rendering of the first image can correspond to the rendering using the first sample number, or by further adaptively adjusting the rendering using the first sample number. Therefore, this way of rendering the first and second images is a particularly efficient sampling process for obtaining the first and second images, because the rendering of both is optimized to be performed in a single run in the system. For example, the system can set the first number to, for example, n = 128 samples, and start the sampling process. Then, the image corresponding to the intermediate step of performing the sampling process (i.e., s = 64 samples) is stored in memory as the second image; next, the system continues sampling until n samples are reached. Then, the image obtained using the first number of n samples is further adaptively adjusted by removing the 64 samples s corresponding to the second image. For example, this adaptive adjustment can be performed by multiplying the image corresponding to n = 128 samples by 2 and subtracting the second image corresponding to 64 samples s. The arithmetic operation is understood as an operation performed using a matrix of images. The result is an image with samples s' = ns, corresponding to the rendering of the first image after adaptive adjustment to the first number of samples. The resulting first image (after adaptive adjustment) is independent of the second image's samples s (corresponding to s' samples), i.e., by construction, the noise between the two images is uncorrelated. This achieves the same result as rendering two independent images with the same number of samples, but avoids restarting the sampling process. Therefore, this method allows rendering the first and second images without restarting the sampling process to perform rendering, requiring only further storage of the intermediate results of the sampling process (i.e., the second image), making this method better integrated into existing rendering architectures. For example, uniform rendering of Monte Carlo images is an iterative process where an additional sample is accumulated to each pixel of the image in each iteration. This means that at each point in the sampling time, according to uniform sampling, only the accumulated version of the image exists. Therefore, by obtaining the second image through the same process as the first image, the second image can be obtained simply by removing samples from the first image. If the first image is adaptively adjusted by removing samples from the second image, the first and second images are independent, which in turn makes it easier to calculate the variance and thus improves the rendering quality.

[0147] In the example, a first Monte Carlo ray-traced image can be rendered based on a first random seed, and a second Monte Carlo ray-traced image can be rendered based on a second random seed. The random seed is the initial condition used to begin any sampling strategy chosen for performing sampling. As is known in the Monte Carlo rendering field, a first random seed, different from the second random seed, produces different samples of the same shot, resulting in different renderings of the same shot. The number of first Monte Carlo samples can be equal to the number of second Monte Carlo samples. Because the first and second random seeds are different, this ensures that noise present in the first image after the sampling process is independent of noise present in the second image; however, the same number of samples is still required.

[0148] Examples of forming datasets

[0149] Refer to sections 8 through 13 for a discussion of combinations of the examples discussed below.

[0150] First, we discuss the metric used to compute the error map. This metric uses features from the VGG19 network (pre-trained on ImageNet) to compare images. The VGG19 network consists of a convolutional feature extractor and a fully connected classifier. Figure 8 A flowchart of the layers of the VGG19 architecture is shown. The metric computes features from a dataset containing images with varying levels of detail, along with their corresponding denoised and reference images. In the tested implementation, layers 7, 9, and 12 of the VGG19 architecture were found to be the layers that best extracted features of structural detail.

[0151] Before calculating the squared difference between the denoised image and the reference image, the corresponding features are normalized to balance the size of the outputs from different layers. Figure 9 The calculation of the difference in normalized features is shown. This applies to all layers after layer 12 in the VGG-19 architecture (i.e., Figure 9 The medium gray layer is discarded from the measurement and is no longer used.

[0152] Because the network includes a downsampling step, the spatial resolution of different layers differs. By averaging to 1 / 32 of the input resolution, the spatial resolution of different layers is reduced. The averaged output is then averaged by channel to obtain a coarse error map. Figure 10 The flowcharts for downsampling at different layers and the resulting rough error plots are shown.

[0153] Figure 11-13 An error graph calculated based on the metric is shown. Figure 11The diagram shows a reference image, a noisy image with fewer samples compared to the reference image, a denoised image as the output of the denoiser, and a coarse error map calculated based on the normalized difference of features from the VGG19 image. The error map shows areas of low error in darker regions and areas of high error in brighter regions. This metric identifies the blurring caused by denoising on the blue curtain. Figure 12 The corresponding zoomed-out portion of the same image is shown, revealing the loss of detail in areas on the curtains and in fine structural features on the floor, such as cracks in the floor. Figure 13 This shows a scaled version of an object (statue) located away from the camera view. The error graph indicates the loss of detail in such objects.

[0154] Error plot estimation

[0155] Refer to 14 to 18 for a discussion of the combinations of examples discussed below.

[0156] Since reference images are typically not present in the real-world scene, an error map needs to be generated in the absence of a reference image. A convolutional neural network is trained to estimate the error map. To train the network, a dataset with error maps computed based on a metric is created. The dataset contains 2437 shots, encompassing various objects in different lighting scenes with all kinds of structural detail, such as leather, carpet, stonework, wallpaper, rough metal, metal floors, or fine details in plastic. Each shot is rendered using a reference image with a different sample count (1-512) and 131072 samples per pixel (in iterations of rendering). The metric described above is used to generate ground truth outputs. The network is then trained in a supervised manner for approximately 24 hours. In the tested implementation, the training settings offer a good trade-off between the time spent training the dataset and the accuracy of the trained network architecture. Figure 14 The network architecture is shown. The network architecture is similar to the VGG19 feature extractor, using layers 0 through 12 of the feature extractor in the metric. The main difference lies in the number of input channels, as the denoised image and the same inputs (noise, albedo, and normal buffer) used by the denoiser are fed into the network. Figure 15 Supervised learning is illustrated. The trained network directly outputs an error map corresponding to the input image. Figure 16-18 The estimated error plot obtained through the tested implementation is shown. Figure 16 The images correspond to shots of interior scenes. The image on the left corresponds to a noisy image with low sample counts. Continuing clockwise, the second image corresponds to the denoised image. For ease of comparison, an error map corresponding to comparing the denoised image with a reference error map is also shown in the lower right, while the estimated error map (also known as the prediction error map) is shown in the lower left. Figure 16-18 The good consistency between the two error plots is shown.

[0157] Adaptive noise reduction system

[0158] Refer to 3 and Figures 19 to 20 Discuss combinations of the examples discussed below.

[0159] The following example illustrates an adaptive denoiser for adaptive image denoising. This example uses a low sample count for the variance estimator but requires a sufficiently high sample count to render the sample map, e.g., enough to achieve approximately 50+ dB in relevant regions. (See reference...) Figure 3 In step P10, the system uses a low sample count (e.g., 64 SPP) to uniformly render the scene. During rendering, intermediate images are also stored at half the sample count required to estimate the variance (32 in this case). The 32 and 64 SPP images are fed to the variance estimator P50 to estimate the noise level of the image. The 64 SPP image, along with the corresponding normal and albedo feature buffers, is fed to the denoiser P20 to generate denoising input for the error estimation network. The resulting error map and the variance estimation output from P50 are then used to compute the adaptive sample map and the final blending map. The tanh function is applied to the squared error map to normalize the errors in the error map to the real range [0,1]. Next, the normalized error map is converted into a blending map containing the required decibel level for blending. The peak signal-to-noise ratio (PSNR) in decibels is calculated based on the target root mean square error (rmse) value using the following formula: -20*log10(rmse). Depending on the rendering computation budget, these decibel levels may be below our blending threshold, thus preventing blending in regions that are not sufficiently convergent. Different decibel levels are used to define the blending limits. This implementation performs linear blending starting at 40 dB (0 blending) and continuing up to 50 dB (full blending). The tested implementation compared several noise decibel levels to define the blending threshold. Figure 19 An example of a noisy image with a signal-to-noise ratio (SNR) of dB relative to a reference image is shown. Images with 40 dB or less of noise have higher noise levels, and details increase as the dB value increases.

[0160] A sample map P60, indicating the number of samples needed to reach the desired decibel level, can be calculated using a variance estimator and Monte Carlo convergence properties. The sample map P60 is then fed to the renderer, which adaptively continues rendering based on it. The renderer's output is denoised; the denoised image, along with the rendered output, is provided to a mixing step P90. The mixing step P90 blends the rendered output and the denoised version based on a mixing map. With roughly the same computational cost, this system achieves significantly better image quality compared to denoising only. Figure 20 In the lens shown, the error is reduced by approximately 40% relative to the measurement. Figure 21 In China, the decrease was approximately 27%.

Claims

1. A computer-implemented machine learning method for learning a convolutional neural network (CNN) architecture for estimating degradation generated from ray-traced images by a denoiser, the method comprising: - Provide the dataset obtained through the following operations: • Provide image pairs, each pair including a reference image and a corresponding denoised image, wherein the reference image is a ray-traced image rendered in Monte Carlo, and the corresponding denoised image is the result of inputting a corresponding noisy image of the reference image into a denoiser; For each image pair: ○ The image pairs are fed into a pre-trained CNN architecture, which is similar to the learned CNN architecture used to estimate the degradation generated by the denoiser on the ray-traced images; ○ Calculate the difference between a first normalized feature of the denoised image and a second normalized feature of the reference image, wherein the first normalized feature and the second normalized feature are the outputs of the same layer of the pre-trained CNN architecture; ○ Calculate an error graph representing the calculated difference; as well as ○ Add the corresponding denoised image and the error map to the corresponding noise image of the dataset and the reference image, wherein the noise image is the image from which the denoised image has been obtained; - The CNN architecture is learned based on the provided dataset, the learning process including: taking the image generated by the denoiser and the corresponding noisy image in the dataset as input, and outputting an error map.

2. The method according to claim 1, wherein, The learned CNN architecture also takes an albedo map and a normal buffer as input for each denoised image in the dataset.

3. The method according to any one of claims 1 to 2, wherein, The learning described is supervised learning.

4. The method according to any one of claims 1 to 2, wherein, The CNN architecture being learned is the same as the pre-trained CNN architecture and has the same number of layers. All layers of the CNN architecture that were learned after the layers of the pre-trained CNN architecture that generated the first normalized feature and the second normalized feature are removed.

5. The method according to any one of claims 1 to 2, wherein, Calculating the error graph includes: - Downsample the resolution of the calculated difference; and - The error map is calculated using the downsampled resolution of the difference, and each pixel of the error map has a color that follows the color scale of the difference calculated by the penalty, thereby obtaining a coarse error map.

6. The method according to any one of claims 1 to 2, wherein, The pre-trained CNN architecture and the CNN architecture being learned are Visual Geometry Group (VGG) architectures.

7. The method according to claim 6, wherein, The Visual Geometry Group (VGG) architecture is the VGG-19 architecture, and wherein the first normalized feature of the denoised image and the second normalized feature of the reference image are extracted from the 7th and / or 9th and / or 12th layers of the VGG-19 architecture.

8. A client computer system, comprising: Central processing unit; as well as A storage device for storing a CNN architecture, including a visual geometry group (VGG) architecture, wherein the visual geometry group (VGG) architecture is learned by the method according to any one of claims 1 to 7.

9. A computer-implemented method for presenting a ray-traced image, the method comprising: - Provide a first Monte Carlo ray-traced image and a second Monte Carlo ray-traced image rendered from the same lens, wherein the first Monte Carlo ray-traced image and the second Monte Carlo ray-traced image are different; - Provides a first denoised image of the first Monte Carlo ray-traced image; - An error map is obtained by applying the first Monte Carlo ray-traced image and the first denoised image to the CNN architecture; - Calculate a blending map based on the obtained error map, the blending map representing the signal-to-noise ratio (SNR) level in dB for each region of the error map; - Estimate the variance from the first Monte Carlo ray-traced image and the second Monte Carlo ray-traced image; - Calculate the sample map indicating the number of Monte Carlo samples required to achieve the signal-to-noise ratio level based on the estimated variance between the first Monte Carlo ray-traced image and the second Monte Carlo ray-traced image; - Render a third Monte Carlo ray-traced image of the lens using the number of Monte Carlo samples identified from the sample image; - Provide a rendered third Monte Carlo ray-traced image as input to the denoiser to obtain the corresponding denoised image; as well as - The third Monte Carlo rendered ray-traced image is blended with the corresponding denoised image obtained, the blending being weighted for each region in the blended image having a signal-to-noise ratio level above a threshold in Bass.

10. The method according to claim 9, wherein, Calculating the mixture map based on the obtained error map includes: - Normalize the error map by applying the sigmoid function to the error map; and - The hybrid map is calculated by calculating the decibel value of each region of the error map based on the target root mean square error between the normalized error map and the target error map.

11. The method according to any one of claims 9 or 10, wherein, The provided first Monte Carlo ray-traced image has been rendered using a first number of Monte Carlo samples, and the second Monte Carlo ray-traced image has been rendered using a second number of Monte Carlo samples, wherein the first number of Monte Carlo samples is greater than or equal to the second number of Monte Carlo samples.

12. The method according to claim 11, wherein, The number of the second Monte Carlo sample is half the number of the first Monte Carlo sample.

13. The method according to claim 11, wherein, The provided first Monte Carlo ray tracing image has been rendered according to a first random seed, and the provided second Monte Carlo ray tracing image has been rendered according to a second random seed, wherein the number of the first Monte Carlo samples is equal to the number of the second Monte Carlo samples.

14. A computer program product comprising computer instructions, wherein the computer instructions, when executed by a processor, perform the method according to any one of claims 1 to 7 or the method according to any one of claims 9 to 13.

15. A computer-readable data storage medium having a computer program product recorded thereon according to claim 14.

16. The computer-readable data storage medium of claim 15, wherein the learned CNN architecture is stored on the computer-readable data storage medium.

Images