Radiation-based up-sampling of digital material models

Abstract

In implementations of techniques for up-sampling a digital material model based on radiation, a computing device implements an up-sampling system to receive an input digital material model having a first resolution. An up-sampling system generates bilinear up-sampled texels based on an input digital material model having a first resolution. The upsampling system then generates texels having a second resolution higher than the first resolution based on the bilinear upsampled texels using a machine learning model trained on training data to generate texels. Based on the texels having the second resolution, the up-sampling system generates an output digital material model having a resolution higher than the first resolution.

Description

Background Art

[0001] In computer graphics, materials are described by a set of graphs, which are two-dimensional (2D) images used to model the visual details of virtual three-dimensional (3D) objects. For example, materials are applied to the surface of a virtual 3D object to simulate the appearance of different physical materials (including wood, metal, fabric, or stone). Digital material models customize the spatial variation characteristics of materials in a virtual environment and are created using image editing applications and then applied to virtual 3D objects within a 3D rendering or game engine. Different digital material models are combined to obtain the visual illusion of complex materials without modeling every small detail. However, conventional techniques for generating digital material models often lead to errors and cause visual inaccuracies and computational inefficiencies in real-world scenes. Summary of the invention

[0002] Techniques and systems for upsampling a digital material model based on radiation are described. In one example, an upsampling system receives an input digital material model having a first resolution. In some examples, the input digital material model having the first resolution includes a base color map, a normal map, a metal map, a roughness map, and a height map. Based on the input digital material model having the first resolution, the upsampling system generates bilinear upsampled texels.

[0003] The upsampling system also generates texels having a second resolution higher than the first resolution based on the bilinearly upsampled texels using a machine learning model trained on training data to generate texels. For example, the training data includes renderings calculated based on texels sampled from the image upsampler at different light positions.

[0004] The upsampling system then generates an output digital material model based on the texels having the second resolution, the output digital material model having a resolution higher than the first resolution. The high-resolution digital material model is then output for display in a user interface. In some examples, the output digital material model is applied to a three-dimensional geometry.

[0005] This Summary introduces a series of concepts in a simplified form, which are further described in the Detailed Description below. Therefore, this Summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The detailed description is described with reference to the accompanying drawings. The entities represented in the drawings refer to one or more entities, and thus singular or plural forms of the entities may be referred to interchangeably in the discussion.

[0007] Figure 1is an illustration of a digital media environment in an example implementation operable to employ techniques and systems for radiometrically upsampling a digital material model as described herein;

[0008] Figure 2 Depicted is a system in an example implementation showing operation of an upsampling module for upsampling a digital material model based on radiometry;

[0009] Figure 3 An example of an input digital material model is depicted;

[0010] Figure 4 An example of the architecture of an upsampling module is depicted;

[0011] Figure 5 An example is depicted including an output table including different high resolution digital material models;

[0012] Figure 6 Depicts a process in an example implementation of radiosity-based upsampling of a digital footage model;

[0013] Figure 7 depicts a process in an additional example implementation for upsampling a digital footage model based on radiosity; and

[0014] Figure 8 An example system is shown that includes various components of an example device that can be implemented as described in reference Figure 1-Figure 7 Any type of computing device described and / or used to implement embodiments of the techniques described herein. DETAILED DESCRIPTION

[0015] Overview

[0016] The digital material model is composed of multiple two-dimensional (2D) image maps used to define the appearance of a three-dimensional (3D) object in a virtual environment. The digital material model is applied to the surface of the 3D model to provide details of the surface that simulate the texture of real-world materials, including wood, metal, fabric, stone, or other materials on the 3D object. In this example, the digital material model is a compilation of maps that specify base colors, normal properties, metallicity, roughness, and height, which drive a standard microfacet reflection model.

[0017] The digital material model is applied to the 3D object by mapping the pixels of the 2D image onto the surface of the 3D object, the mapping including assigning coordinates to the vertices of the 3D object that determine how the digital material model wraps around the surface of the 3D object. Because the digital material model is a low-resolution sample of the material, applying the digital material model to the 3D object involves upsampling the digital material model. Upsampling the digital material model increases the resolution of the digital material model.

[0018] A conventional technique for upsampling digital material models employs bilinear upsampling. Bilinear upsampling estimates the value of a new pixel by considering the surrounding pixels in a 2×2 neighborhood. For example, each pixel in the original image is replaced with a 2×2 grid of new pixels, resulting in an image that is twice as wide and tall. The value of the new pixel is determined by averaging the values ​​of the four nearest pixels to the original image. Bilinear upsampling increases resolution (which measures the total number of pixels) but does not increase sharpness (which measures the number of pixels per square unit of measurement).

[0019] However, conventional bilinear upsampling does not take into account the accurate exposure of the digital asset model to different radiances, which are based on the exposure of the digital asset model to different virtual lighting environments. Therefore, the bilinearly upsampled digital asset model using conventional techniques behaves unrealistically when exposed to various light sources and lighting directions in the virtual environment, thereby revealing the low-resolution characteristics of the original asset. For example, after bilinear upsampling the digital asset model, the digital asset model with a virtual texture representing a tile appears blurry in terms of gloss, shadows, and highlights from different light directions.

[0020] Techniques and systems for upsampling digital material models based on radiation are described, which address technical challenges of conventional upsampling techniques. In this example, an upsampling system begins by receiving an input digital material model, which includes but is not limited to a base color map, a normal map, a metal map, a roughness map, and a height map.

[0021] The upsampling system then renders and upsamples multiple images representing different radiances of the input digital material model. The radiance corresponds to the response of the material relative to the input digital material model at a specific light position. The upsampling system then uses an RGB upsampler to generate a super-resolution digital material model based on each of the multiple upsampled images representing different radiances of the input digital material model. The machine learning model is then trained on the collection of material examples coupled to the super-resolution digital material model.

[0022] The upsampling system also generates a bilinear upsampled digital material model based on the input digital material model, the model including bilinear upsampled versions of a base color map, a normal map, a metal map, a roughness map, and a height map. Based on a window of the bilinear upsampled digital material model corresponding to a given texel of the bilinear upsampled digital material model, the upsampling system uses a multi-layer perceptron (MLP) upsampler to reconstruct a high-resolution value of the bilinear upsampled texel, which is a set of digital material characteristics corresponding to a pixel of the digital material model. The upsampling system then applies the machine learning model to the bilinear upsampled texels to generate high-resolution texels. The machine learning model is trained on a super-resolution digital material model, which means that the machine learning model is trained on multiple upsampled versions of radiation rendered from the digital material model at a set of different light positions.

[0023] Based on the high-resolution texels, the upsampling system generates a high-resolution digital material model. The high-resolution digital material model produces realistic material when exposed to different light positions in the virtual environment. The upsampling module then generates an output including the high-resolution digital material model for display in the user interface. For example, the high-resolution digital material model is incorporated into a three-dimensional virtual object to add a detailed virtual texture to the three-dimensional virtual object presented in the user interface.

[0024] Upsampling the digital asset model based on radiation in this manner overcomes the disadvantage of conventional bilinear upsampling applications that do not account for different super-resolution radiation. For example, upsampling the digital asset model based on radiation can generate high-resolution texels that also exhibit higher clarity using a machine learning model trained on multiple upsampled renderings of the input digital asset model, the multiple upsampled renderings corresponding to different light positions relative to the input digital asset model. This results in the high-resolution digital asset model appearing realistic when exposed to different lighting directions in the virtual environment, which is not possible when using conventional bilinear upsampling applications that do not account for different super-resolution radiation.

[0025] In the following discussion, an example environment is described that employs the techniques described herein. Example procedures that can be performed in the example environment as well as other environments are also described. Therefore, the execution of the example procedures is not limited to the example environment, and the example environment is not limited to the execution of the example procedures.

[0026] Example Environment

[0027] Figure 1 is an illustration of a digital media environment 100 in an example implementation operable to employ techniques and systems for radiometrically based upsampling of digital material models as described herein. The illustrated digital media environment 100 includes a computing device 102 that is configurable in a variety of ways.

[0028] For example, computing device 102 may be configured as a desktop computer, a laptop computer, a mobile device (e.g., assuming a handheld configuration such as a tablet or mobile phone), an augmented reality device, etc. Thus, computing device 102 ranges from full-resource devices with large amounts of memory and processor resources (e.g., personal computers, game consoles) to low-resource devices with limited memory and / or processing resources, such as mobile devices. Furthermore, although a single computing device 102 is shown, computing device 102 also represents a plurality of different devices, such as Figure 8 Multiple servers described in used by an enterprise to perform "on-cloud" operations.

[0029] The computing device 102 also includes an image processing system 104. The image processing system 104 is implemented at least in part in hardware of the computing device 102 to process and represent digital content 106, which is shown as being maintained in a storage device 108 of the computing device 102. Such processing includes creating the digital content 106, representing the digital content 106, modifying the digital content 106, and rendering the digital content 106 for display in a user interface 110 for output by, for example, a display device 112. Although shown as being implemented locally at the computing device 102, the functionality of the image processing system 104 may also be configurable in whole or in part through functionality available via a network 114, such as part of a web service or "in the cloud."

[0030] Computing device 102 also includes upsampling module 116, which is shown as being incorporated by image processing system 104 to process digital content 106. In some examples, upsampling module 116 is separate from image processing system 104, such as in examples where upsampling module 116 is accessible via network 114.

[0031] The upsampling module 116 is configured to receive an input 118 including an input digital material model 120 having a first resolution 122, and generate a high-resolution digital material model 124 having an updated resolution 126 higher than the first resolution 122. The input digital material model 120 is a collection of two-dimensional (2D) images that include the visual appearance and properties of a three-dimensional (3D) surface or object rendered in the user interface 110. For example, the input digital material model 120 is used to simulate the properties of a texture when applied to the surface of a 3D object, including base color, normal properties, metallicity, roughness, and height. Additionally or alternatively, the input digital material model 120 includes an embedding of another image describing material properties. However, the input digital material model 120 in this example has a first resolution 122, which is a low resolution and does not produce a realistic rendering when applied to the surface of a 3D object in the user interface 110.

[0032] To generate a high-resolution digital asset model 124 having an updated resolution 126 that is higher than the first resolution 122, the upsampling module 116 generates a super-resolution digital asset model using a super-resolution algorithm based on super-resolution radiances corresponding to different light positions. For example, the super-resolution digital asset model includes information related to a plurality of up-sampled versions of the high-resolution digital asset model 124, each up-sampled version corresponding to a different light position relative to the high-resolution digital asset model 124.

[0033] The upsampling module 116 also generates bilinear upsampled texels. A texel is a basic unit of the input digital material model 120, including a single pixel in some examples. The upsampling module 116 generates bilinear upsampled texels by increasing the size and resolution of the texels of the input digital material model 120.

[0034] The upsampling module 116 then generates high resolution texels using the machine learning model. The machine learning model is trained on a super-resolution digital asset model, which means that the machine learning model is trained on multiple upsampled renderings corresponding to different light positions relative to the input digital asset model 120. Therefore, based on the upsampled radiance generated from the renderings of the input digital asset model 120 relative to the different light positions, high resolution texels are generated to mimic the behavior relative to the different light positions.

[0035] Then, the upsampling module 116 generates a high-resolution digital asset model 124 based on the high-resolution texels. For example, a plurality of high-resolution texels are generated using a machine learning model and stitched together to generate the high-resolution digital asset model 124. The high-resolution digital asset model 124 produces realistic assets when exposed to different light positions in the virtual environment.

[0036] The upsampling module 116 then generates an output 128 including the high-resolution digital asset model 124 for display in the user interface 110. For example, the high-resolution digital asset model 124 is incorporated onto a three-dimensional virtual object to add a detailed virtual texture to the three-dimensional virtual object presented in the user interface 110.

[0037] Generally, in the context of the example processes described in this section, the functions, features, and concepts described in connection with the examples above and below are employed. In addition, the functions, features, and concepts described with respect to the different figures and examples in this document are interchangeable with each other and are not limited to implementation in the context of a particular figure or process. In addition, the blocks and corresponding figures associated with different representative processes herein may be applied together and / or combined in different ways. Therefore, the individual functions, features, and concepts described herein with respect to different example environments, devices, components, figures, and processes may be used in any suitable combination and are not limited to the specific combinations represented by the examples listed in this specification.

[0038] Upsampling of digital material models based on radiosity

[0039] Figure 2 Depicted is a system 200 in an example implementation, which shows in more detail Figure 1 The following discussion describes techniques that can be implemented using the previously described systems and devices. Aspects of each of these processes are implemented in hardware, firmware, software, or a combination thereof. These processes are shown as a set of blocks that specify operations performed and / or caused by one or more devices, and are not necessarily limited to the order shown for performing the operations by the corresponding blocks. In portions of the following discussion, reference is made to Figure 1-Figure 8 .

[0040] In this example, to begin with, the upsampling module 116 receives an input digital material model 120. For example, the input digital material model 120 is a collection of 2D images used to define the appearance of a 3D object in a 3D rendering. The input digital material model 120 is applied to the surface of the 3D model to provide details to simulate the appearance of real-world materials (including wood, metal, fabric, or stone) on the digital object. In this example, the input digital material model 120 is a compilation of maps including a base color map specifying a base color, a normal map specifying a normal property, a metal map specifying a metallic property, a roughness map specifying a roughness, and a height map specifying a height. In one example, a radiation image is generated during rendering using the input digital material model 120. The input digital material model 120 in this example has a first resolution 122, which is a low resolution that does not produce realistic radiation when applied to the surface of a 3D object in the user interface 110.

[0041] The upsampling module 116 also includes a bilinear upsampling module 202. In this example, the bilinear upsampling module 202 generates bilinear upsampled texels 204. To this end, the bilinear upsampling module 202 operates on a texel-by-texel basis based on the input digital material model 120. A texel is a basic unit of the input digital material model 120. For example, the bilinear upsampling module 202 isolates a texel from the input digital material model 120 and generates the bilinear upsampled texel by increasing the size and resolution of the texel.

[0042] The upsampling module 116 also includes a super-resolution module 206. The super-resolution module 206 generates a super-resolution digital material model 208 based on the radiance corresponding to the different light positions using a super-resolution algorithm. The super-resolution algorithm generates the super-resolution digital material model 208 having a higher resolution level and a higher level of detail than the input digital material model 120. Figure 4 The super-resolution algorithm is described in detail. In addition, the super-resolution digital material model 208 includes information related to multiple upsampled versions of the input digital material model 120, each upsampled version corresponding to a different light position relative to the input digital material model 120. In other examples, the super-resolution module 206 generates the super-resolution digital material model 208 before the bilinear upsampling module 202 generates the bilinear upsampled texels 204.

[0043] The upsampling module 116 also includes a texel upsampling module 210. The texel upsampling module 210 uses a machine learning model 212 to generate high-resolution texels 214. The machine learning model 212 is trained on the super-resolution digital material model 208. Therefore, the machine learning model 212 is trained on multiple upsampled versions of radiation corresponding to different light positions relative to the input digital material model 120. Based on the behavior of the input digital material model 120 relative to the different light positions, the high-resolution texels 214 are generated to display realistic behavior when exposed to the different light positions. In some examples, the high-resolution texels 214 are generated using a transformer model that includes filters optimized for a range of radiation.

[0044] The upsampling module 116 also includes a digital material model generation module 216. The digital material model generation module 216 generates a high-resolution digital material model 124 based on the high-resolution texels 214. For example, a plurality of high-resolution texels are generated using the machine learning model 212 and incorporated into the high-resolution digital material model 124. The high-resolution digital material model 124 produces realistic material when exposed to different light positions in the virtual environment. The upsampling module 116 then generates an output 128 including the high-resolution digital material model 124 for display in the user interface 110. For example, the high-resolution digital material model 124 is incorporated into a three-dimensional virtual object to add a detailed virtual texture to the three-dimensional virtual object presented in the user interface 110. In some examples, parameter optimization is performed on the high-resolution digital material model 124.

[0045] Figure 3-Figure 5 Depicted are the stages of upsampling a digital material model based on radiosity. In some examples, the stages shown in these figures are performed in a different order than described below.

[0046] Figure 3 An example 300 of an input digital asset model 120 is depicted. The input digital asset model 120 is a 2D image that defines the visual characteristics of the surface of a 3D object. The input digital asset model 120 provides information about the color, reflectivity, transparency, roughness, and other surface characteristics of the object. The digital asset model is applied to the corresponding 3D model by mapping the pixels of the 2D image onto the surface of the 3D object. This process is called "UV mapping," where UV coordinates are assigned to each vertex of the 3D model and determine how the digital asset model wraps around the surface of the 3D object.

[0047] The input digital material model 120 includes multiple digital material models that define specific characteristics, including a base color map, a normal map, a metal map, a roughness map, and a height map. The base color map defines the color and overall appearance of the surface, and specifies the base color or texture applied to the 3D object. The normal map encodes information about the surface normal of the 3D object. Each pixel in the normal map represents a perturbation of the surface normal, which creates the illusion of fine details and surface bumps without actually modifying the geometry. The metal map specifies reflective highlights on the surface of the 3D object to indicate areas that are more or less reflective to create a visual illusion of gloss or glossiness. The roughness map simulates surface details. Instead of storing surface normals, the roughness map encodes height information as grayscale values ​​to create a visual illusion of bumps on the surface of the 3D object. The height map modifies the geometry of the 3D object by displacing the vertices of the 3D object based on the values ​​of the map. The height map adds additional depth and detail to the surface to create a visual illusion of height variation.

[0048] The input digital asset model 120 also stores information about how light interacts with the surface of the 3D object. Different channels of the input digital asset model 120 represent aspects of light interaction, including reflection, refraction, transparency, and roughness. For example, different lighting positions relative to the input digital asset model 120 result in different levels and positions of reflections, glare, reflections, shadows, and highlights on the input digital asset model 120, and information related to these interactions is stored in the input digital asset model 120.

[0049] In this example, the input digital asset model 120 features a user-designed pattern applied to the surface of a 3D object. For example, the input digital asset model 120 includes a base color map 302 , a normal map 304 , a metallic map 306 , a roughness map 308 , and a height map 310 .

[0050] Figure 4 An example 400 of the architecture of the upsampling module 116 is depicted. Figure 4 yes Figure 3 A continuation of the example described in .

[0051] At step 402, the upsampling module 116 receives an input digital material model 120, which includes a base color map, a normal map, a metal map, a roughness map, and a height map. Figure 3 For example, the input includes a set of low-resolution (LR) spatially varying bidirectional reflectance distribution (SVBRDF) maps M LR , whose size is 128 pixels square.

[0052] At step 404, the upsampling module 116 renders a plurality of images representing different radiances of the input digital material model 120. The radiances correspond to specific light positions relative to the input digital material model 120. To do this, the upsampling module 116 uses the radiances applied to the low-resolution input R(M LR ) of the shading model R. For example, the upsampling module 116 recovers the following high-resolution SVBRDFM SR SVBRDF parameters, including base color map, normal map, metal map, roughness map and height map.

[0053] At step 406, the super-resolution module 206 generates a super-resolution digital material model 208 for each of the multiple images representing different radiations of the input digital material model 120. The super-resolution module 206 uses an RGB upsampler or other natural image upsampler to obtain a super-resolution digital material model 208 for each radiation, which has a size of 512 pixels squared. The RGB upsampler is an algorithm for increasing the resolution of an image represented in an RGB (red, green, and blue) color format. The upsampling process includes analyzing existing RGB pixel values ​​and generating new pixels based on surrounding information. The super-resolution digital material model 208 is used to train the machine learning model 212, as described in detail below. In some examples, the super-resolution algorithm is SwinIR (Liang et al., 2021) or SRFlow (Lugmayr et al., 2020). For SwinIR, the learning weights of the large model SwinIRL_x4_GAN are trained for real-world images. For SRFlow, the weights are trained using SRFlow_DF2K_4X.

[0054] At step 408, the bilinear upsampling module 202 generates the bilinear upsampled texels 204. To this end, the bilinear upsampling module 202 first generates a bilinear upsampled digital material model based on the input digital material model 120, which includes bilinear upsampled versions of the base color map, normal map, metal map, roughness map, and height map.

[0055] At step 410, the bilinear upsampling module 202 identifies a window of size N×N corresponding to a given texel of the bilinear upsampled digital material model. The window includes channels corresponding to the base color map, normal map, metal map, roughness map, and height map connected as input to a multi-layer perceptron (MLP). The MLP generates the bilinear upsampled texels 204, which are a set of digital material models corresponding to 1×1 pixels.

[0056] At step 412, the texel upsampling module 210 generates high-resolution texels 214 using the machine learning model 212. The machine learning model 212 is trained on the super-resolution digital asset model 208, which means that the machine learning model 212 is trained on multiple upsampled versions of the input digital asset model 120 corresponding to different light positions relative to the input digital asset model 120. Based on the behavior of the input digital asset model 120 relative to the different light positions, the high-resolution texels 214 are generated to display realistic behavior when exposed to the different light positions.

[0057] At step 416, the digital asset model generation module 216 generates a high-resolution digital asset model 124 based on the high-resolution texels 214. For example, a plurality of high-resolution texels are generated using the machine learning model 212 and incorporated into the high-resolution digital asset model 124. The high-resolution digital asset model 124 produces realistic assets when exposed to different light positions in the virtual environment. The upsampling module 116 then generates an output 128 including the high-resolution digital asset model 124 for display in the user interface 110. For example, the high-resolution digital asset model 124 is incorporated into a three-dimensional virtual object to add a detailed virtual texture to the three-dimensional virtual object presented in the user interface 110.

[0058] The intermediate step 418 represents loss function training. The weights are trained so that the high-resolution digital asset model 124 matches how light interacts with the super-resolution digital asset model 208.

[0059] Here, the loss function for a given pixel p is:

[0060]

[0061] in

[0062] Ω is a set of selected light source directions ω2, and is the regularization loss.

[0063] Given a set of texels P = {p} (a number of pixels of one or more materials), the corresponding loss function are collectively minimized. While the upscaled renderings mostly present common structures, outliers occur. To address outliers, a two-level fitting strategy is applied to output a clean SVBRDF at both the pixel and asset levels. At the pixel level, a sparsity-induced loss is used that allows the relevant BRDF signal to be preserved even in the presence of such outliers, as described in detail below. At the asset level, the functions mapping inputs to outputs are co-optimized to achieve consistent upsampling of similar regions.

[0064] Regarding the loss due to sparsity, given P, the following loss is minimized:

[0065]

[0066] in is the sum of the l1 norm over all data fitting terms in P and the l2 squared norm over all regularization terms in P:

[0067] / / Data fitting caused by sparsity

[0068] / / Structural regularization prior

[0069] Regarding MLP-based optimization, a multi-layer perceptron (MLP) is used to compute the high-resolution graph M SR (p) for each texel p, where the entry is the above sampled bilinear A square window of size N×N centered on p of the input material. The MLP calculates the difference from the center texel. The machine learning model 212 then learns a filter that takes as input a bilinear upsampled digital material model and outputs a super-resolution texel, which is optimized for a specific pair of shading model R and radiation upsampler S.

[0070] Regarding height map reconstruction, the geometric mesostructure is captured by the normal map, on which the height map is reconstructed following the method of Martin et al. A normal integration scheme [Durou and Courteille 2007] is applied, which uses a Poisson solution. The low-resolution input height map is synthesized by high-pass filtering of the upsampled normal-based height channel with bilinear upsampling of the input, providing a dense anchor against low-frequency noise, as described by Nehab et al.

[2005] .

[0071] Regarding SVBRDF parameter regularization, the metalness map encodes the values ​​that deviate from their original low-resolution counterparts during the optimization process. To prevent further deviation from the original values, a regularization term is applied:

[0072]

[0073] Where α is the weighting parameter, Represents a low-resolution metal image m LR Bilinear upsampling of , and m SR is the optimized metallic map. This will produce values ​​that mimic the original metallic map while remaining sharper than the bilinearly upsampled input.

[0074] Conventional per-pixel optimization suffers from performance issues because existing RGB supersamplers make it difficult to process large inputs (e.g., 4096×4096 pixels), and such inputs cannot be optimized simultaneously by this approach due to the large amount of VRAM required during gradient descent. For this reason, a tile-based strategy is used to decompose the material into small tiles (128×128 pixels in size in this example) for processing. This results in noisy output maps and visible seams between adjacent tiles. Independent of the tile-based decomposition, material that exhibits self-similarity can be consistently processed by the optimization. However, optimizing texels independently results in self-similar patches that provide inconsistent results, where structures of similar colors tend to result in smooth or sharp visual transitions.

[0075] A fully connected MLP is used to implement spatial regularization. The fully connected MLP bilinearly upsamples the low-resolution SVBRDF takes as input a small texel patch in and outputs a high-resolution SVBRDF M SR The SVBRDF is then fed to the differentiable renderer and used to compute the sparsity-induced loss

[0076] To generate data to train the machine learning model 212, the upsampling module 116 takes as input 128×128 pixels of the low-resolution SVBRDF and computes multiple renderings under varying point light positions. In some examples, the renderings are sampled from the Fibonacci set to evenly cover the hemisphere. For the renderings, the standard microfacet model [Cook and Torrance, 1982] is implemented with the GGX [Trowbridge and Reitz 1975; Walter et al. 2007] normal distribution function in PyTorch, omitting energy decay to preserve information from the upsampler. Two different super-resolution algorithms are applied, as described above with reference to the super-resolution module 206. This provides a pair of low-resolution SVBRDF and high-resolution renderings for training the SVBRDF upsampling network.

[0077] The MLP consists of 6 fully connected hidden layers of size 512, which take 17×17 texel tiles as input. In this example, the above loss function uses α=10 -1 The weights of the machine learning model 212 are optimized using gradient descent using the Adam optimizer with default parameters.

[0078] Figure 5 An example 500 including an output table including different high-resolution digital material models is depicted. For example, the upsampling module 116 generates an output 128 including a high-resolution digital material model 124 for display in the user interface 110. In this example, orthographic rendering illustrates the high-resolution digital material model. Perspective rendering illustrates the high-resolution digital material model applied to 3D geometry. For example, the high-resolution digital material model 124 is incorporated into a three-dimensional virtual object to add a detailed virtual texture to the three-dimensional virtual object presented in the user interface 110. As shown, the digital material model obtained by the radiation-based upsampling has a higher level of detail than the digital material model generated using conventional bilinear upsampling.

[0079] Example Process

[0080] The following discussion describes techniques that can be implemented using the previously described systems and devices. Aspects of each of these processes can be implemented in hardware, firmware, software, or a combination thereof. These processes are shown as a set of blocks that specify operations performed by one or more devices and are not necessarily limited to the order shown for performing the operations by the respective blocks. In portions of the following discussion, reference is made to Figure 1-Figure 5 .

[0081] Figure 6 A process 600 in an example implementation of upsampling a digital material model based on radiosity is depicted. At block 602, an input digital material model 120 having a first resolution is received. In some examples, the input digital material model 120 having a first resolution includes at least one of a base color map 302, a normal map 304, a metallic map 306, a roughness map 308, or a height map 310.

[0082] At block 604 , bilinear upsampled texels 204 are generated based on the input digital material model 120 having a first resolution.

[0083] At box 606, texels having a second resolution higher than the first resolution are generated based on the bilinear upsampled texels 204 using a machine learning model 212, the machine learning model 212 being trained on training data to generate texels. In some examples, the training data includes renderings calculated based on texels upsampled at different light positions by an image upsampler. For example, the rendering is generated using a microfacet model. Additionally or alternatively, the machine learning model 212 includes a multilayer perceptron model. In some examples, the texels having the second resolution are generated by a transformer model. Additionally or alternatively, the transformer model includes filters optimized for a range of radiations.

[0084] At block 608, an output digital material model is generated based on the texels having the second resolution, the output digital material model having a resolution higher than the first resolution. Some examples also include performing parameter optimization on the output digital material model. Additionally or alternatively, the output digital material model is applied to the three-dimensional geometry.

[0085] Figure 7 A procedure 700 is depicted in an additional example implementation of upsampling a digital material model based on radiance. At block 702, an input digital material model 120 is received. In some examples, the input digital material model 120 includes at least one of a base color map 302, a normal map 304, a metallic map 306, a roughness map 308, or a height map 310.

[0086] At block 704 , upsampled texels are generated by bilinearly upsampling a portion of the input digital material model 120 .

[0087] At block 706, texels having a higher resolution than the input digital material model 120 are generated based on the upsampled texels using a machine learning model 212 trained on training data to generate texels based on radiances corresponding to different light positions. In some examples, the training data includes renderings computed from texels upsampled at different light positions by an image upsampler. In some examples, the machine learning model is a multilayer perceptron model. Additionally or alternatively, the texels are generated by a transformer model. For example, the transformer model includes filters optimized for a range of radiances.

[0088] At block 708, an output digital material model is generated based on texels having a higher resolution than the input digital material model 120. Some examples also include performing parameter optimization on the output digital material model.

[0089] Example Systems and Devices

[0090] Figure 8 An example system is shown generally at 800 and includes an example computing device 802, which represents one or more computing systems and / or devices that implement the various techniques described herein. This is illustrated by the inclusion of an upsampling module 116. For example, the computing device 802 may be configured as a server of a service provider, a device associated with a client (e.g., a client device), a system on a chip, and / or any other suitable computing device or computing system.

[0091] The example computing device 802 shown in the figure includes a processing system 804, one or more computer-readable media 806, and one or more I / O interfaces 808 that are communicatively coupled to each other. Although not shown, the computing device 802 also includes a system bus or other data and command transmission system that couples various components to each other. The system bus includes any one or a combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus that utilizes any of the various bus architectures. Various other examples are also contemplated, such as control lines and data lines.

[0092] Processing system 804 represents functionality for performing one or more operations using hardware. Thus, processing system 804 is shown to include hardware elements 810, which may be configured as processors, functional blocks, and the like. This includes hardware implementations as application specific integrated circuits or other logic devices formed using one or more semiconductors. Hardware elements 810 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor may be configured as (multiple) semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor executable instructions are electronic executable instructions.

[0093] Computer-readable storage media 806 is shown to include memory / storage 812. Memory / storage 812 represents memory / storage capacity associated with one or more computer-readable media. Memory / storage 812 includes volatile media (such as random access memory (RAM)) and / or non-volatile media (such as read-only memory (ROM), flash memory, optical disks, magnetic disks, etc.). Memory / storage 812 includes fixed media (e.g., RAM, ROM, fixed hard drives, etc.) and removable media (e.g., flash memory, removable hard drives, optical disks, etc.). Computer-readable media 806 can be configured in a variety of other ways, as further described below.

[0094] Input / output interface(s) 808 represent functionality for allowing a user to input commands and information to computing device 802 and also allowing information to be presented to the user and / or other components or devices using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone, a scanner, touch functionality (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., using visible or invisible wavelengths such as infrared frequencies to identify movement as gestures that do not involve touch), etc. Examples of output devices include a display device (e.g., a monitor or projector), a speaker, a printer, a network card, a tactile response device, etc. Thus, computing device 802 can be configured in various ways as further described below to support user interaction.

[0095] Various techniques are described herein in the general context of software, hardware elements, or program modules. Typically, such modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The terms "module," "function," and "component" as used herein generally refer to software, firmware, hardware, or a combination thereof. Features of the techniques described herein are platform-independent, meaning that the techniques can be configured on a variety of commercial computing platforms with a variety of processors.

[0096] The implementation of the above modules and techniques is stored on or transmitted via some form of computer-readable media. Computer-readable media includes various media accessed by computing device 802. By way of example and not limitation, computer-readable media include "computer-readable storage media" and "computer-readable signal media."

[0097] "Computer-readable storage media" refers to media and / or devices that are capable of persistent and / or non-transitory storage of information, as compared to simple signal transmission, carrier waves, or signals themselves. Therefore, computer-readable storage media refers to non-signal bearing media. Computer-readable storage media include hardware, such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information, such as computer-readable instructions, data structures, program modules, logic elements / circuits, or other data. Examples of computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical storage, hard disk, cassette, magnetic tape, disk storage or other magnetic storage devices, or other storage devices, tangible media or articles suitable for storing desired information and accessible by a computer.

[0098] "Computer-readable signal media" refers to a signal-bearing medium that is configured to transmit instructions to the hardware of the computing device 802, such as via a network. Signal media typically include computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, data signal, or other transmission mechanism. Signal media also include any information transfer media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in a manner that encodes information in the signal. By way of example and not limitation, communication media include wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0099] As previously described, hardware elements 810 and computer-readable media 806 represent modules, programmable device logic, and / or fixed device logic implemented in hardware, which in some embodiments are used to implement at least some aspects of the techniques described herein, such as executing one or more instructions. Hardware includes components of integrated circuits or systems on chips, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations of silicon or other hardware. In this case, the hardware is used as a processing device that performs program tasks defined by the instructions and / or logic included in the hardware and the hardware for storing instructions for execution (e.g., the previously described computer-readable storage medium).

[0100] Combinations of the above may also be used to implement the various techniques described herein. Therefore, software, hardware, or executable modules are implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 810. Computing device 802 is configured to implement specific instructions and / or functions corresponding to software and / or hardware modules. Therefore, the implementation of a module executable as software by computing device 802 is at least partially implemented in hardware, such as by using a computer-readable storage medium and / or a hardware element 810 of a processing system 804. Instructions and / or functions are executable / operable by one or more articles (e.g., one or more computing devices and / or processing systems 804) to implement the techniques, modules, and examples described herein.

[0101] The techniques described herein are supported by various configurations of computing device 802 and are not limited to the specific examples of the techniques described herein. The functionality may also be implemented using a distributed system, such as on the "cloud" 1114 via platform 816 as described below.

[0102] Cloud 814 includes and / or represents a platform 816 for resources 818. Platform 816 abstracts the underlying functionality of the hardware (e.g., servers) and software resources of cloud 814. Resources 818 include applications and / or data that can be used when computer processing is executed on a server remote from computing device 802. Resources 818 can also include services provided over the Internet and / or over a subscriber network such as a cellular or Wi-Fi network.

[0103] The platform 816 abstracts resources and functionality to connect the computing device 802 with other computing devices. The platform 816 is also used to abstract the scaling of resources to provide a corresponding scaling level to the demand for resources 818 encountered that is implemented via the platform 816. Therefore, in an interconnected device embodiment, the implementation of the functionality described herein may be distributed throughout the system 800. For example, the functionality may be implemented in part on the computing device 802 and via the platform 816 that abstracts the functionality of the cloud 814.

Claims

1. A method comprising: Receiving, by a processing device, an input digital material model having a first resolution; generating, by the processing device, bilinear upsampled texels based on the input digital material model having the first resolution; generating, by the processing device, texels having a second resolution higher than the first resolution based on the bilinearly upsampled texels using a machine learning model trained on training data to generate texels; as well as An output digital material model is generated by the processing device based on the texels having the second resolution for display in a user interface, the output digital material model having a resolution higher than the first resolution.

2. The method of claim 1, wherein the training data comprises renderings computed from texels upsampled by an image upsampler at different light positions. The method of claim 2 , wherein the rendering is generated using a microfacet model. 4 . The method of claim 1 , wherein the input digital material model having the first resolution comprises at least one of: a base color map, a normal map, a metalness map, a roughness map, or a height map.

5. The method of claim 1, wherein the machine learning model comprises a multi-layer perceptron model. The method of claim 1 , wherein the texels having the second resolution are generated by a transformer model.

7. The method of claim 6, wherein the converter model comprises a filter optimized for a range of radiation.

8. The method of claim 1, further comprising performing parameter optimization on the output digital material model.

9. The method of claim 1, wherein the output digital material model is applied to a three-dimensional geometric body.

10. A system comprising: Memory components; as well as A processing device, coupled to the memory component, the processing device configured to perform operations, the operations comprising: receiving an input digital material model; generating upsampled texels by bilinearly upsampling a portion of the input digital material model; generating texels having a higher resolution than a resolution of the input digital footage model based on the upsampled texels using a machine learning model trained on training data to generate texels based on radiance corresponding to different light positions; and An output digital material model is generated based on the texels whose resolution is higher than the resolution of the input digital material model.

11. The system of claim 10, wherein the training data comprises renderings computed from texels upsampled by an image upsampler at different light positions.

12. The system of claim 10, wherein the input digital material model comprises at least one of: a base color map, a normal map, a metallic map, a roughness map, or a height map.

13. The system of claim 10, wherein the machine learning model is a multi-layer perceptron model.

14. The system of claim 10, wherein the texels are generated by a transformer model.

15. The system of claim 14, wherein the converter model comprises a filter optimized for a range of radiation.

16. The system of claim 10, further comprising performing parameter optimization on the output digital material model.

17. A non-transitory computer-readable storage medium storing executable instructions, which when executed by a processing device causes the processing device to perform operations comprising: receiving an input digital material model having a first resolution; generating bilinear upsampled texels based on the input digital material model having the first resolution; generating texels having a second resolution higher than the first resolution based on the bilinearly upsampled texels using a machine learning model trained on training data to generate texels; as well as An output digital material model is generated based on the texels having the second resolution for display in a user interface, the output digital material model having a resolution higher than the first resolution.

18. The non-transitory computer-readable storage medium of claim 17, wherein the training data comprises renderings computed from texels upsampled at different light positions by an image upsampler.

19. The non-transitory computer-readable storage medium of claim 18, wherein the rendering is generated using a microfacet model.

20. The non-transitory computer-readable storage medium of claim 17, wherein the input digital material model having the first resolution comprises at least one of: a base color map, a normal map, a metalness map, a roughness map, or a height map.

Images