Machine learning technique to create higher resolution data structure representing texture from lower resolution data structure

Abstract

To provide a machine learning method for creating a higher resolution data structure representing a texture from a lower resolution data structure, and an assembly.SOLUTION: A method generates a first mipmap of a texture having a first compression on the basis of a second mipmap of the same texture having a second compression without using compression or decompression in generating the first mipmap using machine learning (ML), and renders a computer graphics object using the first mipmap. ML engine training uses the blocks decompressed from originally compressed blocks as input.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] This application relates to technically inventive and unconventional solutions that give rise to concrete technical improvements that are necessarily rooted in computer technology. [Background technology]

[0002] In computer simulations, such as computer games, objects are rendered in part using "texture" data that describes the surfaces of the objects. The more texture data there is for a given object, the higher the resolution of the rendering can be. However, for bandwidth purposes, it is desirable not to send large texture data structures to the rendering device. Summary of the Invention

[0003] As understood herein, so-called "mipmaps" may be used to conserve bandwidth in the following manner: A mipmap is a series of texture data structures, each of which is a progressively lower resolution representation of the previous texture data structure. Typically, the reduction is by a factor of two in each dimension. Bandwidth is conserved by using a higher resolution mipmap to render objects close to the viewer and a lower resolution mipmap to render objects further away from the viewer. Typically, the mipmap level is chosen to be the level that best matches the pixel density of the image. Ideally, one texture pixel is desired for every screen pixel. A texture pixel may also be called a "texel" (a combination of texture and pixel).

[0004] However, as will be understood herein, to conserve memory, texture data is typically compressed into one of a variety of block compression (BCn) modes that can be sampled natively on a GPU. The maximum resolution of a texture is typically limited by storage space constraints and artist authoring time. To generate a high-resolution image from a low-resolution one, such as using existing machine learning-based techniques, the BCn compressed texture data must first be decompressed, upsampled, and recompressed. This is undesirable because extra storage space is required for uncompressed versions of both the low-resolution and high-resolution images, and the process of BCn compression is very complex and computationally expensive.

[0005] Thus, the assembly includes at least one processor configured with instructions to train at least one machine learning (ML) engine to create high-resolution computer graphics textures (CGTs) from low-resolution CGTs, where the training is accomplished by compressing a plurality of training blocks having a first number of texels to create compressed training blocks, decompressing the compressed training blocks to create decompressed training blocks, and inputting the decompressed training blocks to the ML engine such that the ML engine up-res the decompressed training blocks into predicted high-resolution training blocks.

[0006] In some example implementations, the instructions may be executable to input a high-resolution block of ground truth corresponding to each decompressed training block to the ML engine for comparison with the predicted high-resolution training block.

[0007] In some implementations of the first aspect, the instructions may be executable to input a respective one of at least a portion of the plurality of training blocks to N compressors, each compressor operating according to a compression mode of a respective Nth block to generate a respective N outputs for each of the at least a portion of the plurality of training blocks. Each of the N outputs is decompressed to create N decompressed training blocks for each one of the at least a portion of the plurality of training blocks. The N decompressed training blocks are input to the ML engine for the ML engine to up-rezo into a predicted high-resolution training block.

[0008] In other embodiments, instead of inputting all N outputs to the ML engine for training, a prediction error may first be obtained for each of the N outputs, the prediction error may be used to select a compression mode for each Nth block for compressing and decompressing each training block, and only the resulting decompressed training block may be input to the ML engine for training. In addition to the prediction error, the processing cost associated with each mode may also be considered when selecting the Nth compression mode.

[0009] In a non-limiting example, the training blocks may include one or more mipmaps, the CGT may include physically based rendering (PBR) and material data, and the high resolution CGT may include a mipmap that is one mip level higher than the low resolution CGT.

[0010] In another aspect, an assembly includes at least one computer storage, the at least one computer storage including instructions executable by at least one processor to input decompressed texture blocks associated with a first resolution, rather than the temporal signal, to at least one machine learning (ML) algorithm. The instructions are executable to input ground truth texture blocks associated with a second resolution, higher than the first resolution, to the ML algorithm and execute the ML algorithm to train on the decompressed texture blocks and the ground truth texture blocks. After training, the instructions are further executable to execute the ML algorithm to receive a low-resolution texture from a source and generate a high-resolution texture from the low-resolution texture without decompressing or compressing the low-resolution texture.

[0011] In another aspect, the method includes accessing at least one machine learning (ML) engine and training the ML engine using the decompressed texture block to generate a high-resolution texture from a low-resolution texture.

[0012] In an example embodiment of this aspect, the ML engine generates high-resolution textures from low-resolution textures without using any decoding or encoding.

[0013] In some embodiments of this aspect, the method may include training the ML engine using N versions of each one of at least a portion of the decompressed texture block, the N versions being generated by each of the N compression modes.

[0014] In other embodiments of this aspect, the ML engine may be trained using less than N at least one version of each one of at least a portion of the decompressed texture block produced by each of the N compression modes. The version(s) may be selected based at least in part on a prediction error associated with each of the N modes, and optionally also based on processing constraints associated with each of the N modes.

[0015] The details of the present application, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference characters refer to like parts, and in which: [Brief description of the drawings]

[0016] [Figure 1] 1 is a block diagram of an exemplary system including an example according to the principles of the present invention; [Diagram 2] 4 shows an example of a texture communication path. [Diagram 3] Here are some examples of PBR material components: [Figure 4] 1 illustrates example logic in the form of an example flowchart for providing ground truth training data. [Diagram 5] 1 illustrates example logic in the form of an example flowchart for training a texture rendering machine learning engine. [Figure 6] 1 illustrates exemplary logic in the form of an exemplary flowchart for providing textures to a renderer. [Figure 7] 1 illustrates exemplary logic in the form of an exemplary flowchart for upscaling an input texture using a machine learning engine. [Figure 8] 1 illustrates, in flow chart form, exemplary logic for upscaling two texture data types together. [Figure 9]1 illustrates exemplary logic in the form of an exemplary flowchart for training a machine learning engine using decompressed textures. [Figure 10] 1 illustrates an exemplary training system consistent with present principles. [Figure 11] 1 illustrates exemplary logic in an exemplary flow chart form for predicting the best compression mode. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] The present disclosure generally relates to a computer ecosystem that includes aspects of user information based on consumer electronics (CE) devices in the computer ecosystem. The system herein may include a server component and a client component connected through a network such that data may be exchanged between the client component and the server component. The client component may include one or more computing devices, including portable computers such as portable televisions (e.g., smart televisions, Internet-enabled televisions), laptop computers, and tablet computers, as well as smartphones and other mobile devices, including additional examples described below. These client devices may operate in a variety of operating environments. For example, some of the client computers may use operating systems obtained from Microsoft or Unix or Apple Inc. or Google, as examples. These operating environments may be used to execute one or more browsing programs, such as browsers created by Microsoft or Google or Mozilla, or other browser programs that can access web applications hosted by Internet servers, as described below.

[0018] A server may include one or more processors that execute instructions that configure the server to send and receive data over a network such as the Internet. Alternatively, clients and servers may be connected through a local intranet or a virtual private network. A server or controller may be instantiated by a gaming console such as a Sony PlayStation®, a personal computer, or the like.

[0019] Information may be exchanged between the clients and the servers over a network. For this purpose and for security, the servers and / or clients may include firewalls, load balancers, temporary storage, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form an apparatus for implementing a method for providing a secure community, such as an online social website, for network members.

[0020] As used herein, instructions refer to computer-implemented steps for processing information within a system. Instructions may be implemented in software, firmware, or hardware and may include any type of programmed steps performed by components of the system.

[0021] The processor may be a single-chip processor or a multi-chip processor capable of executing logic through various lines, such as address lines, data lines, and control lines, as well as registers and shift registers. The processor may be implemented by or include one or more graphics processing units (GPUs).

[0022] The software modules described herein by flowcharts and user interfaces may include various subroutines, procedures, etc. Without limiting the disclosure, logic defined to be performed by a particular module may be redistributed to other software modules and / or aggregated together in a single module and / or made available in a shareable library.

[0023] The inventive principles described herein may be implemented as hardware, software, firmware, or a combination thereof. Accordingly, the illustrative components, blocks, modules, circuits, and steps have been described in terms of their functionality.

[0024] Further, of those mentioned above, the logic blocks, modules, and circuits described below can be implemented or performed by digital signal processors (DSPs), field programmable gate arrays (FPGAs), or other programmable logic devices such as application specific integrated circuits (ASICs), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor can be implemented by a controller or state machine, or combination of computing devices.

[0025] The functions and methods described below, when implemented in software, can be written in a suitable language, such as, but not limited to, Java, C#, or C++, and can be stored on or transmitted by a computer-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), other optical disk storage, such as compact disk read-only memory (CD-ROM) or digital versatile disk (DVD), magnetic disk storage or other magnetic storage devices, including removable thumb drives, etc. A computer-readable medium may be established by a connection. Such connections may include, by way of example, hardwire cables, including fiber optic and coaxial wire, as well as digital subscriber line (DSL) and twisted pair wire.

[0026] Components included in one embodiment may be used in other embodiments in any suitable combination, for example, any of the various components described herein and / or illustrated in the figures may be combined, interchanged, or eliminated from other embodiments.

[0027] A "system having at least one of A, B, and C" (and similarly "a system having at least one of A, B, or C" and "a system having at least one of A, B, C") includes systems having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.

[0028] Referring now specifically to FIG. 1, an exemplary ecosystem 10 is shown, which may include one or more of the exemplary devices described above and in detail below in accordance with the principles of the present invention. The first exemplary device included in the system 10 is an exemplary primary display device, which in the illustrated embodiment is an audio-video display device (AVDD) 12, such as, but not limited to, an Internet-enabled TV. Thus, the AVDD 12 may alternatively be an appliance or household item, such as a computer-controlled Internet-enabled refrigerator, washer, or dryer. Alternatively, the AVDD 12 may also be a computer-controlled Internet-enabled ("smart") phone, a tablet computer, a notebook computer, a wearable computer-controlled device (e.g., a computer-controlled Internet-enabled watch, a computer-controlled Internet-enabled bracelet, etc.), other computer-controlled Internet-enabled devices, a computer-controlled Internet-enabled music player, a computer-controlled Internet-enabled headphones, a computer-controlled Internet-enabled implantable device such as an implantable skin device, etc. Nonetheless, it should be understood that AVDD12 is configured to implement the principles of the present invention (e.g., to communicate with other CE devices to implement the principles of the present invention, to execute logic described herein, and to perform any other functions and / or operations described herein).

[0029] Thus, to implement such principles, AVDD 12 may be established by some or all of the components shown in FIG. 1. For example, AVDD 12 may include one or more displays 14, which may be implemented by high resolution or ultra-high resolution, i.e., "4K" or "8K" (or high resolution), flat screens, and may be touch-enabled to receive consumer input signals by touching on the display. AVDD 12 may include one or more speakers 16 for outputting sound in accordance with the principles of the present invention, and at least one additional input device 18, such as a keyboard or keypad or audio receiver / microphone, for inputting audible commands to AVDD 12 to control AVDD 12. An exemplary AVDD 12 may also include one or more network interfaces 20 for communicating over at least one network 22, such as the Internet, a WAN, a LAN, etc., under the control of one or more processors 24. Thus, interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface. It should be understood that processor 24 controls AVDD 12 to implement the principles of the present invention, including other elements of AVDD 12 described herein, e.g., controlling display 14 to present images thereto, receiving input therefrom, etc. It should further be noted that network interface 20 may be, for example, a wired or wireless modem or router, or other suitable interface, such as, for example, a wireless telephony transceiver or a Wi-Fi transceiver as described above.

[0030] In addition to the foregoing, AVDD 12 may also include one or more input ports 26, such as, for example, a USB port for physically connecting (e.g., using a wired connection) to another CE device, and / or a headphone port for connecting headphones to AVDD 12 to present audio from AVDD 12 to the consumer through the headphones. AVDD 12 may further include one or more computer memories 28 that are not signal transitory, such as disk-based or solid-state storage (including, but not limited to, flash memory). Also, in some embodiments, AVDD 12 may include a position or location receiver, such as, but not limited to, a cellular receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographic location information from at least one satellite or cellular tower and provide the information to processor 24 and / or determine the altitude at which AVDD 12 is located in conjunction with processor 24. However, it should be understood that another suitable position receiver other than a cellular receiver, a GPS receiver, and / or an altimeter may be used in accordance with the principles of the present invention to determine the location of AVDD 12, for example, in all three dimensions.

[0031] Continuing with the description of AVDD 12, in some embodiments, AVDD 12 may include one or more cameras 32, which may be, for example, digital cameras such as infrared imaging cameras, webcams, and / or cameras integrated into AVDD 12 and controllable by processor 24 to collect pictures / images and / or videos in accordance with the principles of the present invention. AVDD 12 may also include a Bluetooth transceiver 34 and other NFC elements 36 for communicating with other devices using Bluetooth and / or Near Field Communication (NFC) technology, respectively. An exemplary NFC element may be a Radio Frequency Identification (RFID) element.

[0032] Furthermore, AVDD 12 may include one or more auxiliary sensors 37 (e.g., motion sensors such as accelerometers, gyroscopes, cyclometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, gesture sensors (e.g., sensors for detecting gesture commands, etc.)) that provide input to processor 24. AVDD 12 may include still other sensors, such as, for example, one or more climate sensors 38 (e.g., barometers, humidity sensors, wind sensors, light sensors, temperature sensors, etc.) and / or one or more biometric sensors 40, that provide input to processor 24. In addition to the above, it is noted that AVDD 12 may also include an IR transmitter and / or an IR receiver and / or an IR transceiver 42, such as an infrared (IR) data association (IRDA) device. A battery (not shown) may be provided for powering AVDD 12.

[0033] 1, in addition to the AVDD 12, the system 10 may include one or more other CE device types. In one example, a first CE device 44 may be used to send messages to a second CE device 46, which may include similar components as the first CE device 44 and will not be described in detail. In the example shown, only two CE devices 44, 46 are shown, although of course a fewer or greater number of devices may be used.

[0034] An exemplary, non-limiting, first CE device 44 may be established by any one of the devices mentioned above, such as a portable wireless laptop computer or tablet computer or notebook computer or mobile phone, and may therefore have one or more of the components described below. The second CE device 46 may be established by, but is not limited to, a wireless telephone. The second CE device 46 may implement a portable handheld remote control (RC). The second CE device 46 may implement a virtual reality (VR) and / or augmented reality (AR), head mounted display (HMD). The CE devices 44, 46 may include some or all of the components shown for AVDD12.

[0035] At least one server 50 may include at least one server processor 52, at least one computer memory 54, such as disk-based or solid-state storage, and at least one network interface 56 that allows communication with other devices of Figure 1 over network 22 under the control of server processor 52, and may, in fact, facilitate communication between the server and client devices in accordance with the principles of the present invention. It should be noted that network interface 56 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface, such as, for example, a wireless telephony transceiver.

[0036] Thus, in some embodiments, server 50 may be an Internet server and may include and perform "cloud" functionality such that devices of system 10 may access a "cloud" environment via server 50, in an exemplary embodiment, or server 50 may be implemented by a gaming console or other computer in or near the same room as the other devices shown in FIG.

[0037] The devices described herein may, where appropriate, include some or all of the various components shown in FIG.

[0038] Before referring to Figure 2, a "texture" is a data structure that can be mapped to an image to characterize the surface of a rendered object. The basic data element of a texture data structure is a texture element or texel (a combination of texture and pixel). A texture is represented by an array of texels that represent a texture space. The texels are mapped to pixels in the image being rendered to define the surface of the rendered image.

[0039] Thus, because textures are data and not images, the neural network (NN) training described below does not necessarily rely on the principle of perceptual error. There may be exceptions for albedo (discussed further below), but in general, training on texture data uses an error metric specific to the objective on the particular data.

[0040] Some types of data, such as normals and roughness, can be paired together because they are interrelated and the two data can mitigate each other's errors. More specifically, in PBR rendering, a relationship exists between normal map data and roughness data (sometimes called gloss data). Roughness essentially represents the distribution of normals across texture pixels. Thus, techniques exist to modify the roughness to account for data lost when reducing the resolution of the normal map when generating mipmaps, essentially anti-aliasing the normal map. In this case, there is a close relationship between the normal map and the corresponding roughness map.

[0041] Additional details on textures, physically based rendering (PBR) and materials are a general set of guidelines for rendering realistic material and light interactions, including using various types of data stored in textures. Light is modeled as either diffuse or specular. Diffuse light is generally view independent and generally does not change when viewing a material from any angle. Specular light, on the other hand, is view dependent to emulate, for example glare.

[0042] Typically, PBR textures include an "albedo" parameter that characterizes the diffuse light response of a material. For example, for an emulated surface of polished wood, the albedo texture contains grain patterns and color variations, but does not contain shape information and is very flat. Metals have no albedo, since all of their light response comes from reflection.

[0043] As mentioned above, PBR textures may also contain "normal" parameters. A normal map defines small geometric detail of a surface, specifically representing the surface normal of the underlying surface. This may or may not be separate from the geometric normal of the triangle data used in rendering. It is vector data stored as a texture, and although it can be displayed, it is not, in itself, an image. In the wood example, this would be mostly flat where the wood is smooth, but may contain etched or carved detail into the surface of the wood.

[0044] As mentioned above, PBR textures may also contain a gloss / roughness parameter that defines how rough the surface is being rendered. Generally, this is thought of as the variance of sub-pixel normal data (detail smaller than a texel in the normal map). In the wood example, this may include data about scratches in the wood (think of rubbing wood with sandpaper). Sanded wood is relatively smooth, and therefore has a low roughness.

[0045] Another PBR parameter is reflectance, which describes the response of reflected light. For most non-metallic materials, reflectance is colorless. The reflectance of almost all non-metals is constant at 2% (the amount of light that reflects directly). For mirrors, reflectance approaches 100%.

[0046] Some PBR techniques attempt to take advantage of the fact that non-metals have a constant reflectance, and that metals have no albedo. This usually involves repurposing the albedo texels where the surface is metallic, and instead representing the reflectance (with a color, gold, brass, etc.), and storing additional information in a separate texture channel to identify which texels are metal and which are not. Roughly speaking, this information is called "metallicity". This encoding is generally done to save texture memory by not storing three channels for albedo and another three channels for reflectance.

[0047] In general, textures can have different purposes and can differ from game engine to game engine.

[0048] In view of the above discussion regarding textures, multiple versions of the same texture are typically generated with different compressions (and therefore different resolutions). Specifically, a single texture may be represented by a full mipmap chain of textures. Mip-mapping takes an image and successively reduces the resolution by a factor of two. Thus, for a given texture (e.g., 1024x1024, etc.), its 512x512 version and its 256x256 version, etc. are also stored in memory. This improves performance, visual fidelity, and facilitates texture streaming. Texture streaming attempts to conserve memory by loading only the "tail" of the mipmap (the lower mip level and all lower resolution mip levels below it) that is needed for a given texture based on the screen size of the rendered object. For example, the 256x256 mip level and lower than the mip chain of a 1024x1024 texture may be all that is needed to load a distant object. Which mip levels are loaded is changed on demand based on the changing game environment and the viewer's position. As an object gets closer, higher resolution mip levels or "levels of detail" are loaded into memory to render the now nearby object.

[0049] Textures may use various types of compression. One type is block compression, sometimes referred to as BCn compression. This can be a lossy texture compression that can be decompressed in-place by a graphics processing unit (GPU). Because block compression does not require the entire image to be decompressed, the GPU can decompress the data structures while sampling the texture as if it were not compressed at all.

[0050] Block compression techniques compress a 4x4 block of pixels into a single (smaller) data packet. Typically this involves choosing two or more "endpoint" colors (depending on the BC compression type), along with some per-pixel information about how to blend these two colors at each pixel. The endpoint colors are shared across the entire 4x4 pixel block. For example, for an image with just red, blue, and purple pixels, the compressor is likely to choose one endpoint to be red and the other endpoint to be blue. The purple pixels will have a value that blends the two together.

[0051] Most of the different BC types differ in the number of texture channels they have (e.g. BC4 is one channel grayscale, i.e. "black and white"). BC6 and BC7 are special because they introduce the concept of mode, which determines the interpretation of each block. For the other BC modes, all blocks are encoded in the same way, with the same number of bits allocated to the endpoint colors and blend values. Different BC6 / 7 modes allocate their bits differently per block. This allows the compressor to make different quality tradeoffs in different areas of the texture.

[0052] 2 shows a texture source 200 that transmits textures for rendering according to the above principles over a communication path 202, such as a local data bus or a wireless / wired network link, to a texture renderer 204. The texture renderer 204 typically includes one or more GPUs with memory to render an image according to the image data and texture data on a display.

[0053] 3 illustrates that in an embodiment herein, only the tail texture 300 needs to be sent from the source 200 to the renderer 204. The renderer 204 can run a machine learning engine 302 on the texture to upscale it to the next higher level of resolution texture 304 without the need for decoding / encoding and thus without the need for a codec. The machine learning engine 302 may include one or more trained neural networks, such as generative, noise-based, and possibly adversarial networks.

[0054] Therefore, machine learning is used to generate mipmaps for textures that are one level higher than the input mipmaps for all streaming textures (e.g., from a mipmap chain created on disk). New (higher resolution) mipmaps are similarly introduced as if they existed on disk, but are simply generated procedurally instead. Since textures can be stored on disk in BCN compressed format, the network generates new compressed mip levels from the highest of the existing compressed mip levels in memory. If the cost of generating mip levels at runtime is too high, the mip levels can be generated offline and stored on disk using the same method.

[0055] 4 shows an example of the training principle of the machine learning engine 302 of FIG. 3. An existing texture library may be accessed in block 400 for ground truth training and compressed in block 402 to establish the next mip level down (again by half). In other words, in block 400, a full uncompressed (and thus highest resolution) mipmap may be accessed and compressed in half in block 402 to create a compressed (and thus lower resolution) mipmap. The mipmaps generated in block 402 may be successively compressed according to the principles herein to create a full set of mipmaps for each input uncompressed mipmap, generating the ground truth in block 404.

[0056] 5 shows that a simple network may access one compressed BC block data packet from a ground truth set in block 500 and generate four BC block data packets for higher mip levels, e.g., going efficiently from a 4×4 block of pixels to an 8×8 block of pixels, in block 502. Alternatively, instead of or in addition to taking one BC block as input, a neighborhood of blocks surrounding the block (e.g., the surrounding eight blocks) may also be provided as input to better inform the network of the features.

[0057] Training may be performed using backpropagation, gradient descent. Training may be performed using an 80 / 20 training test split, where a random sample of 80% of the ground truth data is used to train or set the weights in the NN, and then an additional 20% of the test data downscaled files are input and compared to those generated by the NN. In other words, the output of the NN can be compared to the 20% full resolution files of the ground truth that were not input during the training session.

[0058] Since there are mostly different types of data between different material attribute textures, a different network may be trained for each material attribute type, for example, one network may be trained to upscale reflectance texture data and another network may be trained to upscale albedo.

[0059] In the case of normals and roughness where similar data is stored and thus there is some interrelationship, a single network can be trained to upscale them together. The same considerations can be applied to other related pairs or groups of material attributes. In the case of a shared normal and roughness up-res network, one may input one BC block with normal data and one BC block of roughness data in the same relative location, and the network may output four BC blocks of normal data and four BC blocks of roughness data.

[0060] FIG. 6 shows that textures may be compressed at block 600 and sent to a renderer at block 602. In one example, the lowest resolution (most compressed) "tail" textures in the mipmap chain are sent to the renderer. The renderer can upscale them on the fly using a trained machine learning engine as described herein, without the need for a codec. In other embodiments, relatively less compressed high-resolution textures (or the uncompressed base texture itself) may be provided as input to a renderer that can use machine learning to generate more compressed lower-resolution textures.

[0061] 7 illustrates the renderer logic. At state 700, an input texture is received. At block 702, the texture is processed by a trained machine learning engine, which outputs a different compressed texture at block 704 that is either more compressed (low resolution) or less compressed (high resolution) than the input texture. The texture at block 704 is used to render an image on a display.

[0062] 8, for a shared normal and roughness up-res network, one BC block with normal data is shown received by the machine learning engine at block 800, and one BC block of roughness data at the same relative location is received at block 802. At block 804, the machine learning engine upscales the two input blocks together to output four BC blocks of normal data and four BC blocks of roughness data.

[0063]

[0031] Referring now to Figure 9, a technique for training an ML engine in addition to those suggested above is illustrated. Beginning at block 900, a ground truth texture or texture portion, such as a 2x2 texel block, and a corresponding high resolution block, such as a 4x4 texel block, are received. Moving to block 902, the low resolution (e.g., 2x2) block is compressed using one or more block compression modes and then decompressed at block 904 using the same mode(s). The decompressed block, along with the corresponding high resolution ground truth block, are input to an ML engine or algorithm at block 906 for training.

[0064] The above technique of using decompressed blocks instead of the original uncompressed blocks allows for consideration of the compromises that a particular compression mode(s) may make, providing more realistic training.

[0065] 10 and 11 show additional techniques related to the overall logic of FIG. 9. In FIG. 10, an uncompressed NxN texture block, such as a 2x2 texel block, may be sent to multiple block compressors 1002, 1004, each operating in a respective block compression mode. The compressors 1002, 1004 compress the blocks and send the compressed blocks to respective decompressors 1006, 1008, which operate in the same mode as the compressor that received the compressed block. The decompressed blocks (i.e., N versions of each decompressed block, one for each compression mode) can be used in block 906 of FIG. 9 to train an ML engine or algorithm, or the decompressed blocks can be sent to an error prediction module 1010 to prune which of the N versions of each decompressed block to use in block 906 of FIG. 9.

[0066] 11 illustrates a non-limiting example error prediction module 1010. Beginning at block 1100, for a particular frame (e.g., a 2x2 texel block) of a first resolution (typically a low resolution) from which an uncompressed training texture block is obtained, a processing budget or cost base may be established, for example, in terms of time. Thus, depending on the type of frame, different (higher or lower) processing budgets may be defined. For example, a short or less complex frame may have a larger processing budget than a long, large, or more complex frame.

[0067] Moving to block 1102, for each mode of each input training block, the model predicts the error of each of the modes of that block in accurately outputting a high resolution block (e.g., a 4x4 block). Without limitation, a recurrent neural network (RNN) such as a long short-term memory (LSTM) may be used for error prediction.

[0068] Decision diamond 1104 indicates that the modes for the block under test may be ranked according to prediction error, and if the “best” mode, in terms of having the lowest prediction error, is not within a threshold prediction error difference of the next best mode(s), then that mode is output as the “best” mode at state 1106 for use for the corresponding decompressed block that is rendered according to the “best” mode of block 906 of FIG. 9 .

[0069] On the other hand, if the second or third best mode (or additional modes) in terms of error prediction are within the threshold prediction error difference of the best mode, the logic proceeds from decision diamond 1106 to decision diamond 1108 to identify the processing budgets of the best and next best mode(s) within the prediction error difference. If the processing budget of the best mode is not within the threshold budget difference of the next best mode(s) and the budget(s) of the next best mode(s) is lower than the budget of the best mode (meaning that a significant processing budget can be saved by using the next best mode(s) instead of the best mode), the logic moves to block 1110 to use the next best mode(s) as input to block 906 of FIG. 9. In this way, the next best mode(s) in terms of lowest prediction error may be used instead of the best mode in FIG. 9 if the "expensive" mode is best in terms of prediction error but is slightly better compared to the next best mode(s) and a significant processing budget can be saved by using the next best mode(s).

[0070] On the other hand, if decision diamond 1108 determines that the processing budget savings from using the next best mode(s) instead of the best mode is insignificant (e.g., the budget of the best mode is within a threshold difference of the budget of the next best mode(s)), the logic moves to block 1112 to compress and decompress the block under test using the best mode and input the decompressed block to block 906 of FIG. 9. Decision diamond 1114 tests as to whether the block under test is the last block of the current frame, and if not, at block 1116 the next block of the frame is obtained and processing returns to block 1102. On the other hand, if the last block of the frame has been processed, the logic moves to block 1118 to obtain the next frame and then returns to block 1100 to determine the processing budget for the blocks of that frame.

[0071] It should be appreciated that N versions of each training uncompressed block (one for each of the N block compression modes) may be pruned using a simpler technique than that shown in Figure 11. For example, the top M versions in terms of lowest error prediction may be used (1 < M < N-1, where N is an integer).

[0072] Although particular techniques have been shown and described in detail herein, it should be understood that the subject matter encompassed by this application is limited only by the scope of the claims.

Claims

1. An assembly including at least one processor, The at least one processor At least one machine learning (ML) engine is trained to derive high-resolution computer graphics textures (CGTs), at least in part, from low-resolution CGTs. compressing a plurality of training blocks comprising a first number of texels to create a compressed training block; decompressing the compressed training block to create a decompressed training block; inputting at least the decompressed training block to the ML engine for up-resolving the decompressed training block into a predicted high resolution training block; The instruction to create the assembly.

2. The instruction: Inputting a high-resolution block of ground truth corresponding to each decompressed training block into the ML engine for comparison with the predicted high-resolution training block. The assembly of claim 1 , wherein the assembly is operable as follows:

3. The instruction: inputting respective ones of the at least a portion of the plurality of training blocks to N compressors, each compressor operating according to a respective Nth block compression mode, to generate N respective outputs for each of the at least a portion of the plurality of training blocks; decompressing each of the N outputs to produce N decompressed training blocks for each one of the at least a portion of the plurality of training blocks; and For each one of said at least a portion of said plurality of training blocks, inputting said N decompressed training blocks to said ML engine for said ML engine to up-rezo into a predicted high resolution training block. The assembly of claim 1 , wherein the assembly is operable as follows:

4. The instruction: inputting respective ones of the at least a portion of the plurality of training blocks to N compressors, each compressor operating according to a respective Nth block compression mode to generate N respective outputs for each of the at least a portion of the plurality of training blocks; predicting each error for each of the N outputs; and selecting each Nth block compression mode based at least in part on each of said errors; The assembly of claim 1 , wherein the assembly is operable as follows:

5. Each one of the at least some of the plurality of training blocks is associated with a respective Nth block compression mode, and the instructions compressing a respective one of said at least a portion of said plurality of training blocks using each of said Nth block compression modes to create a respective one of said compressed training blocks. It is feasible to 5. The assembly of claim 4.

6. The instruction: selecting each Nth block compression mode based at least in part on each of the errors and each of the processing costs associated with each of the modes; 5. The assembly of claim 4, wherein the assembly is operable as follows:

7. The assembly of claim 1 , wherein the plurality of training blocks comprises mipmaps.

8. The assembly of claim 1 , wherein the CGT includes physically based rendering (PBR) and material data.

9. The assembly of claim 1 , wherein the high resolution CGT includes a mipmap that is one mip level higher than the low resolution CGT.

10. 1. An assembly including at least one computer storage device that is not a transitory signal, the at least one computer storage device comprising: inputting the decompressed texture blocks associated with the first resolution into at least one machine learning (ML) algorithm; inputting ground truth texture blocks associated with a second resolution higher than the first resolution into the ML algorithm; running the ML algorithm to train on the decompressed texture block and the ground truth texture block; and Following training, executing the ML algorithm to receive a low resolution texture from a source and generate a high resolution texture from the low resolution texture without decompressing or compressing the low resolution texture. comprising instructions executable by at least one processor such that assembly.

11. The instruction: inputting respective ones of the at least some of the decompressed texture blocks to N compressors, each compressor operating according to a respective Nth block compression mode, to generate N respective outputs for each of the at least some of the decompressed texture blocks; decompressing each of the N outputs to produce N decompressed training blocks for each one of the at least some of the decompressed texture blocks; and for each one of said at least a portion of said decompressed texture blocks, inputting said N decompressed training blocks to said ML algorithm for said ML algorithm to up-rezo a predicted high resolution training block; The assembly of claim 10, wherein the assembly is operable as follows:

12. The instruction: inputting respective ones of the at least a portion of the decompressed texture block to N compressors, each compressor operating according to a respective Nth block compression mode to generate N respective outputs for each of the at least a portion of the decompressed texture block; predicting each error for each of the N outputs; and selecting each Nth block compression mode based at least in part on each of said errors; The assembly of claim 10, wherein the assembly is operable as follows:

13. Each one of the at least some of the decompressed texture blocks is associated with a respective Nth block compression mode, and the instructions compressing a respective one of the at least a portion of the decompressed texture blocks using each of the Nth block compression modes to create a respective compressed training block; and Decompressing the compressed training block to create the decompressed texture block. The assembly of claim 12, operable as follows:

14. The instruction: selecting each Nth block compression mode based at least in part on each of the errors and each of the processing costs associated with each of the modes; The assembly of claim 12, operable as follows:

15. accessing at least one machine learning (ML) engine; training said ML engine using the decompressed texture blocks to generate high resolution textures from low resolution textures; A method comprising:

16. The method of claim 15 , wherein the ML engine generates the high resolution texture from the low resolution texture without using any decoding or encoding.

17. 16. The method of claim 15, comprising training the ML engine using N versions of each one of at least a portion of the decompressed texture block, the N versions being generated by each of N compression modes.

18. 16. The method of claim 15, comprising training the ML engine using less than N at least one version of each one of at least a portion of the decompressed texture block, the N versions being generated by each of N compression modes.

19. 20. The method of claim 18, wherein the less than N at least one version of each one of at least a portion of the decompressed texture block is selected based at least in part on a prediction error associated with each of the N modes.

20. 20. The method of claim 19, wherein the less than N at least one version of each one of at least a portion of the decompressed texture block is selected based at least in part on processing constraints associated with each of the N modes.

Images