Machine learning techniques to create high-resolution data structures representing textures from low-resolution data structures.

Abstract

To provide a machine learning method for creating a higher resolution data structure representing a texture from a lower resolution data structure, and an assembly.SOLUTION: A method generates a first mipmap of a texture having a first compression on the basis of a second mipmap of the same texture having a second compression without using compression or decompression in generating the first mipmap using machine learning (ML), and renders a computer graphics object using the first mipmap. ML engine training uses the blocks decompressed from originally compressed blocks as input.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] This application relates to a technically inventive non-conventional solution that necessarily results in specific technical improvements rooted in computer technology.

Background Art

[0002] In computer simulations such as computer games, an object is rendered by partially using "texture" data representing the surface of the object. As the texture data of a given object increases, the rendering resolution may increase. However, for bandwidth purposes, it is desirable not to transmit large texture data structures to the rendering device.

Summary of the Invention

[0003] As understood herein, so-called "mipmaps" can be used to save bandwidth in the following manner. A mipmap is a series of texture data structures, each of which represents a gradually reduced resolution of the previous texture data structure. Usually, this reduction is by half in each dimension. By using a high-resolution mipmap to render an object near the viewer and a low-resolution mipmap to render an object far from the viewer, bandwidth is saved. Usually, the mipmap level is chosen to be the level that most closely matches the pixel density of the image. Ideally, one texture pixel per screen pixel is desired. A texture pixel may also be referred to as a "texel" (a combination of texture and pixel).

[0004] However, as will be understood herein, to conserve memory, texture data is typically compressed into one of several block compression (BCn) modes that can be natively sampled by the GPU. The maximum resolution of a texture is usually limited by storage space constraints and the artist's authoring time. To generate high-resolution images from low-resolution images using existing machine learning-based techniques, etc., it is first necessary to decompress the BCn-compressed texture data, upsample it, and recompress it. This is undesirable because it requires extra storage space for both the uncompressed versions of the low-resolution and high-resolution images, and the BCn compression process is very complex and computationally expensive.

[0005] Therefore, the assembly includes at least one processor comprising instructions for training at least one machine learning (ML) engine to create high-resolution computer graphics textures (CGTs) from low-resolution CGTs. Training is achieved by compressing multiple training blocks having a first number of texels to create compressed training blocks, decompressing the compressed training blocks to create decompressed training blocks, and inputting the decompressed training blocks into the ML engine so that the ML engine can up-res (up-res) the decompressed training blocks into predicted high-resolution training blocks.

[0006] In some exemplary embodiments, the instruction may be executable to input a high-resolution block of ground truth corresponding to each decompressed training block into the ML engine for comparison with a predicted high-resolution training block.

[0007] In some embodiments of the first aspect, the instruction may be executable to input one of each of at least a subset of a plurality of training blocks to N compressors, each compressor operating according to the compression mode of its respective Nth block to produce N outputs for each of at least a subset of the plurality of training blocks. Each of the N outputs is decompressed to create N decompressed training blocks for each of at least a subset of the plurality of training blocks. The N decompressed training blocks are input to the ML engine for up-resolution to predict high-resolution training blocks.

[0008] In other embodiments, instead of inputting all N outputs into the ML engine for training, the prediction error may be obtained first for each of the N outputs, and the prediction error may be used to select a compression mode for each Nth block to compress and decompress each training block, and only the resulting decompressed training blocks may be input into the ML engine for training. When selecting the Nth compression mode, the processing cost associated with each mode may also be considered in addition to the prediction error.

[0009] In an unrestricted example, multiple training blocks may contain one or more mipmaps, a CGT may contain physically based rendering (PBR) and material data, and a high-resolution CGT may contain mipmaps one mip level higher than a low-resolution CGT.

[0010] In another embodiment, the assembly includes at least one computer storage, the at least one computer storage, which includes instructions executable by at least one processor to input decompressed texture blocks associated with a first resolution, rather than transient signals, to at least one machine learning (ML) algorithm. The instructions are executable to input ground truth texture blocks associated with a second resolution higher than the first resolution to the ML algorithm, and to run the ML algorithm to train with the decompressed texture blocks and the ground truth texture blocks. After training, the instructions are further executable to run the ML algorithm to receive a low-resolution texture from a source, and generate a high-resolution texture from the low-resolution texture without decompressing or compressing the low-resolution texture.

[0011] In another embodiment, the method includes accessing at least one machine learning (ML) engine and training the ML engine using decompressed texture blocks to generate high-resolution textures from low-resolution textures.

[0012] In this exemplary embodiment, the ML engine generates a high-resolution texture from a low-resolution texture without using decoding or encoding.

[0013] In some embodiments of this model, the method may include training an ML engine using N versions of at least a portion of the decompressed texture blocks, each of which is generated by one of N compression modes.

[0014] In other embodiments of this model, the ML engine may be trained using at least one version of at least one of the decompressed texture blocks generated by each of the N compression modes, with a maximum of N versions. The versions(s) can be selected at least in part on the prediction error associated with each of the N modes, and optionally also on the processing constraints associated with each of the N modes.

[0015] Details of this invention, both in terms of its structure and operation, can best be understood by referring to the attached drawings, in which similar reference numerals indicate similar parts. [Brief explanation of the drawing]

[0016] [Figure 1] This is a block diagram of an exemplary system including an example that follows the principles of the present invention. [Figure 2] An example of a texture communication path is shown. [Figure 3] An example of a PBR material component is shown. [Figure 4] This diagram illustrates the exemplary logic in flowchart format for providing ground truth training data. [Figure 5] This diagram illustrates the exemplary logic in flowchart format for training a texture rendering machine learning engine. [Figure 6] This example flowchart illustrates the logic for providing textures to the renderer. [Figure 7] This example flowchart illustrates the exemplary logic for upscaling an input texture using a machine learning engine. [Figure 8] An illustrative flowchart shows the logic for upscaling two texture data types together. [Figure 9]This diagram illustrates the exemplary logic in flowchart form for training a machine learning engine using decompressed textures. [Figure 10] An exemplary training system consistent with this principle is shown. [Figure 11] This example flowchart illustrates the logic for predicting the best compression mode. [Modes for carrying out the invention]

[0017] This disclosure generally relates to a computer ecosystem that includes a form of user information based on consumer electronics (CE) devices in a computer ecosystem. The systems herein may include server and client components connected via a network so that data can be exchanged between client and server components. Client components may include one or more computing devices, including portable computers such as portable televisions (e.g., smart TVs, internet-enabled televisions), laptop computers, and tablet computers, as well as other mobile devices, including smartphones and additional examples described below. These client devices may operate in a variety of operating environments. For example, some client computers may use operating systems obtained from Microsoft or Unix® or Apple or Google, for example. Using these operating environments, one or more browsing programs may run, such as browsers created by Microsoft, Google, or Mozilla, or other browser programs that can access web applications hosted by internet servers as described below.

[0018] The server may include one or more processors that execute instructions configuring the server to transmit and receive data through a network such as the Internet. Alternatively, the client and the server can be connected through a local intranet or a virtual private network. The server or the controller can be instantiated by a gaming console such as Sony PlayStation (registered trademark), a personal computer, or the like.

[0019] Information can be exchanged between the client and the server through the network. For this purpose and for security, the server and / or the client may include a firewall, a load balancer, temporary storage, and a proxy, as well as other network infrastructure for reliability and security. One or more servers can form an apparatus that implements a method for providing a secure community such as an online social website to network members.

[0020] As used herein, instructions refer to computer-implemented steps for processing information within a system. The instructions can be implemented in software, firmware, or hardware, and can include any type of programmed steps performed by components of the system.

[0021] The processor can be a single-chip processor or a multi-chip processor that can execute logic through various lines such as address lines, data lines, and control lines, as well as registers and shift registers. The processor can be implemented by or include one or more graphics processing units (GPUs).

[0022] Software modules described herein by flowcharts and user interfaces may include various subroutines, procedures, etc. Without limiting this disclosure, logic specified to be executed by a particular module can be redistributed to other software modules and / or aggregated into a single module and / or made available in a shareable library.

[0023] The principles of the present invention described herein can be implemented as hardware, software, firmware, or a combination thereof. Accordingly, exemplary components, blocks, modules, circuits, and steps are described in terms of their function.

[0024] Furthermore, with respect to the items mentioned above, the logic blocks, modules, and circuits described below can be implemented by digital signal processors (DSPs), field-programmable gate arrays (FPGAs), or application-specific integrated circuits (ASICs), other programmable logic devices such as discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The processor can be implemented by a controller or state machine, or a combination of computing devices.

[0025] The functions and methods described below, when implemented in software, can be written in a suitable language such as Java®, C#, or C++, and can be stored in or transmitted through computer-readable storage media such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM), or other optical disk storage such as digital versatile disks (DVDs), magnetic disk storage, or other magnetic storage devices such as removable thumb drives. Computer-readable media can be established by connections. Such connections may include, for example, hardwire cables including optical fibers and coaxial wires, as well as digital subscriber lines (DSLs) and twisted pair wires.

[0026] Components included in one embodiment can be used in any suitable combination in other embodiments. For example, any of the various components described herein and / or depicted in the figures can be combined, replaced, or excluded from other embodiments.

[0027] A "system having at least one of A, B, and C" (similarly, a "system having at least one of A, B, or C" and a "system having at least one of A, B, and C") includes systems having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.

[0028] Referring specifically to Figure 1, an exemplary ecosystem 10 is shown, which may include one or more exemplary devices described above and detailed below, according to the principles of the present invention. The first exemplary device included in system 10 is an exemplary primary display device, which in the shown embodiments is an audio-video display device (AVDD) 12, such as an internet-enabled TV, but is not limited thereto. Thus, AVDD 12 may alternatively be an electrical appliance or household item, e.g., a computer-controlled internet-enabled refrigerator, washing machine, or dryer. Alternatively, AVDD 12 may also be a computer-controlled internet-enabled ("smart") phone, tablet computer, notebook computer, wearable computer-controlled devices (e.g., a computer-controlled internet-enabled watch, a computer-controlled internet-enabled bracelet, etc.), other computer-controlled internet-enabled devices, computer-controlled internet-enabled implantable devices such as a computer-controlled internet-enabled music player, computer-controlled internet-enabled headphones, or an implantable skin device. Nevertheless, it should be understood that AVDD12 is configured to carry out the principles of the present invention (for example, to communicate with other CE devices to carry out the principles of the present invention, to execute the logic described herein, and to perform any other functions and / or operations described herein).

[0029] Therefore, to implement such a principle, AVDD12 can be established by some or all of the components shown in Figure 1. For example, AVDD12 may include one or more displays 14, which may be implemented by high-resolution or ultra-high-resolution, i.e., "4K" or "8K" (or high resolution) flat screens, and may be touch-enabled to receive consumer input signals by touch on the display. AVDD12 may also include one or more speakers 16 for outputting sound according to the principles of the present invention, and at least one additional input device 18, such as a keyboard or keypad or an audio receiver / microphone, for inputting audible commands to AVDD12 to control AVDD12. An exemplary AVDD12 may also include one or more network interfaces 20 for communicating over at least one network 22, such as the Internet, WAN, LAN, etc., under the control of one or more processors 24. Therefore, the interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface. It should be understood that the processor 24 controls AVDD 12 to implement the principles of the present invention, including other elements of AVDD 12 as described herein, such as controlling the display 14 to present images to it and receiving input from it. Furthermore, it should be noted that the network interface 20 may be, for example, a wired or wireless modem or router, or other suitable interface such as a wireless telephony transceiver or the Wi-Fi transceiver described above.

[0030] In addition to the foregoing, AVDD12 may also include one or more input ports 26, such as a USB port for physically connecting to another CE device (e.g., using a wired connection) and / or a headphone port for connecting headphones to AVDD12 to present audio from AVDD12 to a consumer through headphones. AVDD12 may further include one or more non-transient computer memories 28, such as disk-based storage or solid-state storage (including, but not limited to, flash memory). Also, in some embodiments, AVDD12 may include, but not limited to, a location receiver or place receiver such as a cell phone receiver, GPS receiver, and / or altimeter 30, which is configured to receive geographic location information from, for example, at least one satellite or cell phone tower and provide the information to a processor 24 and / or determine the altitude at which AVDD12 is positioned together with the processor 24. However, it should be understood that another suitable location receiver other than a cell phone receiver, GPS receiver, and / or altimeter may be used, for example, in all three dimensions, according to the principles of the present invention.

[0031] Continuing the description of AVDD12, in some embodiments, AVDD12 may include one or more cameras 32, the cameras 32 may be, for example, digital cameras such as infrared imaging cameras and webcams, and / or cameras integrated into AVDD12 and controllable by a processor 24 for collecting photographs / images and / or videos in accordance with the principles of the present invention. AVDD12 may also include a Bluetooth® transceiver 34 and other NFC elements 36 for communicating with other devices using Bluetooth® and / or Near Field Communication (NFC) technology. An exemplary NFC element may be a radio-frequency identification (RFID) element.

[0032] Furthermore, AVDD12 may include one or more auxiliary sensors 37 that provide input to the processor 24 (e.g., motion sensors such as accelerometers, gyroscopes, cyclometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed sensors and / or cadence sensors, gesture sensors (e.g., sensors for detecting gesture commands)). AVDD12 may also include other sensors that provide input to the processor 24, such as one or more climate sensors 38 (e.g., barometers, humidity sensors, wind sensors, light sensors, temperature sensors, etc.) and / or one or more biometric sensors 40. In addition to the foregoing, it should be noted that AVDD12 may also include an IR transmitter and / or IR receiver and / or IR transceiver 42, such as an infrared (IR) data association (IRDA) device. A battery (not shown) may be provided to power AVDD12.

[0033] Referring further to Figure 1, in addition to AVDD 12, system 10 may include one or more other CE device types. In one example, a first CE device 44 may be used to send a message to a second CE device 46, and the second CE device 46 may include components similar to those of the first CE device 44, so it will not be described in detail. In the example shown, only two CE devices 44 and 46 are shown, but of course, fewer or more devices may be used.

[0034] An exemplary, non-limiting first CE device 44 may be established by any one of the above-described devices, e.g., a portable wireless laptop computer or tablet computer or notebook computer or mobile phone, and therefore may have one or more components as described below. A second CE device 46 may be established by a wireless telephone, but is not limited to this. A second CE device 46 may implement a portable handheld remote control (RC). A second CE device 46 may implement a virtual reality (VR) and / or augmented reality (AR), or a head-mounted display (HMD). CE devices 44, 46 may include some or all of the components shown in the case of AVDD 12.

[0035] At least one server 50 may include at least one server processor 52, at least one computer memory 54 such as disk-based storage or solid-state storage, and at least one network interface 56 that, under the control of the server processor 52, enables communication with other devices in Figure 1 via the network 22 and, in fact, facilitates communication between the server and client devices in accordance with the principles of the present invention. Note that the network interface 56 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface such as a wireless telephony transceiver.

[0036] Therefore, in some embodiments, the server 50 may be an internet server and may include and perform a “cloud” function such that, in exemplary embodiments, devices of the system 10 can access the “cloud” environment via the server 50. Alternatively, the server 50 may be implemented by a game console or other computer in the same room as or near the other devices shown in Figure 1.

[0037] The devices described herein may include some or all of the various components shown in Figure 1, as necessary.

[0038] Before referring to Figure 2, it's important to understand that a "texture" is a data structure that can be mapped to an image to characterize the surface of a rendered object. The basic data elements of a texture data structure are texture elements, or texels (combinations of texture and pixels). A texture is represented by an array of texels that represent the texture space. Texels are mapped to pixels in the rendered image to define the surface of the rendered image.

[0039] Therefore, since textures are data and not images, the neural network (NN) training described below does not necessarily rely on the principle of perceptual error. While there may be exceptions for albedo (which will be discussed further below), generally, training texture data uses an error metric specific to the particular data.

[0040] Some types of data, such as normals and roughness, can be paired together because they are interrelated, allowing the two sets of data to mitigate each other's errors. More specifically, in PBR rendering, there is a relationship between normal map data and roughness data (sometimes called gloss data). Roughness essentially represents the distribution of normals across the entire texture pixel. Therefore, there is a technique to correct the roughness to account for the data lost when reducing the resolution of the normal map when generating mipmaps, essentially anti-aliasing the normal map. In this case, there is a close relationship between the normal map and its corresponding roughness map.

[0041] The additional details regarding textures, physically based rendering (PBR), and materials are a general set of guidelines for rendering realistic material and light interactions, including the use of various types of data stored in textures. Light is modeled as either diffuse or reflected. Diffuse light is generally view-independent and generally does not change regardless of the angle from which the material is viewed. Reflected light, on the other hand, is view-dependent to emulate; glare is an example.

[0042] Generally, PBR textures include an "albedo" parameter that characterizes the diffuse light reaction of a material. For example, in the case of an emulated surface of polished wood, the albedo texture includes the wood grain pattern and color variations, but does not include shape information and is very flat. Metals have no albedo because all of their light reactions result from reflection.

[0043] As mentioned above, PBR textures can also include a "normal" parameter. A normal map defines the details of small shapes on a surface and specifically represents the surface normals of the underlying surface. This may or may not be separated from the geometric normals of the triangular data used in rendering. This is vector data stored as a texture and can be displayed, but it is not an image in itself. In the wood example, this is mostly flat where the wood is smooth, but the surface of the wood may include etching or carving details.

[0044] As mentioned above, PBR textures may also include a gloss / roughness parameter that defines the degree of roughness of the surface being rendered. Generally, this can be seen as the dispersion of subpixel normal data (details smaller than texels in the normal map). In the example of wood, this could include data about scratches on the wood (imagine rubbing wood with sandpaper). Polished wood will have a lower roughness because it is relatively smooth.

[0045] Another PBR parameter is reflectance, which represents the reaction of reflected light. For most nonmetallic materials, reflectance is colorless. The reflectance of almost all nonmetals is constant at 2% (the amount of light directly reflected). In the case of mirrors, the reflectance approaches 100%.

[0046] Some PBR techniques attempt to leverage the fact that nonmetals have a constant reflectivity and metals have no albedo. This typically involves using albedo texels, which are metallic surfaces, for a different purpose, and instead representing reflectivity (e.g., color, gold, brass), by storing additional information in a separate texture channel to identify which texels are metallic or not. Generally, this information is referred to as "metallicity." This encoding is commonly done to conserve texture memory by not storing three channels for albedo and another three channels for reflectivity.

[0047] In general, textures can serve various purposes and may differ from one game engine to another.

[0048] Considering the above explanation of textures, typically multiple versions of the same texture are generated with various compressions (and therefore, various resolutions). Specifically, a single texture can be represented by a full mipmap chain of the texture. Mipmapping takes an image and successively reduces the resolution by half. Therefore, for a given texture (e.g., 1024x1024), its 512x512 version and its 256x256 version, etc., are also stored in memory. This improves performance, visual fidelity, and facilitates texture streaming. Texture streaming attempts to conserve memory by loading only the "tails" of mipmaps (lower mip levels and all lower resolution mip levels below them) required for a given texture, based on the screen size of the rendered object. For example, a 256x256 mip level and anything lower than the mip chain of a 1024x1024 texture might be all that's needed to load an object that's far away. Which mip levels are loaded changes on demand based on the changing game environment and viewer position. As you get closer to an object, the mip level, or "detail level," resolution loaded into memory to render the nearby object increases.

[0049] Textures can use various types of compression. One type is block compression, sometimes referred to as BCn compression. This can be a lossy texture compression that can be decompressed in-place by the graphics processing unit (GPU). Because block compression does not require decompressing the entire image, the GPU can decompress the data structure while sampling the texture as if it were completely uncompressed.

[0050] Block compression techniques compress 4x4 blocks of pixels into a single (smaller) data packet. Generally, this involves selecting two or more "endpoint" colors (depending on the BC compression type), and these "endpoint" colors carry some per-pixel information about how to blend these two colors at each pixel. The endpoint colors are shared across the entire 4x4 pixel block. For example, in an image with only red, blue, and purple pixels, the compressor is likely to choose one endpoint for red and the other for blue. The purple pixels would have values ​​that blend the two together.

[0051] Most different BC types differ in the number of texture channels they have (for example, BC4 is one channel grayscale, i.e., "black and white"). BC6 and BC7 are special because they introduce the concept of modes, which determine the interpretation of each block. For other BC modes, all blocks are encoded in the same way, with the same number of bits assigned to the color and blend values ​​of the endpoints. With different BC6 / 7 modes, those bits are assigned differently to each block. This allows the compressor to make different quality trade-offs in different areas of the texture.

[0052] Figure 2 shows a texture source 200 that transmits textures for rendering according to the above principle via a communication path 202, such as a local data bus or a wireless / wired network link, to a texture renderer 204. The texture renderer 204 typically includes one or more GPUs with memory to render images according to image data and texture data on a display.

[0053] Figure 3 shows that in the embodiments of this specification, only the tail texture 300 needs to be sent from the source 200 to the renderer 204. The renderer 204 runs a machine learning engine 302 on the texture to upscale it to the next higher resolution texture 304 without requiring decoding / encoding, and therefore without requiring a codec. The machine learning engine 302 may include one or more trained neural networks, such as generative, noise-based, and possibly adversarial networks.

[0054] Therefore, machine learning is used to generate mipmaps for textures that are one level higher than all input mipmaps for streaming textures (e.g., from a mipmap chain created on disk). The new (high-resolution) mipmaps are then introduced as if they were present on disk, and are simply generated procedurally instead. Since textures can be stored on disk in BCN compression format, the network generates new compressed mip levels from the highest existing compressed mip levels in memory. If the cost of generating mip levels at runtime is too high, mip levels can be generated offline and stored on disk using the same method.

[0055] Figure 4 shows an example of the training principle of the machine learning engine 302 in Figure 3. An existing texture library may be accessed in block 400 for ground truth training, compressed in block 402, and then the next mip level down (again, by half) may be established. In other words, in block 400, a fully uncompressed (and therefore highest resolution) mipmap may be accessed, and in block 402, it may be compressed by half to create a compressed (and therefore lower resolution) mipmap. The mipmaps generated in block 402 may be sequentially compressed according to the principles herein to create a full set of mipmaps for each input uncompressed mipmap, and in block 404, the ground truth is generated.

[0056] Figure 5 shows that a simple network can access a single compressed BC block data packet from the ground truth set at block 500 and generate four BC block data packets for a higher mip level at block 502, for example, efficiently progressing from a 4x4 block pixel to an 8x8 block pixel. Alternatively, instead of taking a single BC block as input, or in addition to that, the surrounding block neighborhood (e.g., the eight surrounding blocks) can also be provided as input, which can better inform the network of its features.

[0057] Training can be performed using backpropagation or gradient descent. Training may be performed using an 80 / 20 training-test split, in which 80% of the ground truth data is used to train or set the weights in the NN using random samples, and then an additional 20% of the test data downscaled files are input and compared to what the NN produces. In other words, the output of the NN can be compared to the 20% of the ground truth full-resolution files that were not input during the training session.

[0058] Because there are almost entirely different types of data between different material attribute textures, a different network may be trained for each material attribute type. For example, one network may be trained to upscale reflective texture data, and another network may be trained to upscale albedo.

[0059] For normals and roughness, where similar data is stored and therefore has some interrelationship, a single network can be trained to upscale them together. The same considerations can be applied to other related pairs or groups of material attributes. In the case of a shared normal and roughness upscaling network, one BC block with normal data and one BC block of roughness data in the same related location may be input, and the network may output four BC blocks of normal data and four BC blocks of roughness data.

[0060] Figure 6 shows that textures can be compressed in block 600 and sent to the renderer in block 602. In one example, the lowest resolution (most compressed) "tail" texture in the mipmap chain is sent to the renderer. The renderer can upscale them on the fly using a trained machine learning engine described herein, without requiring a codec. In other embodiments, relatively uncompressed high-resolution textures (or the uncompressed base texture itself) may be provided as input to a renderer, which can use machine learning to generate more compressed, lower-resolution textures.

[0061] Figure 7 shows the renderer logic. In state 700, the input texture is received. In block 702, the texture is processed by a trained machine learning engine, and in block 704, a different compressed texture is output, either with a higher compression ratio (lower resolution) or a lower compression ratio (higher resolution) than the input texture. The texture from block 704 is used to render the image on the display.

[0062] Figure 8 shows that in the case of a shared normal and roughness up-rescaling network, at block 800, one BC block with normal data is received by the machine learning engine, and at block 802, one BC block of roughness data is received at the same related location. At block 804, the machine learning engine upscales the two input blocks together to output four BC blocks of normal data and four BC blocks of roughness data.

[0063] Referring to Figure 9, in addition to what was suggested above, the technique for training the ML engine is shown. Starting at block 900, ground truth textures or texture portions, such as 2x2 texel blocks, and corresponding high-resolution blocks, such as 4x4 texel blocks, are received. Moving to block 902, the low-resolution (e.g., 2x2) blocks are compressed using one or more block compression modes, and then decompressed at block 904 using the same mode(s). The decompressed blocks, along with the corresponding high-resolution ground truth blocks, are input into the ML engine or algorithm at block 906 for training.

[0064] The above technique, which uses decompressed blocks instead of the original uncompressed blocks, allows us to consider the compromises that a particular compression mode(s) may make, providing more realistic training.

[0065] Figures 10 and 11 illustrate additional techniques related to the overall logic of Figure 9. In Figure 10, uncompressed NxN texture blocks, such as 2x2 texel blocks, may be sent to multiple block compressors 1002, 1004, each operating in its respective block compression mode. Compressors 1002, 1004 compress the blocks and send the compressed blocks to their respective decompressors 1006, 1008, each operating in the same mode as the compressor that received the compressed blocks. The decompressed blocks (i.e., N versions of each decompressed block, one for each compression mode) can be used in block 906 in Figure 9 to train an ML engine or algorithm, or the decompressed blocks can be sent to an error prediction module 1010 to prune which of the N versions of each decompressed block to use in block 906 in Figure 9.

[0066] Figure 11 shows an unrestricted example of the error prediction module 1010. For a particular frame (e.g., a 2x2 texel block) at a first resolution (generally low resolution) from which uncompressed training texture blocks are obtained, starting with block 1100, the processing budget or cost base may be established, for example, in terms of time. Thus, different (higher or lower) processing budgets may be stipulated depending on the type of frame. For example, a short or less complex frame may have a larger processing budget than a long, large, or more complex frame.

[0067] Moving to block 1102, for each mode of each input training block, the model predicts the error for each mode of that block when accurately outputting a high-resolution block (e.g., a 4x4 block). Recurrent neural networks (RNNs), such as Long Short-Term Memory (LSTM), may be used for error prediction, rather than being limited.

[0068] The decision diamond 1104 indicates that the modes of the block under test can be ranked according to the prediction error, and if the “best” mode, in terms of having the lowest prediction error, is not within the threshold prediction error difference range of the next best mode(s), then that mode is output as the “best” mode in state 1106 for the use of the corresponding decompression block, which will be rendered according to the “best” mode of block 906 in Figure 9.

[0069] On the other hand, if the second or third best mode (or additional mode) in terms of error prediction is within the threshold predicted error difference of the best mode, the logic moves from decision diamond 1106 to decision diamond 1108 to identify the processing budgets of the best and second best modes (or more) within the predicted error difference. If the processing budget of the best mode is not within the threshold budget difference of the second best modes (or more), and the budget of the second best modes (or more) is lower than the budget of the best mode (meaning that a considerable processing budget can be saved by using the second best modes (or more) instead of the best mode), the logic moves to block 1110 to use the second best modes (or more) as input to block 906 in Figure 9. Thus, if an "expensive" mode is the best in terms of predicted error, but only slightly better than the second best modes (or more), and a considerable processing budget can be saved by using the second best modes (or more), then the second best mode (or more) in terms of the lowest predicted error may be used instead of the best mode in Figure 9.

[0070] On the other hand, if the decision diamond 1108 determines that the budget savings from using the next best mode instead of the best mode are minimal (for example, the budget of the best mode is within the threshold difference of the budget of the next best mode), the logic moves to block 1112, compresses and decompresses the block under test using the best mode, and inputs the decompressed block into block 906 in Figure 9. The decision diamond 1114 tests whether the block under test is the last block of the current frame; if not, block 1116 takes the next block of the frame, and processing returns to block 1102. On the other hand, if the last block of the frame has been processed, the logic moves to block 1118, takes the next frame, and then returns to block 1100 to determine the processing budget for the blocks in that frame.

[0071] It should be understood that a simpler technique than the one shown in Figure 11 may be used to prune N versions of each training uncompressed block (one for each of the N block compression modes). For example, the top M versions may be used in terms of lowest error prediction (1 ≤ M ≤ N-1, where N is an integer).

[0072] While certain technologies are shown and described in detail herein, it should be understood that the subject matter covered by this application is limited solely to the claims.

Claims

1. An assembly comprising at least one processor, The aforementioned at least one processor is Train at least one machine learning (ML) engine to convert high-resolution computer graphics textures (CGTs) from low-resolution CGTs, at least partially. Compressing multiple training blocks containing a first number of texels using N different block compression modes to create N compressed training blocks, The process involves decompressing the aforementioned N compressed training blocks using their respective compression modes to create N decompressed training blocks. The system obtains a prediction error for each decompression training block, and selects a compression mode to be used for each training block, based at least partially on the prediction error. The decompression training block in the selected compression mode is input to the ML engine, and the ML engine inputs the decompression training block to predict high-resolution training blocks. It consists of instructions created by assembly.

2. The instruction is, Identify the processing cost associated with each compression mode, and select the compression mode to use for each training block, based at least partially on the predicted error and the processing cost. The assembly according to claim 1, which is executable in this manner.

3. The aforementioned instruction, To compare with the aforementioned high-resolution training blocks, a high-resolution block of ground truth corresponding to each decompressed training block is input to the ML engine. The assembly according to claim 1, which is executable in this manner.

4. The assembly according to claim 1, wherein the plurality of training blocks include mipmaps.

5. The assembly according to claim 1, wherein the CGT includes physically based rendering (PBR) and material data.

6. The assembly according to claim 1, wherein the high-resolution CGT includes a mipmap that is one mip level higher than the low-resolution CGT.

7. An assembly comprising at least one computer storage that is not a transient signal, wherein the at least one computer storage is Multiple training blocks associated with the first resolution are compressed using N different block compression modes to create N compressed texture blocks. Each of the aforementioned N compressed texture blocks is decompressed to create N decompressed texture blocks. For each decompressed texture block, obtain the prediction error, and based at least partially on the prediction error, select the compression mode to be used for each training block. The decompressed texture block, using the selected compression mode, is input into at least one machine learning (ML) algorithm. A ground truth texture block associated with a second resolution higher than the first resolution is input to the ML algorithm. The ML algorithm is executed to train the decompressed texture block and the ground truth texture block, and, Following training, the ML algorithm is executed to receive a low-resolution texture from the source, and a high-resolution texture is generated from the low-resolution texture without decompressing or compressing it. The instructions include instructions that can be executed by at least one processor, assembly.

8. The instruction is, Identify the processing cost associated with each compression mode, and select the compression mode to use for each training block, based at least partially on the predicted error and the processing cost. The assembly according to claim 7, which is executable in this manner.

9. Access to at least one machine learning (ML) engine, Compressing multiple training blocks using N different block compression modes to create N compressed texture blocks, The process involves decompressing each of the aforementioned N compressed texture blocks to create N decompressed texture blocks, The system obtains a prediction error for each decompressed texture block, and selects a compression mode to use for each training block based at least partially on the prediction error. The ML engine is trained using decompressed texture blocks with the selected compression mode to generate high-resolution textures from low-resolution textures. Methods that include...

10. The method according to claim 9, wherein the processing cost associated with each compression mode is identified, and the compression mode to be used for each training block is selected based at least in part on the predicted error and the processing cost.

11. The method according to claim 9, wherein the ML engine generates the high-resolution texture from the low-resolution texture without using decoding or encoding.

Images