Method and system for generating super-resolution of block compressed textures for texture mapping applications

Abstract

A method for super-resolution of block compressed textures is provided. The method includes receiving a first texture block having a first block size. Based on applying a first block compression (BC) scheme to the received first texture block, coded texture values ​​are generated in a compressed domain. Additionally, a first machine learning model is applied to the generated coded texture values ​​to generate super-resolution coded texture values ​​in the compressed domain. The generated super-resolution coded texture values ​​are processed to generate a second texture block having a second block size. The second block size is larger than the first block size.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS / INCORPORATED BY REFERENCE

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 003,953, filed on April 2, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Various embodiments of the present disclosure relate to three-dimensional (3D) rendering systems, image super-resolution, and machine learning. More specifically, various embodiments of the present disclosure relate to methods and systems for performing super-resolution of block compressed textures for texture mapping applications. Background Art

[0004] Advances in 3D rendering systems and computer graphics have led to the adoption of high-resolution textures for various 3D rendering applications such as games, augmented reality, virtual reality, and 2D / 3D animation. Traditionally, developers or studios (e.g., game studios) want to render computer graphics (e.g., in games) with high fidelity, but want to avoid releasing texture libraries with high resolution. This can keep the file size of the content (e.g., game files) as small as possible. Without texture compression, it may be difficult to reasonably reduce the file size. For example, existing methods compress texture images into encoded bit streams. The encoded bit stream can be decoded into a low-resolution image. The low-resolution image can be scaled up with known methods (such as bilinear or bicubic interpolation) to obtain a high-resolution image, which can be re-encoded into another bit stream and sent to a graphics processing unit for rendering. Due to the high-fidelity requirements, existing interpolation (e.g., bilinear, bicubic) methods may not be applicable or suitable because they produce blurred images. Moreover, the computational cost associated with existing methods is higher, which makes them less useful for real-time or near real-time rendering applications.

[0005] Limitations and disadvantages of conventional and traditional approaches will become apparent to those skilled in the art by comparing the described systems with certain aspects of the present disclosure, as set forth in the remainder of this application and with reference to the accompanying figures. Summary of the invention

[0006] There is provided a method and system for performing super-resolution of block compressed textures for texture mapping applications, substantially as shown in and / or described in connection with at least one of the figures as more fully set forth in the claims.

[0007] These and other features and advantages of the present disclosure may be understood by reading the following detailed description of the disclosure and the accompanying drawings, in which like reference numerals refer to like parts throughout. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a block diagram illustrating an exemplary network environment for super-resolution of block compressed textures for texture mapping applications according to an embodiment of the present disclosure.

[0009] Figure 2 is a block diagram illustrating an exemplary system for super-resolution of block compressed textures for texture mapping applications according to an embodiment of the present disclosure.

[0010] Figure 3 is a block diagram illustrating an exemplary implementation of a system as a codec chipset for super-resolution of block compressed textures according to an embodiment of the present disclosure.

[0011] Figure 4A and Figure 4B Together, exemplary operations of super-resolution of block compressed textures for texture mapping applications according to embodiments of the present disclosure are illustrated.

[0012] Figure 5 is a diagram illustrating an exemplary architecture of a machine learning model for super-resolution of block compressed textures according to an embodiment of the present disclosure.

[0013] Figure 6 is a diagram illustrating an exemplary operation of training a machine learning model for super-resolution of block compressed textures according to an embodiment of the present disclosure.

[0014] Figure 7 is a diagram illustrating an exemplary architecture of an exemplary pre-trained machine learning model for training a first machine learning model according to an embodiment of the present disclosure.

[0015] Fig. 8A and Figure 8B 1 and 2 collectively illustrate flow charts of exemplary methods for super-resolution of block compressed textures according to embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] The embodiments described below may be found in the disclosed system and method for super-resolution of block compressed textures for texture mapping applications. Exemplary aspects of the present disclosure provide a system that may be configured to receive a texture image (e.g., a texture image associated with a gaming application). The system may divide the received texture image into a plurality of texture blocks, each of which may have a first block size (e.g., 4×4 texels). Based on applying a first block compression (BC) scheme to the first texture block, the system may generate a coded texture value in a compressed domain. The system may apply a first machine learning (ML) model to the generated coded texture value to generate a super-resolution coded texture value in a compressed domain. The system may process the generated super-resolution coded texture value to generate a second texture block having a second block size (e.g., 8×8 texels). The second block size may be larger than the first block size.

[0017] Traditionally, a developer or studio (e.g., a game studio) wants to render computer graphics with high fidelity (e.g., in a game), but wants to avoid releasing texture libraries with very high resolutions to keep file sizes as small as possible (storage and transmission costs). The present disclosure allows content (e.g., game content or extended reality (XR) content, or animated content) to be delivered with lower resolution textures, but rendered in real time or near real time with higher resolution textures. The present disclosure does not require decoding and re-encoding of the bitstream or modification of the software development environment (such as a game engine). Instead, it directly leverages machine learning in the compressed domain to achieve super-resolution at runtime.

[0018] The present disclosure solves the unmet needs of developers and studios. For example, for game studios, such requirements include providing texture super-resolution for games at runtime with high fidelity and high speed (real-time or near real-time rendering). Due to the high fidelity requirements, existing methods based on interpolation (e.g., bilinear, bicubic) may not be applicable because they produce blurred images. Due to the complex mapping functions (i.e., machine learning models) in the compressed domain, the present disclosure not only produces better image quality than those existing methods. Known existing methods can use encoded bit streams, decode them into images, and scale up lower resolution images with known algorithms (such as bilinear or bicubic interpolation). Thereafter, the high-resolution image can be re-encoded into another bit stream and sent to a graphics processing unit (GPU) for rendering. However, this process may be computationally expensive, and the interpolation method used may cause unsatisfactory image quality, such as blurred edge boundaries.

[0019] Figure 1 is a block diagram illustrating an exemplary network environment for super-resolution of block compressed textures for texture mapping applications according to an embodiment of the present disclosure. Figure 1, a network environment 100 is shown. The network environment 100 may include a system 102 and a display device 106. A plurality of machine learning (ML) models 104 on the system 102 are also shown. The plurality of ML models 104 may include a first ML model 104A, a second ML model 104B, ..., and an Nth ML model 104N. A texture image 108 and a first texture block 110 of the texture image 108 are also shown.

[0020] System 102 may include suitable logic, circuitry, and interfaces that may be configured to generate a super-resolution texture from an input low-resolution texture (e.g., texture image 108). The generated super-resolution texture may be used in one or more operations of a graphics rendering pipeline. By way of example and not limitation, such operations may be a shader operation or part of a shader pipeline on a graphics processing unit (GPU) of system 102.

[0021] For image super-resolution, the system 102 may apply a first block compression (BC) scheme on a first texture block 110 (having a first block size) of the texture image 108 to generate an encoded texture value (in a compressed domain). The system 102 may apply a first ML model 104A to the generated encoded texture value to generate a super-resolution encoded texture value in the compressed domain. Based on the processing of the super-resolution encoded texture value, the system 102 may also generate a second texture block 112 having a second block size (which may be larger than the first block size). Examples of the system 102 may include, but are not limited to, a gaming device (such as a gaming console, a wearable headset (such as an extended reality (XR) headset), or an in-vehicle gaming system), a personal computer, a server (such as a gaming server or a cloud-based gaming streaming server (which provides gaming as a service)), a gaming engine, a GPU (or a cluster of connected GPUs), a personal computer, a smartphone, a mobile computer, a gaming monitor with an integrated GPU, a mainframe, a computer workstation, and / or any computing device capable of rendering graphics.

[0022] Each of the plurality of ML models 104 may be a regression model that may be trained to interpolate information (or additional coded texture values ​​in a compressed domain), which may be required to upscale or upsample a texture block having a first block size (e.g., 4×4 texels) such as the first texture block 110 to a texture block having a second block size (e.g., 8×8 texels) such as the second texture block 112. In an embodiment, each of the plurality of ML models 104 may be trained separately to handle coded texture values ​​associated with a particular mode of the BC compression scheme.

[0023] Each ML model (e.g., the first ML model 104A) can be defined by its hyperparameters (e.g., the number of weights, the cost function, the input size, the number of layers, etc.). The hyperparameters of the first ML model 104A can be adjusted before or after each training cycle of the corresponding ML model. During several rounds of training, parameters such as the weights of the corresponding ML model can be updated until a global minimum of the cost function for the first ML model 104A is reached. After training the input features of the training data set for several rounds, the first ML model 104A can be determined to be trained.

[0024] The first ML model 104A may include electronic data that may be implemented as a software component of an application executable, for example, on a system. The first ML model 104A may rely on a library, an external script, or other logic / instructions for execution by a processing device such as the system 102. The first ML model 104A may include code and routines configured to enable a computing system such as the system 102 to perform one or more operations, such as generating super-resolution encoded texture values. Additionally or alternatively, the first ML model 104A may be implemented using hardware, including but not limited to a processor, a microprocessor (e.g., for performing one or more operations or controlling the performance of one or more operations), a field programmable gate array (FPGA), a coprocessor (such as an artificial intelligence (AI) accelerator), or an application specific integrated circuit (ASIC). Alternatively, in some embodiments, the first ML model 104A may be implemented using a combination of both hardware and software.

[0025] In some embodiments, the first ML model 104A may be a neural network-based model. A neural network model may be a computing network or system of artificial neurons arranged as nodes in multiple layers. The multiple layers of the neural network model may include an input layer, one or more hidden layers, and an output layer. Each of the multiple layers may include one or more nodes or artificial neurons (e.g., represented by circles). The outputs of all nodes in the input layer may be coupled to at least one node of the (one or more) hidden layers. Similarly, the input of each hidden layer may be coupled to the output of at least one node in the other layers of the neural network. The output of each hidden layer may be coupled to the input of at least one node in the other layers of the neural network. The (one or more) nodes in the last layer may receive input from at least one hidden layer to output a result. The number of layers and the number of nodes in each layer may be determined based on the hyperparameters (and topological information) of the neural network. Such hyperparameters may be set before training, during training, or after training the neural network on a training data set.

[0026] Each node of a neural network model may correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) with a set of parameters that are adjustable during training of the network. The set of parameters may include, for example, weight parameters, regularization parameters, etc. Each node may use a mathematical function to compute an output based on one or more inputs from nodes in other layers (e.g., previous layers) of the neural network. All or some of the nodes of a neural network may correspond to the same or different mathematical functions. For example, in Figure 5 An exemplary implementation of the first ML model 104A as a neural network based model is described in .

[0027] In the training of the neural network model, one or more parameters of each node of the neural network can be updated based on whether the output of the last layer for a given input (from the training data set) matches the correct result based on the loss function for the neural network. The above process can be repeated for the same or different inputs until the minimum value of the loss function can be reached and the training error can be minimized. Several methods for training are known in the art, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, meta-heuristics, etc. For example, in Figure 6 The details of training the first ML model 104A are further described. Other ML models (such as the second ML model 104B and the Nth ML model 104N) can be functionally (parameter-wise and topologically) similar to the first ML model 104A (e.g., Figure 1 For the sake of brevity, the present disclosure omits descriptions of other ML models in the plurality of ML models 104.

[0028] The display device 106 may include suitable logic, circuit systems, and interfaces that may be configured to display three-dimensional or two-dimensional graphics including super-resolution textures (generated by the system 102). The display device 106 may be implemented by several known technologies, such as, but not limited to, at least one of a liquid crystal display (LCD) display, a light emitting diode (LED) display, a plasma display, or an organic LED (OLED) display technology or other display device. According to an embodiment, the display device 106 may refer to a display screen of an electronic device, a smart phone, a computer, a head mounted device (HMD), a smart glasses device, a perspective display, a projection-based display, an electrochromic display, or a transparent display.

[0029] exist Figure 1 , the display device 106 is shown as being separate from the system 102. However, the present disclosure may not be limited thereto, and in some embodiments, the display device 106 may be integrated into the system 102 without departing from the scope of the present disclosure.

[0030] In operation, the system 102 may receive a texture image 108. For example, the texture image 108 may be received from a server or may be retrieved locally from a storage unit of the system 102. The texture image 108 may be associated with a graphics rendering application such as, but not limited to, games, augmented reality, virtual reality, and 2D / 3D animation. The system 102 may divide the received texture image 108 into a plurality of texture blocks. The plurality of texture blocks may include, for example, a first texture block 110. Each of the plurality of texture blocks may have a first block size. As an example and not limitation, the first block size may be 4×4 texels (also referred to as texture pixels or texture elements). In many scenarios, the first block size may not be sufficient for graphics rendering applications that require higher texture resolution. As described herein, each texture block may need to be rescaled or upsampled (e.g., 4 times) to meet the requirements of the graphics rendering application.

[0031] The system 102 may select a first block compression (BC) scheme from a plurality of block compression schemes. According to an embodiment, the first BC scheme specifies a first BC format, which is one of several BC formats, such as a BC1 format, a BC2 format, a BC3 format, a BC4 format, a BC5 format, a BC6 format, or a BC7 format. In some embodiments, the first BC format specified in the first BC scheme is in a first mode, which may be one of fourteen modes of the BC6 format or eight modes of the BC7 format.

[0032] After selecting the BC scheme, the system 102 may apply the first BC scheme to the received first texture block 110. Based on the application of the first BC scheme, the system 102 may generate a first compressed bitstream. In some embodiments, the system 102 may also convert the first block of code bits of the generated first compressed bitstream to generate a coded texture value in the compressed domain. For example, Figure 4A Further details on the generation of encoded texture values ​​are provided in.

[0033] The system 102 may also select an ML model (e.g., a first ML model 104A) from the plurality of ML models 104 based on determining that the first compressed bitstream may be associated with a first mode (e.g., mode 1) of a first BC format (BC7 format) specified in a first BC scheme. Each ML model in the plurality of ML models 104 may be associated with a corresponding mode of the first BC format. The system 102 may apply the selected first ML model 104A to the generated coded texture values ​​to generate super-resolution coded texture values ​​in a compressed domain. The super-resolution coded texture values ​​may include a set of coded texture values ​​that may be required to enlarge (or upsample) a first texture block 110 having a first block size (e.g., 4×4 texels) to a second texture block 112 having a second block size (e.g., 8×8 texels). For example, in Figure 4BDetails of applying the first ML model 104A to generate super-resolution encoded texture values ​​are further provided in.

[0034] After generating the super-resolution coded texture values, the system 102 may process the generated super-resolution coded texture values ​​to generate a second texture block 112 having a second block size (e.g., 8×8 texels, 16×16 texels, etc.). The second block size may be several times larger than the first block size (e.g., 4, 16, etc.). In some embodiments, the generated second texture block 112 may be a super-resolution texture block, which may mean that the resolution of the second texture block 112 may be greater than the first texture block 110. Thus, in general, the term “super-resolution” may refer to a higher resolution. According to an embodiment, the processing may include converting the super-resolution coded texture values ​​in the compressed domain into a second compressed bitstream. The second compressed bitstream may include a set of second blocks of coded bits, each second block of coded bits representing a corresponding portion (e.g., one of four 4×4 blocks) of the second texture block 112 (e.g., 8×8 texels) in the compressed domain. The processing may also include decoding (e.g., using a GPU) the second compressed bitstream to generate the second texture block 112. In one or more embodiments, decoding may include applying a first block decoding scheme to the set of second blocks of coded bits in a block-wise manner to generate corresponding portions of the second texture block 112 in a first-in-first-out (FIFO) array. Figure 4B Details of processing the super-resolution encoded texture values ​​to generate the second texture block 112 are further provided in.

[0035] Figure 2 is a block diagram illustrating an exemplary system for super-resolution of block compressed textures for texture mapping applications according to an embodiment of the present disclosure. Figure 2 Combination Figure 1 The elements in are explained. Figure 2 , a block diagram 200 of a system 102 is shown. The system 102 may include a circuit system 202 and a memory 204. The memory 204 may store a plurality of ML models 104. The system 102 may also include a graphics processing unit (GPU) 206 and an input / output (I / O) device 208. For example, the I / O device 208 may include a display device 106. The system 102 may also include a network interface 210.

[0036] The circuit system 202 may include appropriate logic, circuit systems, and interfaces that may be configured to execute program instructions associated with different operations to be performed by the system 102. For example, some of the operations may include generating coded texture values ​​in a compressed domain based on applying a first BC scheme to a first texture block 110 having a first block size. The operations may also include applying a first ML model 104A to the generated coded texture values ​​to generate super-resolution coded texture values ​​in the compressed domain. Based on processing the super-resolution coded texture values, the circuit system 202 may also generate a second texture block 112 having a second block size.

[0037] The circuit system 202 may include one or more special processing units that may be implemented as separate processors. In an embodiment, the one or more special processing units may be implemented as an integrated processor or a cluster of processors that jointly perform the functions of the one or more special processing units. The circuit system 202 may be implemented based on a variety of processor technologies known in the art. Examples of implementations of the circuit system 202 may be an X86-based processor, a GPU, a field programmable gate array (FPGA), a programmable logic device (PLD), an application specific integrated circuit (ASIC), a programmable ASIC (PL-ASIC), a dedicated integrated component (ASSP), a system on a chip (SOC) based on a standard microprocessor (MPU) or a digital signal processor (DSP), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a microcontroller, a coprocessor (such as an AI accelerator, a visual processing unit or a neuromorphic chip), a central processing unit (CPU) and / or a combination thereof.

[0038] The memory 204 may include suitable logic, circuitry, and interfaces that may be configured to store program instructions executable by the circuitry 202. The memory 204 may be configured to also store the plurality of ML models 104, texture images in compressed or uncompressed form (such as the texture image 108), texture blocks of the texture image 108 (such as the first texture block 110), intermediate results (such as compressed domain values), and super-resolution texture blocks (such as the generated second texture block 112). Examples of implementations of the memory 204 may include, but are not limited to, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a hard disk drive (HDD), a solid-state drive (SSD), a CPU cache, and / or a secure digital (SD) card.

[0039] The GPU 206 may include suitable logic, circuitry, and interfaces that may be configured to perform operations related to graphics rendering, processing super-resolution encoded texture values ​​to generate super-resolution texture blocks (which may be used in a shader pipeline), and other auxiliary computations. Example implementations of the GPU 206 may include, but are not limited to, an integrated GPU, a discrete GPU, a hybrid GPU or multi-GPU implementation, a general purpose GPU (GPGPU), and an external GPU.

[0040] I / O devices 208 may include suitable logic, circuit systems, and interfaces that may be configured to receive input from a user and provide output based on the received input. I / O devices 208, which may include various input and output devices, may be configured to communicate with circuit system 202. For example, system 102 may receive user input via I / O devices 208 to render 3D graphics associated with a graphics rendering application (e.g., a gaming application or gaming software) on display device 106. Examples of I / O devices 208 may include, but are not limited to, a touch screen, a display device, a keyboard, a mouse, a joystick, a microphone, and a speaker.

[0041] The network interface 210 may comprise suitable logic, circuitry, and interfaces that may be configured to facilitate communication between the system 102 and a server (e.g., a server on which the texture image 108 is stored) via a communication network ( Figure 2 The network interface 210 may also be configured to facilitate communication between the system 102 and an electronic device associated with a user of the graphics rendering application. The network interface 210 may be implemented using various known technologies to support wired or wireless communication between the system 102 and a communication network.

[0042] The network interface 210 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec (CODEC) chipset, a subscriber identity module (SIM) card, or a local buffer circuit system. The network interface 210 may be configured to communicate with a network (such as the Internet, an intranet, or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and a metropolitan area network (MAN))) via wireless communication. Wireless communication can be configured to use one or more of a variety of communication standards, protocols and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), 5th Generation New Radio (5G NR), Long Term Evolution (LTE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Worldwide Interoperability for Microwave Access (Wi-MAX), email protocol, instant messaging and short message service (SMS).

[0043] like Figure 1 The functions or operations performed by system 102 may be performed by circuit system 202. For example, in Figure 4A and Figure 4B The operations performed by circuit system 202 are described in detail in FIG.

[0044] Figure 3 is a block diagram illustrating an exemplary implementation of a system as a codec chipset for super-resolution of block compressed textures according to an embodiment of the present disclosure. Figure 3 Combination Figure 1 and Figure 2 The elements in are explained. Figure 3 , a block diagram 300 depicting an exemplary embodiment of system 102 as a codec (coder-decoder) chipset is shown. System 102 may include circuit system 202, which may include input / output (I / O) interface 302, on-chip memory 304, encoder 306A, and decoder 308A. Circuit system 202 may also include bit-to-value (B2V) converter 306B and value-to-bit (V2B) converter 308B. Block diagram 300 of system 102 may also include processor 310, external memory 312, and GPU 206 communicatively coupled to circuit system 202 via system bus or I / O interface 302. In some embodiments, processor 310, external memory 312, and GPU 206 may be integrated into circuit system 202 without departing from the scope of the present disclosure.

[0045] The I / O interface 302 may include suitable logic, circuitry, and interfaces that may be configured to manage a plurality of I / O ports to facilitate the exchange of communication data from other circuitry or devices (e.g., external memory 312, processor 310, or GPU 206) via a system bus network. The communication data may include, for example, bitstreams of encoded textures, control signals, CPU instructions, etc. The configuration of the I / O ports may depend on the specifications, form factor, or design of the circuitry 202.

[0046] The on-chip memory 304 may include suitable logic, circuitry, and interfaces that may be configured to store different operational data (e.g., texture image 108 and first texture block 110) that may be utilized by different components of the circuitry 202 to rescale or upscale image blocks (such as first texture block 110). Examples of operational data that the on-chip memory 304 may be configured to store may include, but are not limited to, encoded texture values, information related to the BC scheme, compressed bitstreams, and / or multiple ML models 104. The on-chip memory 304 may be implemented as a specific memory with defined memory specifications (such as R / W speed, memory size, manufacturing factors, etc.). Examples of the on-chip memory 304 may include, but are not limited to, static random access memory (SRAM), dynamic access memory (DRAM), and static dynamic RAM (SD-RAM).

[0047] The encoder 306A may include suitable logic, circuit systems, and interfaces that may be configured to generate a first compressed bitstream by applying a block compression (BC) scheme to a texture block (e.g., a 4×4 texel block, an 8×8 texel block, etc.). The texture block may be stored in a memory, such as an on-chip memory 304 or an external memory 312. The encoder 306A may be optimized to reduce the number of bits that may be used to encode the first texture block 110 to improve compression efficiency and / or the compression factor of the texture block. In some embodiments, the encoder 306A may be a hardware encoder chip that may be implemented based on one of an ASIC, a programmable logic device (PLD), a SOC, an FPGA, a digital signal processor (DSP), or other dedicated circuits. In other embodiments, the encoder 306A may be implemented as a combination of hardware and an instruction set (e.g., instructions based on VHDL or HDL logic) stored in the on-chip memory 304.

[0048] The B2V converter 306B may include suitable logic, circuit systems, and interfaces that may be configured to convert a first block of code bits of the generated first compressed bitstream to generate a coded texture value. The conversion may include, for example, a unique hot encoding of the code bits. In some embodiments, the B2V converter 306B may be a hardware chip that may be implemented based on one of an ASIC, a programmable logic device (PLD), a SOC, an FPGA, a digital signal processor (DSP), or other dedicated circuits. In other embodiments, the B2V converter 306B may be implemented as a combination of hardware and an instruction set (e.g., instructions based on VHDL or HDL logic) stored in the on-chip memory 304.

[0049] The decoder 308A may include suitable logic, circuit systems, and interfaces that may be configured to decode a second compressed bitstream associated with a super-resolution encoded texture value (obtained from the first ML model 104A). The decoder 308A may have pre-stored quantization tables and encoding tables that may serve as references for the decoder 308A to decode the second compressed bitstream. In some embodiments, the decoder 308A may be a hardware encoder chip that may be implemented based on one of an ASIC, a programmable logic device (PLD), a SOC, an FPGA, a digital signal processor (DSP), or other dedicated circuits. In other embodiments, the decoder 308A may be implemented as a combination of hardware and an instruction set (e.g., instructions based on VHDL or HDL logic) stored in the on-chip memory 304.

[0050] The V2B converter 308B may include suitable logic, circuit systems, and interfaces that may be configured to convert a set of coded texture values ​​associated with the super-resolution coded texture values ​​into a set of second blocks of coded bits. The decoder 308A may apply a first block decoding scheme to the set of second blocks of coded bits in a block-by-block manner to generate a corresponding portion of the second texture block 112 in a first-in, first-out (FIFO) array. In some embodiments, the V2B converter 308B may be a hardware chip that may be implemented based on one of an ASIC, a programmable logic device (PLD), a SOC, an FPGA, a digital signal processor (DSP), or other dedicated circuits. In other embodiments, the V2B converter 308B may be implemented as a combination of hardware and an instruction set (e.g., instructions based on VHDL or HDL logic) stored in the on-chip memory 304.

[0051] The processor 310 may include suitable logic, circuit systems, and interfaces that may be configured to execute defined instructions stored in a memory (e.g., external memory 312 or on-chip memory 304). The processor 310 may be implemented based on a variety of processor technologies known in the art. Examples of the processor 310 may include, but are not limited to, a graphics processing unit (GPU), a coprocessor for a processor (such as a dedicated image coprocessor), a dedicated digital signal processor, a central processing unit (CPU), an x86-based processor, an x64-based processor, a reduced instruction set computing (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computing (CISC) processor.

[0052] The external memory 312 may include suitable logic, circuitry, and interfaces that may be configured to store the texture image 108 (uncompressed raw or encoded) or a plurality of texture blocks including the first texture block 110. The external memory 312 may also be configured to act as an image buffer to store the plurality of texture blocks retrieved from the GPU 206. Additionally, the external memory 312 may store instructions associated with a sequential encoding / decoding scheme that may be selectively applied by the circuitry 202 to generate a compressed bitstream.

[0053] In an embodiment, the external memory 312 may be a dynamic random access memory (DRAM) that may be externally interfaced with the circuit system 202. In another embodiment, the external memory 312 may be a static random access memory (SRAM) that may be externally interfaced with the circuit system 202. Additional examples of implementations of the external memory 312 may include, but are not limited to, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a hard disk drive (HDD), a solid-state drive (SSD), a CPU cache, or a secure digital (SD) card.

[0054] Figure 4A and Figure 4B are diagrams that collectively illustrate exemplary operations for super-resolution of block compressed textures according to embodiments of the present disclosure. Figure 4A and Figure 4B Combination Figure 1 , Figure 2 and Figure 3 The elements of Figure 4A and Figure 4B , a diagram 400 is shown including exemplary operations from 402 to 412 , starting at 402 .

[0055] At 402, a texture image 402A may be received. According to an embodiment, the circuit system 202 may be configured to receive the texture image 402A. The texture image 402A may be required in one or more operations (e.g., texture and shader operations) associated with a graphics rendering application, such as, but not limited to, games, game development engines, XR applications, and 2D / 3D animations. In an embodiment, the texture image 402A may be decoded from a collection of texture files, which may be included in a software package or texture library of the graphics rendering application.

[0056] In many cases, the texture resolution of texture image 402A or the block size of individual texture blocks of texture image 402A may be insufficient for operations associated with graphics rendering applications. In such cases, such operations may require a higher texture resolution. Texture image 402A or each texture block may need to be rescaled or upsampled to meet the requirements of operations associated with graphics rendering applications. Circuitry 202 may be configured to divide received texture image 402A into a plurality of texture blocks, each of which may have a first block size, e.g., 4×4 texels.

[0057] At 404, a first texture block 404A may be selected from a plurality of texture blocks. According to an embodiment, the circuit system 202 may select the first texture block 404A having a first block size. According to an embodiment, the circuit system 202 may be configured to select the first block compression (BC) scheme from a plurality of BC schemes. Such schemes may specify a particular BC format (and mode of the BC format) that may be used to compress the plurality of texture blocks of the texture image 402A.

[0058] In some embodiments, the selected first BC scheme may specify a first BC format, which may be one of several compression formats specified in S3 texture compression (which includes a set of lossy texture compression algorithms). Examples of the first BC format may include, but are not limited to, a BC1 format, a BC2 format, a BC3 format, a BC4 format, a BC5 format, a BC6 format, or a BC7 format. As an example and not a limitation, a set of BC formats and the individual texture blocks that each BC format can handle are provided in Table 1, which are given as follows:

[0059] BC format Texture formats handled by BC format BC1 RGB + optional 1-bit alpha BC2 RGB+4-bit alpha BC3 RGBA BC4 Grayscale BC5 2-channel grayscale BC6 RGB, floating point BC7 RGB or RGBA

[0060] Table 1: BC format and texture format

[0061] The BC1 format of the first BC scheme may encode the three-component color data of the first texture block 404A using a 5:6:5 representation of the color data of the first texture block 404A (such as 5 bits of red, 6 bits of green, and 5 bits of blue). In addition, the BC1 format may encode the alpha data of the first texture block 404A using 1 bit. The alpha data may be used to determine the opacity level of each texel of the first texture block 404A. The BC1 format may allow for the storage of 16 colors (based on the 5:6:5 representation of the color data) and 16 2-bit color indexes of the first texture block 404A. The 2-bit color index may be used to identify the color of each texel element of the first texture block 404A.

[0062] The BC2 format may encode color information (such as red-green-blue (RGB) information) of the first texture block 404A as a 5:6:5 representation of the color information (such as 5 bits of red, 6 bits of green, and 5 bits of blue). The BC2 format may encode alpha data of the first texture block 404A as 4 bits per texel.

[0063] The BC3 format may encode the color information of the first texture block 404A using a 5:6:5 representation of the color information (such as 5 bits of red, 6 bits of green, and 5 bits of blue). The BC3 format may encode the alpha data of the first texture block 404A in one byte. The BC4 format may encode single-component color information (e.g., grayscale color information) of the first texture block 404A using 8 bits. In an embodiment, the BC4 format may allow for the storage of two reference colors, such as "0" corresponding to a bit when red is not present, and "1" corresponding to a bit when red is present in a texel of the first texture block 404A. The BC4 format may also allow for the storage of 16 3-bit color indices for the first texture block 404A.

[0064] The BC5 format may encode the two-component color information of the first texture block 404A using 8 bits of a texel for each color of the color information. In an embodiment, the BC5 format may allow for the storage of two reference colors for each component. The BC5 format may allow for the storage of 16 3-bit color indices for each component of the two-component color information of the first texture block 404A.

[0065] According to an embodiment, the first BC format specified in the first BC scheme may be in a first mode, which may be one of fourteen modes of the BC6 format or eight modes of the BC7 format. A set of eight different modes (represented by index i) of the BC7 format and the corresponding texture data handled by each mode and the number of subsets for each corresponding mode are provided in Table 2, as shown below:

[0066] Mode i data Subset 0 RGB 3 1 RGB 2 2 RGB 3 3 RGB 2 4 RGBA 1 5 RGBA 1 6 RGBA 1 7 RGBA 2

[0067] Table 2: BC7 modes and features

[0068] The BC6 format and the BC7 format may allow different modes (such as the first mode) that may change the details of the BC format. The BC6 format and the BC7 format may specify endpoints (e.g., 2 endpoints) to describe color fragments used for interpolation of texels (or pixels) in a texture block. The BC6 format and the BC7 format may also specify indices, each of which may correspond to a weight for each pixel used for interpolation. For example, for a 4×4 texture block, mode 7 of the BC7 format may produce a set of 16 indices, as shown below:

[0069] 11 10 10 01 10 01 11 01 01 10 11 01 00 11 01 00

[0070] Table 3: 16 2-bit indices

[0071] Each mode can have different characteristics, such as the colors supported (color components and alpha channel), the number of subsets per block, the endpoint array (e.g., RGBP 4.4.4.1 endpoints with a unique P bit for each endpoint), the bit size of each index (e.g., 3-bit indexes), and the number of partitions (e.g., 16 partitions). Each mode of the BC6 format (e.g., each of the fourteen modes) or each mode of the BC7 format (e.g., each of the eight modes) can be used for block compression of the first texture block 404A. Different modes of the BC6 format and the BC7 format can be selected to trade off between different factors, such as between desired accuracy and compression speed. The BC-n format is well known to those of ordinary skill in the art; therefore, for the sake of brevity, further details of the BC-n format are omitted in this disclosure.

[0072] At 406, the selected first BC scheme may be applied to the first texture block 404A to generate a first compressed bitstream. According to an embodiment, the encoder 306A of the circuit system 202 may be configured to apply the selected first BC scheme to the first texture block 404A to generate a first compressed bitstream. The first compressed bitstream may be formatted according to a first BC format of the selected first BC scheme.

[0073] For example, the encoder 306A may select mode 7 of the BC7 format as the first BC scheme. The encoder 306A may use the selected mode 7 of the BC7 format for compressing the first texture block 404A. The selected mode of the BC7 format may be represented as 7BCi, where "i" represents one of the modes of the BC7 format. The representation of the generated first compressed bitstream (based on the selected mode 7 (7BC7) of the BC7 format) is shown in Table 4, as follows:

[0074]

[0075] Table 4: First compressed bit stream based on BC7 format - Mode 7 (7BC7) The "6-bit partition" bit in Table 4 indicates the partition type of Mode 7 of the BC7 format. The bits "5-bit R0" to "1-bit P3" in Table 4 indicate the endpoints of Mode 7 of the BC7 format. R corresponds to red, G corresponds to green, B corresponds to blue, A corresponds to alpha, and P corresponds to the lowest bit of all channels of Mode 7 of the BC7 format. "30 bits" indicates 16 indices, each of which is a weight for interpolation of texels using the endpoints.

[0076] According to an embodiment, the generated first compressed bitstream can be input to the B2V converter 306B. The B2V converter 306B can convert the first block of code bits of the generated first compressed bitstream to generate a coded texture value. For example, as part of the conversion, a 6-bit partition in the first compressed bitstream can be converted into a 64×1 unique hot encoding vector. The 4×RGBA endpoints of the first compressed bitstream can be converted into 16×1 floating point values ​​(normalized to (0,1)). The B2V converter 306B can convert a 16×2-bit pixel index represented by "30 bits" into a 16×1 floating point value (normalized to (0,1)). It can be assumed that each of the two subsets within the block starts with bit "0". The circuit system 202 can generate a coded texture value in a compressed domain based on applying the first BC scheme to the received first texture block 404A.

[0077] At 408, the first ML model 104A may be applied to the generated coded texture values ​​to generate super-resolution coded texture values ​​in the compressed domain. In some embodiments, the first ML model 104A may be selected from the plurality of ML models 104. The selection of the first ML model 104A may be based on the selected first BC scheme. In one or more embodiments, the selection of the first ML model 104A may be selected based on determining that the generated first compressed bitstream is associated with a first mode (e.g., mode 7) of a first BC format (e.g., BC7 format) specified in the first BC scheme.

[0078] In some embodiments, the plurality of ML models 104 may include ML models, each model being used for a specific BC format, such as a BC1 format, a BC2 format, ... or a BC7 format. In one or more embodiments, each ML model in the plurality of ML models 104 may be associated with a corresponding mode in a plurality of modes of the first BC format. For example, the plurality of ML models 104 may include fourteen ML models, each model being used for a corresponding mode of the BC6 format. Alternatively, the plurality of ML models 104 may include eight ML models, each model being used for a corresponding model of the BC7 format. Each ML model in the plurality of ML models 104 may be trained separately according to a corresponding BC format. For example, in Figure 6 The training of the first ML model 104A is further described in .

[0079] According to an embodiment, the application of the selected first ML model 104A may include feeding the generated coded texture values ​​as input to the selected first ML model 104A. For example, if the first ML model 104A is associated with mode 7 of the BC7 format, then the coded texture values ​​generated based on the selected mode 7 of the BC7 format may be input to the first ML model 104A. Based on the input, the first ML model 104A may generate super-resolution coded texture values. The super-resolution coded texture values ​​may include a set of coded texture values ​​(four sets of coded texture values) that may be required to enlarge the first texture block 404A to the second texture block 412A. For example, it may be necessary to enlarge the first texture block 404A having a first block size by "M" times to generate a second texture block 412A having a second block size. The ratio of the second block size to the first block size may be a positive integer (M) greater than one.

[0080] In an embodiment, the first block size of the first texture block 404A may be 4×4 texels. The first texture block 404A may have to be scaled up by a factor of 2 to obtain a second texture block (which may be 8×8 texels). In this case, the set of coded texture values ​​that may be required to scale up the first texture block 404A to the second texture block may be obtained by applying the first ML model 104A. The first ML model 104A may be trained to generate four sets of coded texture values ​​(as part of super-resolution coded texture values) based on input coded texture values ​​associated with the first texture block 404A. Each of the four sets of generated coded texture values ​​may be used to generate a 4×4 texture block, and the four sets of coded texture values ​​together may be used to generate an 8×8 texture block.

[0081] According to an embodiment, a mipmap level required by a shader of the GPU 206 may be determined. A mipmap may be an optimized sequence of images, where each image in the sequence represents a lower resolution than the previous image. Mipmaps are typically used to speed up rendering time, either by reducing the number of texels sampled to render each pixel, or by increasing the memory locality of the samples taken. Each image in the sequence of images is associated with a mipmap level. Each mipmap level may be a power of two less than the previous mipmap level. The shader of the GPU 206 may be utilized to color or provide textures for a three-dimensional (3D) mesh. The mipmap level (required by the shader) may be determined to appropriately set a second block size (or a scaling factor by which a texture block should be rescaled or enlarged). The circuit system 202 may signal the determined mipmap level to a selected first ML model 104A of the plurality of ML models 104. Based on the signaled mipmap level, the first ML model 104A may set a second block size of a second texture block (such as the second texture block 412A). For example, the first ML model 104A may set the second block size to 8×8 texels or a scaling factor of 4. Based on the set second block size, the first ML model 104A may generate a super-resolution coded texture value in a compressed domain. The circuit system 202 may also process the generated super-resolution coded texture value as described herein. As part of the processing, the super-resolution coded texture value may be input to the V2B converter 308B.

[0082] At 410, the V2B converter 308B may convert the super-resolution coded texture values ​​in the compressed domain into a second compressed bitstream. The generated second compressed bitstream may include a set of second blocks of coded bits, each of which may represent a corresponding portion (e.g., a 4×4 block) of a second texture block (e.g., second texture block 412A) in the compressed domain.

[0083] In an embodiment, for each set of coded texture values ​​in the super-resolution coded texture values, the V2B converter 308B may convert a 64×1 one-hot encoded vector of super-resolution coded texture values ​​into a 6-bit partition. The V2B converter 308B may also convert the 16×1 floating point value normalized to (0,1) into a 4×RGBA endpoint and convert the 16×1 floating point value normalized to (0,1) into a 16×2 (32) bit index to generate a second compressed bitstream. Since it is assumed that each of the two subsets within the block starts with bit "0", it may be considered that 30 bits are used for the index instead of 32 bits.

[0084] At 412, a second texture block 412A may be generated. According to an embodiment, the second compressed bitstream may be input to the decoder 308A to generate the second texture block 412A. The processing of the super-resolution encoded texture value may include inputting the second compressed bitstream to the decoder 308A. The decoder 308A may decode the second compressed bitstream to generate a second texture block 412A having a second block size. The second block size may be larger than the first block size. For example, the second block size of the generated second texture block 412A may be 8×8 texels.

[0085] According to an embodiment, decoding of the second compressed bitstream may include applying the first block decoding scheme to a set of second blocks of encoded bits in a block-by-block manner in a first-in-first-out (FIFO) array to generate a corresponding portion (e.g., a 4×4 block) of the second texture block 412A. Alternatively, the decoder 308A may decode the second texture block 412A pixel by pixel (in a defined array).

[0086] Figure 5 is a diagram illustrating an exemplary architecture of a machine learning model for super-resolution of block compressed textures according to an embodiment of the present disclosure. Figure 5 Combination Figure 1 , Figure 2 , Figure 3 , Figure 4A and Figure 4B The elements in are explained. Figure 5 , shows the architecture 500 of a first ML model 502, which may be Figure 1 An exemplary implementation of the first ML model 104A.

[0087] The first ML model 502 may include multiple fully connected layers, such as fully connected layer 504A, fully connected layer 504B, and fully connected layer 504C. The first ML model 502 may also include multiple activation layers, such as activation layer 506A, activation layer 506B, and activation layer 506C. Examples of multiple activation layers may include, but are not limited to, exponential linear units, linear activation layers, rectified linear units, sigmoid functions, or softmax functions.

[0088] The output of B2V converter 306B may be input to fully connected layer 504A of first ML model 502. Fully connected layer 504A may be connected to activation layer 506A. Activation layer 506A may also be connected to fully connected layer 504B. Additionally, fully connected layer 504B may be connected to activation layer 506B. Activation layer 506B may be connected to fully connected layer 504C. Fully connected layer 504C may be connected to activation layer 506C. The output of activation layer 506C may be input to V2B converter 308B of circuit system 202.

[0089] The coded texture values ​​generated by the B2V converter 306B may be input to the fully connected layer 504A. The coded texture values ​​may pass through a series of multiple fully connected layers and multiple activation layers, such as activation layer 506A and activation layer 506B and activation layer 506C, to generate super-resolution coded texture values ​​(which may be referred to as interpolation of the input coded texture values). The output of activation layer 506C may be fed as input to the V2B converter 308B of the circuit system 202. The V2B converter 308B may output a second compressed bitstream based on the received input.

[0090] Figure 6 is a diagram illustrating exemplary operations for training a machine learning model for super-resolution of block compressed textures according to an embodiment of the present disclosure. Figure 6 Combination Figure 1 , Figure 2 , Figure 3 , Figure 4A , Figure 4B and Figure 5 The elements of Figure 6 , a diagram 600 is shown. The diagram 600 may include the first ML model 104A, the encoder 306A, the B2V converter 306B, the decoder 308A, and the V2B converter 308B. The diagram 600 may also include a first sample texture block 602, a pre-trained ML model 604, a second sample texture block 606, a loss function 608, a multiplier function 612, a ground truth texture block 610, a multiplier function 612, and a super-resolution texture block 614.

[0091] According to an embodiment, the first ML model 104A may be a neural network-based model. For training of the first ML model 104A, the first ML model 104A may receive a first sample texture block 602 having a first block size. For example, the first block size may be 4×4 texels. In some embodiments, the circuit system 202 may receive a sample texture image and may divide the sample texture image into a plurality of sample texture blocks (including the first sample texture block 602).

[0092] The first sample texture block 602 may be input to the encoder 306A of the circuit system 202, which may generate a first sample compressed bitstream by applying a first BC scheme to the received first sample texture block 602. The first BC scheme may specify a first BC format, which may be one of a BC1 format, a BC2 format, a BC3 format, a BC4 format, a BC5 format, a BC6 format, or a BC7 format. The generated first sample compressed bitstream may be input to the B2V converter 306B. The B2V converter 306B of the circuit system 202 may convert the sample compressed bitstream into a first sample coded texture value.

[0093] The circuit system 202 may apply the first ML model 104A to the first sample coded texture value to generate a second sample coded texture value. The generated second sample coded texture value may be input to the pre-trained ML model 604. In one or more embodiments, the circuit system 202 may include the pre-trained ML model 604. The pre-trained ML model 604 may reconvert the second sample coded texture value into a second sample compressed bitstream. The pre-trained ML model 604 may also decode the second sample compressed bitstream to generate a second sample texture block 606 having a second block size. In some embodiments, the second block size may be 8×8 texels. For example, in Figure 7 Details of the architecture of the pre-trained ML model 604 are provided in.

[0094] The circuit system 202 may also generate a ground truth texture block 610 from the first sample texture block 602. For example, the circuit system 202 may generate the ground truth texture block 610 based on applying a multiplier function 612 to the first sample texture block 602.

[0095] The ground truth texture block 610 and the generated second sample texture block 606 may be passed as inputs to a loss function 608. The loss function 608 may be used to calculate a first loss between the second sample texture block 606 and the ground truth texture block 610 having a second block size. The first loss may be provided to the first ML model 104A and the circuit system 202 may update the weights of the first ML model 104A based on the calculated first loss. Similar to the training of the first sample texture block 602, the training of the first ML model 104A may be performed on the remaining sample texture blocks.

[0096] For the generation of the super-resolution texture block 614, the generated second sample coded texture value may be input to the V2B converter 308B. The V2B converter 308B may convert the second sample coded texture value into a second sample compressed bitstream. The second sample compressed bitstream may be input to the decoder 308A, which may generate the super-resolution texture block 614 having the second block size.

[0097] Figure 7 is a diagram illustrating an exemplary architecture of a pre-trained machine learning model for training a first machine learning model according to an embodiment of the present disclosure. Figure 7 Combination Figure 1 , Figure 2 , Figure 3 , Figure 4A , Figure 4B , Figure 5 and Figure 6 The elements of Figure 7, a graph 700 is shown. Graph 700 may include a pre-trained ML model 604. The pre-trained ML model 604 may include a plurality of fully connected layers and a plurality of activation layers.

[0098] The plurality of fully connected layers may include fully connected layer 704A, fully connected layer 704B, fully connected layer 704C, fully connected layer 704D, and fully connected layer 704E. The plurality of activation layers may include activation layer 706A, activation layer 706B, activation layer 706C, activation layer 706D, and activation layer 706E. Examples of the plurality of activation layers may include, but are not limited to, exponential linear units, linear activation layers, rectified linear units, sigmoid functions, or softmax functions.

[0099] Multiple fully connected layers and multiple exponential linear units may be connected sequentially. For example, fully connected layer 704A may be connected to activation layer 706A. Activation layer 706A may be connected to fully connected layer 704B. Fully connected layer 704B may be connected to activation layer 706B. Activation layer 706B may be connected to fully connected layer 704C. Fully connected layer 704C may also be connected to activation layer 706C. Activation layer 706C may be connected to fully connected layer 704D. Fully connected layer 704D may be connected to activation layer 706D. Activation layer 706D may be connected to fully connected layer 704E. Fully connected layer 704E may be connected to activation layer 706E.

[0100] The pre-trained ML model 604 may receive a second sample encoded texture value (e.g., Figure 6 The second sample coded texture value may be passed through a plurality of fully connected layers and a plurality of activation layers to reconvert the second sample coded texture value into a second sample compressed bitstream. The pre-trained ML model 604 may also decode the second sample compressed bitstream to generate a second sample texture block 606. The second sample texture block 606 may be output via the activation layer 706E. The second sample texture block 606 output by the pre-trained ML model 604 may be input to the loss function 608.

[0101] Fig. 8A and Figure 8B 1 and 2 collectively illustrate flow charts of exemplary methods for super-resolution of block compressed textures according to embodiments of the present disclosure. Fig. 8A and Figure 8B Combined from Figure 1 , Figure 2 , Figure 3 , Figure 4A , Figure 4B , Figure 5 , Figure 6 and Figure 7 The elements of Fig. 8A and Figure 8B, a flowchart 800 is shown. The method shown in flowchart 800 may be performed by any computing system, such as by system 102 or circuit system 202. The method may start at 802 and proceed to 804.

[0102] At 804, a texture image (such as texture image 108) may be received. According to an embodiment, circuit system 202 may be configured to receive texture image 108. In some embodiments, texture image 108 may be decoded from a collection of texture files that may be included in a software package or texture library of a graphics rendering application. For example, in Figure 4A Details of the reception of texture image 108 are further provided in .

[0103] At 806, the received texture image 108 may be divided into a plurality of texture blocks. According to an embodiment, the circuit system 202 may be configured to divide the received texture image 108 into a plurality of texture blocks. Each of the plurality of texture blocks may have a first block size (e.g., 4×4 texels). The first texture block 110 may be included in the plurality of texture blocks.

[0104] At 808, a first BC scheme may be selected from among the plurality of BC schemes. According to an embodiment, the circuit system 202 may be configured to select the first BC scheme. The first BC scheme may specify a first BC format, which may be specified in S3 texture compression (S3TC) and may be one of a BC1 format, a BC2 format, a BC3 format, a BC4 format, a BC5 format, a BC6 format, or a BC7 format. For example, in Figure 4A The details of the selection of the first BC scheme are further described in .

[0105] At 810, the selected first block compression scheme may be applied to the first texture block 110 to generate a first compressed bitstream. According to an embodiment, the circuit system 202 may be configured to apply the selected first block compression scheme to the first texture block 110 to generate a first compressed bitstream. The first compressed bitstream may be formatted according to a first BC format of the selected first BC scheme. For example, Figure 4A Details of applying the selected first BC scheme to the first texture block 110 to generate a first compressed bitstream are further described in.

[0106] At 812, the first block of code bits of the generated first compressed bitstream may be converted to generate an encoded texture value. According to an embodiment, the circuit system 202 may be configured to convert the first block of code bits of the generated first compressed bitstream to generate an encoded texture value. For example, Figure 4B Details of converting the first block of code bits to generate an encoded texture value are further provided in.

[0107] At 814, a first ML model (e.g., the first ML model 104A) may be selected from a plurality of ML models (e.g., the plurality of ML models 104). According to an embodiment, the circuit system 202 may be configured to select the first ML model 104A from the plurality of ML models 104 based on determining that the first compressed bitstream is associated with a first mode of a first BC format specified in a first BC scheme. Each ML model in the plurality of ML models 104 (e.g., the first ML model 104A, the second ML model 104B, and the Nth ML model 104N) may be associated with a corresponding mode in a plurality of modes of the first BC format. For example, in Figure 4B Details of selecting the first ML model 104A from the plurality of ML models 104 are further provided in .

[0108] At 816, it may be determined whether a mipmap level is specified. According to an embodiment, the circuit system 202 may be configured to determine whether a mipmap level is specified. In the case where a mipmap level is specified, control may be passed to 818. Otherwise, control may be passed to 828.

[0109] At 818, a mipmap level required by a shader of a GPU (e.g., GPU 206) may be determined based on the determination of the specified mipmap level. According to an embodiment, circuit system 202 may be configured to determine the mipmap level required by a shader of GPU 206. For example, in Figure 4B Details related to mipmap levels are provided in .

[0110] At 820, the determined mipmap level may be signaled to the first ML model 104A. According to an embodiment, the circuit system 202 may be configured to signal the determined mipmap level to the first ML model 104A. For example, Figure 4B Details of signaling determined mipmap levels are further described in .

[0111] At 822, a second block size of the second texture block 112 may be set by the first ML model 104A based on the signaled mipmap level. According to an embodiment, the first ML model 104A may set the second block size of the second texture block 112 based on the signaled mipmap level.

[0112] At 824, a super-resolution coded texture value may be generated in the compressed domain based on the set second block size. According to an embodiment, the first ML model 104A may generate a super-resolution coded texture value in the compressed domain based on the set second block size. For example, Figure 4B Details of the generation of super-resolution encoded texture values ​​are further provided in.

[0113] At 826, the first ML model 104A may be applied to the generated coded texture values ​​to generate super-resolution coded texture values ​​in the compressed domain. According to an embodiment, the circuit system 202 may be configured to apply the first ML model 104A to the generated coded texture values ​​to generate super-resolution coded texture values ​​in the compressed domain. For example, Figure 4B Details of applying the first ML model 104A to generate super-resolution encoded texture values ​​are further provided in.

[0114] At 828, the generated super-resolution encoded texture values ​​may be processed to generate a second texture block 112 having a second block size. According to an embodiment, the circuit system 202 may be configured to process the generated super-resolution encoded texture values ​​to generate a second texture block 112 having a second block size. The second block size may be larger than the first block size. For example, Figure 4B Details of processing the generated super-resolution encoded texture values ​​to generate the second texture block 112 are further provided in. Control may be passed to end.

[0115] Although flowchart 800 is illustrated as discrete operations, such as 802, 804, 806, 808, 810, 812, 814, 816, 818, 820, 822, 824, 826, and 828, the present disclosure is not limited thereto. Thus, in certain embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without departing from the essence of the disclosed embodiments.

[0116] Various embodiments of the present disclosure may provide a non-transitory computer-readable medium and / or storage medium having computer-executable instructions stored thereon, the instructions being executable by a machine and / or computer to operate a system (such as system 102). The computer-executable instructions may cause the machine and / or computer to perform operations that may include receiving a first texture block (such as first texture block 110) having a first block size. The operations may also include generating coded texture values ​​in a compressed domain based on applying a first block compression (BC) scheme to the received first texture block 110. The operations may also include applying a first machine learning model (such as first ML model 104A) to the generated coded texture values ​​to generate super-resolution coded texture values ​​in a compressed domain. In addition, the operations may include processing the generated super-resolution coded texture values ​​to generate a second texture block (such as second texture block 112) having a second block size. The second block size may be larger than the first block size.

[0117] Exemplary aspects of the present disclosure may provide a system (such as Figure 1The circuit system 202 may be configured to receive a first texture block (such as the first texture block 110) having a first block size. The circuit system 202 may also generate coded texture values ​​in a compressed domain based on applying a first block compression (BC) scheme to the received first texture block 110. The circuit system 202 may also be configured to apply a first machine learning model (such as the first ML model 104A) to the generated coded texture values ​​to generate super-resolution coded texture values ​​in a compressed domain. In addition, the circuit system 202 may be configured to process the generated super-resolution coded texture values ​​to generate a second texture block (such as the second texture block 112) having a second block size. The second block size may be larger than the first block size.

[0118] According to an embodiment, the method may include receiving a texture image 402A. The method may also include dividing the received texture image 402A into a plurality of texture blocks, each texture block having a first block size, and the first texture block 404A is included in the plurality of texture blocks.

[0119] According to an embodiment, the method further comprises selecting a first BC scheme from among a plurality of block compression schemes. The method further comprises applying the selected first block compression scheme to the first texture block 404A to generate a first compressed bitstream. The first compressed bitstream may be formatted according to a first BC format of the selected first BC scheme.

[0120] According to an embodiment, the method further comprises converting a first block of code bits of the generated first compressed bitstream to generate an encoded texture value.

[0121] According to an embodiment, the method further includes selecting a first ML model 104A from the plurality of ML models 104 based on determining that the first compressed bitstream is associated with a first mode of a first BC format specified in a first BC scheme. Each machine learning model in the plurality of ML models 104 may be associated with a corresponding mode of the plurality of modes of the first BC format.

[0122] According to an embodiment, the application of the selected first ML model 104A may further include feeding the generated coded texture value as an input to the selected first ML model 104A. The application may also include generating, by the selected first ML model 104A, a super-resolution coded texture value based on the input. The super-resolution coded texture value may include a set of coded texture values ​​that may be required to enlarge the first texture block 404A to the second texture block 412A.

[0123] According to an embodiment, the processing may further include converting the super-resolution coded texture values ​​in the compressed domain into a second compressed bitstream. The second compressed bitstream may include a set of second blocks of coded bits, each second block of coded bits representing a corresponding portion of the second texture block 412A in the compressed domain. The processing may also include decoding the second compressed bitstream to generate the second texture block 412A.

[0124] According to an embodiment, decoding may include applying the first block decoding scheme to the set of second blocks of coded bits in a block-by-block manner to generate the corresponding portion of the second texture block 412A in a first-in-first-out (FIFO) array.

[0125] According to an embodiment, the first ML model 104A may be a neural network based model.

[0126] According to an embodiment, the first BC scheme may specify a first BC format, which is one of a BC1 format, a BC2 format, a BC3 format, a BC4 format, a BC5 format, a BC6 format, or a BC7 format.

[0127] According to an embodiment, the first BC format specified in the first BC scheme is in a first mode, which is one of fourteen modes of the BC6 format or eight modes of the BC7 format.

[0128] According to an embodiment, the first block size may be 4×4 texels, and a ratio of the second block size to the first block size is a positive integer greater than one.

[0129] According to an embodiment, the method may further include determining a mipmap level required by a shader of the GPU 206. The method may further include signaling the determined mipmap level to the first ML model 104A. Based on the signaled mipmap level, the method may include setting, by the first ML model 104A, a second block size of the second texture block 412A. The method may further include generating, by the first ML model 104A, a super-resolution encoded texture value in a compressed domain based on the set second block size.

[0130] According to an embodiment, the method may include training the first ML model 104A. The training may include receiving a first sample texture block 602 having a first block size. The training may also include generating a first sample compressed bitstream by applying a first BC scheme to the received first sample texture block 602. The training may also include converting the sample compressed bitstream into a first sample coded texture value. In addition, the training may include applying the first ML model 104A to the first sample coded texture value to generate a second sample coded texture value. The training may also include reconverting the second sample coded texture value into a second sample compressed bitstream. Moreover, the training may include reconverting the second sample coded texture value into a second sample compressed bitstream. The training may also include decoding the second sample compressed bitstream to generate a second sample texture block 606 having a second block size. The training also includes calculating a first loss between the second sample texture block 606 and a ground truth texture block 610 having a second block size. The training may also include updating the weights of the first ML model 104A based on the calculated first loss.

[0131] The present disclosure may be implemented using hardware or a combination of hardware and software. The present disclosure may be implemented in a centralized manner, in at least one computer system, or in a distributed manner, where different elements may be distributed across several interconnected computer systems. A computer system or other device suitable for performing the methods described herein may be suitable. The combination of hardware and software may be a general-purpose computer system with a computer program that, when loaded and executed, can control the computer system so that it performs the methods described herein. The present disclosure may be implemented using hardware that includes a portion of an integrated circuit that also performs other functions.

[0132] The present disclosure may also be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein and which, when loaded into a computer system, is capable of carrying out these methods. In this document, a computer program refers to any expression of a set of instructions in any language, code or notation, which is intended to cause a system with information processing capabilities to perform a specific function either directly or after one or both of the following: a) conversion into another language, code or notation; b) reproduction in a different material form.

[0133] Although the present disclosure is described with reference to certain embodiments, it will be appreciated by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present disclosure. In addition, many modifications may be made to adapt specific circumstances or materials to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, it is intended that the present disclosure is not limited to the specific embodiments disclosed, but that the present disclosure will include all embodiments falling within the scope of the appended claims.

Claims

1. A rendering method, comprising: receiving a first texture block having a first block size; generating coded texture values ​​in a compressed domain based on applying a first block compression (BC) scheme to the received first texture block; applying a first machine learning model to the generated coded texture values ​​to generate super-resolution coded texture values ​​in a compressed domain; and processing the generated super-resolution encoded texture values ​​to generate a second texture block having a second block size, wherein the second block size is larger than the first block size, The processing also includes: converting the super-resolution coded texture values ​​in the compressed domain into a second compressed bitstream, wherein the second compressed bitstream comprises a set of second blocks of coded bits, each second block of coded bits representing a corresponding portion of the second texture block in the compressed domain; as well as The second compressed bitstream is decoded to generate a second texture block.

2. The method according to claim 1, further comprising: receiving a texture image; as well as The received texture image is divided into a plurality of texture blocks, each texture block having a first block size, and a first texture block is included in the plurality of texture blocks.

3. The method according to claim 1, further comprising: Selecting a first BC scheme from among a plurality of block compression schemes; as well as applying the selected first block compression scheme to the first texture block to generate a first compressed bitstream, The first compressed bit stream is formatted according to a first BC format of a selected first BC scheme. 4 . The method of claim 3 , further comprising converting a first block of code bits of the generated first compressed bitstream to generate an encoded texture value.

5. The method of claim 4, further comprising selecting a first machine learning model from among a plurality of machine learning models based on determining that the first compressed bitstream is associated with a first mode of a first BC format specified in a first BC scheme, Each of the multiple machine learning models is associated with a corresponding pattern among the multiple patterns of the first BC format.

6. The method of claim 5, wherein applying the selected first machine learning model further comprises: feeding the generated encoded texture value as input to the selected first machine learning model; as well as generating a super-resolution encoded texture value based on the input by the selected first machine learning model, The super-resolution coded texture value includes a set of coded texture values ​​required to enlarge the first texture block to the second texture block.

7. The method of claim 1, wherein decoding comprises applying a first block decoding scheme to the set of second blocks of coded bits in a block-by-block manner to generate corresponding portions of the second texture block in a first-in-first-out (FIFO) array.

8. The method of claim 1, wherein the first machine learning model is a neural network based model.

9. The method of claim 1, wherein the first BC scheme specifies a first BC format, the first BC format being one of a BC1 format, a BC2 format, a BC3 format, a BC4 format, a BC5 format, a BC6 format, or a BC7 format. 10 . The method of claim 9 , wherein the first BC format specified in the first BC scheme is a first mode, and the first mode is one of fourteen modes of the BC6 format or eight modes of the BC7 format.

11. The method of claim 1, wherein the first block size is 4x4 texels, and a ratio of the second block size to the first block size is a positive integer greater than one.

12. The method according to claim 1, further comprising: Determine the mipmap level required by the graphics processing unit (GPU) shader; signaling the determined mipmap level to the first machine learning model; setting, by the first machine learning model, a second block size of the second texture block based on the signaled mipmap level; as well as Super-resolution encoded texture values ​​are generated in a compressed domain by the first machine learning model based on the set second block size.

13. The method of claim 1 , further comprising training the first machine learning model, wherein the training comprises: receiving a first sample texture block having a first block size; generating a first sample compressed bitstream by applying a first BC scheme to the received first sample texture block; converting a sample compressed bitstream into a first sample coded texture value; applying a first machine learning model to the first sample encoded texture value to generate a second sample encoded texture value; reconverting the second sample coded texture value into a second sample compressed bitstream; decoding the second sample compressed bitstream to generate a second sample texture block having a second block size; calculating a first loss between a second sample texture block and a ground truth texture block having a second block size; as well as A weight of the first machine learning model is updated based on the calculated first loss.

14. A rendering system, comprising: A circuit system is configured to: receiving a first texture block having a first block size; generating coded texture values ​​in a compressed domain based on applying a first block compression (BC) scheme to the received first texture block; applying a first machine learning model to the generated coded texture values ​​to generate super-resolution coded texture values ​​in a compressed domain; and processing the generated super-resolution encoded texture values ​​to generate a second texture block having a second block size, wherein the second block size is larger than the first block size, The circuit system is configured to process the super-resolution encoded texture value by: converting the super-resolution coded texture values ​​in the compressed domain into a second compressed bitstream, wherein the second compressed bitstream comprises a set of second blocks of coded bits, each second block of coded bits representing a corresponding portion of the second texture block in the compressed domain; as well as The second compressed bitstream is decoded to generate a second texture block.

15. The system of claim 14, wherein the circuit system is further configured to: Selecting a first BC scheme from among a plurality of block compression schemes; and applying the selected first block compression scheme to the first texture block to generate a first compressed bitstream, wherein the first compressed bitstream is formatted according to a first BC format of the selected first BC scheme; and A first block of code bits of the generated first compressed bitstream is transformed to generate an encoded texture value.

16. The system of claim 15, wherein the circuit system is further configured to select a first machine learning model from among a plurality of machine learning models based on determining that the first compressed bitstream is associated with a first mode of a first BC format specified in a first BC scheme, Each of the multiple machine learning models is associated with a corresponding pattern among the multiple patterns of the first BC format.

17. The system of claim 14, wherein the circuit system is further configured to apply the first machine learning model by feeding the generated encoded texture value as an input to the selected first machine learning model, wherein The first machine learning model generates super-resolution encoded texture values ​​based on the input, and The super-resolution coded texture values ​​include a set of coded texture values ​​required to upscale a first texture block to a second texture block.

18. A computer program product having computer executable instructions stored thereon, the instructions, when executed by a system, causing the system to perform operations comprising: receiving a first texture block having a first block size; generating coded texture values ​​in a compressed domain based on applying a first block compression (BC) scheme to the received first texture block; applying a first machine learning model to the generated coded texture values ​​to generate super-resolution coded texture values ​​in a compressed domain; processing the generated super-resolution encoded texture values ​​to generate a second texture block having a second block size, wherein the second block size is larger than the first block size, The processing also includes: converting the super-resolution coded texture values ​​in the compressed domain into a second compressed bitstream, wherein the second compressed bitstream comprises a set of second blocks of coded bits, each second block of coded bits representing a corresponding portion of the second texture block in the compressed domain; as well as The second compressed bitstream is decoded to generate a second texture block.

Images