SYSTEM AND / OR METHOD FOR IMPLEMENTING A STATISTICAL APPROACH TO TEXTURE COMPRESSION - Patent application

Abstract

To provide a method based on a statistical technique, a texture encoding system, and a storage medium for texture compression suitable for use in subsequent lossless compression according to rate-distortion optimization (RDO).SOLUTION: A method includes: encoding each of a plurality of blocks obtained by dividing a texture into a plurality of bitstreams: selecting a block bitstream having a smallest distortion value for each block; obtaining data chunks such that, for each selected block bitstream, each data chunk is a contiguous section of the respective selected block bitstream from which it was obtained; and detecting a collision between the obtained data chunks; and based on the detected collision, forming match chains having redundant data chunks such that, for each match chain, the redundant data chunks therein represent a common value of the associated block bitstream.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] Certain example embodiments described herein relate to texture compression that may be used in three-dimensional virtual environments, etc. More particularly, certain example embodiments described herein relate to statistical techniques for texture compression that result in lossy encoded textures suitable for use in subsequent lossless compression according to rate-distortion optimization (RDO). [Background technology]

[0002] Textures are one of the most important assets in a virtual environment, such as those used in video games. A texture is a material that acts as a layer covering the surface of a three-dimensional (3D) object to give it a desired appearance in the virtual environment (e.g., a desired in-game appearance). Most textures are stored as files that contain a two-dimensional (2D) array of pixels, sometimes called texels. The process of mapping these pixels onto an object is called texture mapping. Summary of the Invention

[0003] Figure 1 illustrates wrapping a three-dimensional object with a two-dimensional texture to produce a texture-mapped object. More specifically, Figure 1 illustrates a three-dimensional spherical object 102 and a two-dimensional texture 104 that is a world map. The texture mapping makes the texture-mapped object 106 appear as a sphere because the two-dimensional texture 104 is wrapped around the three-dimensional spherical object 102. For example, the example of Figure 1 takes into account lighting effects included in the three-dimensional object 102, as seen in the generated texture-mapped object 106.

[0004] Textures differ in the type of data they store. A "normal map", for example, stores the X, Y, and Z coordinates of the surface normal (a vector perpendicular to the local surface of an object). These components can be used to calculate how light reflects off different areas of an object, giving it a greater sense of detail and therefore increased "realism". "Realism" in this sense may reflect a more immersive experience of a real-world environment, or for example a setting that is obviously not "real" (such as some virtual game environments). Specular maps, on the other hand, typically store grayscale values ​​that can be used to control the shininess of an object. In such cases, the higher the value, the more reflective the area is. Other common types of textures include diffuse maps, albedo maps, displacement maps, light maps, ambient occlusion, etc.

[0005] Texture data is generally not stored "as is." In other words, textures used in a virtual environment, such as a game world, are typically not stored in a format that is directly usable in such an environment. Instead, the texture data is typically stored in a lossy compressed format. Storing texture data in a compressed format provides several technical advantages, such as, for example, a reduced amount of GPU or other memory required and reduced memory bandwidth usage, as compared to, for example, "raw" texture data.

[0006] Compression formats used for textures are different from those used for image compression formats (such as JPEG) and video formats (such as H.264). One difference is that compression formats used for textures typically encode blocks of pixels independently, producing a fixed-size output for each block. For example, the BC1 format used to compress textures encodes blocks of 4x4 pixels and outputs a bitstream of 8 bytes per block. Thus, each pixel is encoded with a fixed number of bits (or bits per pixel (BPP)), which is well suited for random GPU access. That is, to access a particular pixel, the GPU calculates the address of the block to which it belongs. The GPU then decodes the block on the fly to obtain the pixel data. Because the bitrate is fixed, such formats generally result in lower compression ratios compared to formats with variable bitrates. Common and widely used texture compression formats include block compression (BC) formats such as BC1, BC3, BC4, BC5, BC7, and Adaptive Scalable Texture Compression (ASTC), with typical bit rates for such formats ranging from 8 BPP to 1 BPP.

[0007] Texture-related assets typically make up a large portion of a game's total storage size. In many situations, limiting their size is crucial. For example, some games are stored on a physical medium such as a cartridge. Smaller games can use smaller media (e.g., smaller cartridges), which can lead to lower manufacturing costs and more flexibility in the types of media that can be used. As another example, downloading a digital game can use less server bandwidth, require fewer computing resources on both the server and client side, and potentially reduce maintenance requirements on the server side.

[0008] There are various ways to control texture size. One approach involves creating smaller textures when possible. For example, a texture with a resolution of 2048x2048 pixels may actually only have a "useful area" of 1024x1024 because, for example, data outside this area may not be fetched by the GPU (e.g., a game or other application may not use it). In this case, data outside the useful area is not rendered on the screen and is not used in any rendering calculations. In such cases, there may be no need to store such data. In the same vein, some channels (e.g., color channels, coordinate channels, etc.) may be redundant with other channels or, again, may not be accessed at all by the GPU.

[0009] Another common technique that can be combined with the techniques mentioned above is to store (compressed) texture files in a lossless format. In other words, lossy compressed texture files can be further compressed using lossless compression. This is sometimes called "ultra-compression". Lossless formats tend to be quickly restored, which typically adds little or negligible overhead to the load time (in this case the time it takes for the texture to move from storage memory (e.g., cartridge, SSD, or other location) to GPU memory (e.g., VRAM) or other memory location). Lossless compression formats (e.g., LZ4 and ZStandard) provide a technically "interesting" tradeoff between compression ratio and restoration speed.

[0010] While a variety of compression techniques (including ultra-compression and other techniques) are widely used for virtual environments (including virtual environments provided in connection with games), further improvements remain possible. For example, it would be desirable to better manage the trade-off between the amount of compression and quality, while potentially allowing for reasonable compression and decompression times.

[0011] Certain exemplary embodiments improve texture compression techniques, for example, by providing improvements to texture compression tools, for example, certain exemplary embodiments provide a statistical approach for texture compression that results in lossy encoded textures suitable for use in subsequent lossless compression following rate-distortion optimization (RDO).

[0012] One aspect of an exemplary embodiment relates to providing a "generic" encoding technique (including a "generic" RDO algorithm) for compression (including texture compression). The encoding technique includes an RDO algorithm that is generic in the sense that the bitstream details of the format used to encode the texture do not need to be known in advance, meaning that it is almost completely independent of the underlying encoder (e.g., the underlying texture encoder). The encoding technique includes: First, each block in the texture is encoded in a number of different ways (e.g., using an optimal compression format, e.g., BC1, ASTC, etc.). The generic RDO algorithm looks for collisions, post-encoding, among all encoding configurations of all blocks, and recognizes that from a statistical perspective, such collisions are likely to occur quite frequently, and can be exploited in the overall compression technique.

[0013] Another aspect of certain exemplary embodiments relates to decoupling the generation of block encoding configurations from downstream processing, so that the algorithm of certain exemplary embodiments is parallelizable and can rapidly handle many different tradeoffs between quality and size (as controlled by hyper-parameters).

[0014] Another aspect of certain exemplary embodiments relates to the ability to advantageously use a wide variety of lossy compression formats. Indeed, due to the comprehensiveness described herein, the algorithms of certain exemplary embodiments can perform RDO using compression standards such as ASTC, where the bitstream is quite complex.

[0015] In an example embodiment, a method for encoding a texture is provided. The texture is read from a data store. The texture is divided into a plurality of blocks. Each of the blocks is encoded into a plurality of block bitstreams, each of the block bitstreams corresponding to a respective encoding configuration and having a distortion value associated therewith. A block bitstream with the smallest associated distortion value for each block is selected, such that for each block, up to a predetermined number of distinct block bitstreams are selected. For each selected block bitstream, a plurality of data chunks are obtained, such that each data chunk is a contiguous section of the respective selected block bitstream from which it was obtained. The data chunks for each selected block bitstream start at a respective position of the respective selected block bitstream and extend to an end of the respective selected block bitstream, such that each data chunk has a minimum data chunk size. Data chunks of different sizes can be obtained for the selected bitstreams. Collisions between the obtained data chunks are detected. Based on the detected collisions, a match chain is formed for each match chain with redundant data chunks, such that the redundant data chunks therein represent a common truncated value of the associated block bitstream. A best match chain is iteratively selected from the formed match chains, the best match chain being determined using hyper-parameters, and a lossy encoding of the texture is obtained from the iteratively selected best match chain.

[0016] In one example embodiment, the lossy encoded texture may be encoded using a lossless compressor.

[0017] In an example embodiment, the data chunks may start at different byte locations.

[0018] In an example embodiment, for each block, as many distinct block bitstreams as possible may be selected, up to a predetermined number. An indicator may be provided for each block for which the number of selected block bitstreams is less than the predetermined number.

[0019] In an example embodiment, an input may be received that identifies a maximum distortion value for one or more identified blocks of the plurality of blocks, and for each of the one or more identified blocks, only block bitstreams having a distortion value better than the maximum distortion value may be selectable for the respective identified block.

[0020] In an example embodiment, the minimum data chunk size may be at least as large as the minimum size available for a lossless compressor to apply lossy encoding of the texture.

[0021] In an example embodiment, obtaining the data chunks may comprise extracting a subset of the valid data chunks for the given selected block bitstream and generating residual data chunks for the given selected block bitstream from the extracted subset.

[0022] In one example embodiment, the hyperparameters may sacrifice rate or distortion for the other and may be part of a defined function that determines which match chain is best.

[0023] In one example embodiment, the texture may be divided into slices, each of which includes a predetermined number of blocks, and the slices may be treated as separate textures that are separately divided into blocks.

[0024] In an example embodiment, match chains that are determined to be unselectable in the iterative selection may not be considered for the formation of match chains. For example, a given match chain may be discarded in response to a determination that another match chain has been or will be formed that is equal to or better than the given match chain. Application of a function of hyperparameters to the match chains may allow different match chains to be compared to one another, and / or iterative selection may select the best match chain.

[0025] In an example embodiment, a method for encoding a texture is provided. Each of a plurality of blocks obtained by dividing the texture is encoded into a plurality of block bitstreams, each of the block bitstreams corresponding to a respective encoding configuration and having a distortion value associated therewith. A block bitstream with the smallest associated distortion value is selected for each block, such that for each block up to a predetermined number of distinct block bitstreams are selected. For each selected block bitstream, a plurality of data chunks are obtained, such that each data chunk is a contiguous section of the respective selected block bitstream from which it is obtained. For the selected bitstreams, data chunks of different sizes can be obtained. Collisions between the obtained data chunks are detected. Based on the detected collisions, a match chain is formed for each match chain with redundant data chunks, such that the redundant data chunks therein represent common values ​​of the associated block bitstreams. A best match chain is selected from the formed match chains. The best match chain is determined using a hyperparameter. A lossy encoding of the texture is obtained from the selected best match chain.

[0026] In an example embodiment, a system is provided for encoding textures, the system comprising: a data store having a texture stored therein; a memory provided, and at least one processor configured to perform operations comprising: reading the texture from the data store; dividing the texture into a plurality of blocks; and encoding each of the blocks into a plurality of block bitstreams, each of the block bitstreams corresponding to a respective encoding configuration and having a distortion value associated with each of the block bitstreams; the operations further comprising: selecting for each block a block bitstream having a lowest associated distortion value, such that for each block, up to a predetermined number of distinct block bitstreams are selected; and for each selected block bitstream, obtaining a plurality of data chunks, each data chunk being a contiguous section of the respective selected block bitstream from which it was obtained; the data chunks for the frame start from a respective position of the respective selected block bitstream and extend to an end of the respective selected block bitstream such that each data chunk has a minimum data chunk size, and data chunks of different sizes are obtainable for the selected bitstreams, the operations further comprising: detecting collisions between the obtained data chunks; forming, for each match chain, match chains with redundant data chunks based on the detected collisions, such that the redundant data chunks therein represent a common truncated value of the associated block bitstream; and iteratively selecting a best match chain from among the formed match chains, the best match chain being determined using a hyper-parameter; and the operations further comprising obtaining a lossy encoding of the texture from the iteratively selected best match chain. In one example embodiment, the lossy encoded texture may be encoded using a lossless compressor.

[0027] In an example embodiment, the data chunks may start at different byte locations.

[0028] In an example embodiment, for each block, as many distinct block bitstreams as possible may be selected, up to a predetermined number. An indicator may be provided for each block for which the number of selected block bitstreams is less than the predetermined number.

[0029] In an example embodiment, an input may be received that identifies a maximum distortion value for one or more identified blocks of the plurality of blocks, and for each of the one or more identified blocks, only block bitstreams having a distortion value better than the maximum distortion value may be selectable for the respective identified block.

[0030] In an example embodiment, obtaining the data chunks may comprise extracting a subset of valid data chunks for a given selected block bitstream, and generating residual data chunks for the given selected block bitstream from the extracted subset.

[0031] In an example embodiment, a method for encoding a texture is provided. Each of a plurality of blocks obtained by dividing the texture is encoded into a plurality of block bitstreams. For each block, up to a predetermined number of separate block bitstreams are selected. For each selected block bitstream, a plurality of data chunks are obtained, such that each data chunk is a contiguous section of the respective selected block bitstream from which it is obtained. Collisions between the obtained data chunks are detected. Based on the detected collisions, a match chain is formed with redundant data chunks for each match chain, such that the redundant data chunks therein represent common values ​​of the associated block bitstreams. Based on a hyperparameter, a set of match chains is iteratively selected from among the formed match chains. At least some potential match chains are determined to be non-selectable in the iterative selection and are not considered in forming the match chain. A lossy encoding of the texture is obtained from the selected set of match chains.

[0032] In an example embodiment, a non-transitory computer-readable storage medium is provided that stores instructions that, when executed by a processor, cause a computer to perform operations comprising: encoding each of a plurality of blocks obtained by dividing a texture into a plurality of block bitstreams, selecting for each block up to a predetermined number of separate block bitstreams, obtaining for each selected block bitstream a plurality of data chunks such that each data chunk is a contiguous section of the respective selected block bitstream from which it is obtained, detecting collisions between the obtained data chunks, forming for each match chain with redundant data chunks based on the detected collisions such that the redundant data chunks therein represent common values ​​of associated block bitstreams, iteratively selecting a set of match chains from among the formed match chains based on a hyper-parameter, where at least some potential match chains are determined to be unselectable in the iterative selection and are not considered in forming the match chains, and further comprising obtaining a lossy encoding of the texture from the selected set of match chains.

[0033] In an example embodiment, a texture encoding system is provided, wherein a texture is stored in a data store, and a memory is provided, and wherein at least one processor is configured to perform operations, the operations comprising: for each block, selecting up to a predetermined number of distinct block bitstreams, for each selected block bitstream, obtaining a plurality of data chunks, such that each data chunk is a contiguous section of the respective selected block bitstream from which it is obtained, detecting collisions between the obtained data chunks, and forming, for each match chain, match chains having redundant data chunks therein based on the detected collisions, such that the redundant data chunks therein represent common values ​​of associated block bitstreams, iteratively selecting a set of match chains from among the formed match chains based on a hyperparameter, wherein at least some potential match chains are determined to be unselectable in the iterative selection and are not considered in forming the match chains, and the operations further comprise obtaining a lossy encoding of the texture from the selected set of match chains.

[0034] In an example embodiment, in response to a determination that another match chain is formed or will be formed that is equal to or better than the given match chain, the given match chain may not be considered.

[0035] In one example embodiment, application of a function of hyperparameters to match chains may allow different match chains to be compared to each other.

[0036] In an example embodiment, an iterative selection may select the best match chain. In one example embodiment, a lossless compressor may be used to encode the lossy encoded texture.

[0037] In one example embodiment, the block bitstream with the smallest associated distortion value for each block may be selected such that up to a predetermined number of distinct block bitstreams are selected.

[0038] In an example embodiment, for a selected bitstream, data chunks of different sizes may be available.

[0039] In one example embodiment, a method for providing a virtual environment in connection with a computing system is provided, wherein an encoded texture is read from a non-transitory computer-readable storage medium, the texture being encoded using the techniques (e.g., methods) described herein, and the encoded texture is provided to at least one processor of the computing system for decoding the texture and use of the decoded texture in the virtual environment.

[0040] In an example embodiment, a non-transitory computer-readable storage medium stores instructions that, when executed by a processor, cause a computer to perform operations corresponding to the techniques (eg, methods) described herein.

[0041] In an example embodiment, a computing system is provided through which a virtual environment can be displayed. The system comprises a memory coupled to one or more processors configured to perform operations comprising reading an encoded texture from a non-transitory computer-readable storage medium, the texture being encoded using a technique (e.g., a method) disclosed herein, and further comprising providing the encoded texture to at least one of the at least one processor of the computing system for decoding the texture and use of the decoded texture in the virtual environment.

[0042] The features, aspects, advantages, and example embodiments described herein may be used separately and / or applied in various combinations to achieve still further embodiments of the present invention.

[0043] These and other features and advantages may be more fully and completely understood by reference to the following detailed description of exemplary illustrative embodiments taken in conjunction with the drawings. [Brief description of the drawings]

[0044] [Figure 1] It shows a 3D object being wrapped with a 2D texture to produce the object after texture mapping. [Diagram 2] Three encodings of the same texture are shown, along with the size and peak signal-to-noise ratio (PSNR) for each. [Diagram 3] 1 is an example illustrating the extraction of different encoding configurations according to an example embodiment. [Figure 4] 1 illustrates an extraction of chunks from an exemplary encoding configuration, according to an exemplary embodiment. [Diagram 5] 13 illustrates a numerical example of match chain identification, according to an example embodiment. [Figure 6] 11 helps illustrate how optimal match chains are created in one exemplary embodiment. [Figure 7] 11 helps illustrate how optimal match chains are created in one exemplary embodiment. [Figure 8] 11 helps illustrate how optimal match chains are created in one exemplary embodiment. [Figure 9] 1 illustrates one iteration of a match chain resolution procedure of an example embodiment. [Figure 10] 1 illustrates a schematic diagram of a texture divided into four slices according to a texture slicing technique of an example embodiment. [Figure 11]In the context of one exemplary embodiment, it helps illustrate when a match chain may be broken. [Figure 12] 1 illustrates an example of an algorithmic factorization in accordance with an illustrative embodiment. [Figure 13] 5 is a flowchart illustrating a process for generating an encoded texture according to an example embodiment. [Figure 14] 14 is a block diagram illustrating a computing architecture that may be used in connection with the example flowchart of FIG. 13, according to an example embodiment. [Figure 15] 1 is a flowchart illustrating a process for decoding a texture encoded via the techniques described herein, in accordance with an example embodiment. [Figure 16] 16 is a block diagram illustrating a computing device that may be used in connection with the example flowchart of FIG. 15, according to an example embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] Certain example embodiments relate to texture compression techniques, and more particularly to systems and / or methods for texture compression suitable for use in virtual worlds, such as those used in video games and the like.

[0046] Texture encoding is the process of applying a texture encoder to a texture. A texture encoder encodes a texture by processing pixels into separate blocks (typically 4x4 square windows) and outputs a bitstream for each of these blocks (typically 8 or 16 bytes, depending on the format). A bitstream is the binary sequence that a texture encoder produces as a result of encoding a texture. It is the binary sequence that you get after encoding a single block of pixels from a texture, or after encoding the entire texture.

[0047] Compression and encoding are related and may be used interchangeably in some situations. Encoding data generally refers to the process of changing one data representation into another. For example, the value "15" can be encoded in hexadecimal as "0×F". Compressing data, on the other hand, means effectively reducing the size of the data. Encoding data often makes it smaller. So, from this perspective, encoding is a form of compression, but it does not have to be. Encoding and compression techniques can be lossy or lossless. Lossless compression (encoding) means that the compressed (encoded) data can be restored (decoded) to allow perfect reconstruction of the original data. Common lossless compressors include LZMA, ZStandard, and LZ4. Texture encoders are generally not lossless. The PNG format is an example of lossless encoding followed by lossless compression.

[0048] The inventors have recognized that texture encoders typically have multiple ways to encode a given block of pixels, often at very little additional cost in terms of quality. This is due to the fact that such texture compression formats use a fixed bit rate per block, which is often unnecessarily high for the complexity of the blocks being encoded. Secondly, the inventors have recognized that lossless compressors generally achieve better compression if more redundancy appears in the data. Thus, by carefully choosing the encoding configuration of texture blocks, it is possible to maintain high texture quality while achieving good compression ratios. Certain exemplary embodiments provide the advantageous technical effects of high quality and good compression for texture compression techniques.

[0049] An exemplary embodiment uses a rate-distortion optimization (RDO) technique. RDO is a technique used in lossy image / video compression to jointly optimize two criteria in the encoding process: rate (size of data) and distortion (quality of the asset). Distortion is generally measured using a metric. A smaller rate generally means a higher distortion and therefore a desire for optimization.

[0050] As applied to texture compression, RDO encodes textures not by optimizing quality alone, but rather by optimizing both the size of the texture (to be understood as the size after compression by a lossless compressor) and the quality together. These two "axes" are the rate and distortion mentioned earlier. The optimization process can be represented by hyperparameters, which represent a trade-off between quality and memory. Changing the hyperparameter value is a matter of prioritizing one of the two axes over the other, resulting in a texture with better quality at the expense of more memory, or a texture with lower quality but smaller size. For example, depending on the cost function, a higher value of the hyperparameter can mean that rate is prioritized over distortion. In the context of texture compression, where the encoder produces a fixed-size bitstream (fixed rate), the rate can be influenced by leveraging a lossless compressor.

[0051] "Rate" in the RDO format refers to the size of the data after encoding / compressing the asset. In contrast, in the context of texture encoding with RDO, an attempt is made to estimate the rate at the level of the blocks of the texture. Thus, the rate implies how much storage size savings are expected when choosing a given encoding configuration for a given block after applying lossless compression to the texture. As a concrete example, it can be estimated that choosing encoding configuration number 3 for block number 24 will reduce the texture size by 2 units (e.g., bytes), so the rate is 2. "Distortion" in the RDO format refers to the quality of the asset after encoding / compression, the idea being that the higher the distortion, the worse the quality of the asset. In the context of texture encoding with RDO, distortion is measured at the level of the blocks of the texture. Thus, the distortion implies how much quality loss occurs as a result of choosing a given encoding configuration for a given block. As a concrete example, it can be estimated that choosing encoding configuration number 3 for block number 24 will reduce the block quality by 2 units (e.g., decibels), so the distortion is 2. Further details regarding rate and distortion in the context of RDO in an example embodiment are provided below.

[0052] FIG. 2 shows three encodings of the same texture, along with their size and peak signal-to-noise ratio (PSNR) for each. PSNR is a widely used metric for evaluating image quality. The unit of PSNR is decibels (dB). The higher the PSNR value, the higher the overall quality of the texture. In FIG. 2, the top encoding is a standard block compression using BC1 encoding (a lossy format) without RDO, while the next two encodings are BC1 encoding with RDO using different values ​​for the hyperparameter (called lambda). As the PSNR shows, the quality of the textures after RDO encoding is lower. However, their size is smaller after compression with the lossless ZStandard format (denoted as Zstd in FIG. 2). It is acknowledged that the images, formats, and figures are given for illustrative purposes only. These and / or other lossy and / or lossless encoders may be used in conjunction with these and / or other images / textures, with these and / or other lambda values, which may result in different metrics (measured in PSNR or some other metric) and / or different size reduction.

[0053] As described in more detail below, an exemplary embodiment provides a statistical approach to RDO in conjunction with a texture collision algorithm. The texture collision algorithm of an exemplary embodiment is advantageous in several ways. First, the approach of an exemplary embodiment uses a statistical principle known as the Birthday Paradox. This is a statistical result that states that repeatedly generating values ​​from variables leads to "collisions" (previously obtained values) much faster than common sense would suggest would happen, hence the paradox. For texture compression, an exemplary embodiment extracts a large number of encoding configurations for a given block, taking advantage of the high collision frequency that results from surrounding blocks that themselves have a large number of possible encoding configurations. As alluded to above, an encoding configuration is a possible valid way of encoding a block of pixels (i.e., resulting in a valid bitstream for this block), given a texture encoder. The algorithmic approach becomes almost completely independent of the underlying texture encoder. That is, once the algorithm has a "catalog" of encoding configurations for each block, it can work with only very basic information about the encoder, such as block bitstream size and block dimensions. More precisely, an exemplary embodiment does not need to know details about block bitstream structure such as headers, colors, weights, delimiters, etc., whereas a traditional RDO approach would require precise description of this and / or other information. Thus, the approach of an exemplary embodiment paves the way for compatibility with a wide range of lossy encoders / texture formats, including (for example) ASTC, which has been an obstacle for many encoders for many years. Also, the algorithmic approach of an exemplary embodiment can essentially find matches with completely unrelated data. A "match" in the context of traditional lossless compression is defined as a collision (repetition) of some chunk of data in the source to be compressed.A lossless compressor exploits such redundancy, for example by replacing one or some of the duplicated chunks with a reference to another chunk, resulting in smaller data. Unlike some conventional approaches, an exemplary embodiment, as noted, can find matches with completely unrelated data. For example, it can match the color of one block with the weight of another block, provided it determines that it is "interesting" or useful to do so.

[0054] A second advantage relates to decoupling the texture encoder. The texture encoder of an example embodiment is not "RDO aware" in the sense that it extracts multiple encoding configurations for each texture block. This is in contrast to conventional encoders that typically only pick the best quality configuration. That is, an example embodiment allows the generation of block encoding configurations to be decoupled from other downstream operations in the encoding pipeline. In this way, the approach of an example embodiment is easily parallelizable and can rapidly handle several different tradeoffs between quality and size, i.e., several different values ​​of the hyperparameters.

[0055] The third advantage below is related to (lambda) factorization. The decoupling between texture encoding and RDO allows the texture encoding to be "factorized" to produce multiple RDO encoded versions of a texture, each with different hyper-parameter values ​​(and therefore different quality vs. compression tradeoffs). As will be shown in more detail below, it allows most of the core steps of the texture collision algorithm of an exemplary embodiment to be factorized, thereby minimizing the lambda-dependent parts. In other words, it allows multiple versions of an encoded texture to be output with very little additional time compared to outputting a single texture.

[0056] The Birthday Paradox in Texture Compression This section helps illustrate the birthday paradox as it applies to texture compression. Suppose K random numbers between 1 and N are chosen (with equal probability, and with replacement, meaning that the same number can be chosen several times). Then the expected collision frequency E (the average number of choices that result in a duplicate number) is given by:

[0057]

number

[0058] Assume that a texture of 256x256 pixels is encoded in ASTC format with a block size of 4x4 pixels. This corresponds to 4,096 blocks. With ASTC format, each block is encoded with 16 bytes. For the sake of this example, let us count the number of expected 4-byte matches that will be present in the encoded texture. Since the encoded size is 65,536 bytes (4,096x16 bytes), this represents a total of 65,533 consecutive 4-byte chunks. Applying the formula above, we determine the expected collision frequency of these chunks, or how many chunks are expected to have the same value as at least one other chunk. In particular, applying the formula above with N=2^32 (the range of integer values ​​that a 4-byte chunk can take) and K=65,533 (the number of 4-byte chunks) shows that E=0.9988≈1.

[0059] Now, assume that the texture is "enlarged" to 1,024 times its original size, which corresponds to a texture of 8,192x8,192 pixels, i.e. 4,194,304 blocks of 4x4 pixels. If E is computed on the encoded texture, N still equals 2^32, but now K is 6,7108,861, which gives E=1,040,426.

[0060] In summary, by multiplying the texture size by 1,024, the expected collision frequency is now multiplied by over 1 million.

[0061] As a general matter, it is not really useful to actually increase the texture size / texture dimensions. However, a similar effect can be achieved in a different way: if we can extract multiple encoding configurations for each block, it will be possible to recreate conditions similar to the illustrated scenario, while still working with the original texture dimensions. Essentially, one exemplary embodiment proceeds in this way.

[0062] Note that this example has been simplified in a number of ways. In practice, not all 4-byte values ​​are achievable possibilities, since some would correspond to unrelated (and sometimes invalid) bitstreams. Furthermore, textures typically exhibit strong spatial correlation patterns in some areas, which would result in a significant increase in the probability of a match in such regions. This example and the formula for E assume uncorrelated values. Typical supercompression ratios for ASTC-encoded textures (defined as the ratio between the size of the ASTC-encoded texture and the size of the ASTC-encoded and losslessly compressed texture) range from 1 to 1.3, which in practice tends to indicate a higher collision frequency than computed. Thus, practical realities can make the techniques disclosed herein perform even better than a strict mathematical interpretation of the Birthday Paradox would suggest.

[0063] Texture Collision Algorithm of an Example Embodiment The texture collision algorithm of an example embodiment can be decomposed into different (conceptual) stages, each of which is described in more detail below.

[0064] Stage 1: Multi-component texture encoding As this first stage, the texture encoder encodes the texture and returns for each block not only the best encoding configuration (i.e., the block bitstream), but rather a top-N list of the separate best encoding configurations. In an exemplary embodiment, "best" should be understood to mean least distortion. Typical values ​​of N can range, for example, from 128 to 4,096, depending on the texture encoder at hand as well as the memory consumption required to store these configurations. The "Texture Slices" section below provides some considerations regarding memory consumption. These values ​​of N have been found to work well, but different implementations may use larger or smaller values.

[0065] In addition to the bitstream, a distortion value is stored for each of these encoding configurations. The "Rate and Distortion" section below provides a discussion of examples of possible metrics that may be used in an exemplary implementation.

[0066] For some blocks, it may not be possible to completely fill the list. One example where this can happen is when some regions in a texture need to be treated with special care. For example, textures often have completely flat regions, and any visual artifacts introduced in such regions are generally very noticeable and therefore hardly desired. In such cases, it may be desirable to simply maintain configurations with the least distortion. More generally, for some blocks, it may be desirable to explicitly control the distortion value and thus the number of encoding configurations. In any case, if the encoder will not return the maximum number of configurations it is allowed to obtain, an indicator may be provided in the storage buffer immediately after the last valid configuration to indicate the end of the list.

[0067] FIG. 3 is an example showing extraction of different encoding configurations according to an example embodiment. In FIG. 3, an area 302 is a texture divided into M texture blocks, and the encoding configurations for these blocks are represented by gray boxes with bent corners. Each block has N possible encoding configurations. i,j is block number i(b i ) for the encoding configuration number j, d i,j is the distortion value associated with this encoding structure. "END" is a marker that indicates the end of the encoding structure if the list is not completely filled. i For, the encoding configurations are sorted by increasing distortion (d i,1 ≦d i,2 ≦ ≦ d i,N ).

[0068] Stage 2: Chunk Extraction This stage extracts contiguous groups of bytes (also called chunks) from the bitstream collected through the previous stage. For a given bitstream, chunks are extracted starting from every byte position and extending to the end of the bitstream. In an exemplary embodiment, each chunk size is larger than a given value specified as MIN_MATCH_SIZE, a modifiable parameter.

[0069] To help understand the operation of this stage, consider the following example where the texture encoder is BC1. BC1 produces a bitstream of 8 bytes for each block. In this example, MIN_MATCH_SIZE is set to 4. Continuing with this illustration, FIG. 4 shows the extraction of chunks from an exemplary encoding configuration, according to an exemplary embodiment. As can be seen from FIG. 4, five chunks are extracted from the BC1 bitstream, ranging in size from 4 bytes to 8 bytes. For each of these extracted chunks, additional information elements are also stored that may be useful for subsequent stages. These additional information elements include the encoding configuration index (j in FIG. 4), the block index (i in FIG. 4), and the starting position of the chunk in the bitstream (0 means that the chunk starts at the first byte).

[0070] There is no need to extract more chunks, because from these five chunks it is possible to obtain all values ​​in the bitstream by removing some of the terminating bytes from these chunks (e.g., via a masking process). For example, the value v2v3v4v5 can be obtained by taking a 7-byte chunk (with values ​​v2v3v4v5v6v7v8) and masking its last 3 bytes (v6, v7, v8).

[0071] The value of MIN_MATCH_SIZE may be different in different exemplary embodiments. For example, it has been found that MIN_MATCH_SIZE may depend on the lossless compressor used. Typically, LZ4 does not consider matches truly smaller than 4 bytes, so it would be counterproductive if chunks of this size were extracted. It is desirable to create redundancy in the final texture bitstream so that the lossless compressor can compress the texture more efficiently. There is little point in looking for a 3-byte match if it is known or expected that the compressor will not see 3 bytes. ZStandard may consider a 3-byte match, but efficiency is not guaranteed. In fact, most lossless compressors actually start to utilize matches starting at 4-byte size, so that may be the default value used in one exemplary embodiment.

[0072] MAX_MATCH_SIZE is defined to be the maximum match size that the texture collision algorithm will consider. By default, you may set this to be equal to the bitstream size, since you cannot extract chunks larger than this value. You may choose a smaller value. There may be some cases where you have specific reasons to avoid such a match size, so the value may be lowered from this default.

[0073] Stage 3: Creating a matching chain This stage contains several sub-stages, including a sub-stage for collision detection and for creating optimal match chains.

[0074] Stage 3.1: Collision detection Once the bitstream chunks are extracted for all encoding configurations of all blocks, we detect collisions between them and identify matches. To do this, we can process the chunks by increasing the match size from MIN_MATCH_SIZE to MAX_MATCH_SIZE.

[0075] For the smallest match size (MIN_MATCH_SIZE), all chunks are sorted by value truncated to MIN_MATCH_SIZE bytes. For example, if MIN_MATCH_SIZE=4, then in the example in Figure 4, the 8-byte chunks would be truncated to v1v2v3v4, the 7-byte chunks would be truncated to v2v3v4v5, and so on. Since these values ​​are integers (32-bit values ​​for 4-byte chunks), an efficient implementation of radix sort may be used to sort the chunk buffer in linear time. This immediately gives duplicate values, and therefore matches.

[0076] It is recognized that other sorting algorithms may be used in different exemplary embodiments. Furthermore, additional optimizations may be implemented to help speed up collision detection. For example, to find matches of higher sizes, the following and / or other optimizations are possible: First, only chunks that are known to have overlaps at the previous chunk size need to be sorted. For example, to sort chunks at a match size of 5 bytes, only chunks that had overlaps at a match size of 4 bytes may be considered, because if a given chunk does not match at 4 bytes, it cannot match at 5 bytes. In practice, this approach has been found to dramatically reduce the number of chunks to be sorted.

[0077] Second, we can discard chunks that are not large enough to handle the current match size. Referring back to Figure 4 for illustration, we cannot consider a 4 byte chunk for a match size of 5 bytes because we cannot "extend" it, as doing so would take us outside the bitstream data. We use the start position to detect and discard these "unextendable" chunks.

[0078] Third, since all chunks are already sorted by the previous match size, an incremental sorting algorithm can be used to sort the chunks by only additional bytes. In practice, this technique has been shown to dramatically reduce processing time as well. For example, assuming the chunk buffer is already sorted by the first 4 bytes (and assuming the sort is stable), sorting a chunk by 5 bytes would only involve sorting according to the 5th byte. For example, an efficient radix sort using a single pass can perform this procedure again in linear time.

[0079] A group of overlapping values ​​may be considered as a match chain. Figure 5 shows a numerical example of match chain identification, according to an example embodiment. The top part of Figure 5 shows an extraction of collisions of 4-byte chunks. When moving to higher match sizes (as in the bottom part of Figure 5), we see that some chunks still collide at 5 bytes. However, some chunks cannot extend above 5 bytes (chunks marked with a cross on the left) or no longer collide (chunks marked with a cross on the right).

[0080] It is recognized that some conflicting chunks may refer to the same block but use different encoding configurations. An exemplary embodiment operates such that only one encoding configuration per block is ultimately chosen, so that such types of matches are not actually made. When such cases occur, only the chunks that originate from the encoding configuration with the least distortion are kept in the match chain, while the other chunks are removed.

[0081] Stage 3.2: Creating an optimal match chain 6 through 8 help illustrate how an optimal match chain is created in one example embodiment. In FIG. 6, the match chain contains all redundant chunks for a given value X, where X is the associated encoding configuration e of index j for block i. i,jThe chunk start position is not needed here, so it is omitted from this representation. However, the distortion (d i,j ) is added. Note that this is the associated encoding configuration for this block (e i,j ) is the distortion that occurs in the block as a result of choosing

[0082] All these redundant chunks form a chain of length N. However, from this chain we can create smaller match chains with fewer chunks. For example, we can group the first three chunks together, which would also form another chain, as shown for example in Figure 7.

[0083] As can be seen from Figure 7, there are many ways to form a chain. N -N-1 ways. All these newly formed chains have their own rates and distortions, which makes each of them potentially interesting to consider, but keeping them all may not be feasible, since N could potentially be any value. For example, if N=16, there would be 65,519 chains to remember.

[0084] This number can be reduced due to suboptimality. In one exemplary embodiment, a match chain is treated as suboptimal if another chain exists that represents a better tradeoff between rate and distortion. However, if no such chain exists, one exemplary embodiment considers the chain to be the optimal chain.

[0085] To see this, assume in Figure 7 that the chunks are sorted from left to right with increasing skewness, i.e., d i1,j1 ≦d i2,j2 ≦ ≦ d iN,jN Assume that.

[0086] Now, suppose the chain of the first two chunks in Figure 7 is "applied". Applying the chain in this context means setting encoding configuration j1 ​​to block i1 and encoding configuration j2 to block i2. A lossless compressor will see a value (X) in the texture bitstream that repeats twice, and then such redundancy will result in a certain storage gain R (rate). The quality loss (distortion) caused by choosing these two encoding configurations is the sum of the two distortions D=d i1,j1 +d i2,j2 It becomes.

[0087] Now consider instead the chain consisting of the second and third chunks of Figure 7. After setting the two encoding configurations j2 and j3 for blocks i2 and i3 respectively, the situation of the lossless compressor is the same as before: it will see the same overlap value (X) and will then yield the same rate R. However, the distortion is now D=d i2,j2 +d i3,j3 ≧d i1,j1 +d i2,j2 It is.

[0088] Thus, an exemplary embodiment prefers the first chain over the second chain because it has the same rate while having less distortion. In other words, the first chain "outperforms" the second chain. The second chain is therefore considered a suboptimal chain. Based on the above, it is inferred that in practice there are only N-1 optimal chains, i.e., one consisting of the first two chunks, a second one consisting of the first three chunks, etc., up to one consisting of all chunks.

[0089] It should be noted that the use of terms such as "optimal", "optimized" and the like here and elsewhere does not necessarily mean "most optimal", "fully optimized", and the like. For example, it is not strictly accurate to call these chains optimal here, since it is not yet known whether they are in fact most optimal. In fact, it may happen that some of all the chains (derived from all chunks of all encoding configurations of all blocks) that will be formed are better than these. For example, there may be another chain that is found to have a better tradeoff between rate and distortion than any of the N-1 chains here. However, at this point, it is known that the discarded chain is suboptimal, since at least one of the N-1 chains considered outperforms them all, and therefore these N-1 chains may be treated as optimal, at least at this point in the process.

[0090] As can be seen from Figure 8, the number of chains is 2 N-N-1 to N-1. An exemplary embodiment does not lose anything by considering only these chains and discarding the others. As will be seen below in the description of the next stage, the chains are applied according to a specific order (controlled by the hyperparameter lambda). Each time a chain is applied, some blocks are set to a specific encoding configuration, and once the blocks are set, they can no longer be changed. Returning to the example above, we see that the chain formed by the first and second chunks and the chain formed by the second and third chunks compete with each other. That is, they both refer to block i2, which means that applying one makes the other no longer applicable. However, due to optimality, it would be suboptimal to apply the second chain instead of the first chain. This makes the second chain unnecessary and it can be discarded. If block i1 is already set, the chain consisting of the first and second chunks is no longer applicable (because it refers to block i1), and logically, the chain consisting of the second and third chunks becomes the optimal chain. But there is a problem. Because that chain has not been kept and is therefore inapplicable. In this case, once block i1 is set, this chain is "naturally" available. Indeed, if we consider the optimal chain consisting of the first three chunks (length 3), once block i1 is set, it consists only of the second and third chunks, which is the chain of interest. Thus, keeping only the optimal chains is indeed sufficient, since these chains are decomposed into smaller chains that are optimal in the current context, and therefore there is no need to keep the other chains that were discarded.

[0091] It should be noted that the hyperparameter lambda controls the trade-off between rate and distortion. Without the hyperparameter, an exemplary embodiment would be unable to determine which of the chains represents a better trade-off than the other chains and thus which chain is truly optimal compared to the other chains. For example, the chain of the first two chunks has a lower distortion than the chain of the first three chunks. However, the former has a worse rate than the latter (because the rate increases with the length of the chain). As will be seen below, the use of the hyperparameter and the introduction of the concept of cost makes it possible to answer the question of which chain represents the best trade-off.

[0092] Stage 4: Match Resolution This stage helps to efficiently apply the chains that are kept during the previous stage. As there can be multiple (possibly optimal) chains, all of which reference different (and potentially common) blocks, a decision is made as to the order in which they should be applied. The hyperparameter λ (lambda) helps in this regard. It controls the trade-off between rate and distortion. One can think of this as a kind of "market price" between exchanging one unit of rate for one unit of distortion. The concept of cost is also introduced. The cost (C) is defined as: C=D-λR where D represents the distortion and R represents the rate. The distortion is calculated as the sum of all the distortions that make up the chain (of length N).

[0093]

number

[0094] Note that C=R-λD is another formula that can be used to define the cost, however in the following paragraphs in the remainder of this section we will assume that the cost function is equal to D-λR.

[0095] As discussed above, the rate R corresponds to the storage gain realized by applying the chain. This can be difficult to compute since it depends on the behavior of the lossless compressor. Thus, an exemplary embodiment provides an estimator of how many bytes the lossless compressor allows to "save", taking into account the matches introduced into the texture bitstream as a result of applying the chain. The Rate and Distortion section below provides further details in this regard.

[0096] While the formula for C given above correctly models the trade-off between rate and distortion, improvements are possible. That is, the encoding configuration of the blocks is set once for all, each time a chain is applied. This means that these blocks cannot be configured anymore. However, it may be desirable to avoid favoring longer chains over smaller chains. Otherwise, an exemplary embodiment may "wear out" blocks very quickly. Thus, an exemplary embodiment may improve the cost per block (C block ) may be considered.

[0097]

number

[0098] In one example embodiment, the procedure works as follows: First, each block is marked as “unset”. Each time, all chains are processed and their cost C blockis calculated. Positive costs imply an invalid rate vs. distortion trade-off, so these chains are skipped. The chain with the smallest (negative) cost is taken and applied. As discussed above, applying a chain involves updating the texture and setting the encoding configuration that the chain references for each relevant block. Each of these referenced blocks is then marked as "set". From each block, it is possible to reference all the chains to which it belongs. Thus, once a block is "set", each chain that references it is updated by "removing" that block from it. Removing a block from a chain involves decreasing its length by one while also subtracting the distortion of this block.

[0099] This procedure is repeated until one of the following conditions is met: (1) all blocks are marked as "set"; (2) all chains have been applied; or (3) the remaining chains are either inapplicable (length strictly less than 2) or have a positive cost. It is acknowledged that a chain with a positive cost does not necessarily mean that it will never remain in that state forever. As discussed above, it is possible that a chain can be decomposed into smaller chains, and one of these smaller chains may have a negative cost (and thus represent an interesting rate vs. distortion tradeoff) and then be selected with the above procedure.

[0100] Figure 9 illustrates one iteration of the match chain resolution procedure of an example embodiment. Figure 9 begins with choosing a chain to apply in the left-most portion of the diagram. It will be appreciated that the numerical values ​​in the example of Figure 9 are given by way of example and not by way of limitation.

[0101] To maximize cache efficiency, the chain is stored as a very lightweight object (taking up very little space in memory) and contains three pieces of information: length, match size, and sum of skewness (which is the sum of the skewness of all blocks the chain references). Thus, from this "lightweight chain" alone, an exemplary embodiment cannot identify the blocks it references. In an exemplary embodiment, such information is kept in a separate buffer. Indexing allows the data to be referenced. For example, in an exemplary embodiment, the following indexing procedure may be implemented:

[0102] Given the index position of a chain in the chain buffer, one can access the list of paired information it references (e.g., in the form (block index, encoding configuration index)).

[0103] Given an encoding configuration index, its distortion value can be accessed.

[0104] Given a block index, you can access the block status ('set' or 'not set').

[0105] Given a block index, we can access the list of chains it participates in.

[0106] These four links enable the procedure of FIG. 9 to be implemented in one exemplary embodiment.

[0107] Finally, if the procedure ends without all blocks being in the "set" state, an example embodiment applies an encoding configuration with zero distortion to each of these remaining blocks. As detailed in the next section, such an encoding configuration always exists and corresponds to the best quality configuration that exists for a block. Note that doing so can also be seen as applying a fictitious chain consisting of only one block with zero distortion but also zero rate (because for the rate to be non-zero, a chain must reference at least two blocks). Such a fictitious chain would therefore have a cost of zero, which is the highest cost that the RDO can consider.

[0108] Exemplary Rate and Distortion Considerations Rate and distortion are discussed above. To measure quality, a metric is used. Possible metrics that can be used in connection with an exemplary embodiment include, for example, the sum of squared differences (SSD) in a given color space (RGB, YUV, CieLab, OkLab, etc.), and some variant of SSD aimed at reducing block noise artifacts, which may include (1) weighted SSD that gives more weight to pixels on the boundaries of blocks, or (2) SSD normalized by pixel variance (noisy blocks can often be corrupted more aggressively than other blocks, because this is less noticeable), and further metrics include angular SSD of normal maps, structural similarity (SSIM), etc.

[0109] An example embodiment measures quality at the level of texture blocks, thereby obtaining a value that measures the quality of a particular texture block. In an example embodiment, the quality metric includes more than the pixels for a given block. For example, it may also include the pixels surrounding the block (e.g., using the spatial context of the block). The quality metric is additive in the sense that a quality measure for a group of texture blocks is equal to the sum of the quality measures for each block in the group, or can at least be approximated in this way.

[0110] Since distortion is a measure of quality loss, it actually measures the quality of a texture block against a reference. The reference in one exemplary embodiment is, for a given block, the best possible quality that can be obtained for that block among all available encoding configurations. In other words, Q is the quality metric used, and q i,j is the encoding configuration j (or e i,j ), the distortion is d i,j =q i,j -Min j (q i,j ) This formula implies that for a given block, the distortion is a positive value and the best encoding configuration has zero distortion, which is desirable.

[0111] Rate is how much storage size is expected to be saved, and may be more difficult to obtain because (unlike distortion) it cannot be computed directly and is therefore estimated in some cases. In fact, it may be difficult to predict how a lossless compressor will react to changes in the texture bitstream, since this often depends on many things inside the compressor itself, and so it may not be known how the match chain will actually save. Exemplary techniques for estimating storage gains include, for example, closed-form and model-based approaches, as discussed in more detail below.

[0112] Exemplary Closed-Form Approach to Rate Estimation One exemplary embodiment attempts to model the storage gain with a function, which has the advantage that it is very fast to evaluate. One function that can be used is R(Length,MatchSize)=Max(0,(Length-1)*(MatchSize-MIN_MATCH_SIZE+1) It is.

[0113] As discussed above, length and match size are attributes of a match chain. If there are 6 overlapping chunks of 5 bytes in a texture, then there is a match chain with length 6 and (match) size 5. Thus, assuming MIN_MATCH_SIZE is 4, the rate of this match chain given by the above formula is R=10.

[0114] This function also has the predictable property of rate. For MatchSize strictly smaller than MIN_MATCH_SIZE or length < 2, R=0 · R is an increasing function of length, and · R is an increasing function of MatchSize Includes.

[0115] Exemplary Model-Based Approach to Rate Estimation Instead of using an explicit formula to compute the rate, an exemplary embodiment can attempt to estimate the rate from a simplified model of the lossless compressor. There are many ways to approximate the performance of a compressor. One approach that an exemplary embodiment can use is to try to estimate the rate of each possible match chain that the algorithm can consider independently. For example, assume there is a desire to have an estimate of how much a 4-byte match chain of length 3 (i.e., a 4-byte chunk repeated on itself 3 times) will save. It is known that matches are made within slices of a given size (see the Texture Slices section above). It is possible to "simulate" how much this particular match chain will save by running the following experiment many times:

[0116] 1. Generate a random buffer of data the size of the texture slice size you are considering (e.g. a slice of 4,096 ASTC blocks means a 65,536 byte buffer).

[0117] 2. Put the slice into a given "compressibility state": the goal is to modify random data in the buffer to create matches that will bring the buffer to a certain compression level (when compressed with a lossless compressor). That is, matches are randomly created in the buffer until the required compression level (e.g. 60% of the uncompressed buffer size) is reached.

[0118] 3. (In our example, this means a 4-byte match of length 3) Do another match in the buffer using the target match chain, making sure not to overwrite any existing matches made in the previous step.

[0119] 4. Compress the buffer using a lossless compressor. Calculate the difference between the resulting compressed size and the uncompressed size from 2. This gives an estimate of how much storage size the match chain has saved.

[0120] 5. Repeat the "experiment" many times (say 1 to 4), randomizing in each iteration the initial buffer contents (1), the compression level and the type of match inserted to reach that level (2), as well as the location in the buffer of the match chain from which the rate must be estimated (3). The rate estimate is then given by the average of all estimates from 4.

[0121] The above procedure is performed for all types of chains that can possibly be performed by the algorithm. For example, if MIN_MATCH_SIZE is set to 4 and MAX_MATCH_SIZE is set to 8, then with a texture slice of 40,96 blocks, there are (8-5+1)*(4,096-1)=20,475 types of chains for which rates should potentially be estimated. This procedure can be performed "offline" (e.g., outside of the algorithm execution) and the results of all rate estimates can be stored (e.g., in an array) that is accessible during the algorithm execution.

[0122] Exemplary Techniques for Controlling Memory Usage Exemplary Texture Slicing Technique As discussed above, stage 2 of the algorithm of an example embodiment extracts chunks from the encoding configuration. For large texture sizes, this may result in excessive memory consumption and therefore impractical. For example, employing a 1,024×1,024 texture with ASTC format and 1,024 encoding configurations per block to compute stage 2 combination results in nearly 1 billion extracted chunks.

[0123] To help address this issue, an exemplary embodiment divides the texture into slices, e.g., containing a given number of blocks. Each slice is processed independently by an algorithm of an exemplary embodiment. Figure 10 shows a schematic of a texture divided into four slices according to the texture slicing approach of an exemplary embodiment.

[0124] In an exemplary embodiment, typical slice sizes range from 1,024 to 4,096 blocks. Choosing the right slice size correlates with choosing the right number of encoding configurations per block, since both strongly affect memory usage and processing time, which are desirable to keep at controlled levels. One may want to favor larger slice sizes and fewer encoding configurations, or conversely, more but smaller slices. Such parameters can be empirically determined, depending not only on constraints (e.g., memory and processing time budgets) but also on the behavior of the algorithm itself that uses these parameters (including compression level and texture quality to be achieved).

[0125] Breaking a Match Chain in an Exemplary Embodiment In some cases, there may be early chain rejection, for example when it is known in advance that some chains can be discarded (and therefore not stored) because stage 4 of an exemplary embodiment algorithm will not consider them. This means that such a chain is always suboptimal, regardless of what smaller chains it may decompose into. In other words, there is always a match chain that is better than this match chain. Stage 3 described one such situation: some chains are suboptimal because they were rate identical to another while having worse distortion, and the above discussion showed that there is no need to store these chains, since the optimal chains considered would naturally decompose into them if they were to be optimal.

[0126] Likewise, a chain can have the same distortion as another while having a poorer rate.

[0127] FIG. 11 helps illustrate when a match chain may be discarded in the context of an exemplary embodiment. FIG. 11 represents a chain that is assumed to be optimal of length 3. An exemplary distortion of each referenced block is given for illustrative purposes. Moving to a higher match size (5 bytes), all of the chunks that make up the chain also collide at 5 bytes. Thus, even if a 4-byte chain is identified as a potentially optimal chain in stage 3 of an exemplary embodiment's algorithm, it does not need to be stored because it is never considered in stage 4 of an exemplary embodiment's algorithm. In fact, no matter what smaller chains it decomposes into, it will always be a suboptimal chain because a higher sized chain will also decompose into the same smaller chain, with the same distortion but with a larger match size (here 5 bytes) and therefore a better rate. That is, in FIG. 11, the second match chain with match size 5 and length of size 3 "wins" the first match chain with match size 4 and length of size 3.

[0128] More details on breaking match chains There are further optimizations that may allow discarding multiple chains. For example, after stage 1 of an exemplary embodiment algorithm, there are multiple encoding configurations for each block. It is recognized that some distortion values ​​are such that it can already be determined that any chains that reference these encoding configurations will not be considered by stage 4 of an exemplary embodiment algorithm.

[0129] To see this, consider a chain of length N. Its cost per block is

[0130]

number

[0131] In the formula, d iis the distortion of the i-th block referenced in the chain, and R N is the rate. The subscript N is used here to indicate that the rate depends, among other things, on the chain length. The distortion increases with increasing value d1≦d2≦...≦d N Assume that the data is sorted by

[0132] Now, the new cost value C block,N+1 Distortion d N+1 Consider adding another block to this chain using . Considering the numerator of this cost,

[0133]

number

[0134] It is. R N+1 -R N is the increment rate, which corresponds to the additional storage gain by extending a chain of length N to a chain of length N+1. This value ΔR max Suppose we know the dominant series for . Then,

[0135]

number

[0136] It is. Therefore, d N+1 >λΔR max If so,

[0137]

number

[0138] and two cases can occur. First,

[0139]

number

[0140] This means that a chain of length N has a positive cost and is therefore uninteresting to apply. But then, by following the inequality, a chain of length N+1 also has a positive cost and is therefore also uninteresting. Secondly,

[0141]

number

[0142] A chain of length N is applicable. For a chain of length N+1 to be applicable,

[0143]

number

[0144] Dividing the above inequality by N gives us

[0145]

number

[0146] However,

[0147]

number

[0148] Therefore,

[0149]

number

[0150] In other words, 0>C block,N+1 >C block,N It is. In the first case, a chain of length N+1 has a positive cost, and in the second case, a chain of length N wins because it is more costly, and so there are suboptimal chains. i >λΔR max If 0>C, then any chain that references such an encoding configuration will be a suboptimal chain. However, being suboptimal is not a sufficient criterion for rejection in an exemplary embodiment. An exemplary embodiment ensures that such a chain can never be selected in stage 4 of the algorithm so that it can be discarded. In other words, an exemplary embodiment makes sure that it always remains a suboptimal chain. Two conditions 0>C block,N+1 >C block,N and d N+1 >λΔR max Therefore, the chain of length N+1 (or whatever remains of it) will never be optimal and will therefore never be applied. The reason is that as long as it remains, there are at least two blocks in the chain of length N, 0>C block,N+1 >C block,N always holds, so a length N chain would be preferable compared to an N+1 chain. If only one block remains in the length N chain (and therefore the length N chain becomes inapplicable), then we know that the length N+1 chain (now a length 2 chain) has a positive cost. For example, suppose only block 1 remains in the length N chain. Then the cost of the length N+1 chain is

[0151]

number

[0152] (Because λR2=λ(R2-R1)≦λΔR max , because R1 is equal to 0).

[0153] Therefore, chains of length N+1 are not "interesting" and can be discarded. Exemplary Lambda Factorization Related Techniques As discussed above, the algorithm of an exemplary embodiment can be factorized. Furthermore, the use of hyperparameters was mentioned in stage 4. In this way, it is possible to output different versions of the encoded texture, for example corresponding to different tradeoffs between rate and distortion, by performing stages 1 to 3 only once. In such a case, only stage 4 would be executed several times. FIG. 12 shows an example of an algorithm factorization according to an exemplary embodiment. In 10 steps, 11 hyperparameter values ​​from 0 to 100 are fed into the algorithm. As can be seen, stages 1 to 3 are executed only once per slice in the same manner as described above. Only stage 4 is executed as many times as there is an input to the process. The input is a particular slice with a specific lambda value that gives a count of 11*4=44 executions of stage 4. Such a structure allows for efficient multithreading in an exemplary embodiment, since each of these executions can be fully parallelized.

[0154] As discussed in the previous subsection, an optimization that can be applied after stage 1 of the algorithm of one exemplary embodiment to avoid inserting a large number of chains is to reduce the number of chains by a certain threshold λΔR max This threshold depends on the value of lambda, which may contradict the notion that stages 1 to 3 of the algorithm are lambda independent. However, by modifying the threshold to make it lambda independent, e.g., by adding a new threshold λ max ΔR max This can be addressed by adopting it as

[0155] Example Development and Runtime Pipeline From a developer's perspective (e.g., from the perspective of one who may be designing or implementing a portion of a virtual environment, for example for use in a game world), the algorithmic techniques disclosed herein may be provided in association with a development tool. The tool may be a software program executing on a computing system including at least one hardware processor operably coupled to a memory. The tool may include a user interface through which a user interacts with the software program. A data store local to or remote from the computing system that accesses the software program stores a library of textures to be compressed.

[0156] Using the tool, a developer selects one or more textures to compress and identifies one or more target lossy formats, e.g., from a list of possible lossy formats such as those identified above (e.g., BC1, BC3, BC4, BC5, BC7, and ASTC). The developer decides whether to apply RDO, and if so, one or more RDO lambda values ​​are specified.

[0157] An encoding is generated for the selected texture or textures using the specified parameters. It is recognized that multiple encodings can be efficiently performed, for example as discussed above. Once the encodings are complete, the developer can compare quality and size. Quality may be specified using a programmed or selected metric (e.g., PSNR, SSD, SSIM, etc.). Size may be expressed as absolute size, a percentage of original size, etc.

[0158] The tool may display the original texture adjacent to the encoded texture in a user interface, for example for visual comparison. Hovering a cursor, such as a mouse pointer, over the encoded texture may provide summary details, such as, for example, the encoder used, whether RDO was applied, quality, size, and a ranking of quality and / or size if multiple encodings were identified. Other details of the encoding configuration chosen by the texture encoder for each block (e.g., partition index, weight values, color end points, etc.) may also be displayed. Providing this type of information is useful for comparing, reviewing, and analyzing encoding configurations.

[0159] A developer can select a particular encoding for decompression, or a particular encoding can be selected automatically (e.g., based on a set of best reciprocity or other programmed rules). The compressed texture may then be packaged and decompressed.

[0160] Runtime refers to decoding. The decoder applies lossless reconstruction (if applicable). Then, in one exemplary embodiment, the lossy encoded textures are given to a GPU. The GPU then handles lossy decoding and rendering, for example, in presenting a virtual environment.

[0161] Overview of Encoder and Decoder Related Techniques Numerous features and aspects of the disclosed technology are set forth above. The following description provides an overview of certain encoder and decoder related techniques. The following merely highlights some features of certain exemplary embodiments and therefore should not be construed as limiting all embodiments, and the scope of the claimed invention is expressly set forth in the claims.

[0162] Turning first to the encoder, FIG. 13 is a flowchart illustrating a process for generating an encoded texture according to an example embodiment, and FIG. 14 is a block diagram illustrating a computing architecture that may be used in connection with the example flowchart of FIG. 13, according to an example embodiment.

[0163] FIG. 13 illustrates the conceptual three-stage decomposition discussed in detail above, including, for example, stage 1 (multi-composition texture encoding), stage 2 (chunk extraction), and stage 3 (match chain creation). FIG. 13 begins at step 1302, where textures are retrieved from a raw texture repository 1402 via an encoding computing system 1404. The raw texture repository 1402 may be local to the encoding computing system 1404 or remote therefrom in different exemplary embodiments. In the latter case, the raw textures (textures to be encoded) may be retrieved via API calls or the like over a suitable network connection such as the Internet. The computing system 1404 may include one or more computers in a standalone, client-server network, distributed computing (e.g., cloud computing), or other technology environment. In certain exemplary embodiments, networked and distributed computing environments may be advantageous because the encoding operations may be parallelized, as discussed above. The encoding computing system 1404 includes at least one processor 1406 operatively coupled to a memory 1408 and hardware user interface elements 1410. The memory 1408 includes a working area (e.g., working RAM 1412) and an encoding application 1414. A user uses the hardware user interface elements 1410 (e.g., a keyboard, mouse, touch screen, etc.) to interface with and collaborate with the encoding application 1414, for example, to indicate which textures to compress, indicate the lossy and / or lossless encoding to use (e.g., based on what is available from a lossy library 1416 and a lossless library 1418), identify hyper-parameters, monitor the output of the encoding operation, etc.

[0164] Once the raw texture to be encoded is read, the encoding application 1414 divides the texture into a number of blocks (e.g., of a predetermined pixel size) at step 1304. At step 1306, each of these blocks is encoded into a number of block bitstreams. In doing so, each of these block bitstreams is encoded according to an encoding configuration (based in part on lossy encoding), and each block bitstream / encoding configuration will have an associated distortion value. At step 1308, the block bitstream with the lowest associated distortion value is selected for each respective block, such that up to a predetermined number of distinct block bitstreams are selected. Note that the blocks, buffers, and / or other data structures used in the operations depicted in FIG. 14 may be stored in memory 1408 (e.g., in working RAM 1412) in an example embodiment.

[0165] In an example embodiment, as many distinct block bitstreams as possible (e.g., up to a predetermined number specified by a user) may be selected, and an indicator may be provided for each block for which the number of selected block bitstreams is less than the predetermined number. Additionally or alternatively, an example embodiment may allow input to be received to identify a maximum distortion value for one or more identified blocks in the plurality of blocks. In such a case, for each of these identified blocks, only block bitstreams having a distortion value better than the maximum distortion value may be selected for the respective identified block.

[0166] At step 1310, data chunks are obtained. This involves extracting at least some of the data chunks. In an example embodiment, only some of the data chunks are extracted directly from the blocks. In such cases, other chunks may be generated from chunks that are extracted directly (or at least more directly) from the blocks. Thus, in an example embodiment, obtaining the data chunks may comprise extracting a subset of valid data chunks for a given selected block bitstream and generating remaining data chunks for the given selected block bitstream from the extracted subset (e.g., via masking).

[0167] In any case, for each block bitstream selected for low distortion, a number of data chunks are obtained such that each data chunk is a contiguous section of the respective selected block bitstream from which it was obtained. The data chunks of each selected block bitstream start at a respective position (e.g., a respective byte) of the respective selected block bitstream and extend to the end of the respective selected block bitstream such that each data chunk has a minimum data chunk size. In an exemplary embodiment, data chunks of different sizes can be obtained for the selected bitstreams. In an exemplary embodiment, the minimum data chunk size may be at least as large as the minimum size available for the lossless compressor ultimately used.

[0168] At step 1312, collisions between the obtained data chunks are detected (e.g., using an efficient sorting technique and by eliminating non-extended, non-colliding, and / or duplicate colliding data chunks from the higher distortion block bitstream). At step 1314, based on the detected collisions, match chains with redundant data chunks are formed such that for each match chain, the redundant data chunks therein represent a common truncated value of the associated block bitstream.

[0169] During the match chain creation process, many of the match chain candidates (potential match chains) are filtered out, discarded, or otherwise not considered, since it is known in advance that they cannot be selected in the subsequent iterative selection procedure, since it has been detected that for each of these discarded match chains, there is always at least another equal or better match chain. In other words, a given match chain is not considered in response to a determination that another match chain is formed or will be formed that is equal to or better than the given match chain. "Better" in this context should be understood as having a better cost, obtained via a cost function related to the hyperparameters.

[0170] The remaining formed match chains with the best value are iteratively selected. In an exemplary embodiment, the "best value" may be determined in relation to a function related to the hyperparameter, e.g., a hyperparameter that sacrifices rate and distortion to obtain one of the other. That is, in an exemplary embodiment, the hyperparameter is part of a defined function that determines which match chain is best at the expense of rate and distortion to obtain the other.

[0171] At step 1316, a lossy encoding of the texture is obtained using the iteratively selected match chains, and at step 1318, the encoded lossy texture is further encoded using a lossless compressor. The "final" encoded textures are sent to an encoded texture repository 1420, which in different exemplary embodiments may again be co-located with or remote from the encoding computing system 1404. These textures may be used, for example, in defining a virtual environment such as a video game.

[0172] 13 may be performed using the encoding application 1414, for example, based on user input obtained via the hardware user interface element 1410. The encoding application 1414 may be implemented as a series of program modules or other structures as instructions executable by the processor 1406.

[0173] FIG. 15 is a flow chart illustrating a process for decoding textures encoded via the techniques described herein, according to an exemplary embodiment. As shown in FIG. 15, at step 1502, the encoded textures are retrieved. The encoded textures may be retrieved as part of a virtual environment (or a portion of a virtual environment) being rendered on a computing device such as that shown in and described in connection with FIG. 16, as part of a virtual tour (e.g., of a home, a museum, etc.), and / or in other scenarios. The textures may be included in the application, retrieved from a local or remote data store, etc. Once the textures are retrieved, at step 1504, they are provided to a lossless decoder, the output of which is provided to a lossy texture decoder 1506. These decoders may be implemented in software and / or in conjunction with features provided by a processor, such as a CPU, a GPU, etc. In an exemplary embodiment, the retrieved textures may simply be passed to a GPU for decoding and rendering. In any case, at step 1508, a virtual environment is generated based on the decoded textures. The output may be provided to a display device (eg, an LCD, LED, OLED, or other display device) operably connected to the computing device.

[0174] 16 is a block diagram illustrating a computing device 1602 that may be used in connection with the example flowchart of FIG. 15, according to an example embodiment. The computing device 1602 may be a gaming device, such as a handheld gaming device, a stationary gaming device, or the like. Alternatively, the computing device 1602 may be a television, a personal computer, a laptop, a mobile electronic device, or other types of electronic devices. As will be appreciated by those skilled in the art, some or all of the blocks illustrated in FIG. 16 may be provided or omitted in different implementations depending, for example, on the type of electronic device provided.

[0175] 16, computing device 1602 includes one or more processors 1604 operably connected to memory 1606. The one or more processors 1604 may include a CPU, GPU, and / or other processors located within a common device, across several devices (e.g., in a networked environment), etc. Memory may include media such as, for example, SSD, HDD, flash memory, RAM, DRAM, etc.

[0176] The device 1602 can be turned on using a power button 1608, which is connected to a power control circuit 1610. The power control circuit 1610 is in turn connected to a battery 1612 (e.g., a rechargeable battery). The device 1602 can be powered over pin 1614, which can additionally be used to charge the battery 1612 when it is rechargeable.

[0177] A user can interact with the computing device 1602 through a controller coupled thereto. The controller may be integral with the device in an exemplary embodiment. In an exemplary embodiment, a separate controller may be connectable to the computing device through controller terminals 1616 and / or a wireless controller interface 1618 (e.g., operating over a wireless protocol such as Bluetooth). In an exemplary embodiment, the controller may include game system type controls such as buttons, cross switches, joysticks, slide pads, and the like. In an exemplary embodiment, the controller may include a keyboard, mouse, trackball, and the like. In an exemplary embodiment, the user may interact with the device 1602 through a touch panel 1620 that interfaces with the device 1602 in association with a touch panel controller 1622. Video output is provided to a display 1624 that is integral with or otherwise connectable to the device 1602. Audio is provided through an integrated speaker 1624 and / or through an audio jack or other terminal 1626. The audio jack or other terminal 1626 connects to the processor 1604 through a codec circuit 1628.

[0178] The slot 1630 receives physical media, such as, for example, an SD card, a micro SD card, a CD-ROM, a DVD-ROM, etc. The slot interface 1632 provides a connection between the slot 1630 and the processor 1604. In an exemplary embodiment, the media inserted into the slot 1630 may include a program that includes or otherwise accesses a compressed texture to be displayed. In an exemplary embodiment, the program that includes or otherwise accesses a compressed texture to be displayed may be read via the network interface 1634, for example, in connection with downloaded games, cloud gaming scenarios, etc.

[0179] Although one exemplary gaming system is shown, it will be appreciated that the techniques disclosed herein may be used in connection with other gaming systems, and other electronic devices that are not in fact dedicated gaming systems (e.g., PCs, TVs, mobile devices such as smartphones, etc.).

[0180] While the present invention has been described in connection with what are presently considered to be the most practical and preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. 1. A method of encoding a texture, comprising the steps of: encoding each of a plurality of blocks obtained by dividing the texture into a plurality of block bitstreams, each of the block bitstreams corresponding to a respective encoding configuration and having a distortion value associated with each of the block bitstreams; and selecting for each block the block bitstream having the smallest associated distortion value, such that for each block up to a predetermined number of distinct block bitstreams are selected; obtaining a plurality of data chunks for each selected block bitstream, each data chunk being a contiguous section of the respective selected block bitstream from which it was obtained, wherein data chunks of different sizes can be obtained for the selected bitstreams; and detecting collisions between the retrieved data chunks; forming, for each match chain, a match chain having redundant data chunks therein based on the detected collisions, such that the redundant data chunks therein represent common values ​​of associated block bitstreams; selecting a best match chain from among the formed match chains, the best match chain being determined using hyper-parameters; and obtaining a lossy encoding of the texture from the selected best match chain.

2. The method of claim 1 , further comprising encoding the lossy encoded texture with a lossless compressor.

3. The method of claim 1 or claim 2, wherein the data chunks start at different byte locations.

4. for each block, as many distinct block bitstreams as possible are selected, up to said predetermined number; The method of claim 1 or claim 2, further comprising providing an indicator for each block for which a number of selected block bitstreams is less than the predetermined number.

5. receiving an input specifying a maximum distortion value for one or more identified blocks of the plurality of blocks; 3. The method of claim 1 or claim 2, wherein for each of the one or more identified blocks, only block bitstreams having a distortion value better than the maximum distortion value are selectable for the respective identified block.

6. Obtaining the data chunks includes: Extracting a subset of valid data chunks for a given selected block bitstream; and generating residual data chunks for the given selected block bitstream from the extracted subset.

7. 1. A method of providing a virtual environment in connection with a computing system, comprising: reading an encoded texture from a storage medium, said texture being encoded using the method of claim 1 or claim 2; and providing the encoded texture to at least one processor of the computing system for decoding the texture and use of the decoded texture in the virtual environment.

8. A storage medium storing instructions which, when executed by a processor, cause a computer to perform operations corresponding to the method of claim 1 or claim 2.

9. A storage medium storing instructions that, when executed by a processor, cause a computer to perform operations corresponding to the method recited in claim 7.

10. a computing system through which a virtual environment can be displayed, the computing system comprising: a memory coupled to one or more processors configured to perform operations, the operations including: reading an encoded texture from a storage medium, said texture being encoded using the method of claim 1 or claim 2; and At least one of the one or more processors decodes the texture.

11. A program for encoding textures, comprising: The program includes: encoding each of a plurality of blocks obtained by dividing the texture into a plurality of block bitstreams; each of the block bitstreams corresponds to a respective encoding configuration and has a distortion value associated with each of the block bitstreams; and The program causes the at least one processor to: selecting for each block the block bitstream having the smallest associated distortion value, such that for each block up to a predetermined number of distinct block bitstreams are selected; obtaining, for each selected block bitstream, a plurality of data chunks, such that each data chunk is a contiguous section of the respective selected block bitstream from which it is obtained; Data chunks of different sizes are obtainable for the selected bitstream; and The program causes the at least one processor to: detecting collisions between the retrieved data chunks; forming, for each match chain, a match chain having redundant data chunks therein based on the detected collisions, such that the redundant data chunks therein represent common values ​​of associated block bitstreams; selecting a best match chain from among the match chains formed; The best match chain is determined using hyperparameters; and The program causes the at least one processor to: obtaining a lossy encoding of the texture from the selected best match chain.

12. 1. A system for encoding and decoding texture, comprising: The system comprises: a first device for encoding the texture; a second device for decoding the encoded texture; The first device is encoding each of the blocks obtained by dividing the texture into a plurality of block bitstreams; each of the block bitstreams corresponds to a respective encoding configuration and has a distortion value associated with each of the block bitstreams; and The first device selecting for each block the block bitstream having the smallest associated distortion value, such that for each block up to a predetermined number of distinct block bitstreams are selected; obtaining, for each selected block bitstream, a plurality of data chunks, such that each data chunk is a contiguous section of the respective selected block bitstream from which it is obtained; Data chunks of different sizes are obtainable for the selected bitstream; and The first device Detecting collisions between the retrieved data chunks; forming, for each match chain, a match chain having redundant data chunks therein based on the detected collisions, such that the redundant data chunks therein represent common values ​​of associated block bitstreams; Selecting a best match chain from among the formed match chains; The best match chain is determined using hyperparameters; and The first device obtaining a lossy encoding of the texture from the selected best match chain; The second device is receiving an encoded texture from the first device; The system decodes the encoded texture.

Images