Sequential data compression using artificial neural networks.

Abstract

Some aspects of the present disclosure provide techniques for compressing content using neural networks. An exemplary method generally includes receiving content for compression. The content is encoded into a first latent code space via an encoder implemented by an artificial neural network trained to generate a latent space representation of the content. A first compressed version of the encoded content is generated using a first quantization bin size of a series of quantization bin sizes. A refined compressed version of the encoded content is generated by scaling the first compressed version of the encoded content to one or more second quantization bin sizes smaller than the first quantization bin size, conditional at least on values ​​of the first compressed version of the encoded content. The refined compressed version of the encoded content is output for transmission.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Application No. 17 / 648,808, filed January 24, 2022, which claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 141,322, entitled "Progressive Data Compression Using Artificial Neural Networks," filed January 25, 2021, which is assigned to the assignee of this application and the entire contents of each of which are incorporated herein by reference.

[0002] Aspects of the present disclosure relate to machine learning, and more particularly, to using artificial neural networks to compress data such as video content. [Background technology]

[0003] Data compression techniques may be used to reduce the size of content for a variety of reasons, including improving storage and transmission efficiency and conforming to the intended use (e.g., data resolution appropriate to the size of a device's display). Data compression may be performed using lossy techniques such that a decompressed version of the data is an approximation of the original data that was compressed, or by using lossless techniques that result in a decompressed version of the data corresponding to the original data.

[0004] In general, lossless compression may be used when no data should be lost in the compression, such as in compressing a file archive. In contrast, lossy compression may be used when exact reproduction of the original data is not necessary (e.g., in compressing still images, video, or audio, where some data loss can be tolerated, such as loss in fineness of color data, or loss in audio frequencies in the extreme audible spectrum).

[0005] Data compression schemes are often defined or based on fixed-rate compression (e.g., a single bit rate), which makes them inflexible to varying data types and compression needs. That is, for any given compression scheme, data is generally compressed at a particular bit rate, regardless of whether the data is suitable for a higher or lower compression bit rate. For example, for images that do not contain fine details, a fixed bit rate compression scheme may compress these images using more bits than are necessary to represent the information in the image, even if the image is not suitable for compression using a lossier compression scheme (and correspondingly, a lower bit rate). Similarly, a more detailed image may be compressed at a bit rate that is too low to be satisfactorily reproduced. Thus, traditional schemes often involve trade-offs in the design of fixed compression schemes, which are neither dynamic nor adaptive. Summary of the Invention [Problem to be solved by the invention]

[0006] Accordingly, what is needed are improved techniques for adaptively compressing content. [Means for solving the problem]

[0007] Some aspects provide a method for compressing content using a neural network. An exemplary method generally includes receiving content for compression. The content is encoded into a first latent code space via an encoder implemented by an artificial neural network. A first compressed version of the encoded content is generated using a first quantization bin size of a series of quantization bin sizes. A refined compressed version of the encoded content is generated by scaling the first compressed version of the encoded content to one or more second quantization bin sizes smaller than the first quantization bin size, conditional at least on values ​​of the first compressed version of the encoded content. The refined compressed version of the encoded content is output.

[0008] Some aspects provide a method for decompressing compressed content using a neural network. An exemplary method generally includes receiving encoded content for decompression. Approximations of values ​​in a latent code space are recovered from the received encoded content by recovering codes from a series of quantization bin sizes, the series of quantization bin sizes including a first quantization bin size and one or more second quantization bin sizes smaller than the first quantization bin size. A decompressed version of the encoded content is generated by decoding the approximations of values ​​in the latent code space via a decoder implemented by an artificial neural network. A decompressed version of the encoded content is output.

[0009] Other aspects provide a processing system configured to perform the methods described above and further described herein, a non-transitory computer readable medium comprising instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the methods described above and further described herein, a computer program product embodied on a computer readable storage medium comprising code for performing the methods described above and further described herein, and a processing system comprising means for performing the methods described above and further described herein.

[0010] The following description and the related drawings set forth in detail certain illustrative features of the one or more aspects.

[0011] The accompanying drawings illustrate some aspects of the one or more aspects and therefore should not be considered limiting of the scope of the disclosure. [Brief description of the drawings]

[0012] [Figure 1] 1 illustrates an exemplary neural network-based data compression pipeline. [Diagram 2] FIG. 1 illustrates an exemplary pipeline for compressing and decompressing content using an encoder and decoder implemented as an artificial neural network and continuous scaling of the compression bitrate. [Diagram 3] FIG. 13 illustrates an example of latent scaling of quantization widths in continuous control of compression bitrate, according to aspects of the present disclosure. [Figure 4A] FIG. 2 illustrates an example of quantization bin sizes used to achieve different compression bit rates, according to aspects of the present disclosure. [Figure 4B] A diagram illustrating an example of quantization bin size levels used to achieve different compressed bit rates, where the bins in the quantization bin size levels have unequal sizes, in accordance with aspects of the present disclosure. [Diagram 5]FIG. 1 illustrates an example of nested quantization based on quantized codes in a finer quantization bin size conditional on quantized codes in a coarser quantization bin size, in accordance with aspects of the present disclosure. [Figure 6] FIG. 2 illustrates example operations for compressing content received via a compression pipeline using sequential coding, according to aspects of the disclosure. [Figure 7] 5A-5C illustrate example operations for decompressing encoded content according to aspects of the disclosure. [Figure 8] FIG. 2 illustrates an example of channel-wise sequential coding using different bit rates for different channels, according to an aspect of the present disclosure. [Figure 9] FIG. 1 illustrates an effective quantization grid for sequential coding, in accordance with an aspect of the present disclosure. [Figure 10] FIG. 13 illustrates a quantization grid for sequential coding with alignment to the finest quantization grid, in accordance with aspects of the present disclosure. [Figure 11] 1A-1C are diagrams illustrating example results of data compression using sequential coding, according to aspects of the present disclosure. [Figure 12] 1A-1C are diagrams illustrating example results of data compression based on different orderings of coding units, according to aspects of the present disclosure. [Figure 13] 1A-1C are diagrams illustrating example results of data compression using sequential coding and different ordering of coding units, in accordance with aspects of this disclosure. [Figure 14] FIG. 1 illustrates an example neural network-based data compression pipeline in which side information is used to decompress data, according to aspects of the present disclosure. [Figure 15] FIG. 1 illustrates an example implementation of a processing system in which sequential coding and decoding of content may be performed in accordance with aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] For ease of understanding, wherever possible, identical reference numbers have been used to designate identical elements common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated into other embodiments without further recitation.

[0014] Aspects of the present disclosure provide techniques for incrementally compressing content using artificial neural networks, such that a single model can be used to encode content at varying bit rates or levels of quality.

[0015] Neural network-based data compression systems may be used to compress various types of data. For example, neural network-based data compression may be used to compress various types of content suitable for compression. The content may include, for example, video content, image content, audio content, sensor content, and other types of data suitable for compression. In general, neural network-based data compression may compress content using a bit rate determined a priori based on a trade-off between the size of the encoded content and distortion (the difference between the original content and the decompressed content). In many data compression systems, a higher bit rate (e.g., a larger number of bits used to represent the content being compressed) may be associated with lower distortion, while a smaller bit rate may be associated with higher distortion. The trade-off between distortion D and bit rate R is expressed by the formula

number

[0016] However, typical neural network-based data compression is not suitable for large-scale deployment for various reasons. For example, in many machine learning-based compression schemes, models may need to be trained to support various bit rates. That is, a first model may be trained for a low (baseline) bit rate, a second model may be trained for a second bit rate higher than the baseline bit rate, a third model may be trained for a third bit rate higher than the bit rate of the second model, and so on. In other machine learning-based compression schemes, the encoder and decoder networks may depend on a β parameter so that a single model can adapt to various rate-distortion tradeoffs. Other machine learning-based compression schemes may learn to adjust the quantization step size of the generated latencies. However, these models cannot effectively learn compression schemes that allow for variable coding of content. This variable coding may be achieved via a sequential coding scheme, or an encoding scheme that allows data to be compressed using multiple bit rates, so that some data may be encoded and decoded using lower bit rates and other data (e.g., portions of an image having more detail that should be encoded in a manner that results in a more faithful reconstruction of such detail when decoded) may be encoded and decoded dynamically using higher bit rates to accommodate the differences in the data being compressed.

[0017] Aspects of the present disclosure provide techniques that allow for sequential coding (and sequential compression) of content using a single model. In sequential coding of content, higher bit rate codes may be generated based on lower bit rate codes, whereby an encoded, compressed version of input data is sequentially compressed using multiple bit rates without the need to compress the input data using multiple models. The compressed data may then be restored using any of multiple bit rates according to various considerations, such as the processing power of the device that decompresses the data, the amount of detail required in the decompressed data, etc. Furthermore, the sequentially compressed data may be stored in a single file for multiple versions, each compressed at a different bit rate, which may improve storage and transmission efficiency for the compressed data.

[0018] Exemplary Neural Network-Based Data Compression Pipeline 1 illustrates an exemplary neural network-based data compression pipeline 100 in accordance with an embodiment of the present disclosure. As shown, the pipeline 100 receives content x111 for compression and an approximation of the content x111.

number

[0019] At the decoding side 120 of the pipeline 100, an entity decoder 122 recovers the quantized version of the content and dequantizes it using the inverse quantization scheme (Q -1 )124 is the approximate code

number

number

number

number

[0020] The training loss associated with neural network-based data compression is the sum of the amount of distortion (e.g., the approximation of the content x111 and the content x111)

number

[0021] In general, independent models may be trained to obtain various bitrate options for compressing the content. However, such independent models cannot sequentially encode the content since the models are trained separately and are not related. Thus, these independent models are non-sequential models where a single encoder-decoder is used and various parameters may be used to manipulate the rate-distortion tradeoff of the independent models. Furthermore, variable bitrate solutions may involve the generation, transmission, and / or storage of multiple copies of the encoded version of the input at various quality levels, which may increase the amount of data generated, transmitted, and stored in the data compression operation.

[0022] Aspects of the present disclosure provide for sequential compression of data using a single encoder-decoder model. In general, compression of data may be achieved using coding techniques that encode data into a more compact representation. These coding techniques, referred to herein as sequential coding, but also known as encoded coding or scalable coding, allow content to be encoded with multiple bit rates each embedded at a time. By encoding content with multiple bit rates embedded at a time, dynamic control of the compression bit rate is simplified, such that multiple encoded versions of the data do not need to be generated to support different compression bit rates (and therefore different levels of compression quality preservation).

[0023] For example, the bit rate of broadcast content may be dynamically adapted (e.g., in response to available throughput, latency, the complexity of the content, the amount of detail to be preserved in the compression of this content, etc.) Furthermore, incremental coding of content may enable reduced transmission and storage costs by providing a single version of the compressed content that can be decoded using a variety of bit rates, instead of multiple versions of the compressed content generated for each of multiple supported bit rates.

[0024] To enable sequential coding of content, such that the content is encoded with each of multiple bit rates embedded at one time, the latent space code y representing the content x may be encoded using a nested quantization model in which codes associated with finer quantization levels (and correspondingly higher bit rate compression) are embedded within codes associated with coarser quantization levels (and correspondingly lower bit rate compression). As described in further detail herein, nested quantization may allow codes associated with finer quantization levels to be conditional on codes associated with coarser quantization levels, such that data may be encoded into successively finer quantization levels, with corresponding successive increases in bit rate and quality of the decompressed data.

[0025] In a nested quantization model, starting with a high bitrate model, a series of quantization bin sizes may be learned. Each quantization bin size in the series may be associated with a particular parameter (e.g., a value of β). Starting with the coarsest quantization bin (i.e., the quantization bin associated with the lowest bitrate), the latent code space y representing content x may be successively coded into finer quantization bins. As described in more detail herein, the bits for a particular quantization bin may be expressed as the sum of the bits for the coarsest quantization bin and each of the successively finer quantization bins up to the particular quantization bin according to the quantized probability chain rule. In general, at each quantization level in a nested quantization model, a probability may be associated with each code in a population of possible codes, and the code with the highest probability may be selected as the code for which the data is compressed at that quantization level. Based on the chain rule, it can be seen that the code by which data is compressed at any given quantization level N may be expressed as a function of the code by which the data is compressed to quantization levels lower than N (i.e., to quantization levels associated with coarser quantization bins). For example, the bits for the finest quantization bin (i.e., the Nth of N quantization bins) can be expressed as

[0026]

number

[0027] In the above formula, P(y N ) is the probability mass under the distribution curve that the code associated with the compressed input data falls within the N-th quantization bin, and P(y N |y N-1 ) is the probability mass under the distribution curve that the code associated with the input data lies in the Nth quantization bin, given that the code associated with the input data is in the N-1th quantization bin. In other words, the number of bits for the finest quantization bin (i.e.,

number

number

number

number

[0028] Furthermore, as described in more detail below, sequential coding may be used in channel-based latent ordering, in which the quantization bin sizes may be sequentially refined across different channels in the data to be compressed.

[0029] For example, in video content represented by luminance (Y), blue difference (Pb), and red difference (Pr) channels, different bin sizes may be used for the Y, Pb, and Pr channels. In another example, for visual content represented by chrominance channels (e.g., red (R), green (G), and blue (B) color channels), different bin sizes may be used for the R, G, and B color channels. The ordering of these channels may be defined by classifying the channels based on the ratio of distortion difference to rate difference calculated for each channel, such that coarser quantization bins are used for channels for which higher compression does not result in significantly more distortion, and finer quantization bins are used for channels for which higher compression results in significantly more distortion. Thus, multi-channel content is encoded by ordering the channels and encoding them using different quantization bins according to the order in which the channels are ordered, such that the channels that have the greatest impact on the quality of the resulting decompressed data may be compressed using the highest quality compression, and the channels that have a smaller impact on the quality of the resulting decompressed data may be compressed using lower quality compression. This may reduce the size of the resulting compressed representation of the input data, which may reduce storage and transmission costs for the compressed data.

[0030] FIG. 2 shows further details of a pipeline for compressing and decompressing content (eg, as shown in FIG. 1) using an encoder and decoder implemented as an artificial neural network and continuous scaling of the compression bitrate.

[0031] In the pipeline 200, an encoder 202 encodes an input x into a latent space code y. To continuously control the latent bit rate, the latent space code y is processed through a hyper-encoder 204, which may be another network used to generate weights for the encoder neural network, to generate hyper-latents that represent the inputs used to control the weights in the encoder neural network. The hyper-latents may be used as information to capture spatial dependencies in the data, or may be used to round (quantize) y into a rounded (quantized) representation of y, denoted as [y] (a simplified version of quantization when no scaling factor is applied, such as when s=1; in such cases,

number

[0032] On the decoder side, the hyperdecoder 206, which may be a network used to generate weights for the decoder neural network, decodes the hyper-latency to determine the entropy model used to code the rounded (quantized) latent space code [y]. The entropy model may be, for example, a probabilistic model used to generate a probability distribution used on the encoder side of the pipeline 200 to encode y into the code [y]. Based on the entropy model, the decoder 208 derives a

number

number

[0033] In some cases, a scaling factor may be applied to the pipeline 200 to affect the bitrate and amount of compression applied when generating a compressed version of the content x. In this case, a scaling parameter s may be applied in the scaler 210 before quantization (rounding y to a quantized value) and in the rescaler 212 before decompression, such that the quantized and scaled version of the latent space code y representing the content x may be expressed as y / s. By scaling y by the scaling factor s, the quantization bin size used to round (quantize) y may be changed from a reference value (e.g., 1) to a different value that corresponds to a finer or coarser degree of quantization and thus a finer or coarser degree of compression. This allows models to be trained in relation to the quantization bin size, allowing the use of different distortion-rate tradeoffs.

[0034] Exemplary scaling of quantization width in sequential coding of data 3 shows an example of potential scaling of quantization widths in continuous control of compression bitrate. As shown, for a given value y, prior to scaling by s, quantization of y results in rounding to the nearest discrete point [y] 306. To quantize y to [y] 306 and transmit via entropy coding, the system can calculate a probability mass 302 within a probability distribution 300 between the upper and lower bounds of the quantization bin 304 in which y lies. The probability mass can be calculated using the formula

number

[0035] When scaling is applied, the quantization bin size may be changed to a different value. For example, a scaling factor s of 2 may double the width of the quantization bin size and halve the number of possible values ​​that y may be coded into, as shown in the scaled probability distribution 310 (i.e., as indicated by the dots below the probability distribution 310). After scaling by s, for a given value y, the quantization of y results in rounding to a nested discrete point 2[y / 2] 316. To quantize and scale y to 2[y / 2] and transmit via entropy coding, the system may calculate the probability mass 312 within the scaled probability distribution 310 between the upper and lower bounds of the scaled quantization bin 314 in which y resides. The probability mass can be calculated using the formula

number

[0036] This corresponds to a larger quantization interval and a smaller number of bits than those shown in probability distribution 300.

[0037] The dequantized potential is expressed as

number

number

number

number

[0038] 4A and 4B show examples of quantization bin sizes used to achieve different compression bit rates according to aspects of this disclosure.

[0039] In particular, FIG. 4A shows a series 400A of quantization levels having bin sizes s1 and s2 to which a latent code y may be mapped. Because s1 and s2 are different bin sizes, each quantization level may have a different midpoint, for which values ​​may be quantized or rounded, and may have a different number of bins of different sizes. Thus, a quantization level associated with bin size s1 may have a lower effective bit rate than a quantization level associated with bin size s2. Thus, data compressed using quantization level s1 may be smaller than data compressed using quantization level s2, but may have lower quality when decompressed.

[0040] More generally, for a number N of quantization bin sizes in the sequence, the first quantization bin size s1 corresponds to the largest quantization bin size, and subsequent quantization bin sizes are s NWhen the quantization bin size decreases toward s1>s2>s3>...>s N Correspondingly, the bit rate for the quantization bin size can be expressed as β1<β2<β3<...<β N It can be represented by:

[0041] In some aspects, the bin size for any level n of quantization in the set of quantization levels {1, 2, ..., N} need not be consistent throughout the levels of quantization. For example, Figure 4B illustrates a series 400B of quantization levels 402, 404, 406, 408 having different quantization bin sizes. As shown, quantization level 402 may have the coarsest quantization bin size, quantization level 404 may have a first intermediate quantization bin size, quantization level 406 may have a second intermediate quantization bin size that is finer than the bin size of quantization level 404, and quantization level 408 may have the finest quantization bin size.

[0042] Further, FIG. 4B illustrates that quantization bin sizes may vary even within a quantization level. For example, in quantization level 406, the central bin 416 may have a different size than the other bins. Note that while quantization level 404 shows a single bin with a different size than the other bins, a quantization level may include bins of various sizes at various locations. For example, a quantization level may have a large central bin and successively smaller bins on either side of the midpoint (e.g., a smaller non-central bin and a larger central bin). In another example, a quantization level may have larger bins interspersed between smaller bins. In general, selecting a larger central bin may improve compression performance (e.g., improve the quality of the decompressed image by reducing distortion to the original version of the image). This is because, in a Gaussian distribution, most of the probability mass under probability distribution 300 may be centered around the central point of the probability distribution. Therefore, increasing the bin size around the center point of the probability distribution has a larger effect on the rate reduction than increasing the bin size for non-center bins (e.g., where the rate is

number

[0043] FIG. 5 illustrates an example of nested quantization 500. In general, nested quantization may allow the quantization of data using a finer quantization bin size to be defined conditional on the code quantized within a coarser quantization bin size. As explained above, quantization of latent space code y using quantization bin size s1 generally results in a lower bit rate than quantization of latent space code y using a smaller quantization bin size s2. In this case, given a latent space code y, y can be bounded by an upper bound of 0 for a quantization level with a quantization bin size of s1.

number

number

number

number

[0044] More generally, to scale y to any quantization level i, we use i is the formula

number

number

number

number

number

[0045] In nested quantization, the probability mass of y quantized at the lowest quantization level (and thus the largest quantization bin size) may represent the initial quantization of the latent space code y for the content x being compressed. That is, the probability mass of y corresponds to the probability mass associated with one of the codes at the lowest quantization level into which y is mapped. Subsequent quantizations of y to higher quantization levels (and thus smaller quantization bin sizes and higher bit rates) may be calculated as conditional probability masses conditional on the coarser quantization values. For example, to quantize y using a second quantization bin size s2, the probability mass of quantized value y2 conditional on y1 may be calculated as the formula

number

[0046] The number of bits used to represent data compressed using the finest quantization bin (e.g., for the quantization level having the smallest bin size, and therefore the largest bit rate) is given by the formula

number

number

[0047] That is, for any given quantization bin size, the conditional probability for that quantization bin size may be conditioned on the conditional probability calculated for quantization bin sizes larger than that quantization bin size. Since the conditional probability of the quantized code [y] at any given quantization bin size may be conditioned on the conditional probability of the code in the larger quantization bin size, the code at any given quantization bin size may be derived based on the chain rule using the code generated at the larger quantization bin size. Thus, a single model may be used to encode and compress content at any compression bit rate, and multiple supported compression bit rates are embedded within the compressed content. Furthermore, the compressed content may be decompressed from any given compression bit rate, which may allow devices to decompress data based on, for example, the computing capabilities of each device.

[0048] In general, in nested quantization, encoding may occur in N stages, where N represents the number of quantization levels at which y may be encoded. In general, y may be initially quantized using the quantization level associated with the coarsest quantization bin, and the quantization of y may be iteratively refined using successively finer quantization bin sizes. In general, the additional information provided from quantizing y into quantization levels with finer quantization bin sizes is used to refine the conditional probability formula

number

number

number

number

[0049] In some aspects, the naive approach may result in increased complexity (described in more detail below) and codeword lengths that may be longer than the codeword lengths generated at the finest quantization level. For example, taking the naive approach, the codes generated by the N stages form a bitstream that embeds data in N different bit rates, and the total length of the bitstream is determined by the formula:

number

number

number

number

number

[0050] To reduce complexity, as described below, a set of fully nested quantization levels may be defined such that the center points of the quantization bins in a coarser quantization level are a subset of the center points of the quantization bins in a finer quantization level. With fully nested quantization levels, the set of grid points in a coarser quantization bin may be the set of points in a finer quantization bin. That is, the quantization may be defined according to the following formula: I n =I n+1 ∩[y - (s k ), y + (s k )]=[y - (s k ), y + (s k )].

[0051] This may simplify the bitstream length formula discussed above to:

[0052]

number

[0053] By using a fully nested set of quantization levels, the performance of the highest bitrate model within a coding model may be preserved while simplifying the process of compressing the data.

[0054] In some embodiments, the selection of the scaling factor may effectively implement various types of compression. For example, s i-1 =2s i When s, the resulting compression scheme may be binary bit-plane coding. i-1 s i When σ is an integer multiple of σ, calculating the upper and lower bounds of the quantization bins may be a simple calculation and may therefore allow for less processor-intensive compression and decompression of data.

[0055] Exemplary method for sequential coding of data for data compression Figure 6 illustrates example operations 600 that may be performed by a system to compress content received through a compression pipeline, such as pipeline 100 shown in Figure 1 or pipeline 200 shown in Figure 2, using sequential coding. Operations 600 may be performed by a system having one or more processors, such as system 1200 of Figure 12, that implements a compression pipeline that includes a neural network-based encoder and quantizer that implement a learned nested quantization scheme.

[0056] As shown, operations 600 may begin with receiving content for compression at block 610. The received content may be single channel content, such as a stream of data, or may be multi-channel content (i.e., content having multiple channels of data) where different channels may be compressed separately. Multi-channel content may include, for example, audio content including multiple spatial channels (e.g., left / right stereo, surround sound content, etc.), video content including luminance and / or chrominance channels (e.g., YPbPr, RGB, etc.), audiovisual content including separate visual and audio channels, etc.

[0057] In block 620, the content is encoded (e.g., by the encoder 112(g) shown in FIG. a ) into the latent code space. To encode the content into the latent code space, an encoder implemented by an artificial neural network trained to generate a latent space representation of the content may be used. In some aspects, encoding the received content x into a code y in the latent code space may be a lossless mapping of the received content x to the code y. The compression, and the resulting loss (or distortion) relative to the original received content x, may be obtained by quantizing the code y.

[0058] At block 630, a first compressed version of the encoded content is generated (e.g., via quantizer 114(Q) shown in FIG. 1). To generate the first compressed version of the encoded content, the code y into which the content is encoded may be quantized using a first quantization bin size of a set of quantization bin sizes associated with a first bitrate. For example, the first quantization bin size may be a coarsest quantization bin size of multiple quantization bin sizes in the set of quantization bin sizes and may result in compression at the lowest bitrate of the bitrates associated with the multiple quantization bin sizes.

[0059] At block 640, a refined compressed version of the encoded content is generated (e.g., via quantizer 114(Q) shown in FIG. 1). In one example, to generate the refined compressed version of the encoded content, the first compressed version of the encoded content is scaled to one or more second quantization bin sizes that are smaller than the first quantization bin size, conditional on at least the values ​​of the encoded content. In general, each respective quantization bin size of the one or more second quantization bin sizes corresponds to a bit rate that is higher than the first bit rate. That is, each respective quantization bin size of the one or more second quantization bin sizes may be smaller than the first quantization bin size.

[0060] At block 650, the refined, compressed version of the encoded content is output for transmission (eg, via entity coder 116 (EC) shown in FIG. 1).

[0061] Figure 7 illustrates example operations 700 that may be performed by a system for decompressing encoded content. Operations 700 may be performed by a system having one or more processors, such as system 1200 of Figure 12, that implements a compression pipeline that includes a neural network-based encoder and quantizer that implement a learned nested quantization scheme, such as pipeline 100 shown in Figure 1 or pipeline 200 shown in Figure 2.

[0062] As shown, operations 700 may begin at block 710 where encoded content is received for decompression.

[0063] At block 720, an approximation of the code in the latent code space is recovered from the received encoded content.

[0064] In some cases, approximations of the code

number

number

[0065] In general, a series of quantization bin sizes may allow decompression of the content using a single model for any bitrate based on the tolerable amount of distortion in the decompressed version of the encoded content. As explained above, the code representing the compressed data may be reconstructed from any quantization level using the chain rule, where the code at a given quantization level may be defined as the code conditional on the code obtained at a lower quantization level (e.g., a quantization level having a smaller quantization bin size than that at the given quantization level). The amount of distortion in the decompressed version of the encoded content may be inversely proportional to the bitrate associated with the smallest quantization bin size used to reconstruct the approximation of the code. That is, the lowest bitrate associated with the largest quantization bin size of the series of quantization bin sizes may have the highest amount of distortion, and the distortion may decrease as successively smaller quantization bin sizes are used in reconstructing the approximation of the code.

[0066] At block 730, the decompressed version of the encoded content is sent to the decoder 126 (g) shown in FIG. s ) via an artificial neural network implemented decoder, which may be complementary to an artificial neural network implemented encoder, and may be used to encode content into the latent code space, as described above with respect to FIG.

[0067] At block 740, the decompressed version of the encoded content is output. In some aspects, the decompressed version of the encoded content may be output to one or more output devices, such as a display or audio device connected to or integrated with the system, for playback to a user of the device. In some aspects, the decompressed version of the encoded content may be output to one or more other computing systems for output to users of those computing systems.

[0068] In some aspects, sequential coding may be used to compress the multi-channel data using different levels of compression (and thus achieving different levels of distortion) for each channel in the multi-channel data. As described, the channels in the multi-channel data may include luminance and / or chrominance channels in visual content, spatial sound information in multi-channel audio, etc. Each channel may have a different amount of data or a different impact on the final audiovisual rendition of the content when decompressed, and therefore it may be useful to encode (compress) each channel using a different amount of compression. The selection of the bit rate to use for compressing or decompressing the data may be made based on, for example, a congestion control or bandwidth adaptation function controlled by an application layer in the network stack. For example, if the content server detects low bandwidth between the content server and the requesting device, the content server may select a lower bit rate (e.g., compression using a larger quantization bin size), and similarly, if the content server detects high bandwidth between the content server and the requesting device, the content server may select a higher bit rate (e.g., compression using a smaller quantization bin size).

[0069] For example, in multi-channel video data in the YPbPr space (i.e., having a luminance channel and two color channels), the luminance channel may be considered the most important channel because it carries the most visual information in the multi-channel video data. Therefore, it may be desirable to encode the luminance channel using the highest bit rate and to encode the color channels using lower bit rates to balance the quality and the amount of compression applied to the video content. Thus, in encoding multi-channel video data in the YPbPr space, a neural network may encode each of the Y, Pb, and Pr channels separately to generate different latent space codes y Y , y Pb , and y Pr and each of these latent space codes may be encoded separately.

[0070] In another example, in image data carried in multiple color data channels (e.g., in an RGB chrominance color space), some color data may have a greater impact on the visual presentation of the decompressed content than other color data. For example, one color channel may be encoded using a higher bit rate compression than other channels based on a priori known sensitivities to different colors. For example, for RGB data, the green color channel may be compressed using a higher bit rate than those used for the red and blue color channels because it is known that the human eye is more sensitive to green color data than to other color data.

[0071] To perform channel-wise sequential coding, the channels may be ordered according to the amount of compression applied to each channel. The ordering may be determined based on the difference in distortion ΔD and the difference in bit rate ΔR. For example, the ordering may be determined based on the ratio calculated for each channel:

number

[0072] Exemplary Channel-Wise Sequential Coding FIG. 8 shows an example of channel-wise sequential coding 800 using different bit rates for different channels.

[0073] As shown, in coding 800, a bitstream may be generated for each of C channels. For these channels 1 through C, the quantization bin size for the code generated for each of channel c at a given bit rate b is

number

number

number

number

[0074] For each channel, we use the code y c , can be compressed at the coarsest quantization bin size and output for transmission. To achieve nested quantization for each channel, the code y c can be compressed and output for transmission at the finer quantization bin sizes, conditional on the quantized value of y at the coarser bin sizes. c By outputting the additional codes y c (i.e., generated for successively finer quantization bin sizes).

[0075] In some cases, additional sequential coding may be achieved by outputting additional quantization information for some channels c in the set of C channels.

[0076] For example, each of the channels c may be compressed at the coarsest quantization bin size and output for transmission. For channels that have a greater impact on the resulting quality of the decompressed data, additional codes y are generated for progressively finer quantization bin sizes. cmay be output (e.g., for a subset of channels c). For any quantization level beyond the level associated with the coarsest quantization bin size, codes may be generated for the subset of C channels such that no additional compression is performed for channels that have a small impact on the resulting quality of the decompressed data. The subset of C channels generated for any quantization level N may be generated using, for example, a greedy technique (e.g., for each one-level increase in quantization level, reduce by one the number of channels coded using the increased quantization level) or by using the quantization bin size calculated for each channel.

number

[0077] In FIG. 8, each level shown in coding 800 represents a different quantization level and corresponding bit rate used to compress the C channels. As shown, each of the C channels is compressed at the lowest quantization level (e.g., quantization level 1, for which the value is expressed as

number

number

[0078] Exemplary Quantization Grids for Sequential Coding in Data Compression 9 illustrates an example of an effective quantization grid 900 for sequential coding of content. As shown, three quantization levels are used to quantize a latent space code y representing content x. A first quantization level is associated with a quantization bin size s1, a second quantization level is associated with a quantization bin size s2, and a third quantization level is associated with a quantization bin size s3. For a code y generated by quantizing y at the quantization level associated with quantization bin size s1, the number of bits transmitted is given by the formula

number

number

[0079] At a further quantization level, the code y3 may be represented by the intersection 904 of the upper bound of the bin in which y3 lies and the lower bound of y2, given by the formula

number

[0080] Thus, the effective quantization bin size may be finer than the finest bin used to compress the content. Thus, the total bits transmitted are calculated using the formula

number

[0081] When nested quantization is applied thoughtlessly, there may be a performance degradation in compressing the data because the effective quantization bins may be smaller than the finest bins actually used to quantize the latent space representation y of the content x. The performance degradation may be seen in a reduction in the increase in quality, as measured by peak signal-to-noise ratio (PSNR), for an increase in the effective compression bitrate.

[0082] To mitigate the performance degradation due to mindlessly applying nested quantization, the quantization grid 1000 for sequential coding may be aligned to the finest quantization grid 1002, as shown in FIG.

[0083] To align the quantization grid 1004 to the grid 1002 for the finest quantization grid size, multi-pass quantization may be used to quantize multiple coarser quantization bins based on the finest quantization bin size. By applying multi-pass quantization, smaller quantization beam intersections are identified (e.g., effective quantization bin 906 is identified), so the effective quantization bin size is not smaller than the finest quantization bin size to simplify processing and to avoid performance degradation due to the use of variable effective quantization bin sizes. The midpoint 1006 of the quantization grid 1000 may be at the midpoint of the central quantization bin in the grid for each of the quantization levels and may be at the midpoint of the central quantization bin in the effective quantization grid 1004.

[0084] FIG. 11 shows an exemplary result of data compression using sequential coding.

[0085] Graphs 1100A-1100F show the relationship between PSNR per pixel and bit rate for various compression techniques used to compress the sample image. As shown in graph 1100A, compression using the incremental coding technique described herein may provide a PSNR of 26.59 dB at a low bit rate of 0.11 bits per pixel used to compress the sample image. Graph 1100B shows that the incremental coding technique described herein may provide a PSNR of 31.02 dB at a bit rate of 0.34 bits per pixel used to compress the sample image. Graph 1100C shows that the incremental coding technique described herein may provide a PSNR of 33.52 dB at a bit rate of 0.60 bits per pixel used to compress the sample image. Graph 1100D shows that the incremental coding technique described herein may provide a PSNR of 35.99 dB at a bit rate of 0.90 bits per pixel used to compress the sample image. Graph 1100E shows that the sequential coding technique described herein may provide a PSNR of 37.70 dB at a bit rate of 1.21 bits per pixel used to compress the sample image. Finally, graph 1100F shows that the sequential coding technique described herein may provide a PSNR of 39.69 dB at a bit rate of 1.48 bits per pixel used to compress the sample image. In these examples, it can be seen that the quality of the compressed image (represented by a PSNR measurement over the compressed image) is higher for each effective bit rate compared to nested dropout sequential coding. Furthermore, for each effective bit rate, the quality of the compressed image may be comparable to the quality of compressed images produced using various non-sequential coding schemes where an a priori defined model is used for each bit rate used to compress the data.

[0086] Exemplary Coding Unit Ordering for Sequential Data Compression In some aspects, data compressed using the techniques described herein may be split into multiple coding units, and each coding unit may be compressed separately. For example, a coding unit may be a channel, a pixel in an image (e.g., data for each of multiple channels at a particular location in an image or video content), a block of data (e.g., one or more channels for an nxm pixel block in an image or video content), or a single element (e.g., data for a single channel at a particular location in an image or video content). To facilitate sequential coding, and considering that each coding unit may have a different amount of information and a different sensitivity to compression loss, each coding unit may be gradually refined separately. Sequential coding may be divided into two phases. The first phase encodes latent variables (e.g., code y representing input x generated by an artificial neural network-based encoder) from the largest quantization bin (e.g., lowest quantization level) to the smallest quantization bin (e.g., highest quantization level). In the second phase, refinement between adjacent quantization levels may be performed sequentially for each coding unit, so that each truncation point in the resulting embedded bitstream that represents a boundary between coding units is associated with a sequential change in quantization level.

[0087] Operationally, successive latents may be coded from infinitely large quantization bins, whereby the latent variables are each quantized to a given midpoint value. As a result, the inverse quantization latents sent to the decoder may be a pre-average. An ordering of coding units for refinement may be found from the pre-quantization, such that coding units in a current quantization bin may be decoded in the appropriate quantization bin. Based on the ordering of coding units, the coding units may be refined from the largest quantization bin to the smallest quantization bin. To simplify the process, the coding is performed in such a way that coding units coded using the highest quantization level (and corresponding largest quantization bin size) are coded first, and coding units coded using lower quantization levels (and corresponding smaller quantization bin sizes) are coded after the coding units using the highest quantization level.

[0088] In a sequential coding scheme, a code y, which may be a tensor in the latent space, is divided into N coding units {y1, ..., y N The elements in a coding unit may be refined together and may correspond to a truncation point or a point in space that achieves the lowest distortion (e.g., compression loss). 4 Various codings may be defined for tensors in the latent space having size (1, H, W). A single channel coding may correspond to a latent slice with size (1, H, W), a single pixel coding may correspond to a latent slice with size (C, 1, 1), and a single element coding may correspond to a latent slice with size (1, 1, 1).

[0089] Given a compression order ρ = (ρ1, ..., ρ N ), the ordered coding unit y ρ =(y ρ1 , ..., y ρN ) but s n From the magnification of s n-1In a mean space hyperprior model, the priors of the latent elements may be conditioned on the hyper-latencies. The bitrate increase ΔR for refining the t-th coding unit due to the scaling described herein is given by

number

[0090] The reduction in distortion ΔD may be calculated similarly and may depend on the other ordered coding units. The reduction in distortion ΔD is calculated according to the formula ΔD(y ρt |y ρ )=D(y ρ (t-1))-D(y ρ (t) where (y ρ (t)=(y ρ≦t (s n-1 ), y ρ>t (s n )), and where D(y)=MSE(x, g s (y)), which represents the distortion for the code y. Thus, refining the ordered latent using the compression order ρ yields the formula

number

[0091] In general, an optimal ordering of ρ may be an ordering in which the convex hull of H(ρ) is better than the convex hull of other orderings of ρ (eg, is a Pareto optimal compaction ordering).

[0092] 12 illustrates the relationship between the amount of coding loss involved in encoding content using different orderings. In the graph 1200, for simplicity, two coding units y1 and y2 are shown, but it should be understood that the data being encoded may have any number of coding units. As shown, y1 is

number

number

[0093] Distortion line 1202 shows the coding loss if y2 is coded before y1 and y2 is coded using a higher rate than y1. In contrast, distortion line 1204 shows the coding loss if y1 is coded before y2 and y1 is coded using a higher rate than y2. For compression using either (y1, y2) or (y2, y1) ordering, the total distortion / rate loss may be the same. However, because y2 is less sensitive than y1 to changes in compression rate (i.e., the reduction in distortion for any given increase in compression rate for y2 is less than for y1), it may be more efficient to code y1 before coding y2, as shown by distortion line 1204 being lower than distortion line 1202.

[0094] Thus, to compress content using sequential coding so that the content is optimally encoded, the coding units may be sorted in descending order of their respective distortion-rate change factors. Because the amount of distortion loss may depend on the order in which the units are coded, additional complexity differences may be introduced when the coding units are generated by a neural network-based encoder, such as encoder 112 shown in FIG. 1. However, the distortion-rate change factors calculated separately for each coding unit may be treated as approximations for purposes of ordering the coding units for sequential coding.

[0095] FIG. 13 shows exemplary results of data compression using sequential coding and different ordering of coding units.

[0096] Graph 1300 illustrates the relationship between peak signal-to-noise ratio per pixel (PSNR) and bit rate for incremental compression of data based on different coding units and classification criteria.

[0097] As described, coding units may be defined for data of various granularities. For a latency having a shape defined by some of the channels C, the highest dimension H, and the width dimension W (e.g., a still image or video frame defined as multiple color space channels and spatial dimensions), a coding unit may be one of the C channels, a single pixel, a block of pixels for one of the C channels, or a single element in the latency (e.g., the value of one of the C channels at a particular location in the image). The classification criteria shown in graph 1300 are the distortion-rate change ratio for each coding block, the rate difference ΔR, and the prior standard deviation ρ.

number

[0098] As shown, latent ordering and element-wise compression may achieve a higher PSNR at a given compression rate than latent ordering and channel-wise compression, and latent ordering and pixel-wise compression may achieve a significantly lower PSNR except at higher compression bit rates. For compression using channel-based coding unit classification, compression performance may be similar for classification by a priori standard deviation, rate difference, or rate-distortion importance. However, for compression using element-based coding unit or pixel-based coding unit classification, it can be seen that compression performance differs between different types of ordering. For example, it can be seen that for compression using element-based coding unit classification, ordering based on a rate difference metric may achieve better compression performance (e.g., higher PSNR for a given bit rate) than ordering based on a priori standard deviation.

[0099] Ordering the coding units may impose some overhead on compression and decompression. For example, ordering the coding units by a priori standard deviation may allow compression to be performed without needing additional information to reconstruct the compressed data, since the a priori standard deviation may be known to the decoder once the hyperlatencies are decoded. However, ordering the coding units by a rate difference metric or a rate-distortion importance metric may allow more accurate ordering of the coding units, at the expense of imposing bitrate overhead for conveying the order in which the coding units are encoded. In some aspects, if the order in which the coding units are encoded is deemed important enough to accept the additional overhead of conveying the ordering information to the decoder as side information, various optimizations may be used to reduce the overhead associated with conveying this ordering information to the decoder. For example, larger coding units, such as blocks of pixels rather than individual pixels in a still image, may be used to compress the data, which may reduce the amount of side information conveyed. In another aspect, the expected order may be learned by a machine learning model from training data, and the expected order generated by the trained machine learning model may be conveyed to the decoder. In a further aspect, the ordering may be learned from other available information, such as from latencies already decoded using larger quantization bin sizes.

[0100] FIG. 14 shows an example neural network-based data compression pipeline 1400 in which side information is used to decompress data.

[0101] As shown, the compression pipeline includes the elements shown in FIG. 1 and described above, as well as the steps for extracting an approximation of the original content x from the compressed bitstream.

number

[0102] A hyper-analysis transform 1402 (h) is used to generate information for a side channel that may be used to decode a compressed version of a latent space code y representing content x. a ) may generate a hyper-latent z, which is quantized by a quantizer 1404 and coded as a hyper-prior by an entropy coder 1406.

number

number

number

[0103] The prior standard deviation σ and mean μ may be used by the entropy coder 116 and the entropy decoder 122 to encode the quantized version of the code y and to recover the quantized version of the code y from a bitstream representing the encoded quantized version of the code y, while the prior mean μ serves as an approximation of the latent space code y to which the content x is mapped, and to which the content x is mapped.

number

[0104] Exemplary Processing System for Sequential Data Compression FIG. 15 illustrates an example processing system 1500 for performing convolutional neural network processing, for example, as described herein with respect to FIGS.

[0105] The processing system 1200 includes a central processing unit (CPU) 1502, which in some examples may be a multi-core CPU. Instructions executed in the CPU 1502 may be loaded from a program memory associated with the CPU 1502 or from a memory partition 1524, for example.

[0106] The processing system 1500 also includes additional processing components tailored to specific functions, such as a graphics processing unit (GPU) 1504, a digital signal processor (DSP) 1506, a neural processing unit (NPU) 1508, a multimedia processing block 1510, a multimedia processing unit 1510, and wireless connectivity components 1512.

[0107] An NPU, such as 1508, is generally a specialized circuit configured to implement all the necessary control and computational logic to execute machine learning algorithms, such as algorithms for processing artificial neural networks (ANN), deep neural networks (DNN), random forests (RF), etc. An NPU may alternatively be referred to as a neural signal processor (NSP), tensor processing unit (TPU), neural network processor (NNP), intelligence processing unit (IPU), vision processing unit (VPU), or graph processing unit.

[0108] An NPU, such as 1508, is configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-chip (SoC), while in other examples, they may be part of a dedicated neural network accelerator.

[0109] An NPU may be optimized for training or inference, or in some cases may be configured to balance performance between both. In an NPU capable of performing both training and inference, the two tasks may still generally be performed independently.

[0110] NPUs designed to accelerate training are typically configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing dataset (often labeled or tagged), iterating over the dataset, and then adjusting model parameters such as weights and biases to improve model performance. In general, optimization based on erroneous predictions involves propagating backwards through layers of the model and determining gradients to reduce prediction errors.

[0111] NPUs designed to accelerate inference are generally configured to operate on complete models. Thus, such NPUs may be configured to input new data and rapidly process the data through already trained models to generate model outputs (e.g., inferences).

[0112] In one implementation, the NPU 1508 is part of one or more of the CPU 1502, the GPU 1504, and / or the DSP 1506.

[0113] In some examples, the wireless connectivity component 1512 may include subcomponents for, for example, third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing component 1512 is further connected to one or more antennas 1514.

[0114] The processing system 1500 may also include one or more sensor processing units 1516 associated with any type of sensor, one or more image signal processors (ISPs) 1518 associated with any type of image sensor, and / or a navigation processor 1520, which may include satellite-based positioning system components (e.g., GPS or GLONASS), as well as inertial positioning system components.

[0115] The processing system 1500 may also include one or more input and / or output devices 1522, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, a microphone, etc.

[0116] In some examples, one or more of the processors of processing system 1500 may be based on the ARM or RISC-V instruction set.

[0117] Processing system 1500 also includes memory 1524, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, memory 1524 includes computer-executable components that may be executed by one or more of the above-mentioned processors of processing system 1500.

[0118] In particular, in this example, memory 1524 includes a latent space encoding component 1524A, a sequential coding component 1524B, a sequential code recovery component 1524C, and a latent space decoding component 1524D. The illustrated components and other components not illustrated may be configured to perform various aspects of the methods described herein.

[0119] In general, the processing system 1500 and / or its components may be configured to perform the methods described herein.

[0120] Notably, in other aspects, aspects of the processing system 1500 may be omitted, such as when the processing system 1500 is a server computer, etc. For example, the multimedia component 1510, the wireless connectivity 1512, the sensors 1516, the ISP 1518, and / or the navigation component 1520 may be omitted in other aspects. Additionally, aspects of the processing system 1500, such as training a model and using the model to generate inferences, such as a user authentication prediction, may be described.

[0121] Example clauses Clause 1: A method for compressing content using a neural network, the method comprising: receiving content for compression; encoding the content into a first latent code space via an encoder implemented by an artificial neural network; generating a first compressed version of the encoded content using a first quantization bin size of a set of quantization bin sizes; generating a refined compressed version of the encoded content by scaling the first compressed version of the encoded content to one or more second quantization bin sizes smaller than the first quantization bin size, conditional on at least a value of the first compressed version of the encoded content; and outputting the refined compressed version of the encoded content.

[0122] Clause 2: The method of clause 1, wherein the step of generating a refined compressed version of the encoded content includes the steps of: generating a first refined compressed version of the encoded content by scaling the first compressed version of the encoded content to a first finer quantization bin size conditional on values ​​of the first compressed version of the encoded content; and generating a second refined compressed version of the encoded content by scaling the first refined compressed version of the encoded content to a second finer quantization bin size conditional on values ​​of the first refined compressed version of the encoded content and the first compressed version of the encoded content, wherein the second finer quantization bin size is smaller than the first finer quantization bin size.

[0123] Clause 3: The method of clause 1 or 2, wherein a size of each respective quantization bin size in the series of quantization bin sizes is an integer multiple of the first quantization bin size.

[0124] Clause 4: The method of any of clauses 1 to 3, wherein a central bin for one quantization bin size of the series of quantization bin sizes has a bin size that is larger than a non-central bin in the series of quantization bin sizes.

[0125] Clause 5: Any of the methods of clauses 1 to 4, wherein the step of generating the refined compressed version of the encoded content includes the step of generating a bitstream based on a set of conditional probabilities, each conditional probability in the set of conditional probabilities being associated with a respective quantization bin size in the set of quantization bin sizes other than the finest quantization bin size and conditioned on conditional probabilities calculated for quantization bin sizes greater than the respective quantization bin size.

[0126] Clause 6: Any of the methods of clauses 1 to 5, wherein the step of generating a refined compressed version of the encoded content comprises the step of generating a probability mass of the encoded content for each quantization bin size in a series of quantization bin sizes based on a cumulative distribution function of upper and lower bounds of each quantization bin in which the encoded content is located.

[0127] Clause 7: The method of clause 6, wherein the probability mass for each respective quantization bin size in the series of quantization bin sizes is conditional on the probability mass for a quantization bin size in the series of quantization bin sizes that is greater than the respective quantization bin size.

[0128] Clause 8: Any of the methods of clauses 1 to 7, wherein the received content includes content having multiple data channels.

[0129] Clause 9: The method of clause 8, wherein each respective data channel of the plurality of data channels is associated with a compression priority corresponding to an amount of compression used to compress the respective data channel.

[0130] Clause 10: The method of clause 9, wherein the multiple data channels include a luminance channel and multiple chrominance channels in the visual content, and the luminance channel is associated with a compression priority associated with a lower amount of compression than a compression priority associated with the multiple chrominance channels.

[0131] Clause 11: The method of clause 9, wherein the received content includes visual content to be compressed, the multiple data channels include multiple color data channels within the visual content, and a first color data channel of the multiple color data channels that has the greatest impact on the quality of the compressed version of the encoded content is associated with a compression priority associated with a lower amount of compression than a compression priority associated with a color data channel other than the first color data channel.

[0132] Clause 12: The method of clause 11, further comprising identifying a first color data channel based on an amount of luminance data included in each of the plurality of color data channels.

[0133] Clause 13: The method of any of clauses 9 to 12, further comprising determining a compression priority associated with each respective data channel of the plurality of data channels based on calculating a reduction in distortion and calculating an increase in bit rate when each data channel is encoded for each of a plurality of bit rates associated with each quantization bin size in the series of quantization bin sizes.

[0134] Clause 14: The method of clause 13, wherein the step of calculating the reduction in distortion for each respective data channel includes the step of calculating a difference between the distortion produced by decoding the encoded content a first time including the respective data channel and a second time the distortion produced by decoding the encoded content excluding the respective data channel.

[0135] Clause 15: The method of any of clauses 1 to 14, further comprising the steps of dividing the received content into a plurality of coding units and ordering the plurality of coding units based on a compression metric, wherein generating a refined compressed version of the encoded content comprises refining each of the plurality of coding units, whereby each of the plurality of coding units is compressed using a different level of quantization, and wherein coding units having a higher compression metric are compressed using a lower amount of compression than coding units having a lower compression metric.

[0136] Clause 16: The method of clause 15, wherein the step of dividing the received content into a plurality of coding units includes the step of dividing the received content into a plurality of elements, each element representing data for one of a plurality of channels at a particular location within the received content.

[0137] Clause 17: The method of clause 15, wherein the step of dividing the received content into a plurality of coding units includes the step of dividing the received content into a plurality of blocks, each block representing data for one of a plurality of channels at a particular range of locations within the received content.

[0138] Clause 18: The method of clause 15, wherein splitting the received content into a plurality of coding units comprises splitting the received content into a plurality of channels.

[0139] Clause 19: The method of clause 15, wherein the step of dividing the received content into a plurality of coding units includes the step of dividing the received content into a plurality of pixels, each pixel representing data for a plurality of channels at a particular location within the received content.

[0140] Clause 20: The method of clause 15, wherein the compression metric includes a prior standard deviation encoded within a hyper-latency, the hyper-latency including a refined compressed version of an initial portion of the encoded content.

[0141] Clause 21: The method of clause 15, wherein the compression metric includes a distortion-rate ratio, and the refined compressed version of the encoded content includes ordering information for the multiple coding units from highest distortion-rate ratio to lowest distortion-rate ratio.

[0142] Clause 22: The method of clause 15, wherein the compression metric includes a change in rate metric, and the refined compressed version of the encoded content includes ordering information for a plurality of coding units from a change in rate to a minimum change in rate.

[0143] Clause 23: The method of any of clauses 1 to 22, wherein a first quantization bin size is associated with a first bitrate, and each respective quantization bin size of the one or more second quantization bin sizes corresponds to a bitrate higher than the first bitrate.

[0144] Clause 24: A method for decompressing content using a neural network, the method comprising: receiving encoded content for decompression; recovering an approximation of a value in a latent code space from the received encoded content by recovering a code from a set of quantization bin sizes, the set of quantization bin sizes including a first quantization bin size and one or more second quantization bin sizes smaller than the first quantization bin size; generating a decompressed version of the encoded content by decoding the approximation of the value in the latent code space via a decoder implemented by an artificial neural network; and outputting the decompressed version of the encoded content.

[0145] Clause 25: The method of clause 24, wherein a size of each respective quantization bin size in the series of quantization bin sizes is an integer multiple of the first quantization bin size.

[0146] Clause 26: The method of clause 24 or 25, wherein a central bin for one quantization bin size of the series of quantization bin sizes has a bin size that is larger than a non-central bin in the series of quantization bin sizes.

[0147] Clause 27: Any of the methods of clauses 24 to 26, wherein the step of recovering an approximation of a value in the latent code space comprises recovering a code from a bitstream representing the encoded content based on a set of conditional probabilities, each conditional probability in the set of conditional probabilities being associated with a respective quantization bin size in the set of quantization bin sizes other than the finest quantization bin size and conditioned on conditional probabilities calculated for quantization bin sizes greater than the respective quantization bin size.

[0148] Clause 28: Any of the methods of clauses 24 to 27, wherein the step of recovering an approximation of the value in the latent code space comprises identifying a probability mass of the encoded content from each quantization bin size in a range of quantization bin sizes based on a cumulative distribution function of upper and lower bounds of each quantization bin in which the encoded content is located.

[0149] Clause 29: The method of clause 28, wherein the probability mass for each respective quantization bin size in the series of quantization bin sizes is conditional on the probability mass for a quantization bin size in the series of quantization bin sizes that is greater than the respective quantization bin size.

[0150] Clause 30: The method of any of clauses 24 to 29, wherein the received encoded content includes content having multiple data channels.

[0151] Clause 31: The method of clause 30, wherein each respective data channel of the plurality of data channels is associated with a compression priority corresponding to an amount of compression used to compress the respective data channel.

[0152] Clause 32: The method of clause 31, wherein the multiple data channels include a luminance channel and multiple chrominance channels in the visual content, and the luminance channel is associated with a compression priority associated with a lower amount of compression than a compression priority associated with the multiple chrominance channels.

[0153] Clause 33: The method of clause 31, wherein the received encoded content includes visual content to be decompressed, and the multiple data channels include multiple color data channels within the visual content, and a first color data channel of the multiple color data channels that has the greatest impact on the quality of the decompressed version of the encoded content is associated with a compression priority associated with a lower amount of compression than a compression priority associated with a color data channel other than the first color data channel.

[0154] Clause 34: The method of clause 33, further comprising identifying a first color data channel based on an amount of luminance data included in each of the plurality of color data channels.

[0155] Clause 35: Any of the methods of clauses 24 to 34, wherein the encoded content includes a plurality of encoded coding units, and wherein recovering an approximation of a value in the latent code space from the received encoded content includes recovering a code in the latent code space associated with each of the plurality of encoded coding units.

[0156] Clause 36: The method of clause 35, wherein the multiple coding units include multiple elements, each element representing data for one of multiple channels at a particular location within the received content.

[0157] Clause 37: The method of clause 35, wherein the multiple coding units include multiple blocks, each block representing data for one of multiple channels at a particular range of locations within the received content.

[0158] Clause 38: The method of clause 35, wherein the multiple coding units include multiple channels.

[0159] Clause 39: The method of clause 35, wherein the multiple coding units include multiple pixels, each pixel representing data for multiple channels at a particular location within the received content.

[0160] Clause 40: The method of clause 35, wherein recovering an approximation of the value in the latent code space includes recovering a prior standard deviation encoded in a hyper-latent, the hyper-latent including an initial portion of the encoded content.

[0161] Clause 41: The method of clause 35, wherein restoring an approximation of the values ​​in the latent code space includes restoring an order in which the multiple coding units were compressed, the order being included as side information associated with the encoded content.

[0162] Clause 42: The method of any of clauses 24 to 42, wherein a first quantization bin size is associated with a first bitrate, and each respective quantization bin size of the one or more second quantization bin sizes corresponds to a bitrate higher than the first bitrate.

[0163] Clause 43: A processing system comprising a memory containing computer-executable instructions and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method according to any one of clauses 1 to 42.

[0164] Clause 44: A processing system, comprising means for carrying out the method according to any one of clauses 1 to 42.

[0165] Clause 45: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method according to any one of clauses 1 to 42.

[0166] Clause 46: A computer program product embodied on a computer-readable storage medium comprising code for performing the method of any one of clauses 1 to 42.

[0167] Additional Considerations The preceding description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples described herein are not intended to limit the scope, applicability, or aspects described in the claims. Various modifications of these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, changes may be made in the function and arrangement of the described elements without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components, as appropriate. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Also, features described with respect to some examples may be combined in some other examples. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects described herein. In addition, the scope of the disclosure is intended to cover such apparatus or methods practiced using other structures, functionality, or structures and functionality in addition to or other than the various aspects of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of a claim.

[0168] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0169] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items including single members. As an example, "at least one of a, b, or c" is intended to encompass a, b, c, ab, ac, bc, and abc, as well as any combination having multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other permutation of a, b, and c).

[0170] The term "determining" as used herein encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, and the like. "Determining" may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. "Determining" may also include resolving, selecting, choosing, establishing, and the like.

[0171] The methods disclosed herein include one or more steps or acts for achieving the method. The steps and / or acts of the method may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or acts is specified, the order and / or use of specific steps and / or acts may be modified without departing from the scope of the claims. Furthermore, various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs), or processors. In general, where there are operations illustrated in figures, those operations may have means-plus-function components of corresponding counterparts with similar numbering.

[0172] The following claims are not limited to the embodiments set forth herein, but are to be accorded the full scope consistent with the language of the claims. Within the claims, reference to an element in the singular shall mean "one or more," and not "the one and only," unless expressly stated otherwise. The term "several" refers to one or more, unless expressly stated otherwise. No element of a claim is to be construed under the provisions of 35 U.S.C. 112(f) unless the element is expressly recited using the phrase "means for," or, in the case of a method claim, unless the element is recited using the phrase "step for." All structural and functional equivalents of the elements of the various embodiments described throughout this disclosure that are known or later become known to those skilled in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be made public, regardless of whether such disclosure is expressly recited in the claims. [Explanation of symbols]

[0173] 100 Neural Network-Based Data Compression Pipeline 110 The Encoding Side of the Pipeline 111 Contents x 112 Neural network-based nonlinear transformation layer (ga) 113 Latent space code y 114 Quantization method (Q) 116 Entity Coder 120 Decryption Side of the Pipeline 122 Entity Decoder 124 Inverse quantization method (Q-1) Approximation of 125 code 126 Neural network-based nonlinear transformation layer (gs) 127 Approximate value 200 Pipeline 202 Encoder 204 HyperEncoder 206 Hyper Decoder 208 Decoder 210 Scaler 212 Rescaler 300 Probability Distribution 302 Probability Mass 304 quantization bins 306 [y] 310 Probability Distribution 312 Probability Mass 314 scaled quantization bins 316 2[y / 2] 400A Quantized Level Series 400B Quantization Level Series 402 quantization levels 404 quantization levels 406 quantization levels 408 quantization levels 500 nested quantization 800 channel-wise sequential coding 900 Effective Quantization Grids 902 Intersection 904 Intersection 906 smallest effective quantization bins 1000 quantization grid 1002 finest quantization grid 1004 Quantization Grid 1006 Quantization Grid Midpoint 1100A Graph 1100B Graph 1100C Graph 1100D Graph 1100E Graph 1100F Graph 1200 graphs 1202 Strain Wire 1204 Strain Wire 1300 graphs 1400 Neural Network-Based Data Compression Pipeline 1402 HyperAnalysis Transformation 1404 Quantizer 1406 Entropy Coder 1408 Entropy Decoder 1410 Inverse quantizer 1412 Hypersynthetic Transformation 1414 Hypersynthetic Transformation 1500 Processing System 1502 Central Processing Unit (CPU) 1504 Graphics Processing Unit (GPU) 1506 Digital Signal Processor (DSP) 1508 Neural Processing Unit (NPU) 1510 Multimedia Processing Block 1512 Wireless Connectivity Components 1514 Antenna 1516 Sensor Processing Unit 1518 Image Signal Processor (ISP) 1520 Navigation Processor 1522 Input and / or Output Devices 1524 memory 1524A Latent Space Coding Component 1524B Sequential Coding Components 1524C Sequential Code Recovery Component 1524D Latent Space Decoding Component

Claims

1. 1. A method for compressing content using a neural network, comprising: receiving content for compression; encoding the content into a first latent code space via an encoder implemented by an artificial neural network; generating a first compressed version of the encoded content using a first quantization bin size of a series of quantization bin sizes, the series of quantization bin sizes comprising a plurality of quantization bin sizes, the first quantization bin size corresponding to a largest quantization bin size of the series of quantization bin sizes and subsequent quantization bin sizes decreasing towards a smallest bin size; generating a refined compressed version of the encoded content by scaling the first compressed version of the encoded content to one or more second quantization bin sizes in the range of quantization bin sizes that are smaller than the first quantization bin size, conditional on at least values ​​of the encoded content of the first compressed version; outputting the refined, compressed version of the encoded content; A method comprising:

2. generating the refined, condensed version of the encoded content, generating a first refined compressed version of the encoded content by scaling the first compressed version of the encoded content to a first finer quantization bin size conditional on values ​​of the first compressed version of the encoded content; generating a second refined compressed version of the encoded content by scaling the first refined compressed version of the encoded content to a second finer quantization bin size conditional on values ​​of the first refined compressed version of the encoded content and the first compressed version of the encoded content, the second finer quantization bin size being smaller than the first finer quantization bin size; 2. The method of claim 1, comprising:

3. a size of each respective quantization bin size in the series of quantization bin sizes is an integer multiple of the first quantization bin size; and / or a central bin for a quantization bin size of the series of quantization bin sizes has a bin size that is larger than a non-central bin in the series of quantization bin sizes; The method of claim 1.

4. generating the refined, compressed version of the encoded content comprises generating a bitstream based on a set of conditional probabilities; each conditional probability in the series of conditional probabilities is associated with a respective quantization bin size in the series of quantization bin sizes other than the finest quantization bin size, and is conditioned on conditional probabilities calculated for quantization bin sizes greater than the respective quantization bin size; The method of claim 1.

5. generating the refined, condensed version of the encoded content, generating a probability mass for the encoded content for each quantization bin size in the series of quantization bin sizes based on a cumulative distribution function of upper and lower bounds of each quantization bin in which the encoded content falls; the probability mass for each respective quantization bin size in the series of quantization bin sizes is conditional on a probability mass for a quantization bin size in the series of quantization bin sizes that is greater than the respective quantization bin size. The method of claim 1.

6. the encoded content includes content having multiple data channels; each respective data channel of the plurality of data channels is associated with a compression priority corresponding to an amount of compression used to compress the respective data channel; the compression priority associated with each respective data channel of the plurality of data channels is determined based on calculating a reduction in distortion after decoding and calculating an increase in bit rate when the respective data channel is encoded for each of a plurality of bit rates associated with each quantization bin size in the series of quantization bin sizes; wherein calculating the reduction in distortion for each respective data channel includes calculating a difference between the distortion produced by decoding the encoded content a first time including the respective data channel and a second time excluding the respective data channel. The method of claim 1.

7. splitting the received content into a plurality of coding units; ordering the plurality of coding units based on a compression metric; Further comprising: generating the refined compressed version of the encoded content includes refining each of the plurality of coding units, whereby each of the plurality of coding units is compressed using a different level of quantization, such that coding units having a higher compression metric are compressed using a lesser amount of compression than coding units having a lower compression metric. The method of claim 1.

8. 20. The method of claim 1, further comprising: dividing the received content into a plurality of elements, each element representing data for one of a plurality of channels at a particular location within the received content; dividing the received content into a plurality of blocks, each block representing data for one of a plurality of channels at a particular range of locations within the received content; splitting the received content into a plurality of channels; or Dividing the received content into a plurality of pixels, each pixel representing data for a plurality of channels at a particular location within the received content. The method of claim 7, comprising one or more of:

9. The compression metric is a prior standard deviation encoded within a hyper-latency, the hyper-latency including an initial portion of the refined compressed version of the encoded content; or distortion-rate ratios, where the refined compressed version of the encoded content includes ordering information for the plurality of coding units from highest distortion-rate ratio to lowest distortion-rate ratio; or the refined compressed version of the encoded content includes one or more of a change in rate metric, and the refined compressed version of the encoded content includes ordering information for the plurality of coding units from a change in rate to a minimum change in rate.

8. The method of claim 7, comprising:

10. 1. A method for decompressing content using a neural network, comprising: receiving encoded content for decompression; recovering an approximation of a value in a latent code space from the received encoded content by recovering a code from a series of quantization bin sizes, the series of quantization bin sizes including a plurality of quantization bin sizes including a first quantization bin size corresponding to a largest quantization bin size in the series of quantization bin sizes and one or more subsequent second quantization bin sizes smaller than the first quantization bin size and decreasing toward a smallest quantization bin size; generating a decompressed version of the encoded content by decoding the approximations of the values ​​in the latent code space via a decoder implemented by an artificial neural network; outputting the decompressed version of the encoded content; A method comprising:

11. a size of each respective quantization bin size in the series of quantization bin sizes is an integer multiple of the first quantization bin size; and recovering the approximation of the value in the latent code space comprises recovering a code from a bitstream representing the encoded content based on a set of conditional probabilities; each conditional probability in the series of conditional probabilities is associated with a respective quantization bin size in the series of quantization bin sizes other than the finest quantization bin size, and is conditioned on conditional probabilities calculated for quantization bin sizes greater than the respective quantization bin size; The method of claim 10.

12. recovering the approximation of the value in the latent code space includes identifying a probability mass of the encoded content from each quantization bin size in the series of quantization bin sizes based on a cumulative distribution function of upper and lower bounds of each quantization bin in which the encoded content is located; the probability mass for each respective quantization bin size in the series of quantization bin sizes is conditional on a probability mass for a quantization bin size in the series of quantization bin sizes that is greater than the respective quantization bin size. The method of claim 10.

13. the received encoded content includes content having multiple data channels; each respective data channel of the plurality of data channels is associated with a compression priority corresponding to an amount of compression to be used to compress the respective data channel; The method of claim 10.

14. the encoded content includes a plurality of encoded coding units; Recovering the approximation of the value in the latent code space from the received encoded content includes recovering a code in the latent code space associated with each of the plurality of encoded coding units; the plurality of coding units includes a plurality of elements, each element representing data for one of a plurality of channels at a particular location within the received content; the plurality of coding units include a plurality of blocks, each block representing data for one of a plurality of channels at a particular range of locations within the received content; the plurality of coding units includes a plurality of channels; the plurality of coding units include a plurality of pixels, each pixel representing data for a plurality of channels at a particular location within the received content; Recovering the approximation of the value in the latent code space comprises: recovering a prior standard deviation encoded within a hyper-latency, the hyper-latency including an initial portion of the encoded content; or recovering the order in which the plurality of coding units were compressed, the order being included as side information associated with the encoded content; The method of claim 10, comprising one of:

15. A system comprising: a memory having executable instructions stored in the memory; a processor configured to execute the executable instructions, the executable instructions causing the system to perform a method according to any one of claims 1 to 14; A system comprising:

Images