Sequential data compression using artificial neural networks

Abstract

Some aspects of the present disclosure provide techniques for compressing content using neural networks. An exemplary method generally includes receiving content for compression. The content is encoded into a first latent code space via an encoder implemented by an artificial neural network trained to generate a latent space representation of the content. A first compressed version of the encoded content is generated using a first quantization bin size of a series of quantization bin sizes. A refined compressed version of the encoded content is generated by scaling the first compressed version of the encoded content to one or more second quantization bin sizes smaller than the first quantization bin size, conditional at least on values ​​of the first compressed version of the encoded content. The refined compressed version of the encoded content is output for transmission.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims the benefits and priority of U.S. Provisional Patent Application No. 63 / 141,322, “Progressive Data Compression Using Artificial Neural Networks,” filed on 25 January 2021, and claims priority to U.S. Application No. 17 / 648,808, filed on 24 January 2022, which is assigned to the assignee of this application and whose entire contents are incorporated herein by reference.

[0002] The aspects of this disclosure relate to machine learning, and more specifically, to the use of artificial neural networks to compress data such as video content. [Background technology]

[0003] Data compression techniques may be used to reduce the size of content for a variety of reasons, including improving storage and transmission efficiency and conforming to intended use (e.g., appropriate data resolution for the size of a device's display). Data compression may be performed using lossy techniques such that the decompressed version of the data is an approximation of the compressed original data, or by using lossless techniques such that the decompressed version of the data is equivalent to the original data.

[0004] Generally, lossless compression may be used when data should not be lost during compression, such as in the compression of file archives. In contrast, lossy compression may be used when an exact reproduction of the original data is not required (for example, in the compression of still images, videos, or audio, where some data loss may be acceptable, such as loss of detail in color data or loss of audio frequencies in the extreme audible spectrum).

[0005] Data compression methods are often defined or based on a fixed rate of compression (e.g., a single bit rate), which makes these compression methods inflexible with respect to varying data types and compression needs. That is, for any given compression method, data is generally compressed at a specific bit rate regardless of whether the data is more suitable for a higher compression bit rate or a lower compression bit rate. For example, in an image that does not contain fine details, a fixed bit rate compression method may use more bits than necessary to represent the information within the image to compress these images, even when the image is not suitable for compression using a more lossy compression method (and accordingly, a lower bit rate). Similarly, a more detailed image may be compressed at a bit rate that is too low to be reproduced satisfactorily. Thus, traditional methods often involve a trade-off in the design of fixed compression methods, which are neither dynamic nor adaptive. Summary of the Invention Problems to be Solved by the Invention

[0006] Accordingly, what is needed is an improved technique for adaptively compressing content. Means for Solving the Problems

[0007] Some aspects provide a method for compressing content using a neural network. An exemplary method generally includes receiving content for compression. The content is encoded into a first latent code space via an encoder implemented by an artificial neural network. A first compressed version of the encoded content is generated using a first quantization bin size of a series of quantization bin sizes. A refined compressed version of the encoded content is generated by scaling the encoded content of the first compressed version to one or more second quantization bin sizes smaller than the first quantization bin size, conditioned on at least the value of the encoded content of the first compressed version. The refined compressed version of the encoded content is output.

[0008] Some aspects provide a method for decompressing compressed content using a neural network. An exemplary method generally includes receiving encoded content for decompression. An approximation of a value within a latent code space is recovered from the received encoded content by decoding the code from a series of quantization bin sizes, the series of quantization bin sizes including a first quantization bin size and one or more second quantization bin sizes smaller than the first quantization bin size. A decompressed version of the encoded content is generated by decoding the approximation of the value within the latent code space via a decoder implemented by an artificial neural network. The decompressed version of the encoded content is output.

[0009] Other embodiments provide a processing system configured to perform the methods described above and the methods described herein; a non-temporary computer-readable medium comprising instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the methods described above and the methods described herein; a computer program product embodied on a computer-readable storage medium comprising code for performing the methods described above and the methods described herein; and a processing system comprising means for performing the methods described above and the methods described herein.

[0010] The following description and related drawings detail some exemplary features of one or more embodiments.

[0011] The attached figures illustrate some of one or more embodiments and should therefore not be considered as limitations of the scope of this disclosure. [Brief explanation of the drawing]

[0012] [Figure 1] This illustrates an exemplary neural network-based data compression pipeline. [Figure 2] This figure shows an exemplary pipeline for compressing and decompressing content using an encoder and decoder implemented as an artificial neural network, and continuous scaling of the compression bitrate. [Figure 3] This figure shows an example of latent scaling of the quantization width in continuous control of the compressed bitrate according to an aspect of the present disclosure. [Figure 4A] This figure shows an example of quantization bin sizes used to achieve different compression bitrates according to aspects of this disclosure. [Figure 4B] This figure shows an example of quantization bin size levels used to achieve different compression bitrates, where the bins at the quantization bin size level have unequal sizes, according to an aspect of the present disclosure. [Figure 5]This figure shows an example of nested quantization based on quantized code in a finer quantization bin size, subject to quantized code in a coarser quantization bin size, according to an aspect of the present disclosure. [Figure 6] This figure shows an exemplary operation for compressing content received through a compression pipeline using sequential coding, according to an aspect of the present disclosure. [Figure 7] This figure shows an exemplary operation for decompressing encoded content according to an aspect of the present disclosure. [Figure 8] This figure shows an example of channel-wise sequential coding using different bitrates for different channels, according to an aspect of this disclosure. [Figure 9] This figure shows an effective quantization grid for sequential coding according to an aspect of the present disclosure. [Figure 10] This figure shows a quantization grid for sequential coding having alignment to the finest quantization grid according to an aspect of the present disclosure. [Figure 11] This figure shows an exemplary result of data compression using sequential coding according to an aspect of this disclosure. [Figure 12] This figure shows exemplary results of data compression based on different orderings of coding units according to aspects of this disclosure. [Figure 13] This figure shows exemplary results of data compression using sequential coding and different orderings of coding units according to aspects of the present disclosure. [Figure 14] This figure shows an exemplary neural network-based data compression pipeline used to decompress data, according to aspects of this disclosure. [Figure 15] This figure shows an exemplary implementation of a processing system in which sequential coding and decoding of content can be performed according to the aspects of this disclosure. [Modes for carrying out the invention]

[0013] For ease of understanding, the same reference numerals are used to designate identical elements common to multiple drawings, where possible. It is intended that elements and features of one embodiment may be usefully incorporated into other embodiments without further description.

[0014] Aspects of this disclosure provide techniques for sequentially compressing content using an artificial neural network, such that a single model can be used to encode content at varying levels of bitrate or quality.

[0015] Neural network-based data compression systems can be used to compress various types of data. For example, neural network-based data compression can be used to compress various types of content that are suitable for compression. This content may include, for example, video content, image content, audio content, sensor content, and other types of data that are suitable for compression. Generally, neural network-based data compression may compress content using a bitrate determined a priori based on the trade-off between the size of the encoded content and the distortion (the difference between the original content and the decompressed content). In many data compression systems, a higher bitrate (e.g., a larger number of bits used to represent the content being compressed) may be associated with lower distortion, while a lower bitrate may be associated with higher distortion. The trade-off between distortion D and bitrate R is given by the formula:

number

[0016] However, typical neural network-based data compression is not suitable for large-scale deployments for various reasons. For example, in many machine learning-based compression schemes, the models may need to be trained to support various bitrates. That is, the first model may be trained for a low (baseline) bitrate, the second model for a second bitrate higher than the baseline bitrate, the third model for a third bitrate higher than the bitrate of the second model, and so on. In other machine learning-based compression schemes, the encoder and decoder networks may rely on a β parameter so that a single model can adapt to various rate-distortion tradeoffs. Other machine learning-based compression schemes may be trained to adjust the quantization step size of the generated latent. However, these models cannot effectively learn compression schemes that allow for variable coding of content. This variable coding may also be achieved through sequential coding schemes or coding schemes that allow data to be compressed using multiple bitrates, so that some data can be coded and decoded using lower bitrates, while other data (for example, parts of an image with more detail that should be coded in a way that results in a more faithful reconstruction of such details when decoded) can be coded and decoded using higher bitrates dynamically to address the differences in the data being compressed.

[0017] Aspects of this disclosure provide techniques that enable sequential coding (and sequential compression) of content using a single model. In sequential coding of content, higher bitrate codes may be generated based on lower bitrate codes, thereby sequentially compressing the encoded and compressed versions of the input data using multiple bitrates without requiring the input data to be compressed using multiple models. The compressed data can then be restored using any of the multiple bitrates, depending on various considerations such as the processing power of the device decompressing the data and the amount of detail required in the decompressed data. Furthermore, sequentially compressed data may be stored in a single file for multiple versions, each compressed at a different bitrate, which may improve the efficiency of storage and transmission of the compressed data.

[0018] Exemplary neural network-based data compression pipeline Figure 1 shows an exemplary neural network-based data compression pipeline 100 according to an aspect of the present disclosure. As shown in the figure, pipeline 100 receives content x111 for compression and approximates the content x111

number

[0019] In the decoding side 120 of pipeline 100, the entity decoder 122 reconstructs the quantized version of the content and dequantizes it using the inverse quantization scheme (Q -1 )124 is an approximate code

number

number

number

number

[0020] The training loss associated with neural network-based data compression is the sum of the amounts of strain (for example, the approximation of content x111 and content x111).

number

[0021] In general, independent models may be trained to acquire various bitrate options to compress content. However, such independent models cannot sequentially encode content because the models are trained separately and are not related. Therefore, these independent models are non-sequential models in which a single encoder-decoder is used and various parameters can be used to manipulate the rate-distortion tradeoff of the independent models. Furthermore, variable bitrate solutions may involve generating, transmitting, and / or storing multiple copies of the input encoded versions at various quality levels, which may increase the amount of data generated, transmitted, and stored in the data compression operation.

[0022] Aspects of this disclosure provide sequential compression of data using a single encoder-decoder model. Generally, data compression may be achieved using coding techniques that encode data into a more compact representation. These coding techniques, referred to herein as sequential coding, but also known as encoded coding or scalable coding, allow content to be encoded with each of multiple bitrates embedded at once. By encoding content with each of multiple bitrates embedded at once, dynamic control of the compression bitrate is simplified, thereby eliminating the need to generate multiple encoded versions of data to support different compression bitrates (and therefore different levels of compression quality preservation).

[0023] For example, the bitrate of broadcast content may be dynamically adapted (in response to, for example, available throughput, latency, content complexity, and the amount of detail to be preserved in the compression of this content). Furthermore, sequential coding of content may enable reduced transmission and storage costs by providing a single version of compressed content that can be decoded using various bitrates, instead of multiple versions of compressed content generated for each of multiple supported bitrates.

[0024] To enable sequential coding of content such that the content is encoded with each of multiple bitrates embedded simultaneously, the latent space code y representing the content x may be encoded using a nested quantization model in which codes associated with finer quantization levels (and accordingly, higher bitrate compression) are embedded within codes associated with coarser quantization levels (and accordingly, lower bitrate compression). As will be described in more detail herein, nested quantization may also allow codes associated with finer quantization levels to be conditional on codes associated with coarser quantization levels, thereby enabling the data to be sequentially encoded to finer quantization levels, and accordingly, the bitrate and quality of the decompressed data to increase sequentially.

[0025] In nested quantization models, a series of quantization bin sizes may be learned, starting with a high-bitrate model. Each quantization bin size in the series may be associated with a specific parameter (e.g., a value of β). Starting with the coarsest quantization bin (i.e., the quantization bin associated with the lowest bitrate), the latent code space y representing content x may be sequentially coded into finer quantization bins. As will be described in more detail herein, the bits for a particular quantization bin may be represented as the sum of the bits for the coarsest quantization bin and each of the sequentially finer quantization bins, up to a particular quantization bin, according to a chain law of quantized probabilities. In general, at each quantization level in a nested quantization model, probabilities may be associated with each code in a population of possible codes, and the code with the highest probability may be selected as the code to which the data is compressed at that quantization level. Based on the chain law, it can be seen that the code to which data is compressed at any given quantization level N may also be expressed as a function of the code to which data is compressed at a lower quantization level than N (i.e., to a quantization level associated with a coarser quantization bin). For example, the bits for the finest quantization bin (i.e., the Nth of N quantization bins) are given by equation

[0026]

number

[0027] It can also be expressed as follows, and in the above equation, P(y N ) is the probability mass under a distribution curve in which the compressed input data and associated code are located within the Nth quantization bin, and P(y N |y N-1 ) is the probability mass under a distribution curve where the code associated with the input data is located in the Nth quantization bin, given the code associated with the input data in the N-1th quantization bin. In other words, the bit for the finest quantization bin (i.e.,

number

number

number

number

[0028] Furthermore, as will be explained in more detail below, sequential coding may be used in channel-based latent ordering. In channel-based latent ordering, the quantization bin size may be sequentially refined across different channels in the data being compressed.

[0029] For example, in video content represented by luminance (Y), blue difference (Pb), and red difference (Pr) channels, different bin sizes may be used for the Y, Pb, and Pr channels. In another example, for visual content represented by chrominance channels (e.g., red (R), green (G), and blue (B) color channels), different bin sizes may be used for the R, G, and B color channels. The ordering of these channels may be defined by classifying the channels based on the ratio of the calculated distortion difference to the rate difference for each channel, such that coarser quantization bins are used for channels where higher compression does not result in significantly more distortion, and finer quantization bins are used for channels where higher compression results in significantly more distortion. Thus, by ordering the channels and encoding the channels using different quantization bins according to the ordered order, multichannel content can be encoded so that the channel that has the greatest impact on the quality of the resulting decompressed data may be compressed using the highest quality compression, and the channel that has a less impact on the quality of the resulting decompressed data may be compressed using lower quality compression. This may reduce the size of the compressed representation of the input data, which may reduce the storage and transmission costs of the compressed data.

[0030] Figure 2 shows further details of the pipeline for compressing and decompressing content (for example, shown in Figure 1) using an encoder and decoder implemented as an artificial neural network, and continuous scaling of the compression bitrate.

[0031] In pipeline 200, encoder 202 encodes the input x into a latent space code y. To continuously control the latent bitrate, the latent space code y is processed via hyperencoder 204, which may be another network used to generate weights for the encoder neural network, in order to generate a hyperlatent representing the input used to control the weights within the encoder neural network. The hyperlatent may also be used as information to capture spatial dependencies in the data, and may be used to round (quantize) y into a rounded (quantized) representation of y, shown as [y] (a simplified version of quantization when no multiplier is applied, such as s=1, in such cases,

number

[0032] On the decoder side, the hyperdecoder 206, which may be a network used to generate weights for the decoder neural network, decodes the hyperlatency to determine the entropy model used to code the rounded (quantized) latent space code [y]. The entropy model may be, for example, a probabilistic model used to generate the probability distribution used on the encoder side of pipeline 200 to encode y into code [y]. Based on the entropy model, decoder 208 decodes from [y]

number

number

[0033] In some cases, a scaling factor may be applied to pipeline 200 to affect the bitrate and amount of compression applied when generating a compressed version of content x. In this case, the scaling parameter s may be applied in scaler 210 before quantization (rounding y to a quantized value) and in rescaler 212 before decompression, so that the quantized and scaled version of the latent space code y representing content x may be expressed as y / s. By scaling y by the scaling factor s, the quantization bin size used to round (quantize) y may be changed from a baseline value (e.g., 1) to different values ​​corresponding to finer or coarser degrees of quantization, and therefore finer or coarser degrees of compression. This allows the model to be trained in relation to the quantization bin size and enables the use of different strain-rate tradeoffs.

[0034] Exemplary scaling of quantization width in sequential coding of data Figure 3 shows an example of latent scaling of the quantization width in continuous control of the compressed bitrate. As shown in the figure, for a given value y, the quantization of y results in rounding to the nearest individual point [y]306 before scaling by s. To quantize y to [y]306 and transmit it via entropy coding, the system can compute a probability mass 302 within a probability distribution 300 between the upper and lower bounds of the quantization bin 304 in which y exists. The probability mass is given by the formula

number

[0035] When scaling is applied, the quantization bin size may be changed to a different value. For example, a multiplier of 2 s may double the width of the quantization bin size, as shown in the scaled probability distribution 310, and halve the number of possible values ​​that y can code (i.e., shown by the dots below the probability distribution 310). After scaling by s, for a given value y, the quantization of y results in rounding to nested individual points 2[y / 2] 316. To quantize and scale y to 2[y / 2] and transmit it via entropy coding, the system can compute the probability mass 312 in the scaled probability distribution 310 between the upper and lower bounds of the scaled quantization bin 314 in which y resides. The probability mass is given by the formula

number

[0036] This corresponds to a larger quantization interval and a smaller number of bits than those shown within probability distribution 300.

[0037] The inversely quantized latent is given by the equation

number

number

number

number

[0038] Figures 4A and 4B show examples of quantization bin sizes used to achieve different compression bitrates according to aspects of the present disclosure.

[0039] In particular, Figure 4A shows a series of quantization levels 400A with bin sizes s1 and s2 to which the latent code y can be mapped. Since s1 and s2 have different bin sizes, each quantization level may have different midpoints, and with respect to the midpoint, the values ​​may be quantized or rounded, and may have different numbers of different sized bins. Therefore, a quantization level associated with bin size s1 may have a lower effective bitrate than a quantization level associated with bin size s2. Thus, data compressed using quantization level s1 may be smaller than data compressed using quantization level s2, but may have lower quality when decompressed.

[0040] More generally, for a number N of quantization bin sizes in a series of quantization bin sizes, the first quantization bin size s1 corresponds to the largest quantization bin size, and subsequent quantization bin sizes s NWhen decreasing towards the quantization bin size, the quantization bin size can be represented as s1>s2>s3>...>s N Accordingly, the bit rate for the quantization bin size can be represented as β1<β2<β3<...<β N as follows.

[0041] In some embodiments, the bin size for any level n of quantization within the set of quantization levels {1, 2,..., N} need not be the same throughout the quantization levels. For example, FIG. 4B shows a series 400B of quantization levels 402, 404, 406, 408 having different quantization bin sizes. As shown, quantization level 402 has the coarsest quantization bin size, quantization level 404 has a first intermediate quantization bin size, quantization level 406 has a second intermediate quantization bin size finer than that of quantization level 404, and quantization level 408 may have the finest quantization bin size.

[0042] Furthermore, Figure 4B shows that quantization bin sizes can differ even within a single quantization level. For example, at quantization level 406, the central bin 416 may have a different size from the other bins. While quantization level 404 shows a single bin with a different size from the other bins, it should be noted that quantization levels may contain bins of varying sizes at various locations. For example, a quantization level may have a large central bin and progressively smaller bins on either side of the midpoint (e.g., a smaller non-central bin and a larger central bin). In another example, a quantization level may have a larger bin inserted between smaller bins. In general, selecting a larger central bin may improve compression performance (e.g., improve the quality of the decompressed image by reducing distortion relative to the original version of the image). This is because, in a Gaussian distribution, most of the original probability masses of probability distribution 300 may be centered around the center of the probability distribution. Therefore, increasing the bin size around the center of the probability distribution has a greater effect on rate reduction than increasing the bin size for bins that are not in the center (for example, where the rate is given by the formula).

number

[0043] Figure 5 shows an example of nested quantization 500. In general, nested quantization may allow the quantization of data using finer quantization bin sizes to be defined such that the quantization of the code is quantized within a coarser quantization bin size. As described above, quantizing a latent space code y using quantization bin size s1 generally results in a lower bitrate than quantizing a latent space code y using a smaller quantization bin size s2. In this case, given a latent space code y, y has an upper bound for quantization levels with quantization bin size s1.

number

number

number

number

[0044] More generally, to scale y to any quantization level i, i is, formula

number

number

number

number

number

[0045] In nested quantization, the probability mass of y quantized at the lowest quantization level (and therefore the largest quantization bin size) may represent the initial quantization of the latent space code y for the compressed content x. That is, the probability mass of y corresponds to the probability mass associated with one of the codes at the lowest quantization level into which y is mapped. Subsequent quantizations of y for higher quantization levels (and therefore smaller quantization bin sizes and higher bit rates) may be calculated as conditional probability masses conditioned on coarser quantization values. For example, to quantize y using a second quantization bin size s2, the probability mass of the quantized value y2 conditioned on y1 is given by:

number

[0046] The number of bits used to represent data compressed using the finest quantization bin (for example, for the quantization level with the smallest bin size and therefore the largest bitrate) is given by the formula:

number

number

[0047] That is, for any given quantization bin size, the conditional probability for that quantization bin size may be conditional on the conditional probability calculated for a larger quantization bin size. Since the conditional probability of a quantized code [y] at any given quantization bin size may be conditional on the conditional probability of a code in a larger quantization bin size, a code at any given quantization bin size may be derived based on a chain law using a code generated at a larger quantization bin size. Thus, a single model may be used to encode and compress content at any compression bitrate, and multiple supported compression bitrates may be embedded within the compressed content. Furthermore, the compressed content may be decompressed from any given compression bitrate, which may allow devices to decompress the data based, for example, on the computing power of each device.

[0048] In general, in nested quantization, encoding may occur at N levels (where N is the number of quantization levels to which y may be encoded). In general, y may first be quantized using the quantization level associated with the coarsest quantization bin, and the quantization of y may be iteratively refined using successively finer quantization bin sizes. In general, the additional information obtained from quantizing y to quantization levels with finer quantization bin sizes is used in the conditional probability expression.

number

number

number

number

[0049] In some aspects, thoughtless methods can result in codeword lengths that are longer than the codeword lengths generated at the finest quantization levels, given the increased complexity (described in more detail below). For example, with a thoughtless method, the code generated by N stages forms a bitstream that embeds the data into data at N different bitrates, and the total length of the bitstream is given by the formula

number

number

number

number

number

[0050] To reduce complexity, a fully nested set of quantization levels may be defined such that the center points of the quantization bins at the coarser quantization levels are a subset of the center points of the quantization bins at the finer quantization levels, as described below. Using fully nested quantization levels, a set of grid points in the coarser quantization bins may be a set of points in the finer quantization bins. That is, quantization may be defined according to the following equation: I n =I n+1 ∩[y - (s k ), y + (s k )]=[y - (s k ), y + (s k )].

[0051] The bitstream length formula explained above can be simplified to the following formula.

[0052]

number

[0053] By using a fully nested set of quantization levels, the performance of the highest bitrate model within the coding model may be preserved while simplifying the data compression process.

[0054] In some embodiments, the choice of magnification may effectively implement various types of compression. For example, s i-1 =2s i In this case, the resulting compression method may be binary bit plane coding. i-1 ga s i When the number is a multiple of an integer, the calculation of the upper and lower bounds of the quantization bins may be simple calculations, and therefore may enable less processor-intensive data compression and decompression.

[0055] Exemplary method for sequential coding of data for data compression Figure 6 shows an exemplary operation 600 that may be performed by the system to compress content received through a compression pipeline, such as pipeline 100 shown in Figure 1 or pipeline 200 shown in Figure 2, using sequential coding. Operation 600 may be performed by a system having one or more processors, such as system 1200 in Figure 12, which implements a compression pipeline including a neural network-based encoder and quantizer that implement a learned nested quantization scheme.

[0056] As shown in the figure, operation 600 may begin in block 610 by receiving content for compression. The received content may be single-channel content such as a data stream, or it may be multi-channel content (i.e., content with multiple data channels) in which different channels can be compressed separately. Multi-channel content may include, for example, audio content with multiple spatial channels (left / right stereo, surround sound content, etc.), video content with luminance channels and / or chrominance channels (YPbPr, RGB, etc.), audiovisual content with independent visual and audio channels, and so on.

[0057] In block 620, the content is (for example, the encoder 112(g shown in Figure 1) a The content is encoded into a latent code space via ). To encode the content into a latent code space, an encoder implemented by an artificial neural network trained to generate a latent space representation of the content may be used. In some embodiments, encoding the received content x into a code y in the latent code space may be lossless mapping of the received content x to the code y. Compression, and the loss (or distortion) resulting to the original received content x, may be obtained by quantizing the code y.

[0058] In block 630, the encoded content of the first compressed version is generated (for example, via the quantizer 114(Q) shown in Figure 1). To generate the encoded content of the first compressed version, the code y in which the content is encoded may be quantized using a first quantization bin size from a set of quantization bin sizes associated with a first bitrate. For example, the first quantization bin size may be the coarsest quantization bin size from a set of quantization bin sizes, resulting in compression at the lowest bitrate from a set of quantization bin sizes associated with a set of quantization bin sizes.

[0059] In block 640, a refined, compressed version of the encoded content is generated (for example, via the quantizer 114(Q) shown in Figure 1). In one example, to generate a refined, compressed version of the encoded content, the first compressed version of the encoded content is scaled to one or more second quantization bin sizes smaller than the first quantization bin size, provided that at least the values ​​of the encoded content are smaller. Generally, each of the one or more second quantization bin sizes corresponds to a bitrate higher than the first bitrate. That is, each of the one or more second quantization bin sizes may be smaller than the first quantization bin size.

[0060] In block 650, the refined, compressed, encoded content is output for transmission (for example, via entity coder 116 (EC) as shown in Figure 1).

[0061] Figure 7 shows an exemplary operation 700 that may be performed by a system for decompressing encoded content. Operation 700 may be performed by a system having one or more processors, such as system 1200 in Figure 12, which implements a compression pipeline that includes a neural network-based encoder and quantizer that implement a learned nested quantization scheme, such as pipeline 100 shown in Figure 1 or pipeline 200 shown in Figure 2.

[0062] As shown in the figure, operation 700 may begin in block 710 where the encoded content is received for decompression.

[0063] In block 720, an approximation of the code in the latent code space is reconstructed from the received encoded content.

[0064] In some cases, the code is an approximation.

number

number

[0065] In general, a set of quantization bin sizes may allow decompression of content using a single model for any bitrate, based on the tolerance for distortion in the decompressed version of the encoded content. As described above, the code representing the compressed data may be reconstructed from any quantization level using a chain law, where the code at a given quantization level may be defined as the code obtained at a lower quantization level (e.g., a quantization level with a smaller quantization bin size than that at the given quantization level). The amount of distortion in the decompressed version of the encoded content may be inversely proportional to the bitrate associated with the smallest quantization bin size used to reconstruct an approximation of the code. That is, the lowest bitrate associated with the largest quantization bin size among the set of quantization bin sizes may have the highest amount of distortion, and the distortion may decrease as progressively smaller quantization bin sizes are used in reconstructing the approximation of the code.

[0066] In block 730, the decompressed version of the encoded content is sent to decoder 126(g) as shown in Figure 1. s It is generated by decoding an approximation of the code in the latent code space via a decoder implemented by an artificial neural network, such as ). The decoder implemented by an artificial neural network may complement, for example, an encoder implemented by an artificial neural network, and may be used to encode content into the latent code space as described above with respect to Figure 6.

[0067] In block 740, the decompressed version of the encoded content is output. In some embodiments, the decompressed version of the encoded content may be output to one or more output devices, such as a display or audio device connected to or integrated with the system, for playback to the device's user. In some embodiments, the decompressed version of the encoded content may be output to one or more other computing systems for output to their users.

[0068] In some embodiments, sequential coding may be used to compress multichannel data using different levels of compression for each channel in the multichannel data (and thus achieving different levels of distortion). As described, channels in multichannel data may include luminance channels and / or chrominance channels in visual content, spatial sound information in multichannel audio, etc. Each channel may have a different amount of data or a different impact on the final audiovisual representation (rendition) of the content when decompressed, and therefore it may be useful to encode (compress) each channel using different amounts of compression. The selection of the bitrate used to compress or decompress the data may be based, for example, on congestion control or bandwidth adaptive features controlled by the application layer in the network stack. For example, if a content server detects low bandwidth between the content server and the requesting device, the content server may select a lower bitrate (e.g., compression using a larger quantization bin size), and similarly, if a content server detects high bandwidth between the content server and the requesting device, the content server may select a higher bitrate (e.g., compression using a smaller quantization bin size).

[0069] For example, in multi-channel video data within a YPbPr space (i.e., having a luminance channel and two color channels), the luminance channel may be considered the most important channel because it carries the most visual information in multi-channel video data. Therefore, to balance the quality and amount of compression applied to the video content, it may be desirable to encode the luminance channel using the highest bitrate and the color channels using a lower bitrate. Thus, in encoding multi-channel video data within a YPbPr space, a neural network might encode each of the Y, Pb, and Pr channels separately to create different latent space codes y Y , y Pb , and y Pr Even so, each of these latent space codes may be encoded separately.

[0070] In another example, in image data carried across multiple color data channels (for example, within the RGB chrominance color space), some color data may have a greater impact on the visual representation of the decompressed content than others. For example, one color channel may be encoded using a higher bitrate compression than the others, based on a priori known sensitivity to different colors. For instance, in RGB data, the green color channel may be compressed using a higher bitrate than that used for the red and blue color channels, because the human eye is known to be more sensitive to green color data than to other color data.

[0071] To perform channel-wise sequential coding, channels may be ordered according to the amount of compression applied to each channel. The ordering may be determined based on the difference in distortion ΔD and the difference in bitrate ΔR. For example, the ordering may be calculated based on the ratio of each channel.

number

[0072] Exemplary channel-wise sequential coding Figure 8 shows an example of channel-wise sequential coding 800, which uses different bitrates for different channels.

[0073] As shown in the figure, in coding 800, the bitstream may be generated for each of the C channels. For these channels 1 to C, the quantization bin size for the code generated for each of the channels c at a given bitrate b is:

number

number

number

number

[0074] For each channel, the code y in the channel's latent spatial representation. c This can be compressed at the coarsest quantization bin size and output for transmission. To achieve nested quantization for each channel, code y c The quantization value of y at a coarser bin size is subject to the quantization value of y at a coarser bin size, and it can be compressed at a finer quantization bin size and output for transmission. Additional code information for the finer quantization bin size y c By outputting, the quality of the compressed content can be progressively improved from the amount of compression reference corresponding to the compression at the coarsest quantization bin size, and the amount of improvement is the additional code y output for transmission. c It can be controlled based on the amount (i.e., the amount generated for quantization bin sizes that are finer than successive).

[0075] In some cases, additional sequential coding can be achieved by outputting additional quantization information for some of the channels c in the set of C channels.

[0076] For example, each channel c may be compressed at the coarsest quantization bin size and output for transmission. Additional code y is generated for progressively finer quantization bin sizes for channels that have a greater impact on the quality of the resulting decompressed data. cHowever, (for example, for a subset of channel c) may be output. For any quantization level beyond the level associated with the coarsest quantization bin size, the code may be generated for a subset of C channels so that additional compression is not performed on channels that have little impact on the resulting quality of the decompressed data. The subset of C channels generated for any quantization level N may be calculated, for example, using a greedy technique (for example, decreasing the number of channels encoded using the increased quantization level by 1 for each increase in the quantization level) or for each channel

number

[0077] In Figure 8, each level shown in coding 800 represents a different quantization level and the corresponding bitrate used to compress the C channel. As shown in the figure, each C channel has the lowest quantization level (for example, quantization level 1), and for that level, the value is given by the formula

number

number

[0078] Exemplary Quantization Grid for Sequential Coding in Data Compression Figure 9 shows an example of an effective quantization grid 900 for sequential coding of content. As shown in the figure, three quantization levels are used to quantize the latent space code y representing the content x. The first quantization level is associated with quantization bin size s1, the second quantization level is associated with quantization bin size s2, and the third quantization level is associated with quantization bin size s3. For the code y generated by quantizing y at the quantization level associated with quantization bin size s1, the number of bits transmitted is given by the formula

number

number

[0079] At a further quantization level, the code y3 may also be represented by the intersection of the upper bound of the bin where y3 is located and the lower bound of y2, and the equation

number

[0080] Therefore, the effective quantization bin size may be finer than the finest bin used for content compression. Thus, the total number of bits transmitted is given by the formula

number

[0081] When nested quantization is applied thoughtlessly, performance degradation in data compression may occur. This is because the effective quantization bin may be smaller than the finest bin actually used to quantize the latent spatial representation y of content x. Performance degradation may be seen as a reduction in the increase in quality, measured by the peak signal-to-noise ratio (PSNR), for an increase in the effective compression bitrate.

[0082] To mitigate performance degradation caused by the thoughtless application of nested quantization, the quantization grid 1000 for sequential coding may be aligned to the finest quantization grid 1002, as shown in Figure 10.

[0083] To align the quantization grid 1004 to grid 1002 for the finest quantization grid size, multipass quantization may be used to quantize multiple coarser quantization bins based on the finest quantization bin size. By applying multipass quantization, intersections of smaller quantization beams are identified (for example, effective quantization bin 906 is identified), so the effective quantization bin size is not smaller than the finest quantization bin size in order to simplify processing and to avoid performance degradation due to the use of a variable effective quantization bin size. The midpoint 1006 of the quantization grid 1000 may be the midpoint of the central quantization bin in the grid for each quantization level, and may be the midpoint of the central quantization bin in the effective quantization grid 1004.

[0084] Figure 11 shows an example of data compression results using sequential coding.

[0085] Graphs 1100A to 1100F show the relationship between PSNR per pixel and the bitrate for various compression techniques used to compress the sample image. As shown in Graph 1100A, compression using the sequential coding technique described herein may provide a PSNR of 26.59 dB at a low bitrate of 0.11 bits per pixel used to compress the sample image. Graph 1100B shows that the sequential coding technique described herein may provide a PSNR of 31.02 dB at a bitrate of 0.34 bits per pixel used to compress the sample image. Graph 1100C shows that the sequential coding technique described herein may provide a PSNR of 33.52 dB at a bitrate of 0.60 bits per pixel used to compress the sample image. Graph 1100D shows that the sequential coding technique described herein may provide a PSNR of 35.99 dB at a bitrate of 0.90 bits per pixel used to compress the sample image. Graph 1100E shows that the sequential coding technique described herein may provide a PSNR of 37.70 dB at a bitrate of 1.21 bits per pixel used to compress the sample image. Finally, Graph 1100F shows that the sequential coding technique described herein may provide a PSNR of 39.69 dB at a bitrate of 1.48 bits per pixel used to compress the sample image. In these examples, it can be seen that the quality of the compressed image (represented by the PSNR measurement across the compressed image) is higher for each run bitrate compared to nested dropout sequential coding. Furthermore, for each run bitrate, the quality of the compressed image may be comparable to the quality of compressed images produced using various non-sequential coding schemes used for each bitrate used by the a priori defined model to compress the data.

[0086] Exemplary coding unit ordering for sequential data compression In some embodiments, data compressed using the techniques described herein may be divided into multiple coding units, each of which may be compressed separately. For example, a coding unit may be a channel, a pixel in an image (e.g., data for each of multiple channels at a particular location in image or video content), a block of data (e.g., one or more channels for an nxm block of pixels in image or video content), or a single element (e.g., data for a single channel at a particular location in image or video content). To facilitate sequential coding, and considering that each coding unit may have different amounts of information and different sensitivities to compression losses, each coding unit may be refined gradually and separately. Sequential coding may be divided into two phases. The first phase encodes latent variables (e.g., code y representing the input x generated by an artificial neural network-based encoder) from the largest quantization bin (e.g., the lowest quantization level) to the smallest quantization bin (e.g., the highest quantization level). In the second phase, the refinement between adjacent quantization levels may be performed sequentially for each coding unit, so that each truncation point in the resulting embedded bitstream, representing the boundary between coding units, is associated with a sequential change in the quantization level.

[0087] In practice, a continuous latent may be coded from an infinitely large quantization bin, thereby quantizing each latent variable to a given midpoint value. As a result, the inverse quantized latent sent to the decoder may be a prior average. The ordering of coding units for refinement may be discovered from prior quantization so that coding units in the current quantization bin may be decoded in an appropriate quantization bin. Based on the ordering of coding units, coding units may be refined from the largest quantization bin to the smallest quantization bin. To simplify processing, coding is performed such that coding units coded using the highest quantization level (and correspondingly the largest quantization bin size) are coded first, followed by coding units coded using lower quantization levels (and correspondingly smaller quantization bin sizes) after the coding units using the highest quantization level.

[0088] In sequential coding, a code y, which may be a tensor in the latent space, consists of N coding units {y1, ..., y}. N It may be divided into}. Elements within a coding unit may be refined together and may correspond to truncation points or points in space where the lowest strain (e.g., compression loss) is achieved. Shape (C, H, W) 4 Various codings may be defined for a tensor in a latent space having . A single channel coding may correspond to a latent slice of size (1, H, W), a single pixel coding may correspond to a latent slice of size (C, 1, 1), and a single element coding may correspond to a latent slice of size (1, 1, 1).

[0089] Given a compression order ρ=(ρ1, ..., ρ N For the ordered coding unit y ρ =( y ρ1 ,...,y ρN ) but, s n From the magnification s n-1They may be scaled separately up to a factor of . In the mean space hyperprior model, the priors of the latent elements may be conditional on the hyperlatency. The increase in bitrate ΔR for refining the t-th coding unit due to the scaling described herein is given by equation

number

[0090] The reduction ΔD in strain may also be calculated similarly, or it may depend on other ordered coding units. The reduction ΔD in strain is given by: ΔD(y ρt |y ρ ) = D(y ρ (t-1)-D(y ρ (t)) It may also be expressed by, where (y ρ (t)=(y ρ≦t (s n-1 ), y ρ>t (s n )) and here D(y) = MSE(x, g s (y)) represents the strain on code y. Therefore, refining the ordered latent using the compression order ρ is given by equation

number

[0091] In general, the optimal order for ρ may be one in which the convex hull of H(ρ) is better than the convex hull of other orders of ρ (for example, a Pareto-optimal compression order).

[0092] Figure 12 shows the relationship between the amounts of coding loss involved in encoding content using different orderings. In Graph 1200, for simplicity, two coding units y1 and y2 are shown, but it should be understood that the data being encoded may have any number of coding units. As shown in the figure, y1 is

number

number

[0093] Distortion line 1202 shows the coding loss when y2 is encoded before y1 and y2 is encoded using a higher rate than y1. In contrast, distortion line 1204 shows the coding loss when y1 is encoded before y2 and y1 is encoded using a higher rate than y2. For compressions using either the order (y1, y2) or (y2, y1), the total distortion / rate loss may be the same. However, since y2 is less sensitive than y1 to changes in the compression rate (i.e., the reduction in distortion for any given increase in the compression rate for y2 is smaller than that for y1), it may be more efficient to encode y1 before encoding y2, as indicated by distortion line 1204 being lower than distortion line 1202.

[0094] Therefore, in order to compress content using sequential coding so that the content is optimally encoded, coding units may be classified in descending order of their respective strain-rate change rates. Since the amount of strain loss may depend on the order in which the units are coded, additional complexity differences may be introduced if the coding units are generated by a neural network-based encoder, such as encoder 112 shown in Figure 1. However, the strain-rate change rates calculated separately for each coding unit may be treated as approximations for the purpose of ordering the coding units for sequential coding.

[0095] Figure 13 shows illustrative results of data compression using sequential coding and different orderings of coding units.

[0096] Graph 1300 shows the relationship between the peak signal-to-noise ratio (PSNR) per pixel and the bitrate for sequential data compression, based on different coding units and classification criteria.

[0097] As explained, coding units may be defined for data of varying granularity. For a latent having a shape defined by some of the channel C, maximum dimension H, and width dimension W (e.g., a still image or video frame defined as multiple color space channels and spatial dimensions), the coding unit may be one of the C channel, a single pixel, a block of pixels for one of the C channels, or a single element within the latent (e.g., a value for one of the C channels at a particular location in the image). The classification criteria shown in Graph 1300 are the strain-rate change rate for each coding block, rate difference ΔR, and prior standard deviation ρ.

number

[0098] As illustrated, latent ordering and element-wise compression may achieve a higher PSNR at a given compression rate than latent ordering and channel-wise compression, while latent ordering and pixel-wise compression may achieve a significantly lower PSNR, except at higher compression bitrates. For compression using channel-based coding unit classification, the compression performance may be similar for classification by prior standard deviation, rate difference, or rate-distortion importance. However, for compression using element-based or pixel-based coding unit classification, the compression performance differs among different types of ordering. For example, for compression using element-based coding unit classification, ordering based on the rate difference metric may achieve better compression performance (e.g., a higher PSNR at a given bitrate) than ordering based on prior standard deviation.

[0099] Ordering coding units may impose some overhead on compression and decompression. For example, ordering coding units by prior standard deviation may allow compression to be performed without requiring additional information to reconstruct the compressed data, because the prior standard deviation may be known to the decoder once the hyperlatency is decoded. However, ordering coding units by a rate difference metric or a rate-distortion importance metric may allow for more precise ordering of coding units at the cost of imposing a bitrate overhead to carry the order in which the coding units are encoded. In some embodiments, if the order in which the coding units are encoded is considered important enough to accept the additional overhead of carrying the ordering information to the decoder as side information, various optimizations may be used to reduce the overhead associated with carrying this ordering information to the decoder. For example, larger coding units, such as blocks of pixels rather than individual pixels in a still image, may be used to compress the data, which may reduce the amount of side information carried. In another embodiment, the expected order may be learned by a machine learning model from the training data, and the expected order generated by the trained machine learning model may be transported to the decoder. In yet another embodiment, the ordering may be learned from other available information, such as from latents already decoded using a larger quantization bin size.

[0100] Figure 14 shows an exemplary neural network-based data compression pipeline 1400 in which side information is used to decompress the data.

[0101] As shown in the figure, the compression pipeline is shown in Figure 1 and includes the elements described above, as well as an approximation of the original content x from the compressed bitstream.

number

[0102] To generate information for a side channel that may be used to decode a compressed version of the latent space code y representing content x, hyperanalysis transformation 1402(h a ) may generate a hyperlatency z, which is quantized by quantizer 1404 and hyperprior distribution by entropy coder 1406.

number

number

number

[0103] The prior standard deviation σ and mean μ may be used by the entropy coder 116 and entropy decoder 122 to encode the quantized version of code y and to reconstruct the quantized version of code y from the bitstream representing the encoded quantized version of code y. Meanwhile, the prior mean μ is used to quantize y and to approximate the latent space code y to which the content x is mapped.

number

[0104] Exemplary processing system for sequential data compression Figure 15 shows an exemplary processing system 1500 for performing convolutional neural network processing, as described herein, for example, with respect to Figures 6 and 7.

[0105] The processing system 1200 includes a central processing unit (CPU) 1502, which in some examples may be a multi-core CPU. Instructions executed in the CPU 1502 may be loaded, for example, from program memory associated with the CPU 1502, or from a memory partition 1524.

[0106] The processing system 1500 also includes additional processing components tailored to specific functions, such as a graphics processing unit (GPU) 1504, a digital signal processor (DSP) 1506, a neural processing unit (NPU) 1508, a multimedia processing block 1510, a multimedia processing unit 1510, and a wireless connectivity component 1512.

[0107] NPUs, such as the 1508, are generally specialized circuits configured to perform all the necessary control and computational logic for executing machine learning algorithms, including algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), and random forests (RFs). NPUs are sometimes also called neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligence processing units (IPUs), vision processing units (VPUs), or graph processing units.

[0108] NPUs such as the 1508 are configured to accelerate the execution of common machine learning tasks such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-a-chip (SoC), while in others they may be part of a dedicated neural network accelerator.

[0109] An NPU may be optimized for training or inference, or in some cases, configured to balance performance between both. In an NPU capable of performing both training and inference, the two tasks may generally still be performed independently.

[0110] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation involving inputting existing datasets (often labeled or tagged), iterating through the datasets, and then tuning model parameters such as weights and biases to improve model performance. Generally, optimization based on incorrect predictions involves propagating backward through layers of the model and determining gradients to reduce prediction errors.

[0111] NPUs designed to accelerate inference are generally configured to operate on complete models. Therefore, such NPUs may be configured to process new data at high speed through a model that has already been trained to generate model outputs (e.g., inferences).

[0112] In one implementation, the NPU1508 is a part of one or more of the CPU1502, GPU1504, and / or DSP1506.

[0113] In some examples, the wireless connectivity component 1512 may include subcomponents for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing component 1512 is further connected to one or more antennas 1514.

[0114] The processing system 1500 may also include one or more sensor processing units 1516 related to any type of sensor, one or more image signal processors (ISPs) 1518 related to any type of image sensor, and / or a navigation processor 1520 which may include satellite-based positioning system components (e.g., GPS or GLONASS), as well as inertial positioning system components.

[0115] The processing system 1500 may also include one or more input and / or output devices 1522, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, or a microphone.

[0116] In some examples, one or more of the processors in the processing system 1500 may be based on an ARM or RISC-V instruction set.

[0117] The processing system 1500 also includes memory 1524 representing one or more static and / or dynamic memories, such as dynamic random access memory and flash-based static memory. In this example, memory 1524 includes a computer executable component that can be run by one or more of the processors of the processing system 1500 described above.

[0118] In particular, in this example, memory 1524 includes a latent space coding component 1524A, a sequential coding component 1524B, a sequential code recovery component 1524C, and a latent space decoding component 1524D. The illustrated components and other components not illustrated may be configured to perform various embodiments of the methods described herein.

[0119] In general, the processing system 1500 and / or its components may be configured to perform the methods described herein.

[0120] In particular, in other embodiments, embodiments of the processing system 1500 may be omitted, for example, when the processing system 1500 is a server computer. For example, the multimedia component 1510, wireless connectivity 1512, sensors 1516, ISP 1518, and / or navigation component 1520 may be omitted in other embodiments. Furthermore, embodiments of the processing system 1500 that train a model and use the model to generate inferences such as user authentication predictions may be described.

[0121] Exemplary clause Clause 1: A method for compressing content using a neural network, comprising: receiving content for compression; encoding the content into a first latent code space via an encoder implemented by an artificial neural network; generating a first compressed version of the encoded content using a first quantization bin size from a set of quantization bin sizes; generating a refined compressed version of the encoded content by scaling the first compressed version of the encoded content to one or more second quantization bin sizes smaller than the first quantization bin size, subject to the values ​​of at least the first compressed version of the encoded content; and outputting the refined compressed version of the encoded content.

[0122] Clause 2: The method of Clause 1, wherein the step of generating a refined compressed version of encoded content includes the steps of generating a first refined compressed version of encoded content by scaling the first compressed version of encoded content to a first finer quantization bin size, conditional on the values ​​of the first compressed version of encoded content, and generating a second refined compressed version of encoded content by scaling the first refined compressed version of encoded content to a second finer quantization bin size, conditional on the values ​​of the first refined compressed version of encoded content and the first compressed version of encoded content, wherein the second finer quantization bin size is smaller than the first finer quantization bin size.

[0123] Clause 3: The method of Clause 1 or 2, wherein the size of each quantization bin size in a set of quantization bin sizes is an integer multiple of the first quantization bin size.

[0124] Clause 4: Any method of Clauses 1 to 3, wherein the central bin for one of the quantization bin sizes in a set of quantization bin sizes has a larger bin size than the bins that are not central in the quantization bin size.

[0125] Clause 5: Any method of Clauses 1 to 4, wherein the step of generating an encoded content in a refined compressed version includes the step of generating a bitstream based on a set of conditional probabilities, each conditional probability in the set of conditional probabilities being associated with each quantization bin size in the set of quantization bin sizes other than the finest quantization bin size, and subject to a conditional probability calculated for quantization bin sizes greater than each quantization bin size.

[0126] Clause 6: Any method of Clauses 1 to 5, wherein the step of generating a refined compressed version of the encoded content includes the step of generating a probability mass of the encoded content for each quantization bin size in a set of quantization bin sizes, based on the cumulative distribution functions of the upper and lower bounds of each quantization bin in which the encoded content is located.

[0127] Clause 7: The method of Clause 6, wherein the probability mass for each quantization bin size in a set of quantization bin sizes is conditional on the probability mass for a quantization bin size in a set of quantization bin sizes larger than the quantization bin size for which each quantization bin size is located.

[0128] Clause 8: The received content includes content with multiple data channels, in any manner described in Clauses 1 through 7.

[0129] Clause 9: The method of Clause 8, wherein each of the multiple data channels is associated with a compression priority corresponding to the amount of compression used to compress each data channel.

[0130] Clause 10: The method of Clause 9, wherein multiple data channels include luminance channels and multiple chrominance channels in visual content, and the luminance channels are associated with a compression priority that is lower than the compression priority associated with the multiple chrominance channels.

[0131] Clause 11: The method of Clause 9, wherein the received content includes visual content to be compressed, and the multiple data channels include multiple color data channels within the visual content, and the first color data channel among the multiple color data channels that have the greatest impact on the quality of the encoded content of the compressed version is associated with a compression priority lower than the compression priority associated with the other color data channels.

[0132] Clause 12: The method of Clause 11, further comprising the step of identifying a first color data channel based on the amount of luminance data contained in each of a plurality of color data channels.

[0133] Clause 13: Any method of Clauses 9 to 12, further comprising the step of determining the compression priority associated with each of the multiple data channels, based on calculating a decrease in distortion and an increase in bitrate, when each data channel is encoded for each of the multiple bitrates associated with each quantization bin size in a set of quantization bin sizes.

[0134] Clause 14: The method of Clause 13, wherein the step of calculating the reduction in strain for each data channel includes the step of calculating the difference between the strain generated by decoding the encoded content including each data channel and the strain generated by decoding the encoded content excluding each data channel.

[0135] Clause 15: Any method of Clauses 1 to 14, further comprising the steps of dividing received content into a plurality of coding units and ordering the plurality of coding units based on a compression metric, wherein the step of generating a refined compressed version of encoded content includes the step of refining each of the plurality of coding units so that each of the plurality of coding units is compressed using a different level of quantization, with coding units having a higher compression metric being compressed using a lower amount of compression than coding units having a lower compression metric.

[0136] Clause 16: The method of Clause 15, wherein the step of dividing the received content into multiple coding units includes the step of dividing the received content into multiple elements, each element representing data for one of multiple channels at a specific location within the received content.

[0137] Clause 17: The method of Clause 15, wherein the step of dividing the received content into multiple coding units includes the step of dividing the received content into multiple blocks, each block representing data for one of multiple channels in a specific range of locations within the received content.

[0138] Clause 18: The method of Clause 15, wherein the step of dividing the received content into multiple coding units includes the step of dividing the received content into multiple channels.

[0139] Clause 19: The method of Clause 15, wherein the step of dividing the received content into multiple coding units includes the step of dividing the received content into multiple pixels, each pixel representing data for multiple channels at a specific location within the received content.

[0140] Clause 20: The compression metric includes the pre-standard deviation encoded within the hyperlatent, and the hyperlatent includes the initial portion of the encoded content in the manner of Clause 15, in the refined compressed version.

[0141] Clause 21: The compression metric includes the strain-rate ratio, and the encoded content of the refined compressed version includes ordering information for multiple coding units from the highest strain-rate ratio to the lowest strain-rate ratio, in the manner of Clause 15.

[0142] Clause 22: The compression metric includes changes in the rate metric, and the encoded content of the refined compressed version includes ordering information for multiple coding units from a certain change in rate to the lowest change in rate, in the manner of Clause 15.

[0143] Clause 23: The first quantization bin size is associated with the first bitrate, and each of the one or more second quantization bin sizes corresponds to a bitrate higher than the first bitrate in any way of Clauses 1 to 22.

[0144] Clause 24: A method for decompressing content using a neural network, comprising the steps of: receiving encoded content to decompress; recovering approximations of values ​​in a latent code space from the received encoded content by recovering the code from a set of quantization bin sizes, the set of quantization bin sizes comprising a first quantization bin size and one or more second quantization bin sizes smaller than the first quantization bin size; generating a decompressed version of the encoded content by decoding the approximations of values ​​in the latent code space via a decoder implemented by an artificial neural network; and outputting the decompressed version of the encoded content.

[0145] Clause 25: The method of Clause 24, wherein the size of each quantization bin size in a set of quantization bin sizes is an integer multiple of the first quantization bin size.

[0146] Clause 26: The method of Clause 24 or 25, wherein the central bin for one of the quantization bin sizes in a set of quantization bin sizes has a larger bin size than the bins that are not central in the quantization bin size.

[0147] Clause 27: Any method of Clauses 24 to 26, wherein the step of reconstructing an approximation of a value in the latent code space includes the step of reconstructing the code from a bitstream representing the encoded content based on a set of conditional probabilities, where each conditional probability in the set of conditional probabilities is associated with each quantization bin size in a set of quantization bin sizes other than the finest quantization bin size, and is conditional on a conditional probability calculated for a quantization bin size greater than each quantization bin size.

[0148] Clause 28: The method according to any one of Clauses 24 to 27, wherein the step of restoring an approximation of a value in the latent code space includes identifying the probability mass of the encoded content from each quantization bin size among a series of quantization bin sizes based on the cumulative distribution functions of the upper and lower bounds of each quantization bin where the encoded content is located.

[0149] Clause 29: The method of Clause 28, wherein the probability mass for each respective quantization bin size among a series of quantization bin sizes is conditioned on the probability mass for quantization bin sizes among a series of quantization bin sizes larger than each respective quantization bin size.

[0150] Clause 30: The method according to any one of Clauses 24 to 29, wherein the received encoded content includes content having a plurality of data channels.

[0151] Clause 31: The method of Clause 30, wherein each respective data channel among the plurality of data channels is associated with a compression priority corresponding to the amount of compression used to compress each respective data channel.

[0152] Clause 32: The method of Clause 31, wherein the plurality of data channels includes a luminance channel and a plurality of chrominance channels within visual content, and the luminance channel is associated with a compression priority associated with an amount of compression lower than the compression priorities associated with the plurality of chrominance channels.

[0153] Clause 33: The method of Clause 31, wherein the received encoded content includes visual content to be decompressed, the plurality of data channels includes a plurality of color data channels within the visual content, and the first color data channel among the plurality of color data channels having the greatest impact on the quality of the decompressed version of the encoded content is associated with a compression priority associated with an amount of compression lower than the compression priorities associated with the color data channels other than the first color data channel.

[0154] Clause 34: The method of Clause 33, further comprising the step of identifying a first color data channel based on the amount of luminance data contained in each of a plurality of color data channels.

[0155] Clause 35: Encoded content comprises multiple encoded coding units, and the step of reconstructing an approximation of a value in the latent code space from the received encoded content comprises the step of reconstructing the code in the latent code space associated with each of the multiple encoded coding units, in any of the methods in Clauses 24 to 34.

[0156] Clause 36: Multiple coding units include multiple elements, each element representing data for one of multiple channels at a specific location in the received content, in the manner of Clause 35.

[0157] Clause 37: Multiple coding units comprise multiple blocks, each block representing data for one of multiple channels within a specific range of locations in the received content, in the manner of Clause 35.

[0158] Clause 38: Multiple coding units include multiple channels, in the manner of Clause 35.

[0159] Clause 39: Multiple coding units include multiple pixels, each pixel representing data for multiple channels at a specific location in the received content, in the manner of Clause 35.

[0160] Clause 40: The method of Clause 35, wherein the step of restoring an approximation of a value in the latent code space includes the step of restoring the prior standard deviation encoded in the hyperlatent, the hyperlatent including the initial portion of the encoded content.

[0161] Clause 41: The method of Clause 35, which includes the step of restoring an approximation of a value in the latent code space, the step of restoring the compressed order of multiple coding units, the order being included as side information associated with the encoded content.

[0162] Clause 42: The first quantization bin size is associated with a first bitrate, and each of one or more second quantization bin sizes corresponds to a bitrate higher than the first bitrate, in any way of Clauses 24 to 42.

[0163] Clause 43: A processing system comprising memory containing computer executable instructions and one or more processors configured to execute computer executable instructions and cause the processing system to perform the method described in any one of Clauses 1 to 42.

[0164] Clause 44: A processing system comprising means for performing the method described in any one of Clauses 1 to 42.

[0165] Clause 45: Non-temporary computer-readable media comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method described in any one of Clauses 1 to 42.

[0166] Clause 46: A computer program product embodied on a computer-readable storage medium, which includes code for performing the method described in any one of Clauses 1 through 42.

[0167] Additional considerations The preceding descriptions are provided to enable any person skilled in the art to practice the various embodiments described herein. The examples described herein do not limit the scope, applicability, or embodiments set forth in the claims. Various modifications of these embodiments will be readily apparent to a person skilled in the art, and the general principles defined herein may apply to other embodiments. For example, changes may be made to the function and configuration of the elements described without departing from the scope of this disclosure. Various examples may, as appropriate, omit, replace, or add various procedures or components. For example, the methods described may be performed in an order different from the order described, and various steps may be added, omitted, or combined. Also, features described in some examples may be combined in some other examples. For example, an apparatus may be implemented or a method may be practiced using any number of embodiments described herein. In addition, the scope of this disclosure covers apparatus or methods that may be practiced using other structures, functionalities, or structures and functionalities, in addition to or other than the various embodiments of the disclosure described herein. It should be understood that any embodiment of the disclosure disclosed herein may be embodied by one or more elements of the claims.

[0168] As used herein, the term “exemplary” means “to serve as an example, case, or illustration.” No embodiment described herein as “exemplary” should necessarily be construed as being preferable or more advantageous than any other embodiment.

[0169] As used herein, the phrase “at least one of” the list of items refers to any combination of those items that have a single member. For example, “at least one of a, b, or c” shall include a, b, c, ab, ac, bc, and abc, as well as any combination having multiple identical elements (for example, aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other order of a, b, and c).

[0170] As used herein, the term “deciding” encompasses a wide variety of actions. For example, “deciding” may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or other data structure), and confirming. It may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and resolving, selecting, choosing, and establishing.

[0171] The methods disclosed herein include one or more steps or acts for achieving the methods. The steps and / or acts of the methods may be exchanged with each other without departing from the scope of the claims. In other words, unless a specific order of steps or acts is specified, the order and / or use of the specific steps and / or acts may be modified without departing from the scope of the claims. Further, the various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software components and / or modules including, but not limited to, circuitry, application specific integrated circuits (ASICs), or processors. Generally, where there are operations shown in the figures, those operations may have corresponding means-plus-function components with similar numbering.

[0172] The following claims are not limited to the aspects shown herein and should be given the full scope consistent with the claim language. In the claims, the recitation of an element in the singular is not to be construed as meaning "sole" or "exclusive" unless so specified, but rather "one or more." Unless otherwise specified, the term "some" refers to one or more. An element of a claim is not to be construed under 35 U.S.C. § 112, paragraph (f) unless the element is expressly recited using the phrase "means for" or, in the case of a method claim, the element is recited using the phrase "step for." All structural and functional equivalents of the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are hereby expressly incorporated by reference herein and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is expressly recited in the claims.

Description of the Reference Numerals

[0173] 100 Neural Network-Based Data Compression Pipelines 110 Encoding side of the pipeline 111 Content x 112 Neural network-based nonlinear transformation layer (ga) 113 Latent space code y 114 Quantization method (Q) 116 Entity Coder 120 Decryption side of the pipeline 122 Entity Decoder 124 Inverse quantization method (Q-1) Approximate values ​​of 125 codes 126 Neural network-based nonlinear transformation layer (gs) 127 Approximate values 200 pipelines 202 encoders 204 Hyper Encoder 206 Hyper Decoder 208 Decoder 210 Scaler 212 Rescalar 300 probability distributions 302 Probability Mass 304 Quantization Bin 306 [y] 310 Probability Distributions 312 Probability Mass 314 scaled quantization bins 316 2[y / 2] 400A Quantization Level Series 400B Quantization Level Series 402 Quantization Level 404 Quantization Level 406 Quantization Levels 408 Quantization Levels 500 Nested Quantization 800 Channel-Wise Sequential Coding 900 Effective Quantization Grids 902 Intersection 904 Intersection 906 Smallest effective quantization bin 1000 Quantization Grid 1002 Finest quantization grid 1004 Quantization Grid 1006 Midpoint of the Quantization Grid 1100A Graph 1100B Graph 1100C Graph 1100D Graph 1100E Graph 1100F Graph 1200 graphs 1202 Distortion Line 1204 Distortion Line 1300 graphs 1400 Neural Network-Based Data Compression Pipelines 1402 Hyperanalysis Transformation 1404 Quantizer 1406 Entropy Coder 1408 Entropy Decoder 1410 Inverse quantizer 1412 Hypercomposition Transformation 1414 Hypercomposition Transformation 1500 processing systems 1502 Central Processing Unit (CPU) 1504 Graphics Processing Unit (GPU) 1506 Digital Signal Processor (DSP) 1508 Neural Processing Unit (NPU) 1510 Multimedia Processing Block 1512 Wireless Connectivity Component 1514 Antenna 1516 Sensor Processing Unit 1518 Image Signal Processor (ISP) 1520 Navigation Processor 1522 Input and / or Output Devices 1524 memory 1524A Latent Space Coding Component 1524B Sequential Coding Component 1524C Sequential Code Recovery Component 1524D Latent Space Decoding Component

Claims

1. A method for compressing content using a neural network, The steps include receiving content for compression, The steps include encoding the content into a first latent code space via an encoder implemented by an artificial neural network, A step of generating a first compressed version of the encoded content using a first quantization bin size from a set of quantization bin sizes, wherein the set of quantization bin sizes comprises a plurality of quantization bin sizes, the first quantization bin size corresponds to the largest quantization bin size from the set of quantization bin sizes, and subsequent quantization bin sizes decrease toward the smallest bin size. A step of generating a refined compressed version of the encoded content by scaling the first compressed version of the encoded content to one or more second quantization bin sizes in a set of quantization bin sizes that are smaller than the first quantization bin size, based on the values ​​of the first compressed version of the encoded content, The steps include outputting the refined compressed version of the encoded content and Methods that include...

2. The step of generating the refined compressed version of the encoded content is: A step of generating a first refined compressed version of the encoded content by scaling the first compressed version of the encoded content to a first finer quantization bin size, conditional on the values ​​of the first compressed version of the encoded content; A step of generating a second refined compressed version of the encoded content by scaling the first refined compressed version of the encoded content to a second finer quantization bin size, provided that the second finer quantization bin size is smaller than the first finer quantization bin size. The method according to claim 1, including the method described in claim 1.

3. The size of each of the series of quantization bin sizes is a multiple of an integer of the first quantization bin size, and / or The central bin of one of the series of quantization bin sizes has a larger bin size than the bin that is not central in the series of quantization bin sizes. The method according to claim 1.

4. The step of generating the refined compressed version of the encoded content includes the step of generating a bitstream based on a set of conditional probabilities, Each conditional probability in the aforementioned series of conditional probabilities is associated with each quantization bin size in the aforementioned series of quantization bin sizes other than the finest quantization bin size, and is calculated based on the conditional probabilities calculated for quantization bin sizes larger than each of the aforementioned quantization bin sizes. The method according to claim 1.

5. The step of generating the refined compressed version of the encoded content is: The step includes generating a probability mass of the encoded content for each quantization bin size in the set of quantization bin sizes, based on the cumulative distribution functions of the upper and lower bounds of each quantization bin in which the encoded content is located. The probability mass for each of the series of quantization bin sizes is calculated based on the probability mass for a quantization bin size in the series of quantization bin sizes that is larger than each of the aforementioned quantization bin sizes. The method according to claim 1.

6. The encoded content includes content having multiple data channels, Each of the plurality of data channels is associated with a compression priority corresponding to the amount of compression used to compress each data channel. The compression priority associated with each of the plurality of data channels is determined based on calculating the decrease in distortion after decoding the increase in bitrate when each of the plurality of bitrates associated with each of the set of quantization bin sizes is encoded for each of the plurality of bitrates, The step of calculating the reduction in strain for each data channel includes the step of calculating the difference between the strain generated by decoding the encoded content including each data channel the first time, and the strain generated by decoding the encoded content excluding each data channel the second time. The method according to claim 1.

7. The steps include dividing the received content into multiple coding units, The steps of ordering the plurality of coding units based on a compression metric and It further includes, The step of generating the refined compressed version of the encoded content includes refining each of the plurality of coding units so that each of the plurality of coding units is compressed using a different level of quantization, and coding units having a higher compression metric are compressed using less compression than coding units having a lower compression metric. The method according to claim 1.

8. The step of dividing the received content into the plurality of coding units is: A step of dividing the received content into multiple elements, each element representing data for one of multiple channels at a specific location within the received content, A step of dividing the received content into a plurality of blocks, wherein each block represents data for one of a plurality of channels within a specific range of locations in the received content, The steps of dividing the received content into multiple channels, or A step of dividing the received content into a plurality of pixels, wherein each pixel represents data for a plurality of channels at a specific location within the received content. The method according to claim 7, comprising one or more of the above.

9. The aforementioned compression metric is A prior standard deviation encoded within a hyperlatent, wherein the hyperlatent includes the initial portion of the encoded content in the refined and compressed version, A strain-rate ratio, wherein the encoded content of the refined compressed version includes ordering information for the plurality of coding units from the highest strain-rate ratio to the lowest strain-rate ratio, or The refined compressed version of the encoded content includes one or more changes in the rate metric, and the ordered information for the multiple coding units from one change in rate to the lowest change in rate. The method according to claim 7, including the method described in claim 7.

10. A method for decompressing content using a neural network, The steps include receiving encoded content for decompression, A step of recovering an approximation of a value in the latent code space from the received encoded content by recovering the code from a series of quantization bin sizes, wherein the series of quantization bin sizes comprises a first quantization bin size corresponding to the largest quantization bin size among the series of quantization bin sizes, and a smallest quantization bin size smaller than the first quantization bin size. A subsequent one or more second quantization bin sizes that decrease toward the quantization bin size and A step including multiple quantization bin sizes, A step of generating a decompressed version of the encoded content by decoding the approximate value of the recovered value in the latent code space via a decoder implemented by an artificial neural network, The steps include outputting the decompressed version of the encoded content and Includes, The encoded content includes a plurality of encoded coding units, The step of recovering an approximation of the value in the latent code space from the received encoded content includes the step of recovering the code in the latent code space associated with each of the plurality of encoded coding units. method.

11. The size of each of the series of quantization bin sizes is a multiple of the first quantization bin size, The step of restoring an approximation of the value in the latent code space includes the step of restoring the code from a bitstream representing the encoded content based on a set of conditional probabilities, Each conditional probability in the aforementioned series of conditional probabilities is associated with each quantization bin size in the aforementioned series of quantization bin sizes other than the finest quantization bin size, and is calculated based on the conditional probabilities calculated for quantization bin sizes larger than each of the aforementioned quantization bin sizes. The method according to claim 10.

12. The step of reconstructing the approximate value of the value in the latent code space includes the step of identifying the probability mass of the encoded content from each quantization bin size of the set of quantization bin sizes, based on the cumulative distribution functions of the upper and lower bounds of each quantization bin in which the encoded content is located. The probability mass for each of the series of quantization bin sizes is calculated based on the probability mass for a quantization bin size in the series of quantization bin sizes that is larger than each of the aforementioned quantization bin sizes. The method according to claim 10.

13. The received encoded content includes content having multiple data channels, Each of the aforementioned data channels is associated with a compression priority corresponding to the amount of compression used to compress each data channel. The method according to claim 10.

14. The plurality of encoded coding units comprises a plurality of elements, each element representing data for one of a plurality of channels at a specific location in the received content, The plurality of encoded coding units comprises a plurality of blocks, each block representing data for one of a plurality of channels in a specific range of locations within the received content, The plurality of encoded coding units include a plurality of channels, The plurality of encoded coding units include a plurality of pixels, each pixel representing data for a plurality of channels at a specific location in the received content, The step of restoring the approximate value of the value in the latent code space is, A step of restoring a prior standard deviation encoded within a hyperlatent, wherein the hyperlatent includes an initial portion of the encoded content, or A step of restoring the compressed order of the plurality of encoded coding units, wherein the order is included as side information associated with the encoded content. The method according to claim 10, comprising one of the following.

15. It is a system, Memory containing executable instructions stored in memory, A processor configured to execute the executable instruction, wherein the executable instruction causes the system to perform the method described in any one of claims 1 to 14. A system equipped with these features.

Images