Progressive encoding for auto-encoders

Abstract

The invention provides a method, a device and a system for image coding. The present implementations more particularly relate to progressive coding techniques for auto-encoders. In some aspects, an image encoder may encode an image as a tensor having hidden attributes for a plurality of channels based on one or more first layers of a neural network model, and recombine the tensor channels in a prioritized order based on one or more second layers of the neural network model. The image encoder may progressively transmit the recombined tensor channels through a communication channel based on a prioritized order. In some implementations, the image encoder may transmit the recombined tensor channels in a priority order, such that a channel assigned with a higher priority is transmitted prior to a channel assigned with a lower priority.

Description

Technical Field

[0001] This implementation generally involves image encoding, and specifically progressive encoding for autoencoders. Background Technology

[0002] A digital image can be any light pattern that can be reproduced on a digital display. For example, a digital image can be represented by an array (or multiple arrays of pixel values ​​associated with different channels) of pixel values ​​describing the color and intensity of light emitted by each pixel of the display. Some display devices can receive digital images from a source device (such as an image capture device or image data storage library) via a communication channel (such as a wired or wireless medium). Due to the bandwidth limitations of the communication channel, digital image data is typically encoded and / or compressed before being transmitted from the source device. Data compression is a technique used to encode information into smaller units of data. Accordingly, data compression can be used to reduce the bandwidth or overhead required to store or transmit digital images over a communication channel.

[0003] Some modern image coding systems, such as autoencoders, use machine learning to achieve higher levels of data compression. In existing autoencoder architectures, the image decoder (the decoding part that implements the autoencoder) must receive all image data from the compressed image data in order to reconstruct the image. However, changing conditions on the communication channel (such as changes in available bandwidth) may cause data to be discarded or otherwise not transmitted by the image encoder. Therefore, new image compression techniques are needed to adapt to changing channel conditions. Summary of the Invention

[0004] This summary is provided to introduce, in a simplified form, the selected concepts further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] One innovative aspect of the subject matter of this disclosure can be implemented in a method for encoding an image. The method includes the steps of: encoding an image into a tensor having hidden properties with a plurality of first channels based on one or more first layers of a neural network model; recombining the plurality of first channels into a plurality of second channels having a priority order based on one or more second layers of the neural network model; and progressively transmitting the plurality of second channels via a communication channel based on the priority order.

[0006] Another innovative aspect of the subject matter of this disclosure can be implemented in an encoder comprising a processing system and a memory. The memory stores instructions that, when executed by the processing system, cause the encoder to: encode an image into a tensor having hidden properties having a plurality of first channels based on one or more first layers of a neural network model; recombine the plurality of first channels into a plurality of second channels having a priority order based on one or more second layers of the neural network model; and progressively transmit the plurality of second channels via a communication channel based on the priority order. Attached Figure Description

[0007] This implementation is illustrated by way of example and is not intended to be limited by the figures in the accompanying drawings.

[0008] Figure 1 A block diagram of an example communication system for encoding and decoding data is shown.

[0009] Figure 2 A block diagram of an example image encoding and decoding system based on some implementation methods is shown.

[0010] Figure 3 This illustrates example operations for progressively encoding image data, based on some implementation methods.

[0011] Figure 4 This illustrates example operations for progressively decoding encoded image data, based on some implementation methods.

[0012] Figure 5 A block diagram of an example entropy coding system based on some implementations is shown.

[0013] Figure 6 A block diagram of an example entropy decoding system based on some implementations is shown.

[0014] Figure 7 A block diagram of an example image encoder based on some implementations is shown.

[0015] Figure 8 An illustrative flowchart is shown, depicting example operations for encoding image data according to some implementation methods. Detailed Implementation

[0016] In the following description, numerous specific details, such as examples of specific components, circuits, and processes, are set forth to provide a thorough understanding of this disclosure. As used herein, the term “coupled” means directly connected to or connected through one or more intermediate components or circuits. The terms “electronic system” and “electronic device” can be used interchangeably to refer to any system capable of electronically processing information. Furthermore, in the following description, and for purposes of explanation, specific terminology is set forth to provide a thorough understanding of aspects of this disclosure. However, it will be apparent to those skilled in the art that these specific details are not required to practice the exemplary embodiments. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring this disclosure. Some portions of the following detailed description are presented according to other notations of procedures, logic blocks, processes, and operations on data bits within computer memory.

[0017] These descriptions and representations are the means by which those skilled in the art of data processing most effectively communicate the substance of their work to others skilled in the art. In this disclosure, procedures, logic blocks, processes, etc., are considered as a self-consistent sequence of steps or instructions that lead to a desired result. These steps are those that require physical manipulation of physical quantities. Typically, although not always, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated in a computer system. However, it should be remembered that all terms in these and similar terms are to be associated with appropriate physical quantities and are merely convenient notations applied to those quantities.

[0018] Unless otherwise specifically stated, as will be apparent from the following discussion, throughout this application, the use of terms such as “access,” “receive,” “send,” “use,” “select,” “determine,” “normalize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” “derive,” etc., relates to actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities in the registers and memories of the computer system into other data similarly represented as physical quantities in the computer system’s memory or registers or other such information storage, transmission, or display devices.

[0019] In the accompanying drawings, a single block may be described as performing one or more functions; however, in practice, the one or more functions performed by that block may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described below according to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this disclosure. Furthermore, example input devices may include components different from those shown, including well-known components such as processors, memory, etc.

[0020] Unless specifically described as implemented in a particular manner, the techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. Any feature described as a module or component may also be implemented together in an integrated logic device or separately as a discrete but interoperable logic device. If implemented in software, the techniques may be implemented at least in part by a non-transitory processor-readable storage medium comprising instructions that, when executed, perform one or more of the methods described above. The non-transitory processor-readable data storage medium may form part of a computer program product, which may include encapsulation material.

[0021] Non-transitory processor-readable storage media may include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, and other known storage media. Alternatively or additionally, the technique may be implemented at least in part by a processor-readable communication medium that carries or transmits code in the form of instructions or data structures and is accessible, read, and / or executed by a computer or other processor.

[0022] The various illustrative logic blocks, modules, circuits, and instructions described in conjunction with the embodiments disclosed herein can be executed by one or more processors (or processing systems). As used herein, the term "processor" can mean any general-purpose processor, special-purpose processor, conventional processor, controller, microcontroller, and / or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

[0023] As described above, some modern image coding systems (such as autoencoders) utilize machine learning to achieve higher levels of data compression. Machine learning is a technique used to improve the ability of computer systems or applications to perform specific tasks. During the training phase, the machine learning system is provided with multiple "answers" and a large amount of raw input data. The machine learning system analyzes the input data to learn a set of rules (also known as a "machine learning model") that can be used to map the input data to answers. During the inference phase, the machine learning system uses the trained machine learning model to infer answers from new input data.

[0024] Deep learning is a specific form of machine learning where the inference and training phases are performed through multiple layers. Because information is processed in a way that is similar to a biological nervous system, deep learning architectures are often called "artificial neural networks." For example, each layer of an artificial neural network can consist of one or more "neurons." Each layer of neurons can perform different transformations on the output data from the previous layer, such that the final output of the neural network leads to the desired inference. The set of transformations associated with the various layers of the network is called a "neural network model."

[0025] Autoencoders are a type of artificial neural network well-suited for image compression. For example, an autoencoder can be trained to regenerate the same image received at its input at its output. A bottleneck is imposed between the input and output layers of the neural network, reducing the dimensionality of the output at intermediate layers. Due to this bottleneck, the autoencoder is forced to learn a compressed representation of the input image (also known as the image's "hidden attributes"). Therefore, an autoencoder architecture typically consists of an image encoder trained to transform a digital image into a lower-dimensional tensor or vector of hidden attributes, and an image decoder trained to reconstruct the original image from the tensor or vector of hidden attributes.

[0026] The hidden attribute represents the compressed image data sent by the image encoder to the image decoder via a communication channel. For example, the image encoder may implement the encoding portion of an autoencoder, and the image decoder may implement the decoding portion of the autoencoder. In existing autoencoder architectures, the image decoder must receive all image data from the image encoder in the compressed image data to reconstruct the image. However, changing conditions on the communication channel (such as changes in available bandwidth) may cause data to be discarded or otherwise not transmitted by the image encoder. This disclosure recognizes that an autoencoder can progressively encode the channels of a tensor based on the contribution of the channels to image quality, allowing the compression of the image data to adapt to changes in channel conditions.

[0027] Various aspects generally involve image compression, and more specifically, progressive coding techniques for autoencoders. As used herein, the term "progressive coding" refers to various techniques for generating compressed image data (such as tensors of hidden attributes) that allow the reconstruction of the original image from varying amounts of compressed data at different quality levels. In some aspects, an image encoder may encode an image into tensors of hidden attributes with multiple channels based on one or more first layers of a neural network model, and recombine the tensor channels in a priority-ordered manner based on one or more second layers of the neural network model. The image encoder may progressively transmit the recombinated tensor channels over a communication channel based on the priority-ordered arrangement. In some implementations, the image encoder may transmit the recombinated tensor channels in priority order, such that channels with higher priority are transmitted before channels with lower priority.

[0028] Specific implementations of the subject matter described in this disclosure can achieve one or more of the following hidden advantages. By progressively encoding digital images, aspects of this disclosure enable the compression of image data to adapt to changes in channel conditions (such as due to bandwidth limitations). For example, tensor channels more important for image reconstruction can be prioritized for transmission over tensor channels less important for image reconstruction. Thus, when the available bandwidth on a communication channel drops below a threshold level, the image encoder can terminate or truncate tensor transmission before all channels in the channel have been transmitted. Because the highest priority channels are transmitted first, the image decoder is still able to reconstruct the image (at a lower quality level) using only a subset of the channels transmitted by the image encoder.

[0029] Figure 1 A block diagram of an example communication system 100 for encoding and decoding data is shown. Communication system 100 includes an encoder 110 and a decoder 120. In some implementations, the encoder 110 and decoder 120 may be provided in a corresponding communication device (such as, for example, a computer, switch, router, hub, gateway, camera, display, or other device capable of transmitting or receiving communication signals). In some other implementations, the encoder 110 and decoder 120 may be included in the same device or system.

[0030] Encoder 110 receives input data 102 to be transmitted or stored via channel 130. For example, channel 130 may include a wired or wireless communication medium facilitating communication between encoder 110 and decoder 120. Alternatively or additionally, channel 130 may include a data storage medium. In some aspects, encoder 110 may be configured to compress the size of input data 102 to accommodate bandwidth, storage, or other resource limitations associated with channel 130. For example, encoder 110 may encode each unit of input data 102 into a corresponding "codeword" capable of being transmitted or stored via channel 130 as encoded data 104. Decoder 120 is configured to receive encoded data 104 via channel 130 and decode encoded data 104 into output data 106. For example, decoder 120 may decompress or otherwise reverse the compression performed by encoder 110 such that output data 106 is substantially similar (if not identical) to the original input data 102.

[0031] Data compression techniques can generally be classified as "lossy" or "lossless". Lossy data compression may result in some information loss between the encoding and decoding steps. As a result, output data 106 may differ from input data 102. Among other examples, lossy compression techniques include transform coding (such as applications via spatial frequency transformation) and quantization (such as applications via quantization matrices). In contrast, lossless data compression does not result in any information loss between the encoding and decoding steps, as long as channel 130 does not introduce errors into the encoded data 104. As a result, output data 106 is identical to input data 102. Among other examples, lossless compression techniques include entropy coding (such as arithmetic coding, Huffman coding, or Golomb coding) and run-length coding (RLE).

[0032] Entropy coding is a lossless data compression method that encodes data values ​​(or "symbols") into codewords of varying lengths based on the probability of occurrence of each data symbol. For example, a data symbol with a higher probability of occurrence can be encoded into a shorter codeword compared to a data symbol with a lower probability of occurrence. To support the decoding of entropy-coded data, encoder 110 can transmit side information 108 to decoder 120. Side information 108 indicates the entropy model used to encode input data 102 (such as the probability of occurrence of each data symbol). Thus, decoder 120 can use side information 108 to recover output data 106 from encoded data 104.

[0033] In some implementations, encoder 110 and decoder 120 may utilize machine learning to achieve even greater data compression. As described above, machine learning is a technique used to improve the ability of a computer system or application to perform specific tasks. During the training phase, the machine learning system is provided with multiple "answers" and a large amount of original input data. The machine learning system analyzes the input data to learn a set of rules that can be used to map the input data to answers. During the inference phase, the machine learning system uses the trained machine learning model to infer answers from new input data. Deep learning is a specific form of machine learning in which the inference and training phases are performed by neurons in multiple layers (also known as "artificial neural networks"). The neurons in each layer can perform different transformations on the output data from the previous layer, such that the final output of the neural network leads to the desired inference.

[0034] Autoencoders are a type of artificial neural network well-suited for image compression. For example, an autoencoder can be trained to regenerate the same image received at its input at its output. A bottleneck is imposed between the input and output layers of the neural network, reducing the dimensionality of the output at intermediate layers. Due to this bottleneck, the autoencoder is forced to learn a compressed representation of the input image (also known as the image's "hidden attributes"). Therefore, an autoencoder architecture typically consists of an encoder trained to transform frames of image data into lower-dimensional tensors or vectors of hidden attributes, and a decoder trained to reconstruct the original frames of image data from the tensors or vectors of hidden attributes.

[0035] Figure 2 A block diagram of an example image encoding and decoding system 200 according to some implementations is shown. System 200 includes an image encoder 210 and an image decoder 220. In some implementations, the image encoder 210 and the image decoder 220 may be respectively Figure 1 Examples of encoder 110 and decoder 120. Thus, image encoder 210 can be communicated via a communication channel (such as...) Figure 1 The channel 130 is communicatively coupled to the image decoder 220.

[0036] Image encoder 210 is configured to encode raw image data 201 into encoded image data 205 for transmission to image decoder 220. For example, frames of raw image data 201 may include arrays (or multiple arrays of pixel values ​​associated with different color channels) representing digital image or video frames captured or acquired by an image source (such as a camera or other image output device). In some aspects, image encoder 210 may transmit a sequence of frames of encoded image data 205, each representing a corresponding frame of digital video.

[0037] Image decoder 220 is configured to decode encoded image data 205 into reconstructed image data 209 for display on a display device, such as a television, computer monitor, smartphone, or other device including an electronic display. More specifically, image decoder 220 can reverse the encoding performed by image encoder 210, such that the reconstructed image data 209 is substantially similar to the original image data 201. In some aspects, image decoder 220 can display or render a sequence of frames of the reconstructed image data 209 on a display device.

[0038] Image encoder 210 is shown as including analytical transform 212, prioritization transform 214, and tensor encoding component 216. Analytical transform 212 transforms the original image data 201 into a tensor 202 of hidden attributes based on neural network model 204. In some implementations, neural network model 204 may perform encoding operations associated with an autoencoder trained to reduce the dimensionality of the original image data 201. Thus, neural network model 204 may also be referred to as the "inference model" of the autoencoder. As a result, tensor 202 may be a compressed representation of the original image data 201.

[0039] In some implementations, the analysis transformation 212 may include multiple layers of a convolutional neural network (CNN) trained to reduce the dimensionality of the original image data 201. For example, the original image data 201 may be composed of layers with a specific height (h). i Width (w) i ) and depth (d) i The image data 201 is represented by a three-dimensional (3D) array of pixel values. As a result of processing the original image data 201 through various convolutional layers, pooling layers, or any combination thereof that reduce the dimension of the original image data 201, the CNN produces a tensor 202 at its output. For example, the resulting tensor 202 can be a 3D array (h, w, d) with height h, width w, and depth d, where h... <h i w <w i And d>d i The depth d of tensor 202 is defined as the number of “channels” with dimensions h × w.

[0040] Aspects of the present disclosure recognize that some tensor channels contribute more significantly to the reconstruction of an image compared to other tensor channels. Aspects of the present disclosure also recognize that it is possible to reconstruct an image from only a subset of the tensor channels that have the greatest impact on image quality. For example, a threshold number (n) of channels may be required to reconstruct an image at a minimum acceptable level of image quality (where n < d). Image quality can be progressively improved by adding tensor channels to the reconstruction (also referred to as "progressive encoding"). To support progressive encoding, an image encoder must have knowledge regarding the contribution of each tensor channel to image quality and / or reconstruction. However, existing autoencoders are not trained to prioritize certain tensor channels over others.

[0041] In some aspects, the prioritization transform 214 is configured to transform the channels of the tensor 202 to produce a prioritized tensor 203 such that each channel of the prioritized tensor 203 is assigned a priority based on the contribution of the channel to the quality level of the image. For example, the prioritization transform 214 may assign a higher priority to tensor channels that contribute more to image quality and a lower priority to tensor channels that contribute less to image quality. In some implementations, the prioritization transform 214 may arrange (or rearrange) the channels of the tensor 202 in order of priority such that the (one or more) highest-priority tensor channels can be encoded as a bitstream and / or transmitted as a bitstream before the (one or more) lowest-priority tensor channels. In such implementations, each channel of the prioritized tensor 203 may be the same as the corresponding channel of the tensor 202.

[0042] In some other implementations, the channels of the tensor 202 may be recombined through one or more layers of the inference model 204 to produce new tensor channels of the prioritized tensor 203 that have an ordered sequence of priorities. For example, the inference model 204 may include two or more densely connected layers that are trained to produce an ordered sequence of channels based on the contribution of each prioritized channel to the quality level of the image. The dense layers may be connected to the output layer of the analysis transform 212 such that the input of the dense layers is consistent with the output of the analysis transform 212 and the output of the dense layers results in an ordered sequence of channels. More specifically, the densely connected layers may modify the channels and / or data of the tensor 202 such that the ordered sequence of channels can be progressively encoded and / or transmitted to achieve various levels of image quality. As a result, one or more channels of the prioritized tensor 203 may be different from any of the channels of the tensor 202.

[0043] In some implementations, the dense layers of the inference model 204 may be trained to pass through several (N) different quality levels (λ nMinimize the following loss function: The parameters learned include the bits per pixel (bpp) required for a compressed representation of the image. n (Also known as "coding rate") and the amount of distortion in the resulting image (distortion) n It can be measured according to any suitable metric. As shown in Equation 1, each quality level λ n Define the ratio between the coding rate and image distortion in the loss function, where s n It is a scalar constant that can be used to prioritize lower or higher quality values ​​during training. More specifically, the quality metric λ n It applies to the coding rate, not the distortion, and is therefore denoted as the reciprocal term.

[0044] Tensor coding component 216 is configured to encode prioritized tensor 203 into one or more codewords of encoded image data 205 according to one or more coding schemes. In some implementations, tensor coding component 216 may perform entropy coding (or other lossless or lossy compression) on prioritized tensor 203 to further reduce the amount of encoded image data 205 transmitted to image decoder 220. In some implementations, in addition to the encoded tensor, the encoded image data 206 may also include side information (such as references) indicating the entropy model used by tensor coding component 216 to encode prioritized tensor 203. Figure 1 (Described by)

[0045] In some aspects, tensor encoding component 216 may encode and transmit each channel of prioritized tensor 203 in priority order, such that (via priority sorting transformation 214) the channel(s) assigned the highest priority are encoded and / or transmitted before the channel(s) assigned the lowest priority. For example, tensor encoding component 216 may encode the channels of prioritized tensor 203 into a bitstream according to the order in which the channels are arranged and / or output via priority sorting transformation 214. In some aspects, tensor encoding component 216 may dynamically terminate or truncate the bitstream before all channels of prioritized tensor 203 have been transmitted and / or encoded.

[0046] In some implementations, tensor encoding component 216 may terminate the bitstream when the amount of encoded image data 205 transmitted through the communication channel exceeds the available bandwidth of the communication channel. For example, under low-bandwidth channel conditions, tensor encoding component 216 may encode and / or transmit only a subset of the highest priority channels of the prioritized tensor 203, while discarding or retrieving the remaining lower priority channels. In some other implementations, tensor encoding component 216 may encode and / or transmit only a threshold number of channels of the prioritized tensor 203 associated with the desired image quality. For example, tensor encoding component 216 may terminate the bitstream after sufficient tensor channels have been encoded and / or transmitted to achieve the desired image quality.

[0047] In some implementations, the percentage of tensor channels discarded by the tensor encoding component 216 can be correlated with a specific quality level λ. n Related. For example, the percentage of tensor channels discarded by the tensor encoding component 216 can be expressed as the target quality level λ. n Functions: Where g is the mass level λ that can be used to adjust. n With λ used to compress the image to this quality level n A scalar that balances the number of tensor channels.

[0048] As shown in Equation 1, the loss function assumes a trade-off between the maximum achievable image quality and the asymptotic nature of the coding (such as the difference between the minimum and maximum rate and distortion that can be achieved using different parts of the total tensor). A scalar g can alter the balance of this trade-off. For example, if g = 50, then when λ n As the value approaches 0, the percentage of channels discarded tends towards 100%. In other words, the tensor coding component 216 can discard almost all channels of the prioritized tensor 203 to achieve the lowest possible image quality. On the other hand, if g = 25, then when λ n As the value approaches 0, the percentage of channels discarded tends to be 50%. In other words, the tensor coding component 216 can discard only half of the channels of the prioritized tensor 203 to achieve the lowest image quality.

[0049] Image decoder 220 is shown to include tensor decoding component 222, inverse priority sorting transform 224, and synthesis transform 226. Tensor decoding component 222 is configured to decode encoded image data 205 to recover decoded tensor 206. In some implementations, tensor decoding component 222 may reverse the encoding performed by tensor encoding component 216. For example, tensor decoding component 222 may decode encoded image data 205 based on one or more of the same encoding scheme implemented by tensor encoding component 216. In some implementations, tensor decoding component 222 may use side information (included with the encoded image data 205) indicating the entropy model used by tensor encoding component 216 to encode prioritized tensor 203 to recover decoded tensor 206. As a result, decoded tensor 206 may be identical to prioritized tensor 203.

[0050] The inverse priority ordering transform 224 is configured to transform the channels of the decoded tensor 206 to produce a tensor 207 suitable for decoding by the synthesis transform 226. In some implementations, the channels of tensor 207 may be the same as or a subset of the channels of tensor 202. In such implementations, the inverse priority ordering transform 224 may reverse the transform performed by the priority ordering transform 214. For example, the inverse priority ordering transform 224 may rearrange and / or modify the channels of the decoded tensor 206 such that the information in each channel of tensor 207 matches the information in the corresponding channel of tensor 202, and the order of the channels in tensor 207 matches the order of the same channels in tensor 202.

[0051] The synthetic transformation 226 transforms the tensor 207 into reconstructed image data 209 based on the neural network model 208. In some implementations, the synthetic transformation 226 can reverse the compression performed by the analytical transformation 212. For example, the neural network model 208 can perform decoding operations associated with the autoencoder implemented by the analytical transformation 212. Thus, the neural network model 208 can also be referred to as a “generative model” of the autoencoder. As a result, the reconstructed image data 209 can be substantially similar to (if not identical to) the original image data 201.

[0052] In some implementations, the synthetic transformation 226 may include multiple layers of a CNN trained to upsample tensor 207. For example, as a result of processing tensor 207 by various convolutional layers, pooling layers, or any combination thereof that increase the dimension of tensor 207, the CNN may produce reconstructed image data 209. Thus, the reconstructed image data 209 is a decompressed representation of tensor 207. In some implementations, the reconstructed image data 209 may be derived from a CNN with a specific height (h).o )、 width (w o ) and depth (d o ), represented by a 3D array of pixel values, where h o > h, w o > w, and d o < d. In some aspects, the reconstructed image data 209 can be further displayed or rendered as a digital image on an electronic display (not shown for simplicity).

[0053] In some aspects, the inverse prioritization transform 224 can also implement at least a portion of the generative model 208. For example, the transformation of the tensor channels can be performed by one or more layers of the generative model 208. In some implementations, the generative model 208 can include two or more densely connected layers that are trained to reverse the transformation performed by the densely connected layers of the inference model 204. For example, a dense layer can be connected to the input layer of the synthesis transform 226 such that the output of the dense layer is consistent with the input of the synthesis transform 226, and the output of the dense layer results in the tensor 207.

[0054] In some aspects, the analysis transform 212, the prioritization transform 214, the inverse prioritization transform 224, and the synthesis transform 226 can be jointly trained as an autoencoder that reproduces the same image at its output that was received at its input. In such aspects, the inverse prioritization transform 224 may not directly reverse the prioritization performed by the prioritization transform 214, and the synthesis transform 226 may not directly reverse the compression performed by the analysis transform 212. In other words, the channels of the tensor 207 can be different from the channels of the tensor 202.

[0055] However, unlike existing autoencoders, the autoencoder of the present implementation is trained to prioritize the tensor channels in order of the importance of the reconstruction of the image. Due to progressive encoding, the tensor 207 received as the input to the synthesis transform 226 can be different from the tensor 202 generated at the output of the analysis transform 212. For example, due to the bandwidth limitation of the communication channel, the tensor 207 can only include a subset of the channels of the tensor 202. Thus, the quality of the reconstructed image data 209 can depend on the number of tensor channels discarded by the tensor encoding component 216.

[0056] Figure 3 Shows an example operation 300 for progressively encoding image data according to some implementations. The operation 300 can be performed by an image encoder (such as Figure 2The image encoder 210 performs the operation to encode the tensor of hidden attribute 302 into an encoded tensor 306 for transmission over the communication channel. In some implementations, tensor 302 and encoded tensor 306 may be respectively... Figure 2 Examples of tensor 202 and encoded image data 205.

[0057] exist Figure 3 In the example, tensor 302 is depicted as a 3D tensor (h,w,d) with height h=4, width w=4, and depth d=6, where each channel in the channel has a corresponding channel index c for i∈{1,2,3,4,5,6}. i .like Figure 3 As shown, tensor channels c1–c6 are arranged according to their channel indices c. i The order of these tensor channels is such that the first tensor channel c1 is located at the "front" of tensor 302, and the sixth tensor channel c6 is located at the "rear" of tensor 302. (See reference...) Figure 2 Furthermore, it is described that information contained in some tensor channels (such as some information in channel c4) can make a more significant contribution to image reconstruction compared to information contained in some other tensor channels (such as some information in channel c3).

[0058] Priority sorting transformation 214 recombines tensor channels c1–c6 to produce priority-ordered channels representing tensor 304. More specifically, the priority ranking transformation 214 prioritizes the channels based on the magnitude of each channel's contribution to image reconstruction. Arrange them. Figure 3 In the example, the prioritized tensor 304 is shown as having the same number of channels as tensor 302. However, in actual implementations, tensor 302 and the prioritized tensor 304 may have different numbers of channels. In some implementations, each channel of the prioritized tensor 304 may be the same as the corresponding channel of tensor 302. In some other implementations, one or more channels of the prioritized tensor 304 may be different from any channel of tensor 302 (such as reference). Figure 2 (As described). In some implementations, the prioritized tensor 304 can be... Figure 2 An example of a prioritized tensor 203. For example... Figure 3 As shown, channel priority is depicted by color gradient, such that lighter channels have higher priority than darker channels.

[0059] Tensor encoding component 216 progressively encodes the prioritized tensor 304 into an encoded tensor 306 and transmits it as an encoded tensor 306 for transmission over the communication channel. More specifically, tensor encoding component 216 uses the highest priority channel... Begin encoding each channel of the prioritized tensor 304 according to priority order (e.g., from front to back). Figure 3 As shown, the tensor encoding component 216 transmits the first encoded channel of the prioritized tensor 304 at time t1, then transmits the second encoded channel of the prioritized tensor 304 at time t2, then transmits the third encoded channel of the prioritized tensor 304 at time t3, and then transmits the fourth encoded channel of the prioritized tensor 304 at time t4. Figure 3 In the example, tensor encoding component 216 terminates the transmission of the prioritized tensor 304 without affecting the remaining lower-priority channels. and To transmit and / or encode.

[0060] In some implementations, the tensor encoding component 216 may terminate the transmission and / or encoding of the prioritized tensor 304 when the size of the encoded tensor 306 reaches or exceeds the bandwidth limit of the communication channel. In some other implementations, the tensor encoding component 216 may terminate the transmission and / or encoding of the prioritized tensor 304 when the size of the encoded tensor 306 is sufficient to reconstruct the image at the target quality level. As a result of early termination of transmission, the last two channels of the prioritized tensor 304... and (Indicating the lowest priority channel) is effectively "discarded" from the encoded tensor 306. Therefore, the image decoder (such as...) Figure 2 The image decoder 220 must use only a subset of the channels of the prioritized tensor 304 (such as channels ordered by priority). To restore the original image.

[0061] Figure 4 Example operation 400 for progressively decoding encoded image data is shown according to some implementation. Operation 400 can be performed by an image decoder (such as...) Figure 2 The image decoder 220 performs the transformation to convert the channels of tensor 402 into the recovered tensor 404 for input to a synthesis transform (such as...). Figure 2 The composition transformation 226). In some implementations, tensor 402 and the recovered tensor 404 can be respectively Figure 2 Examples of decoded tensors 206 and 207.

[0062] exist Figure 4In the example, tensor 402 is depicted as a 3D tensor (h, w, d) with height h = 4, width w = 4, and depth d = 4, where each channel in the channel has a corresponding channel index for i ∈ {1, 2, 3, 4}. In some implementations, tensor 402 can be Figure 3 The decoded representation of the encoded tensor 306. Therefore, the channels of tensor 402... The channel of tensor 304 representing priority A finite subset of. For example... Figure 4 The tensor channel shown (With the highest priority) it is located at the beginning of tensor 402, followed by tensor channels. (Having the second highest priority), followed by tensor channels. (With the third highest priority), followed by the tensor channel located at the rear of tensor 402. (has the fourth highest priority).

[0063] Inverse priority sorting transformation 224 will transform tensor channels Transformed into tensor channels representing the recovered tensor 404 It can be transformed through synthesis (such as...) Figure 2 The composite transform (226) is used to decompress the image to reconstruct it. Figure 4 In the example, the recovered tensor 404 is shown as having the same number of channels as tensor 402. However, in actual implementations, tensor 402 and the recovered tensor 404 may have different numbers of channels. In some implementations, tensor channels... It can be a subset of channels c1–c6 of tensor 302 (such as in the case where the inverse priority sorting transformation 224 is trained on pairs and the priority sorting performed by priority sorting transformation 214 is reversed). In some other implementations, tensor channels One or more tensor channels in the tensor may be different from any of the channels c1–c6 of tensor 302 (such as when the analysis transform 212, priority sorting transform 214, inverse priority sorting transform 224 and synthesis transform 226 are jointly trained as an autoencoder).

[0064] Figure 5 A block diagram of an example entropy coding system 500 according to some implementations is shown. In some implementations, the entropy coding system 500 can be... Figure 2 An example of tensor encoding component 216. More specifically, entropy encoding system 500 is configured to perform entropy encoding operations on prioritized tensor 501 to produce compressed tensor 508. Reference Figure 2The prioritized tensor 501 can be an example of the prioritized tensor 203, and the compressed tensor 508 can be an example of the encoded image data 205.

[0065] Entropy coding system 500 includes a tensor truncation component 510, a hyperanalysis transform 520, a hyperlatent quantization component 530, a hypersynthetic transform 540, a model trimming component 550, a tensor quantization component 560, and an entropy coding component 570. The tensor truncation component 510 is configured to generate a reduced tensor 502 (such as a reference tensor) by discarding one or more channels from a prioritized tensor 501. Figure 2 and 3 (As described). In some implementations, the tensor truncation component 510 may discard channels from the prioritized tensor 501 based on the bandwidth limitations of the communication channel. In some other implementations, the tensor truncation component 510 may discard channels from the prioritized tensor 501 based on the target or desired quality level of the reconstructed image.

[0066] Tensor quantization component 560 quantizes the reduced tensor 502 into a quantized tensor 507. The elements of the quantized tensor 507 have spatial dependencies that can be modeled by hidden variables conditioned on the assumption that such elements are independent. More specifically, the elements of the quantized tensor 507 can be modeled as a zero-mean Gaussian distribution with a spatially varying standard deviation. Hyperanalysis transformation 520 summarizes the distribution of the standard deviation 503 of the reduced tensor 502 based on a parametric transformation. In some respects, the parametric transformation can be a neural network consisting of modified linear units (ReLU) and convolutional layers trained as a hidden representation of an inference entropy model. Thus, the distribution of the standard deviation 503 is also referred to as “super-hidden”.

[0067] The super-hidden quantization component 530 quantizes the super-hidden 503 into a quantized super-hidden 504. The super-synthetic transform 540 estimates the spatial distribution of the standard deviation 505 (also known as the "entropy model") of the quantized tensor 507 by applying another parametric transform to the quantized super-hidden 504. In some implementations, the superanalysis transform 520 and the super-synthetic transform 540 may be used for compressing and decompressing image data associated with the prioritized tensor 501 (such as...). Figure 2 The autoencoders of the analytical transform 212 and the synthetic transform 226 are jointly trained. As a result, one or more channels of the entropy model 505 may be associated with channels discarded from the prioritized tensor 501 (such as by the tensor truncation component 510).

[0068] In some implementations, model trimming component 550 may further discard one or more channels of entropy model 505 to produce a trimmed model 506 that more accurately reflects the spatial distribution of the standard deviation. For example, the discarded channels from entropy model 505 may be associated with discarded channels from prioritized tensor 501. Entropy encoding component 570 uses the trimmed model 506 to encode the quantized tensor 507 into a compressed tensor 508. For example, entropy encoding component 570 may derive a probabilistic estimate of the encoded values ​​of compressed tensor 508 based on the trimmed model 506. Compressed tensor 508 may be passed to entropy decoder (such as...). Figure 2 (Tensor decoding component 222). In some respects, the quantized super-hidden 504 can also be encoded as side information and / or transmitted as side information to the entropy decoder.

[0069] Figure 6 A block diagram of an example entropy decoding system 600 according to some implementations is shown. In some implementations, the entropy decoding system 600 may be... Figure 2 An example of the tensor decoding component 222. More specifically, the entropy decoding system 600 will... Figure 5 The quantized super-hidden 504 is used to perform entropy decoding on the compressed tensor 508 to recover the decoded tensor 606. (Reference) Figure 2 The quantized super-hidden 504 and the compressed tensor 508 can be examples of the encoded image data 205, and the decoded tensor 606 can be an example of the decoded tensor 206.

[0070] The entropy decoding system 600 includes a supersynthesis transform 610, a model trimming component 620, and an entropy decoding component 630. The supersynthesis transform 610 estimates the entropy model 602 based on the quantized super-hidden 504. In some implementations, the supersynthesis transform 610 can be combined with... Figure 5 The supersynthesis transformation 540 is the same. For example, the supersynthesis transformation 610 can recover the entropy model 602 by applying a parameter transformation (such as a neural network consisting of convolutional layers and ReLU) to the quantized superhidden 504. Thus, the entropy model 602 can be Figure 5 An example of entropy model 505. For example, see reference 505. Figure 5 The entropy model 602 can indicate the spatial distribution of the standard deviation of the quantized tensor 507.

[0071] For reference Figure 5As described, one or more channels of the entropy model 602 may be associated with channels discarded from the compressed tensor 508 (such as by the tensor truncation component 510). Therefore, the model trimming component 620 is configured to discard one or more channels of the entropy model 602 to produce a trimmed model 604 that more accurately reflects the spatial distribution of the standard deviation. In some implementations, the model trimming component 620 may be associated with... Figure 5 The model trimming components are the same as 550. For example, refer to Figure 5 The discarded channels from entropy model 602 can be associated with the discarded channels from priority tensor 501.

[0072] Entropy decoding component 630 uses a trimmed model 604 to recover the decoded tensor 606 from the compressed tensor 508. In some implementations, entropy decoding component 630 may use the same entropy model as entropy encoding component 570 to decode the compressed tensor 508. For example, entropy decoding component 630 may derive a probabilistic estimate of the encoded value of the compressed tensor 508 based on the trimmed model 604. In some aspects, the decoded tensor 606 may be further processed and / or decompressed (e.g., by...). Figure 2 The inverse priority sorting transformation 224 and the synthesis transformation 226 are used to recover the reconstructed image data (not shown for simplicity).

[0073] Figure 7 A block diagram of an example image encoder 700 according to some implementations is shown. In some implementations, the image encoder 700 may be... Figure 2 An example of an image encoder 210. More specifically, an image encoder 700 can be configured to encode image data for transmission over a communication channel.

[0074] In some implementations, the image encoder 700 may include a data interface 710, a processing system 720, and a memory 730. The data interface 710 is configured to receive image data from an image source and output a compressed representation of the image data via a communication channel. In some aspects, the data interface 710 may include an image source interface (I / F) 712 for communicating with the image source and a channel interface 714 for communicating via the communication channel.

[0075] The memory 730 may include a non-transitory computer-readable medium (including one or more non-volatile memory elements, such as EPROM, EEPROM, flash memory, hard disk drive, etc.) that can store at least the following software (SW) modules: ● Tensor transformation SW module 732, which is used to transform based on one or more first layers of a neural network model. The image is encoded as a tensor with multiple hidden properties in the first channel; ● Channel priority sorting SW module 734, which is used to recombine multiple first channels into multiple second channels with a priority order based on one or more second layers of a neural network model; as well as ● Progressive transmission SW module 736, which is used to progressively transmit multiple second channels through the communication channel based on a priority order. Each software module includes instructions that, when executed by the processing system 720, cause the image encoder 700 to perform a corresponding function.

[0076] Processing system 720 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in encoder 700 (such as memory 730). For example, processing system 720 may execute tensor transformation module 732 to encode an image into a tensor with hidden properties having multiple first channels based on one or more first layers of a neural network model. Processing system 720 may also execute channel priority sorting module 734 to recombine the multiple first channels into multiple second channels having a priority order based on one or more second layers of a neural network model. Processing system 700 may further execute progressive delivery module 736 to progressively deliver the multiple second channels via a communication channel based on a priority order.

[0077] Figure 8 An illustrative flowchart depicting example operations 800 for encoding image data according to some implementations is shown. In some implementations, operation 800 may be performed by an image encoder (such as...). Figure 2 and 7 (Each image encoder in the respective image encoder 210 or 700) is executed.

[0078] The image encoder encodes an image into a tensor with hidden properties having multiple first channels, based on one or more first layers of a neural network model (810). In some implementations, one or more first layers of the neural network model may be trained to perform encoding operations associated with an autoencoder. The image encoder recombines the multiple first channels into multiple second channels having a priority order based on one or more second layers of the neural network model (820). The image encoder progressively transmits the multiple second channels via a communication channel based on the priority order (830).

[0079] In some aspects, one or more second layers of the neural network model can be trained to assign priority to each of a plurality of second channels based on the channel's contribution to the image's quality level. In some implementations, progressive transmission of the plurality of second channels may include transmitting each of the plurality of second channels in order of assigned priority, such that the channel assigned the highest priority is transmitted before the channel assigned the lowest priority. In some implementations, progressive transmission of the plurality of second channels may further include terminating the transmission of the plurality of second channels before transmitting one or more of the plurality of second channels via a communication channel. In some implementations, the transmission may be terminated at least in part based on the bandwidth of the communication channel.

[0080] In some aspects, progressive delivery of multiple second channels may include: generating a superhide based on a subset of channels in the multiple second channels; determining an entropy model based on the superhide; and encoding each channel in the subset of channels based on the entropy model before transmitting the channels via the communication channel. In some implementations, the superhide may be a hidden representation of the entropy model. In some implementations, the image encoder may further discard one or more channels of the entropy model before encoding the subset of channels. In some implementations, the subset of channels may exclude one or more channels from the multiple second channels that are not transmitted via the communication channel. In some implementations, the image encoder may further transmit the superhide via the communication channel.

[0081] Those skilled in the art will recognize that information and signals can be represented using any of a variety of different techniques and skills. For example, data, instructions, commands, information, signals, bits, symbols, and chips, which can be referenced throughout the description above, can be represented by voltage, current, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0082] Furthermore, those skilled in the art will recognize that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this hardware-software interchangeability, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this disclosure.

[0083] The methods, sequences, or algorithms described in conjunction with the aspects disclosed herein may be implemented directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor.

[0084] In the foregoing description, embodiments have been described with reference to specific examples thereof. However, it will be apparent that various modifications and changes may be made therein without departing from the broader scope of this disclosure as set forth in the appended claims. Accordingly, the description and drawings are to be viewed in an illustrative rather than a limiting sense.

Claims

1. A method for encoding an image, comprising: Image is encoded into a tensor with hidden properties having multiple first channels based on one or more first layers of a neural network model. The plurality of first channels are recombined into a plurality of second channels having a priority order based on one or more second layers of the neural network model; as well as The plurality of second channels are progressively transmitted through the communication channel based on the priority order.

2. The method as described in claim 1, wherein, The one or more first layers of the neural network model are trained to perform encoding operations associated with an autoencoder.

3. The method as described in claim 1, wherein, The one or more second layers of the neural network model are trained to assign priority to each of the plurality of second channels based on the contribution of the channels to the quality level of the image.

4. The method of claim 3, wherein, The progressive transmission of the plurality of second channels includes: Each of the plurality of second channels is transmitted in order of its assigned priority, such that the channel assigned the highest priority is transmitted before the channel assigned the lowest priority.

5. The method of claim 4, wherein, The progressive transmission of the plurality of second channels further includes: Before transmitting one or more of the plurality of second channels through the communication channel, the transmission of the plurality of second channels is terminated.

6. The method of claim 5, wherein, The transmission is terminated at least in part based on the bandwidth of the communication channel.

7. The method of claim 1, wherein, The progressive transmission of the plurality of second channels includes: Super-hidden information is generated based on a subset of the channels of the plurality of second channels; The entropy model is determined based on the aforementioned super-hiddenness; and Before transmitting the channel through the communication channel, each channel in the subset of channels is encoded based on the entropy model.

8. The method of claim 7, wherein, The super-hidden representation is the hidden representation of the entropy model.

9. The method of claim 7, further comprising: One or more channels of the entropy model are discarded before encoding the subset of channels.

10. The method of claim 7, wherein, The subset of channels excludes one or more of the plurality of second channels that are not transmitted through the communication channel.

11. The method of claim 7, further comprising: The super-stealth is transmitted through the communication channel.

12. An encoder, comprising: Processing system; as well as The memory stores instructions that, when executed by the processing system, cause the encoder to: Image is encoded into a tensor with hidden properties having multiple first channels based on one or more first layers of a neural network model. The plurality of first channels are recombined into a plurality of second channels having a priority order based on one or more second layers of the neural network model; as well as The plurality of second channels are progressively transmitted through the communication channel based on the priority order.

13. The encoder of claim 12, wherein, The one or more first layers of the neural network model are trained to perform encoding operations associated with an autoencoder.

14. The encoder of claim 12, wherein, The one or more second layers of the neural network model are trained to assign priority to each of the plurality of second channels based on the contribution of the channels to the quality level of the image.

15. The encoder of claim 14, wherein, The progressive transmission of the plurality of second channels includes: Each of the plurality of second channels is transmitted in order of its assigned priority, such that the channel assigned the highest priority is transmitted before the channel assigned the lowest priority.

16. The encoder of claim 15, wherein, The progressive transmission of the plurality of second channels further includes: Before transmitting one or more of the plurality of second channels through the communication channel, the transmission of the plurality of second channels is terminated.

17. The encoder of claim 16, wherein, The transmission is terminated at least in part based on the bandwidth of the communication channel.

18. The encoder of claim 12, wherein, The progressive transmission of the plurality of second channels includes: generating a super-hidden feature based on a subset of the plurality of second channels; The entropy model is determined based on the aforementioned super-hiddenness; and Before transmitting the channel through the communication channel, each channel in the subset of channels is encoded based on the entropy model.

19. The encoder of claim 18, wherein, The execution of the instruction further enables the encoder to: One or more channels of the entropy model are discarded before encoding the subset of channels.

20. The encoder of claim 18, wherein, The subset of channels excludes one or more of the plurality of second channels that are not transmitted through the communication channel.

Images