Method and apparatus for encoding / decoding at least part of image using multi-level context model

Abstract

Methods and apparatus are disclosed for encoding / decoding at least a portion of an image using context prediction obtained from one or more lower spatial resolutions of a first tensor, a first context is obtained from at least one or more samples of at least one second tensor obtained from at least one downsampling of the first tensor. At least one entropy parameter is determined based on the first context, and the first tensor is entropy-encoded or entropy-decoded using the determined at least one entropy parameter.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application 63 / 442,506, filed February 1, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] At least one embodiment of the present invention is generally directed to a method or apparatus for compressing images and videos using neural network (NN) based tools. Background Art

[0004] In recent years, deep neural networks have been developed to surpass the compression performance of traditional codecs. For example, the Joint Video Exploration Team (JVET) between ISO / MPEG and ITU is currently researching such tools to replace some modules of the latest standard H.266 / VVC, as well as to replace the entire structure through an end-to-end autoencoder approach.

[0005] More precisely, different approaches can be distinguished. Pure encoder approaches can be seen as NN-based algorithms that are used to enhance or speed up the encoder of an existing codec. In this case, there are no standardized changes and any existing standard can be used. Approaches built on top of existing standards replace one or more modules of the state-of-the-art codec with NN-based methods (e.g. post-filters, prediction modules, etc.). End-to-end NN-based codecs completely destroy traditional compression schemes including prediction, transform, quantization and entropy decoding modules.

[0006] In contrast to traditional methods that apply predefined prediction modes and transformations, NN-based methods rely on a large number of parameters learned on a large dataset during a training phase by iteratively minimizing a loss function. In the case of image and video compression, the loss function is defined by a rate-distortion cost, where the rate represents an estimate of the bitrate of the encoded bitstream and the distortion quantifies the quality of the decoded video relative to the original input. Traditionally, the quality of the decoded input image is optimized, for example, based on a measure of mean squared error or an approximation of human perceived visual quality. Summary of the Invention

[0007] At least one embodiment of the present invention is generally directed to a method or apparatus in the context of compressing images and videos using neural networks.

[0008] At least one embodiment of the present invention generally relates to multi-resolution context prediction for encoding or decoding a latent representing an image or video.

[0009] Some embodiments relate to a method for encoding a tensor or potential representing at least a portion of an image, the encoding comprising multi-resolution context prediction for determining an entropy parameter. Some embodiments relate to a method for decoding a potential representing at least a portion of an image, the decoding comprising multi-resolution context prediction for determining an entropy parameter.

[0010] In some embodiments, the method includes: for at least one sample of a first tensor representing at least a portion of an image, obtaining a first context from at least one or more samples of at least one second tensor, and obtaining the at least one second tensor by downsampling the first tensor at least once; determining at least one entropy parameter based on the first context; and entropy encoding or entropy decoding the first tensor using the determined at least one entropy parameter.

[0011] According to another aspect, an apparatus is provided. The apparatus includes a processor. The processor may be configured to implement the general aspects by performing any of the described methods. According to another general aspect of at least one embodiment, a device is provided, comprising: an apparatus configured to implement the general aspects by performing any of the described embodiments; and at least one of: (i) an antenna configured to receive a signal, the signal including a video or an image, (ii) a band limiter configured to limit the received signal to a frequency band including the video or image, and (iii) a display configured to display an output representing the video or image.

[0012] According to another aspect, there is provided a signal carrying coded data representing at least a portion of an image or video encoded according to any of the embodiments described herein.

[0013] According to another general aspect of at least one embodiment, there is provided a computer program product comprising instructions that, when executed by a computer, cause the computer to implement any of the embodiments or variations.

[0014] These and other aspects, features and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which is to be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A block diagram of a system is shown in which aspects of embodiments of the present invention may be implemented.

[0016] Figure 2 A block diagram of an embodiment of an autoencoder for image or video compression is shown.

[0017] Figure 3A block diagram illustrating an embodiment of training an autoencoder for image or video compression.

[0018] Figure 4 A block diagram of an embodiment of an autoencoder with a hyper-prior architecture is shown.

[0019] Figure 5 A block diagram of an embodiment of an autoencoder with a hyper-prior architecture and context prediction is shown.

[0020] Figure 6 An example of a method for predicting context at different resolutions of a tensor according to an embodiment is shown.

[0021] Figure 7 An example of a method for encoding a tensor using contexts predicted at different resolutions according to an embodiment is shown.

[0022] Figure 8 An example of valid mask positions for decoding specific samples of a tensor for reconstruction is shown, according to one embodiment.

[0023] Figure 9 An example of a method for representing a potential encoding of at least a portion of an image according to one embodiment is shown.

[0024] Figure 10 An example of a method for decoding a potential representing at least a portion of an image according to one embodiment is shown.

[0025] Figure 11 An example of a method for encoding a tensor using contexts predicted at different resolutions according to another embodiment is shown.

[0026] Figure 12 A block diagram of an autoencoder with a super-prior architecture and context prediction at different resolutions is shown, under an embodiment.

[0027] Figure 13 A block diagram of an embodiment of an autoencoder with a super-prior architecture and context prediction at different resolutions is shown according to another embodiment.

[0028] Figure 14 One embodiment of an apparatus for encoding or decoding an image or video according to any of the embodiments described herein is shown.

[0029] Figure 15 Two remote devices are shown communicating over a communication network, according to an example of the present principles.

[0030] Figure 16 The syntax of a signal according to an example of the present principles is shown. DETAILED DESCRIPTION

[0031] This application describes a number of aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described as having specificity, and at least in order to illustrate individual characteristics, are usually described in a manner that may sound limited. However, this is for the purpose of describing clearly, and does not limit the application or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide other aspects. In addition, these aspects can also be combined and interchanged with the aspects described in earlier documents.

[0032] The aspects described and contemplated in this application can be implemented in many different forms. Figure 1-1 7 provides some embodiments, but other embodiments are contemplated and are not intended to be construed as Figure 1-1 The discussion of 7 does not limit the breadth of implementation. At least one aspect generally relates to image or video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having stored thereon instructions for encoding or decoding image or video data according to any of the described methods, and / or computer-readable storage media having stored thereon a bitstream generated according to any of the described methods.

[0033] Figure 1 A block diagram of an example of a system in which various aspects and embodiments can be implemented is shown. System 100 can be implemented as a device including the various components described below, and is configured to perform one or more aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 100 can be implemented individually or in combination in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more aspects described in this application.

[0034] The system 100 includes at least one processor 110 configured to implement instructions loaded therein for implementing, for example, various aspects described herein. The processor 110 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). The system 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, magnetic disk drive, and / or optical disk drive. As non-limiting examples, the storage device 140 may include an internal storage device, an attached storage device, and / or a network-accessible storage device.

[0035] The system 100 includes an encoder / decoder module 130 that is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents a module(s) that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both encoding and decoding modules. Additionally, the encoder / decoder module 130 may be implemented as a separate element of the system 100 or may be incorporated into the processor 110 as a combination of hardware and software as known to those skilled in the art. In some embodiments, the encoder / decoder module 130 is an NN-based autoencoder (e.g., with respect to Figure 2-4 The invention also provides an embodiment of the present invention that can be used to implement an automatic encoder or a modified automatic encoder as described above and implement one or more embodiment transform blocks, as described further below. In this document, for clarity, the transform blocks described in the embodiments are referred to as multi-resolution transform blocks, and other terms may be used without limiting the scope of the embodiments described herein.

[0036] Program code to be loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described herein may be stored in the storage device 140 and subsequently loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during the execution of the processes described herein. These stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0037] In some embodiments, memory within the processor 110 and / or encoder / decoder module 130 is used to store instructions and to provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be memory 120 and / or storage device 140, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations.

[0038] Input to the elements of system 100 may be provided through various input devices as shown in block 105. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives a radio frequency (RF) signal transmitted over the air, for example, by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1 Other examples not shown include composite video.

[0039] In various embodiments, the input device of block 105 has associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements suitable for the following operations: (i) selecting a desired frequency (also known as selecting a signal, or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired data packet stream. The RF section of various embodiments includes one or more elements to perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or down-converting to baseband. In a set-top box embodiment, the RF part and its relevant input processing element receive the RF signal transmitted by wired (for example, cable) medium, and by filtering, down-converting and filtering to the desired frequency band again to perform frequency selection. Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements that perform similar or different functions.Adding element can comprise and insert element between existing element, for example, insert amplifier and analog to digital converter.In various embodiments, the RF part comprises antenna.

[0040] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 100 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, in a separate input processing IC or processor 110, as desired. Similarly, various aspects of USB or HDMI interface processing may be implemented, for example, within a separate interface IC or within processor 110, as desired. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements (including, for example, processor 110 and encoder / decoder 130), which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on an output device.

[0041] The various components of system 100 may be disposed within an integrated housing where they may be interconnected and transmit data using suitable connection means 115 (e.g., an internal bus known in the art, including an inter-IC (I2C) bus, wiring, and printed circuit boards).

[0042] System 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. Communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 190. Communication interface 150 may include, but is not limited to, a modem or a network card, and communication channel 190 may be implemented, for example, within a wired and / or wireless medium.

[0043] In various embodiments, data is streamed or otherwise provided to the system 100 using a wireless network, such as a Wi-Fi network, such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signals of these embodiments are received via a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 of these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments provide streaming data to the system 100 using a set-top box that transmits data via an HDMI connection of the input box 105. Still other embodiments provide streaming data to the system 100 using an RF connection of the input box 105. As described above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0044] System 100 can provide output signals to various output devices, including a display 165, speakers 175, and other peripheral devices 185. Display 165 in various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 165 can be used in a television, tablet computer, laptop computer, cellular phone (mobile phone), or other device. Display 1100 can also be integrated with other components (e.g., as in a smartphone) or standalone (e.g., an external monitor for a laptop computer). In various examples of embodiments, other peripheral devices 185 include one or more of a standalone digital video disk (or digital versatile disk) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments utilize one or more peripheral devices 185 that provide functionality based on the output of system 100. For example, a disk player performs the function of playing the output of system 100.

[0045] In various embodiments, control signals are transmitted between the system 100 and the display 165, speakers 175, or other peripheral devices 185 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to the system 100 via dedicated connections through the respective interfaces 160, 170, and 180. Alternatively, the output devices can be connected to the system 100 via the communication interface 150 using a communication channel 190. The display 165 and speakers 175 can be integrated into a single unit in an electronic device (e.g., a television) along with the other components of the system 100. In various embodiments, the display interface 160 includes a display driver, such as a timing controller (TCon) chip.

[0046] For example, if the RF portion of input 105 is part of a separate set-top box, the display 165 and speaker 175 may alternatively be separate from one or more of the other components. In various embodiments where the display 165 and speaker 175 are external components, the output signals may be provided via dedicated output connections, including, for example, an HDMI port, a USB port, or a COMP output.

[0047] The embodiments may be performed by computer software implemented by the processor 110 or a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 120 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, for example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 110 may be of any type suitable for the technical environment and may include, as non-limiting examples, one or more of the following: a microprocessor, a general-purpose computer, a special-purpose computer, a digital signal processor (DSP), a processor based on a single-core or multi-core architecture, a sequential or parallel architecture, a dedicated circuit such as a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a signal processing device, and other processing circuits.

[0048] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "encoding" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image," "picture," and "frame" are used interchangeably. Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0049] Figure 2An embodiment of an end-to-end compression system is shown in which one or more of the embodiments described below may be implemented. The input x to the encoder portion of the network may comprise an image or frame of a video, or a portion of an image, or a tensor representing a group of images, or a tensor representing a portion (crop) of a group of images. In each case, the input may have one or more components, for example: monochrome, RGB, or YCbCr components. In Figure 2 In the example, the input x has 3 components of size HxW.

[0050] The input x is fed to the encoder network g a , also known as analytical transformation. a Usually it has an activation function ( Figure 2 The sequence of convolutional layers ( Figure 2 conv in

[15] . Convolutions can include mechanisms for spatially downsampling the input, e.g., choosing a convolution with a stride of 2 in both the vertical and horizontal directions will result in an output with half the size of the input in both dimensions. The output of a convolution is a tensor of shape CxHxW, where H and W are the spatial height and width, respectively, and C corresponds to the adjustable number of channels. For example, for an RGB image, g a The first convolution takes as input a tensor of 3 channels corresponding to the color components. This encoder network can be viewed as a learned transformation, i.e. lossy, since the output latent tensor typically has fewer elements than the source 3xHxW input. The output of the analysis (mostly in the form of a 3-way array, called a 3D tensor) is called a tensor of latent representations or latent variables. From a broader perspective, a set of latent variables constitutes a latent space, which is also often used in the context of end-to-end compression based on neural networks. Output y = g a (x) is quantized, producing a tensor It is then entropy coded into a binary stream (bitstream) for storage or transmission.

[0051] At the decoder, the bitstream is entropy decoded (ED) to obtain Decoder network g s (also called synthetic transform) generates the reconstructed input: It is a potential representation derived from quantization The synthetic transformation g is an approximation of the original x. s It is usually a sequence of upsampling convolutions, such as transposed convolutions or convolutions followed by upsampling filters. The decoder network can be viewed as the learned inverse transform, or denoising and generative transform.

[0052] The performance of a compression system is measured as a trade-off between the number of bits that need to be transmitted and the quality of the decoded content. For one of these trade-offs, a compression model can be trained using a loss that follows the Lagrangian form L = R + λD, where R represents the rate or bit rate and D represents the distortion of the decoded content.

[0053] Figure 2 and Figure 3 The autoencoder is shown in a practical inference configuration, which includes an encoder that produces a bitstream that is then transmitted and decoded. During training, since entropy encoding / decoding is non-differentiable, The entropy is relative to the learned probability model p ψ To determine, such as Figure 3 As described in .

[0054] For distortion, in existing methods, several types of losses are used to train such NNs, which can be used alone or in combination. One can use a loss based on an “objective” metric (usually the mean squared error (MSE) Figure 3 Marked as ) or use, for example, a loss based on structural similarity (SSIM). The result may be perceptually inferior to the second type, but the fidelity to the original signal (image) is higher. It is also possible to use a "subjective" (or proxy subjective) based loss, typically using a generative adversarial network (GAN) or a high-level visual metric via a proxy NN during the training phase.

[0055] In the above autoencoders, the entropy encoder and decoder rely on simple fully factorized priors such as Figure 2 This approach typically considers a latent entropy model trained separately for each channel. Because each sample is encoded using the same distribution (i.e., they are assumed to be independent and identically distributed (iid)), the latent However, even through g a After processing, It is also not iid, and recent methods have begun to deal with this particular problem. Figure 4A method called autoencoder with hyperprior is described, which is further described in: [Ballé] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” ArXiv 180201436 Cs Eess Math, May 2018, accessed: August 25, 2020. [Online]. Available: http: / / arxiv.org / abs / 1802.01436, because the model now includes an additional convolution sequence h a and h s , which are respectively in potential z and Medium output for potential The learned distribution parameters for each element of . For example, for the potential For each element of , the learned distribution parameters are the scale or mean and scale of the Gaussian or Laplace distribution. a The output tensor z needs to be encoded and sent as side information for decoding by the decoder The tensor z is thus quantized into the tensor and is entropy coded (EE). At the decoder side, the latent The bitstream is entropy decoded (ED) and fed to the distribution information h s in the synthetic transformation.

[0056] Transferring tensors using the fully factorized method It won't cost too much because Corresponding to further h a Reduced y, and for Each element uses a customized Gaussian distribution to greatly exceed the efficiency of transmitting light. burden.

[0057] Note that in Figure 4 In it, it represents all the operations and elements necessary for the actual decoding and encoding of the image, and the distribution information h s The synthesis of h occurs at both the encoder and the decoder. As with conventional video coding, the encoder includes parts of the decoder to generate the exact same metadata that the decoder will decode to process the rest of the bitstream. s Bit-accurate operations must be performed between the encoder and decoder for the system to work. Slight differences in the generated parameters will completely destroy the Arithmetic decoder.

[0058] For example, the autoregressive context model described in [Minnen] D. Minnen, J. Ballé, and G. D. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” in Advances in Neural Information Processing Systems, 2018, pp. 10771–10780, is used to estimate the entropy parameters μ, σ (mu, sigma) of a Gaussian distribution representing the estimated probability distribution from which the tensor Cx1x1 fiber currently being decoded is extracted.

[0059] Figure 5 An example of an autoencoder with a hyper-prior architecture and context prediction is shown. It is encoded or decoded in a raster scan order from top left to bottom right along the image axis (H, W). At a given (x, y) position, a "context prediction" ( Figure 5 cp in ). Figure 5 An example of a Cx5x5 kernel is shown in the lower left of . Figure 5 In the example, the grayscale samples of the tensor are used to obtain the context of the black center position. Output tensor The context of the position (x, y) is taken as a Cx1x1 vector, which is, for example, a weighted combination of the causal samples covered by the kernel. This vector is then combined with the side information ψ=h for this (x, y) tensor position s (Q(h a (y)))Merge / join( Figure 5 cat in ). It is then fed to an “entropy parameter” (ep) network, which outputs a tensor of entropy parameters μ, σ for the location (x, y). An example of an entropy parameter network is given in Figure 5 The entropy parameter network allows the concatenation of the context and the hyper-prior resulting from the 3xCx1x1 input vector to be transformed into a 2xCx1x1 vector containing μ and σ for each position (x, y) of the tensor. The same operation for determining the entropy parameters μ and σ for each position (x, y) of the tensor is performed on both the encoder and decoder parts.

[0060] In [Minnen], spatial causal neighbors in the same channel are used to obtain contextual predictions.

[0061] The basic autoregressive approach introduces sequential computation, which breaks the rest of the end-to-end NN-based encoder-decoder architecture that relies on highly parallelizable operations.

[0062] In this document, some embodiments provide improvements to the efficiency of autoregressive context modeling while designing an architecture that reduces computational complexity.

[0063] In [Ma] C. Ma, Z. Wang, R. Liao, and Y. Ye, “A Cross Channel Context Model for Latents in Deep Image Compression,” ArXiv210302884Cs Eess, March 2021, accessed: April 28, 2021. [Online]. Available: http: / / arxiv.org / abs / 2103.02884, Ma et al. introduce a context model in which previously decoded “groups” of tensor channels can be used to predict future groups. The idea is to make the “groups” of previously decoded tensor channels correlated with the groups to be decoded. However, this correlation is rather loose; there is no strictly enforced correlation between the groups, other than the fact that the groups are all independently derived from effectively disjoint subspaces of the same source. Instead, in the embodiment provided here, “levels” of the input tensor to be encoded or decoded are directly derived from previous “levels” of the tensor. These levels are defined as representations of the tensor encoded or reconstructed at different resolutions. This forces a more direct correlation between groups / levels.

[0064] Some embodiments described further below provide improvements to the context models introduced in [Ballé] or [Minnen].Some embodiments described below utilize context kernels at different resolutions to capture short and longer spatial correlations for efficient modeling.

[0065] According to one aspect of the present disclosure, a method for obtaining context at different resolutions is provided. Figure 6 An example of a method for predicting context at different resolutions of a tensor according to an embodiment is shown. Figure 6 The input tensor y is reduced (601, 602) by y0, y1 and y2 in the input, where y0 is the input tensor at the input resolution. For the position (x, y) of the tensor, and for each level y0, y1 and y2, CP 00 、CP 01 and CP 02The context predictions are applied to the tensors at that level. Each context prediction outputs a Cx1x1 tensor representing the context for the position (x, y) at the level considered, where C is the number of channels of the input tensor. If the level is a downscaling level, the output of the context prediction is upscaled (603, 604) to the next level and combined (605, 606) with the next level up to the input resolution to provide the final context to be used for entropy encoding the position (x, y) of the input tensor y. Figure 6 ctx on ). Figure 6 In the example, three levels are defined, context prediction CP 02 The output of is upscaled (603) to the next higher resolution and compared with the context prediction CP 01 The output of the combined (605) is then scaled up (604) to the input resolution and compared with the context prediction CP 00 In another variation, each output of the context prediction can be directly upscaled to the input resolution and combined with the context prediction CP 00 output combination.

[0066] The same context prediction is done at both encoder and decoder side in order to entropy decode the stream correctly. Figure 6 Above, three levels have been defined, but the mechanism for context prediction can be applied to more or fewer levels, e.g., only two levels, or more than three levels.

[0067] In some embodiments, when the reduced tensor is not transmitted to the decoder, only the causal portion of the reduced tensor with respect to the position (x, y) currently being encoded / decoded may be used to determine the context prediction, e.g. using Figure 5 The kernel shown determines the context prediction.

[0068] In other embodiments, context prediction performed at lower levels can utilize the full kernel. In these embodiments, the reduced tensor is entropy encoded and transmitted to the decoder, in which case the fully decoded reduced tensor can be used to entropy decode higher-level tensors in context prediction. In these embodiments, on the decoder side, there is no need to reduce the tensor y because the reduced tensor is entropy decoded by the decoder and is therefore available to the decoder.

[0069] Figure 7 Shows the use of Figure 6 The context prediction is determined by using a multi-resolution tensor, and the context prediction is used to determine the entropy parameters for entropy encoding or entropy decoding of the input tensor y. The input potential tensor y is reduced to k+1 different levels of tensors y0, y1, ..., y k, where the spatial resolution of these tensors is (not strictly) monotonically decreasing, and y = y0. Figure 7 In the embodiment, these tensors are compressed, serialized, and their side information is used to derive the tensor ψ k ,…ψ0 is a compressed and serialized representation of pairs of y in reverse order k , ..., y0 are transmitted, and they are decoded in this same order. At each level, the current level tensor y l The entropy parameter μ l , σ l It can be predicted from any or all of the following: (i) the current level tensor y l causal context, (ii) any previously decoded level tensor y l+1 ,...,y k The complete context of , and (iii) the side information derived tensor ψ associated with the current level l l =h s (Q(z l ))=h s (Q(h a (y l )). Figure 7 An example of an architecture of k+1=3 levels using all three of the above-mentioned available information sources is visualized.

[0070] For j∈{l,...,k}, each y j is fed into the “context prediction” network CP lj The resulting output is then scaled up to match the spatial dimension and then aggregated via reduction operations (visualized by “+”) such as summing or concatenation. The aggregation result is fed to the entropy parameter network EP l , which also takes the side information ψ l As input, and output entropy parameter μ l , σ l These entropy parameters represent the Gaussian distribution Using this distribution, we can l Entropy coding is performed to generate a signal with rate R yl The bit stream and the side information tensor z l The associated bit stream rate R zl .

[0071] Figure 8 It is used to visualize Figure 7 In order to maintain causality, CP 00Since the other levels have already been decoded in this embodiment, they do not require further masking, and the entire kernel can be used to determine the context of the input tensor position y0 at these levels.

[0072] exist Figure 7 In the embodiment, in order to maintain causality, the constraint is that if j = l, then CP lj Do not use y l The elements that have not yet been decoded. If y l is being decoded in spatial raster scan order (e.g., top left to bottom right with horizontal scan lines), then as shown in [Minnen], in CP lj Including an initial mask convolution of “L” shape inside is sufficient to ensure causality. Also, note that for j>l, there is no need to lj There are no constraints introduced by causality, and it can initially use standard unmasked convolutions. Figure 8 Context window that visualizes the appropriate mask used during the reconstruction of y0.

[0073] Some embodiments described herein relate to methods and apparatus for encoding or decoding a tensor using context predictions obtained from different resolutions of the tensor.

[0074] In some embodiments, a tensor may be, for example, Figure 2-5 The autoencoder shown generates a latent representation of at least a portion of an image, an image, or a video.

[0075] Figure 9 An example of a method 900 for encoding a potential representing at least a portion of an image is shown, according to one embodiment. The method includes at least the following steps for at least one sample of a first tensor representing at least a portion of an image. At 901, a first context is obtained from at least one or more samples of at least one second tensor, the at least one second tensor being obtained by downsampling the first tensor at least once. For example, the first context is obtained as described above, and then the at least one second context is upsampled to a higher resolution, such as the resolution of the first tensor or a resolution of a next tensor level.

[0076] In one variation, a first context may also be determined using one or more causal samples of the first tensor (i.e., samples at the resolution of the input tensor). The context obtained from the one or more causal samples of the first tensor is combined with the upsampled second context. The combination may be an addition or aggregation of contexts.

[0077] At 902, at least one entropy parameter is determined based on the first context. In some variations, the determination of the entropy parameter may also use side information derived for the first tensor.

[0078] For example, the side information is a hyper-prior learned for the first tensor. In this variant, for example, Figure 4 and Figure 5 As shown, the method further includes encoding the learned hyper-prior for the first tensor.

[0079] At 903, the first tensor is entropy encoded using the determined at least one entropy parameter. In some variants, the at least one second tensor is also encoded so that all kernel samples can be used when determining the context prediction based on its level. In some embodiments, the entropy encoding of the at least one second tensor uses the same context prediction mechanism as the input tensor in a recursive manner, e.g., Figure 7 As stated.

[0080] Relatedly, Figure 10 An example of a method 100 for representing the potential decoding of at least a portion of an image according to one embodiment is shown. The method includes at least the following steps for at least one sample of a first tensor representing at least a portion of the image. At 1001, a first context is obtained from at least one or more samples of at least one second tensor, the at least one second tensor representing the first tensor at a lower resolution. At 1002, at least one entropy parameter is determined based on the first context. For correct entropy decoding of the first tensor, the first context and entropy parameter are obtained in a manner similar to that used on the encoding side. At 1003, the first tensor is entropy encoded using the determined at least one entropy parameter. In some variants, the lower resolution of the first tensor is entropy decoded before determining the first context.

[0081] In some embodiments, the above methods are used for neural network based encoders and decoders, respectively, such as autoencoders.

[0082] The embodiments described herein provide recursive level decoding. Similar to the recursive multi-level strategy used with wavelet-based compression, the pyramid approach described herein allows each level of the pyramid to reference a smaller version of itself during decoding. This can result in rate savings because the "lower resolution" version of a level is often a very good predictor of the "higher resolution" elements within the current level.

[0083] The described embodiments also provide a larger available spatial context window during reconstruction. Information about the larger "global" context can be used to improve the entropy parameter μ l , σ l The predictions of l , resulting in rate savings.

[0084] In some embodiments, the "future" context is fully available at a lower resolution. Compared to a single-stage mask autoregression, information about "future" decoded pixels is already available at a lower resolution. For example, in a single-stage raster scan autoregression, the context is restricted to anything that is spatially above or to the left of the pixel currently being decoded. On the other hand, some embodiments of the methods described herein also provide information about "future" decoded pixels that are spatially below or to the right of the pixel currently being decoded. This can help improve the entropy parameter μ l , σ l The predictions of l , resulting in rate savings.

[0085] The described embodiments provide potentially similar RD performance to heavier methods at a lower computational cost. As described in the variants, any autoregressive method can be selected to decode a given level. For example, by selecting a lighter autoregressive method (i.e., a method other than raster scan autoregressive) at the maximum level, the number of iterations of successive self-referencing operations that are difficult to parallelize can be significantly saved. Furthermore, the context captured is very similar to that of the more expensive autoregressive methods, except for perhaps any high-resolution more like patterns within the tensor, which is quite rare.

[0086] In some variations of this method, the l Any form of autoregression or no form of autoregression. Any level y l All can be reconstructed at once using raster scan autoregression, cross-context channel modeling (e.g., as described in [Ma]), checkerboard autoregressive reconstruction (as described in He et al.: D. He, Z. Yang, W. Peng, R. Ma, H. Qin, and Y. Wang, “ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding.” arXiv, Mar. 29, 2022. doi: 10.48550 / arXiv.2203.10886), or any other such method. Each level does not need to use the same reconstruction method.

[0087] In some variations, the l =G l (y l-1 ) using the gating function G l . Trainable masking operations such as convolutions (or multiplications with attention maps with sigmoid activations) can be included to allow the model to selectively choose what parts of what are important at various levels. This is Figure 11 The convolution operation is applied before the downsampling operation ( Figure 11 If any part of the tensor is deemed too expensive to be accurately reconstructed at any given level, the gating function can output a constant for that part of the tensor. It should be noted that the context prediction module CP lj We need to be able to exploit this. If the gating function is a convolution, a natural choice of gating function is a channel attention mechanism, although regular convolution can also provide its own advantages.

[0088] In a variation, the embodiments described herein may be used for progressive decoding. l Can be combined with the distortion loss term Related, among them It is from The decoded corresponding lower resolution image.

[0089] Figure 12 A block diagram of an autoencoder with a hyper-prior architecture and context prediction at different resolutions is shown, according to an embodiment. Figure 12 In FIG, only two levels of resolution are shown (level 0 and level 1), but more levels can also be used.

[0090] For example, the module of the autoencoder (such as the analysis function g a , synthesis function g s , Super Prior Network EE, Q and ED) have Figure 4 or Figure 5 The corresponding module has similar functions. Figure 12 In

[15] , once the input tensor x is transformed into latent y, latent y is scaled down to a lower spatial resolution than the spatial resolution of latent y. Both latent y and its scaled down version are quantized to produce quantized latents with 0 and 1 at each level. and exist Figure 12 In the embodiment, a hyper-prior network is used to learn the latent For example, a fully factorized prior model is used to encode the quantized latent representation of the side information ψ1ψ1 For potential For each spatial position (x, y), use the potential Causal sample application context prediction cp 11 For example, in Figure 12 In , the position (x, y) is shown as the center block sample in the 5×5 kernel, and the available samples (here, causal samples) are shown by the gray samples. The splicing is performed by the context prediction module cp 11The context and side information ψ1 of the output. The resulting concatenated vector is fed to the entropy parameter network ep1 to output The entropy parameters μ1 and σ1 are used for entropy encoding at the position (x, y).

[0091] On the decoder side, by Perform entropy decoding and transform the super-prior potential Transformed into distribution parameter ψ1 to reconstruct the potential Using the context prediction module cp 11 The output context has a potential The spatial position (x, y) is entropy decoded, and the context prediction module only considers In a similar way to the encoding side, the context is concatenated with the reconstructed side information ψ1, and the context is fed to the entropy parameter network ep1 to output the causal sample for The entropy parameters μ1 and σ1 are used for entropy decoding at the position (x, y).

[0092] For level 0, Figure 12 In one embodiment, a hyper-prior network is used to learn the latent For example, a fully factorized prior model is used to encode the quantized latent information representing the side information ψ0. For potential For each spatial position (x, y), the context prediction cp 00 Use potential Causal samples (using Figure 12 The context prediction cp is determined by using the gray samples in the kernel shown above. 01 Also use potential All samples (such as Figure 12 As shown, all samples on the used kernel are shown in gray, because it is considered that on the decoder side has been reconstructed). By the context prediction module cp 01 The output context is upsampled to the potential The spatial resolution of the , and is combined / added to the context prediction module cp 00 Output context. Concatenate the obtained context with the side information ψ0. Feed the obtained concatenated vector to the entropy parameter network ep0 to output the entropy parameters μ0, σ0 for entropy encoding the position (x, y) of ψ0.

[0093] On the decoder side, by Perform entropy decoding and transform the super-prior potential Transformed into distribution parameter ψ0 to reconstruct the potential The entropy parameters μ0, σ0 determined in a similar way to the encoder side are used to quantize the potential The spatial position (x, y) is entropy decoded. The causal sample determines the context prediction cp 00 . Also uses the decoded potential cp 01 All samples to determine the context prediction By the context prediction module cp 01 The output context is upsampled to the potential The spatial resolution of the , and is combined / added to the context prediction module cp 00 Output context. Concatenate the obtained context with the side information ψ0. Feed the obtained concatenated vector into the entropy parameter network ep0.

[0094] Once the potential is reconstructed It can be fed into the synthesis function g s To reconstruct the input tensor For example, an image, a video, or a portion of an image or video.

[0095] Figure 13 A block diagram of an embodiment of an autoencoder with a super-prior architecture and context prediction at different resolutions is shown according to another embodiment. Figure 13 The embodiment shown is Figure 12 The difference of the embodiment is that the lower level tensors are used The side information ψ1 for entropy coding is not learned, but is learned from the side information of the tensor at the input resolution ψ0 To do this, Figure 13 Above, the side information ψ1 is obtained by downsampling the side information ψ0.

[0096] Then, if Figure 12 As shown, the splicing is performed by the context prediction module cp 11 The determined context and the derived side information ψ1 are used to feed the context to the entropy parameter network ep1 for tensor Perform entropy coding. Figure 13 The rest of the scheme shown is similar to Figure 12 Therefore, in Figure 13 In the embodiment of the present invention, there is no need to perform a super-prior potential Encoding is performed to derive the side information ψ1 by downsampling the reconstructed side information ψ0.

[0097] Figure 14An embodiment of an apparatus 1400 for compressing, encoding, or decoding an image or video using the above-described method is shown. The apparatus includes a processor 1410 and can be interconnected to a memory 1420 via at least one port. Both the processor 1410 and the memory 1420 can also have one or more additional interconnections to external connections. The processor 1410 is also configured to insert or receive information in a bitstream and, when implemented by the processor, compress, encode, or decode using program code instructions that implement the above-described method. The program code to be loaded onto the processor 1410 to execute the various aspects described in this application can be stored in a storage device and subsequently loaded onto the memory 1420 for execution by the processor 1410. The memory 1420 can be of any type suitable for the technical environment and can be implemented using any appropriate data storage technology, such as, for example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 1410 may be of any type suitable to the technical environment and may include one or more of a microprocessor, a general purpose computer, a special purpose computer, a digital signal processor (DSP), a processor based on a single core architecture or a multi-core architecture, a sequential or parallel architecture, a dedicated circuit such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a signal processing device, and other processing circuits, as non-limiting examples.

[0098] According to an example of the present principle, Figure 15 As shown, in the context of a transmission between two remote devices A and B on a communication network NET, device A includes a processor associated with RAM and ROM, the RAM and ROM being configured to implement the method for encoding an image or video as described using the above method, and device B includes a processor associated with RAM and ROM, the RAM and ROM being configured to implement the method for decoding an image or video as described using the above method. According to one example, the network is a broadcast network adapted to broadcast / transmit the encoded image or video from device A to a decoding device including device B.

[0099] The signal to be transmitted by device A carries at least one bitstream including decoded data representing a tensor encoding of an image or video encoded according to the above method. In some embodiments, the signal further includes one or more of the following: decoded data representing one or more reduced versions of the tensor, decoded data representing side information learned for the tensor, and decoded data representing side information derived for the one or more reduced versions of the tensor.

[0100] Figure 16An example of the syntax of such a signal is shown when the decoded data is transmitted via a packet-based transport protocol. Each transmitted packet P comprises a header H and a payload PAYLOAD. According to an embodiment, the payload comprises the decoded data encoded according to any of the above-described embodiments.

[0101] Various methods are described herein, and each method includes one or more steps or actions for realizing the described method. Unless the correct operation of the method requires the steps or actions of a specific order, the order and / or use of the specific steps and / or actions may be modified or combined. In addition, terms such as "first", "second" and the like may be used to modify elements, components, steps, operations, etc. in various embodiments, such as, for example, "first decoding" and "second decoding". Unless otherwise specifically required, the use of these terms does not mean the sequencing of the modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding, and may, for example, be performed before, during, or in a time period overlapping with the second decoding.

[0102] The various methods and other aspects described in this application can be used as additional modules or to modify the Figure 2-5 The modules of the image or video autoencoder of the neural network shown. Unless otherwise indicated or technically excluded, the aspects described in this application can be used alone or in combination.

[0103] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes, and the described aspects are not limited to these specific values.

[0104] Various implementations involve decoding. As used herein, "decoding" may include, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by a decoder of the various implementations described herein.

[0105] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0106] Various implementations involve encoding. In a manner similar to the discussion above regarding "decoding," "encoding," as used in this application, may include, for example, all or part of the processes performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as partitioning, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processes also or alternatively include processes performed by the encoder of the various implementations described herein.

[0107] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0108] Note that the syntax elements used here are descriptive terms. Therefore, they do not exclude the use of other syntax element names.

[0109] This disclosure has described various pieces of information, such as syntax, that can be transmitted or stored, for example. This information can be packaged or arranged in various ways, including, for example, ways common in video standards, such as placing the information in an SPS, PPS, NAL unit, header (e.g., a NAL unit header or a slice header), or SEI message. Other ways, including, for example, ways common to system-level or application-level standards, are also possible, such as placing the information in one or more of the following:

[0110] a. SDP (Session Description Protocol), a format for describing multimedia communication sessions for the purpose of session announcement and session invitation, such as the SDP described in RFC and used in conjunction with RTP (Real-time Transport Protocol) transport.

[0111] b. DASH MPD (Media Presentation Description) descriptor, such as the DASH MPD descriptor used in DASH and transmitted over HTTP, a descriptor is associated with a representation or a representation set to provide additional characteristics to the content representation.

[0112] c. RTP header extensions, such as those used during RTP streaming.

[0113] d. ISO Base Media File Format, such as that used in OMAF, and the ISO Base Media File Format use boxes, also referred to as "atoms" in some specifications, which are object-oriented building blocks defined by a unique type identifier and length.

[0114] e. HLS (HTTP Live Streaming) manifest transmitted over HTTP. The manifest can be associated with a version or set of versions of the content, for example, to provide characteristics of the version or set of versions.

[0115] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0116] The implementations and aspects described herein can be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed can also be implemented in other forms (e.g., a device or program). For example, an apparatus can be implemented with appropriate hardware, software, and firmware. The method can be implemented in, for example, a processor, which generally refers to a processing device that includes, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication equipment, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other equipment that facilitates information communication between end users.

[0117] References to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations in various places throughout this application are not necessarily all referring to the same embodiment.

[0118] Additionally, the present application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, and retrieving information from a memory.

[0119] Furthermore, the present application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, and estimating information.

[0120] Additionally, this application may refer to "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information can include, for example, one or more of accessing information and retrieving information (e.g., from a memory device). Furthermore, during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is often involved in one way or another.

[0121] It should be understood that examples using any of the following “ / ,” “and / or,” and “at least one of” (as in the case of “A / B,” “A and / or B,” and “at least one of A and B”) are intended to encompass selection of only the first-listed option (A), or only the second-listed option (B), or both options (A and B). As a further example, in the case of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to include selection of only the first-listed option (A), or only the second-listed option (B), or only the third-listed option (C), or only the first and second-listed options (A and B), or only the first and third-listed options (A and C), or only the second and third-listed options (B and C), or all three options (A, B, and C). This can be extended to multiple items listed, as will be apparent to one of ordinary skill in this and related arts.

[0122] In addition, as used herein, the term "signaling" refers specifically to indicating certain information to a corresponding decoder. Thus, in one embodiment, the same parameters are used on both the encoder and decoder sides. Thus, for example, the encoder can transmit (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters as well as other parameters, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select the specific parameters. By avoiding the transmission of any actual functionality, bit savings are achieved in various embodiments. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to the corresponding decoder. Although the foregoing relates to the verb form of the term "signaling", the term "signal" can also be used as a noun in this document.

[0123] As will be apparent to one of ordinary skill in the art, implementations can generate various signals formatted to carry information that can be stored or transmitted, for example. The information can include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream of the described embodiments. Such a signal can be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. As is known, the signal can be transmitted over a variety of different wired or wireless links. The signal can be stored on a processor-readable medium.

[0124] We have described several embodiments. Features of these embodiments may be provided individually or in any combination across the various claim classes and types. In addition, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination across the various claim classes and types.

Claims

1. A method comprising, for at least one sample of a first tensor representing at least a portion of an image: obtaining a first context from at least one or more samples of at least one second tensor, the at least second tensor being obtained by downsampling the first tensor at least once; determining at least one entropy parameter based on the first context; as well as The first tensor is entropy encoded using the determined at least one entropy parameter.

2. The method according to claim 1, wherein obtaining the first context comprises: determining at least one second context from the one or more samples of the at least one second tensor; as well as The at least one second context is upsampled to a resolution of the first tensor. 3 . The method of claim 2 , wherein the first context is further obtained based on a third context determined from one or more causal samples of the first tensor. The method of claim 3 , wherein obtaining the first context comprises combining the third context with at least one upsampled second context. The method of claim 1 , wherein the at least one entropy parameter is further determined based on side information derived for the first tensor. 6 . The method of claim 5 , wherein the side information derived for the first tensor is a first super-prior learned for the first tensor, the method further comprising encoding the first super-prior. The method of claim 6 , wherein determining the at least one entropy parameter comprises concatenating the first super-prior and the first context.

8. The method of claim 7, wherein the at least one entropy parameter is determined based on a network comprising at least one convolutional layer.

9. The method of claim 1, further comprising entropy encoding the at least one second tensor.

10. The method of claim 9, wherein the at least one second tensor is entropy encoded based on at least one entropy parameter determined using a context, the context being obtained for at least one sample of the at least one second tensor based on one or more causal samples of the at least one second tensor. 11 . The method of claim 9 , wherein the at least one second tensor is entropy encoded based on at least one entropy parameter determined from side information derived for the at least one second tensor. 12 . The method of claim 11 , wherein the side information derived for the at least one second tensor is a second super-prior derived for the at least one second tensor, the method further comprising encoding the second super-prior.

13. The method of claim 11, wherein the side information derived for the at least one second tensor is a downsampled super-prior derived for the first tensor. The method of claim 1 , wherein the at least one entropy parameter represents a Gaussian distribution of the at least one sample of the first tensor.

15. The method of claim 1, wherein the at least one second tensor is obtained using a masking operation applied to the first tensor prior to the at least one downsampling of the first tensor.

16. The method of claim 15, wherein the masking operation is one of a convolution or a multiplication with a sigmoid activated attention map.

17. The method of claim 1, wherein the first tensor is obtained from an encoding of the at least a portion of the image using a neural network.

18. The method of claim 17, wherein the neural network is an autoencoder.

19. An apparatus comprising one or more processors configured to entropy encode a first tensor representing at least a portion of an image according to the method of claim 1.

20. A method comprising reconstructing a first tensor representing at least a portion of an image, the method comprising, for at least one sample of the first tensor: obtaining a first context from at least one or more samples of at least one second tensor, the at least one second tensor representing the first tensor at a lower resolution; determining at least one entropy parameter based on the first context; as well as The first tensor is entropy decoded using the determined at least one entropy parameter.

21. The method of claim 20, wherein obtaining the first context comprises: determining at least one second context from the one or more samples of the at least one second tensor; as well as The at least one second context is upsampled to a resolution of the first tensor.

22. The method of claim 21, wherein the first context is further obtained based on a third context determined from one or more causal samples of the first tensor. The method of claim 22 , wherein obtaining the first context comprises combining the third context with at least one upsampled second context.

24. The method of claim 20, wherein the at least one entropy parameter is further determined based on side information derived for the first tensor.

25. The method of claim 24, wherein the side information decoded for the first tensor is a first super-prior derived for the first tensor, the method further comprising decoding the first super-prior.

26. The method of claim 25, wherein determining the at least one entropy parameter comprises concatenating the first super-prior and the first context.

27. The method of claim 26, wherein the at least one entropy parameter is determined based on a network comprising at least one convolutional layer.

28. The method of claim 20, further comprising entropy decoding the at least one second tensor.

29. The method of claim 28, wherein the at least one second tensor is entropy decoded based on at least one entropy parameter determined using a context, the context being obtained for at least one sample of the at least one second tensor based on one or more causal samples of the at least one second tensor.

30. The method of claim 28, wherein the at least one second tensor is entropy decoded based on at least one entropy parameter determined from side information derived for the at least one second tensor.

31. The method of claim 30, wherein the side information derived for the at least one second tensor is a second super-prior derived for the at least one second tensor, the method further comprising decoding the second super-prior.

32. The method of claim 30, wherein the side information derived for the at least one second tensor is a downsampled super-prior derived for the first tensor.

33. The method of claim 20, wherein the at least one entropy parameter represents a Gaussian distribution of the at least one sample of the first tensor.

34. The method of claim 20, wherein the at least a portion of the image is reconstructed from the first tensor using a neural network.

35. The method of claim 34, wherein the neural network is a decoder portion of an autoencoder.

36. An apparatus comprising one or more processors configured to reconstruct a first tensor representing at least a portion of an image according to the method of claim 20.

37. A signal comprising first decoded data representing a first tensor encoding at least a portion of an image and second decoded data representing at least one second tensor, wherein the at least second tensor represents the first tensor at a lower resolution, and the at least one second tensor is used to obtain a first context, and the first context is used to determine at least one entropy parameter for entropy decoding the first tensor.

38. The signal of claim 37, further comprising side information, the side information being derived for the first tensor and used to determine the at least one entropy parameter for entropy decoding the first tensor.

39. The signal of claim 38, wherein the side information is a first hyper-prior derived for the first tensor.

40. The signal of claim 37, wherein the signal further comprises side information, the side information being derived for the at least one second tensor and being used to determine at least one entropy parameter for entropy decoding the at least one second tensor.

41. The signal according to claim 40, wherein the side information derived for the at least one second tensor is a second super-prior derived for the at least one second tensor.

42. A computer-readable storage medium having stored thereon the signal of claim 37.

43. A computer-readable storage medium having stored thereon instructions for causing one or more processors to execute the method according to claim 1.

44. A computer-readable storage medium having stored thereon instructions for causing one or more processors to execute the method according to claim 20.

45. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to implement the method according to claim 1.

46. ​​A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to implement the method according to claim 20.

47. An apparatus comprising: The device according to claim 34; as well as At least one of: (i) an antenna configured to receive the signal of claim 37, (ii) a band limiter configured to limit the received signal to a frequency band including the first decoded data and the second decoded data, and (iii) a display configured to display a reconstructed version of the at least a portion of the image.

48. The device of claim 47, comprising a TV, a cell phone, a tablet or a set-top box.

Images