Data processing method, device and medium

Abstract

An embodiment of the present disclosure provides a solution for data processing. A data processing method is proposed, the method comprising the steps of: determining, during conversion between data and a bitstream of the data, a first portion of a first sample of a reconstructed latent representation of the data, the first portion being indicative of a prediction of the first sample; determining a second portion of the first sample, the second portion being indicative of a difference between the first sample and the first portion; and performing the conversion based on the second portion.

Description

[Technical field]

[0001] TECHNICAL FIELD Embodiments of the present disclosure relate generally to data processing techniques, and more specifically to neural network-based data coding. [Background technology]

[0002] In the past decade, deep learning has developed rapidly in various fields, especially in computer vision and image processing. Neural networks were originally invented in the interdisciplinary research of neuroscience and mathematics. Neural networks have demonstrated powerful capabilities in the context of nonlinear transformation and classification. Neural network-based image / video compression technology has made remarkable progress in the past five years. It has been reported that the latest neural network-based image compression algorithms achieve rate-distortion (RD) performance comparable to Versatile Video Coding (VVC). Due to the continuous improvement of neural image compression performance, neural network-based video compression has become an actively developed research field. However, the coding efficiency of neural network-based image / video coding is generally expected to be further improved. Summary of the Invention

[0003] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure provide a solution for data processing.

[0004] In a first aspect, a data processing method is proposed, comprising the steps of: determining, during a conversion between data and a bitstream of said data, a first portion of a first sample of a reconstructed latent representation of said data, said first portion being indicative of a prediction of said first sample, determining a second portion of said first sample, said second portion being indicative of a difference between said first sample and said first portion, and performing said conversion based on said second portion.

[0005] According to the method according to the first aspect of the present disclosure, the reconstructed latent data samples are split into two parts, which allows the sequential entropy coding process to be decoupled from the computationally complex neural network. Compared to the inversion solution, in which the entropy coding process and the neural network operations are interleaved, the proposed method advantageously allows the entropy coding process to be performed independently from the neural network, thus improving the coding efficiency.

[0006] In a second aspect, an apparatus for processing data is proposed, said apparatus for processing data comprising a processor and a non-transitory memory with instructions that, when executed by the processor, cause the processor to perform a method according to the first aspect of the present disclosure.

[0007] In a third aspect, a non-transitory computer readable storage medium is proposed, said non-transitory computer readable storage medium storing instructions for causing a processor to perform the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect, another non-transitory computer-readable storage medium is proposed, the non-transitory computer-readable storage medium storing a bitstream of data generated by a method executed by a data processing apparatus, the method comprising the steps of determining a first portion of a first sample of a reconstructed latent representation of the data, the first portion being indicative of a prediction of the first sample, determining a second portion of the first sample, the second portion being indicative of a difference between the first sample and the first portion, and generating the bitstream based on the second portion.

[0009] In a fifth aspect, a method for storing a bitstream of data is proposed, the method comprising the steps of determining a first portion of a first sample of a reconstructed latent representation of the data, the first portion being indicative of a prediction of the first sample, determining a second portion of the first sample, the second portion being indicative of a difference between the first sample and the first portion, generating the bitstream based on the second portion, and storing the bitstream on a non-transitory computer-readable recording medium.

[0010] This description is provided to introduce a selection of concepts in a simplified form that are more fully described in the detailed description. This description is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. [Brief description of the drawings]

[0011] The above and other objects, features, and advantages of the embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: In the exemplary embodiments of the present disclosure, like reference numerals generally refer to like components.

[0012] [Figure 1] 1 shows a block diagram illustrating an example data coding system according to some embodiments of the present disclosure.

[0013] [Diagram 2] 1 shows a general transform coding scheme.

[0014] [Diagram 3] 1 shows an image from the Kodak dataset and different representations of said image.

[0015] [Figure 4] We show the network architecture of an autoencoder that embodies the hyperprior model.

[0016] [Diagram 5] A block diagram of a composite model is shown.

[0017] [Figure 6] 1 illustrates the encoding process of a composite model.

[0018] [Figure 7] 4 illustrates the decoding process of a composite model.

[0019] [Figure 8] This shows the problem of the decoder network.

[0020] [Figure 9] We present an entropy coding sub-network in a state-of-the-art image decoding architecture.

[0021] [Figure 10] 4 illustrates a decoding process according to some embodiments of the present disclosure.

[0022] [Figure 11] 4 illustrates another decoding process according to some embodiments of the present disclosure.

[0023] [Figure 12] 4 illustrates an encoding process according to some embodiments of the present disclosure.

[0024] [Figure 13] 4 illustrates another encoding process according to some embodiments of the present disclosure.

[0025] [Figure 14] 4 illustrates an example of a data decoding process according to some embodiments of the present disclosure.

[0026] [Figure 15]1 illustrates an example of a data encoding process according to some embodiments of the present disclosure.

[0027] [Figure 16] 1 shows a flowchart of a data processing method according to some embodiments of the present disclosure.

[0028] [Figure 17] FIG. 1 illustrates a block diagram of a computing device capable of implementing various embodiments of the present disclosure.

[0029] Throughout the drawings, the same or similar reference numbers typically refer to the same or similar elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Next, the principles of the present disclosure will be described with reference to some embodiments. It should be understood that these embodiments are provided for illustrative purposes only, to help those skilled in the art understand and embody the present disclosure, and do not imply any limitations on the scope of the present disclosure. The disclosure described herein can be embodied in various ways other than the methods described below.

[0031] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0032] References in this disclosure to "one embodiment," "one embodiment," "exemplary embodiment," and the like indicate that the described embodiment may include a particular feature, structure, or characteristic, but not all embodiments necessarily include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an exemplary embodiment, it is noted that it is within the knowledge of one of ordinary skill in the art to affect such feature, structure, or characteristic in connection with other embodiments, whether or not explicitly described.

[0033] Terms such as "first" and "second" may be used herein to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0034] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly dictates otherwise. It will be further understood that the terms "comprise," "comprising," "having," "having," "containing," and / or "including," as used herein, specify the presence of stated features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0035] Example Environment 1 is a block diagram illustrating an example data coding system 100 that may utilize techniques of this disclosure. As shown, the data coding system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a data encoding device, and the destination device 120 may also be referred to as a data decoding device. In operation, the source device 110 may be configured to generate encoded data, and the destination device 120 may be configured to decode the encoded data generated by the source device 110. The source device 110 may include a data source 112, a data encoder 114, and an input / output (I / O) interface 116.

[0036] Data sources 112 may include sources such as data capture devices, examples of which include, but are not limited to, an interface for receiving data from a data provider, a computer graphics system for generating data, and / or combinations thereof.

[0037] The data may include one or more pictures or one or more images of a video. The data encoder 114 encodes data from the data source 112 to generate a bitstream. The bitstream may include a sequence of bits forming a coded representation of the data. The bitstream may include a coded picture and associated data. A coded picture is a coded representation of a picture. The associated data may include a sequence parameter set, a picture parameter set, and other syntax structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The encoded data may be transmitted directly to the destination device 120 through the network 130A via the I / O interface 116. The encoded data may be stored in a storage medium / server 130B for access by the destination device 120.

[0038] Destination device 120 may include an I / O interface 126, a data decoder 124, and a display device 122. I / O interface 126 may include a receiver and / or a modem. I / O interface 126 may obtain encoded data from source device 110 or storage medium / server 130B. Data decoder 124 may decode the encoded data. Display device 122 may display the decoded data to a user. Display device 122 may be integrated with destination device 120 or may be external to destination device 120 configured to interface with an external display device.

[0039] The data encoder 114 and the data decoder 124 may operate according to a data coding standard, such as a video coding standard or a still picture coding standard, and other current and / or future standards.

[0040] Some exemplary embodiments of the present disclosure are described in detail below. Section headings are used herein for ease of understanding, but it should be understood that they do not limit the embodiments disclosed in a section to that section alone. Furthermore, although certain embodiments are described with reference to versatile video coding or other specific data codecs, the disclosed techniques are applicable to other coding techniques. Furthermore, although some embodiments describe coding steps in detail, it will be understood that the corresponding step of decoding that undoes the coding may be embodied by a decoder. Furthermore, the term data processing encompasses data coding or compression, data decoding or decompression, and data transcoding, where data is represented from one compressed format to another compressed format or at a different compressed bit rate. [1. Overview] A neural network based image and video compression method including an autoregressive sub-network and an entropy coding engine, where the entropy coding is performed independently of the autoregressive sub-network. [2. Background] In the past decade, deep learning has been rapidly developed in various fields, especially in computer vision and image processing. Inspired by the great success of deep learning technology in the field of computer vision, many researchers have shifted their attention from traditional image / video compression technology to neural image / video compression technology. Neural networks were originally invented in the interdisciplinary research of neuroscience and mathematics. Neural networks have demonstrated their powerful capabilities in the background of nonlinear transformation and classification. Neural network-based image / video compression technology has made remarkable progress in the past five years. It has been reported that the latest neural network-based image compression algorithm achieves RD performance comparable to Versatile Video Coding (VVC), the latest video coding standard developed by MPEG and VCEG experts and the Joint Video Experts Team (JVET). Due to the continuous improvement of neural image compression performance, neural network-based video compression has become an actively developed research field. However, due to the unique difficulty of the problem, neural network-based video coding is still in its infancy. [2.1. Image / Video Compression] Image / video compression usually refers to computing techniques that compress images / videos into binary code for easier storage and transmission. The binary code may or may not support lossless reconstruction of the original image / video, which is called lossless compression and lossy compression. Since lossless reconstruction is not necessary in most scenarios, most efforts are directed towards lossy compression. Usually, the performance of an image / video compression algorithm is evaluated from two aspects: compression ratio and reconstruction quality. Compression ratio is directly related to the number of binary codes, the fewer the better. Reconstruction quality is measured by comparing the reconstructed image / video with the original image / video, the higher the better. Image / video compression techniques are divided into two branches: traditional video coding methods and neural network-based video compression methods. Traditional video coding methods employ transform-based solutions, where researchers exploit statistical dependencies in latent variables (e.g., DCT or wavelet coefficients) by carefully hand-engineering entropy codes that model the dependencies in the quantization domain. There are two types of neural network-based video compression: neural network-based coding tools and end-to-end neural network-based video compression. The former is incorporated into existing traditional video codecs as a coding tool and works only as part of the framework, while the latter is a separate framework developed based on neural networks without relying on traditional video codecs. In the past 30 years, a series of traditional video coding standards have been developed to accommodate the increasing visual content. International standardization organization ISO / IEC has two expert groups, Joint Photographic Experts Group (JPEG) and Moving Picture Experts Group (MPEG), while ITU-T also has its own Video Coding Experts Group (VCEG) which aims to standardize image / video coding technologies. Influential video coding standards published by these organizations include JPEG, JPEG 2000, H.262, H.264 / AVC, and H.265 / HEVC. After H.265 / HEVC, the Joint Video Experts Team (JVET) formed by MPEG and VCEG has been working on a new video coding standard, Versatile Video Coding (VVC). The first version of VVC was released in July 2020. VVC has been reported to offer an average bitrate reduction of 50% compared to HEVC for the same visual quality. Neural network-based image / video compression is not a new invention, as there have been many researchers working on neural network-based image coding. However, the network architecture was relatively shallow and the performance was not satisfactory. With the support of abundant data and powerful computing resources, neural network-based methods have been better utilized in various applications. At present, neural network-based image / video compression has shown promising improvements and its feasibility has been confirmed. However, the technology is still far from mature and many challenges need to be addressed. [2.2 Neural Networks] Neural networks, also known as artificial neural networks (ANNs), are computational models used in machine learning techniques that typically consist of multiple processing layers, each layer consisting of multiple simple but nonlinear elementary computational units. One advantage of such deep networks is believed to be their ability to process data at multiple levels of abstraction and transform data into different kinds of representations. It is noted that these representations are not manually designed, but rather deep networks, including the processing layers, are learned from large amounts of data using common machine learning procedures. Deep learning is considered particularly useful for processing natively unstructured data, such as acoustic and visual signals, since it eliminates the need for manually created representations, while processing such data has been a long-standing challenge in the field of artificial intelligence. [2.3 Neural networks for image compression] Existing neural networks used in image compression methods can be classified into two categories: pixel probability modeling and autoencoders. The former belong to predictive coding strategies, while the latter are transform-based solutions. In the literature, these two methods are sometimes combined. [2.3.1 Pixel Probability Modeling] According to Shannon's information theory [6], the optimal method for lossless coding is one that can reach the minimum coding rate -log2p(x), where p(x) is the probability of symbol x. Numerous lossless coding methods have been developed in the literature, among which arithmetic coding is considered to be one of the optimal methods. Given a probability distribution p(x), arithmetic coding guarantees that the coding rate is as close as possible to the theoretical limit -log2p(x), without considering the rounding error. Therefore, the remaining problem is how to determine the probability, which is very challenging for natural images / videos due to the curse of dimensionality. According to a predictive coding strategy, one way to model p(x) is to predict pixel probabilities one by one in raster scan order based on previous observations, where x is the image.

number

number

number

number

number

[0041] Further details of embodiments of the present disclosure related to neural network-based data coding are provided below. As used herein, the term "data" may refer to images, pictures in a video, or other data suitable for coding.

[0042] As mentioned above, existing image compression networks include an autoregressive model (e.g., a context model) to improve compression performance. However, the autoregressive model is interleaved with an entropy decoding process that is inherently serial. In this respect, the decoding process is inherently serial and cannot be efficiently parallelized, making the decoding process very slow.

[0043] To solve the above problems and some other problems not mentioned, the data processing solution described below is disclosed.

[0044] 14 illustrates an example data decoding process 1400 according to some embodiments of the present disclosure. For example, the data decoding process 1400 may be performed by the data decoder 124 as shown in FIG. It should be understood that the data decoding process 1400 may also include additional blocks not shown and / or blocks that are shown may be omitted. The scope of the present disclosure is not limited in this respect.

[0045] As shown in Figure 14, the bitstream may be input to a first entropy decoder 1410. The first entropy decoder 1410 may decode the bitstream based on the probability distribution information generated by the decomposition entropy sub-network 1420. In some embodiments, the decomposition entropy sub-network 1420 may generate the probability distribution information using a predefined template, e.g., in the case of a Gaussian distribution, using predefined mean and variance values. The entropy decoding process performed by the first entropy decoder 1410 may be an arithmetic decoding process, a Huffman decoding process, etc.

[0046] The output of the first entropy decoder 1410 may include a second quantized hyper-latent representation of the data (denoted as z^ in FIG. 14). The second quantized hyper-latent representation may be processed by the hyper-scale decoder sub-network 1424 (also referred to hereinafter as the fifth sub-network) to generate second hyper information. By way of example and not limitation, the second hyper information may include second probability distribution information (also referred to as statistical information or probability parameters) of samples of the latent representation of the data. In the example shown in FIG. 14, the second probability distribution information may include the variance of the latent samples (denoted as σ in FIG. 14). In another example, the second probability distribution information may include the standard deviation of the latent samples. It should be understood that the probability distribution information may include any other suitable information. The scope of the present disclosure is not limited in this respect.

[0047] The second entropy decoder 1412 may decode the bitstream by performing an entropy decoding process on the bitstream based on the second hyper information. In one example, the entropy decoding process may be performed using a zero-mean probability distribution. Additionally or alternatively, the entropy decoding process may be performed using a variance. The entropy decoding process performed by the second entropy decoder 1412 may be an arithmetic decoding process, a Huffman decoding process, etc.

[0048] The output of the second entropy decoder 1412 may include a second portion (denoted as ŵ in FIG. 14) of the first sample of the reconstructed latent representation of the data (i.e., the current sample reconstructed at the decoder). As used herein, the term "reconstructed latent representation" means that the samples in the representation are obtained through a reconstruction process. In one example, the second portion may be decoded from a sub-bitstream of the bitstream. By way of example and not limitation, the second portion may be referred to as a quantized residual or a residual of the first sample.

[0049] At block 1430, the first sample may be reconstructed based on the second portion of the first sample and the first portion (denoted as μ in FIG. 14). By way of example and not limitation, the first sample may be determined to be the sum of the first portion and the second portion. Given that the second portion of the sample of the reconstructed latent representation is quantized in an encoder, which is described in more detail below, the reconstructed latent representation may also be referred to as a quantized latent representation.

[0050] The first portion of the first sample may be determined based on a sample set of the reconstructed latent representation. By way of example and not limitation, the set of samples may include a plurality of decoded adjacent samples of the first sample. In one example, the set of samples may be adjacent to the first sample. In another example, at least one sample in the set of samples may be non-adjacent to the first sample. Alternatively, the set of samples may include only one sample. It should be appreciated that the set of samples may include any other suitable samples of the reconstructed latent representation. The scope of the present disclosure is not limited in this respect.

[0051] As shown in FIG. 14 , the set of samples may be input to a context sub-network 1426, also referred to as a first sub-network hereinafter. In some embodiments, the context sub-network 1426 is autoregressive. The context sub-network 1426 generates intermediate information based on the set of samples. By way of example and not limitation, the intermediate information may reflect an average value of the set of samples. The prediction sub-network 1428 (also referred to as a second sub-network hereinafter) may generate a first portion of the first sample based on the output of the context sub-network 1426. In one example, the first portion may be a prediction of the first sample. In another example, the first portion may be a predicted average value of the first sample. It should be understood that the context sub-network 1426 may also be referred to as a context model, a context model sub-network, and / or the like. Additionally, the prediction sub-network may also be referred to as a fusion sub-network, a fusion sub-network, and / or the like.

[0052] In some additional embodiments, to generate the first portion of the first sample, the prediction sub-network 1428 may utilize further information in addition to the output of the context sub-network 1426. In one example, the prediction sub-network 1428 may generate the first portion of the first sample based on the output of the context sub-network 1426 and the first hyper information, as described in more detail below.

[0053] In such a case, the output of the first entropy decoder 1410 may further include a first quantized hyper-latent representation of the data (denoted as z1^ in FIG. 14). In one example, the first quantized hyper-latent representation may be the same as the second quantized hyper-latent representation. Alternatively, the first quantized hyper-latent representation may be different from the second quantized hyper-latent representation. In this case, the first quantized hyper-latent representation may be decoded from a first sub-bitstream of the bitstream, and the second quantized hyper-latent representation may be decoded from a second sub-bitstream of the bitstream. It should be understood that the first quantized hyper-latent representation and the second first quantized hyper-latent representation may be obtained in any other suitable manner. The scope of the present disclosure is not limited in this respect.

[0054] The first quantized hyper-latent representation may be processed by the hyper-decoder sub-network 1422 (hereinafter also referred to as the third sub-network) to generate first hyper-information. By way of example and not limitation, the first hyper-information may include first probability distribution information (also referred to as statistical information or probability parameters) of samples of the latent representation of the data. In one example, the first probability distribution information may include a mean value of the latent samples. Additionally or alternatively, the first hyper-information may include prediction information of the latent samples. It should be understood that the probability distribution information may include any other suitable information. The scope of the present disclosure is not limited in this respect.

[0055] After obtaining a reconstructed latent representation of the data, a synthesis transform may be performed on the reconstructed latent representation in a synthesis transform sub-network 1432 to obtain reconstructed data 1434, i.e., a reconstruction of the data.

[0056] It can be seen that in the data decoding process 1400, the entropy coding process in the second entropy decoder 1412 is performed without using inputs from the context sub-network 1426 and the prediction sub-network 1428. In this respect, the proposed data decoding process allows for decoupling of the sequential entropy coding process from the computationally complex neural network. Thereby, the proposed decoding process advantageously allows for the entropy coding process to be performed independently from the neural network, thus improving coding efficiency.

[0057] A data decoding process according to some embodiments of the present disclosure has been described above, and a corresponding data encoding process will now be described with reference to FIG.

[0058] 15 illustrates an example data encoding process 1500 according to some embodiments of the present disclosure. For example, the data encoding process 1500 may be performed by the data encoder 114 as shown in FIG. It should be understood that the data encoding process 1500 may also include additional blocks not shown and / or blocks that are shown may be omitted. The scope of the present disclosure is not limited in this respect.

[0059] As shown in FIG. 15, in the analysis transformation sub-network 1512, an analysis transformation may be performed on the data 1510 to obtain a latent representation (denoted as y in FIG. 15) of the data 1510. The data 1510 may include one or more pictures in an image or video. The latent representation may be processed by the hyper-encoder sub-network 1530 (hereinafter also referred to as the fourth sub-network) to generate a hyper-latent representation. In the quantization block 1532, the generated hyper-latent representation may be quantized to obtain a quantized hyper-latent representation. The quantized hyper-latent representation may be encoded into a bitstream that may be a part of the bitstream of data based on the probability distribution information generated by the decomposition entropy sub-network 1536. In one example, the quantized hyper-latent representation may include the above-mentioned second quantized hyper-latent representation. In another example, the quantized hyper-latent representation may further include the above-mentioned first quantized hyper-latent representation. The entropy encoder 1534 may perform an entropy encoding process on the quantized hyper-latent representation to obtain a part of the bitstream. Further, based on the probability distribution information generated by the decomposition entropy sub-network 1536, an entropy decoding process may be performed on a portion of the bitstream in the entropy decoder 1538 to reconstruct the quantized hyper-latent representation.

[0060] At block 1514, a residual may be obtained based on a difference between the second sample of the latent representation and the first portion of the reconstructed second sample, where the second sample corresponds to the first sample, and the latent representation corresponds to the reconstructed latent representation. In other words, the first sample is the reconstructed second sample, i.e., a reconstructed version of the second sample.

[0061] The first portion of the first sample may be generated using the prediction sub-network 1522 and the context sub-network 1524 in a manner similar to the data decoding process 1400. In some embodiments, the second sample may be quantized before being processed in block 1514. Alternatively, the second sample may not be quantized.

[0062] The residual may be quantized in quantization block 1516 to obtain a second portion of the first sample. In this case, the second portion is the quantized residual of the first sample. Alternatively, the residual may not be quantized, and thus block 1516 may be omitted. The first sample may be determined to be the sum of the first and second portions in block 1518.

[0063] To obtain a further portion of the bitstream, the entropy encoder 1520 may perform an entropy encoding process on a second portion of the samples of the reconstructed latent representation based on the second hyper information. The second hyper information may be generated by the hyper-scale decoder sub-network 1528 based on the second quantized hyper-latent representation of the data 1510 in a manner similar to the data decoding process 1400.

[0064] In one example, the entropy encoding process may be performed using a zero-mean probability distribution. Additionally or alternatively, the entropy encoding process may be performed using a variance. The entropy encoding process performed by the entropy encoders 1520, 1534 may be an arithmetic encoding process, a Huffman encoding process, or the like.

[0065] Although an exemplary data coding process is described above with respect to Figures 14 and 15, it should be understood that other suitable variations of the data coding process are also contemplated in light of this disclosure. In another exemplary data coding process, a lightweight hyper-decoder sub-network may be used to generate a first portion of the first samples based on the first quantized hyper-latent representation, and the context sub-network and the prediction sub-network may be removed.

[0066] In a further exemplary data coding process, a multi-stage context model may be employed and the predictive sub-network may be removed. In such a process, the latent representation of the data may be divided into a number of regions, each of which may include four latent samples, hereinafter referred to as the first latent, the second latent, the third latent, and the fourth latent. Since all the regions are processed in parallel, four successive steps are involved in the decoder to progressively reconstruct the latent representation.

[0067] In the first step, only the hyper priors are used to generate entropy parameters of the first latent for entropy decoding and reconstruction. The decoded first latent is then processed with a masked 3x3 convolution to generate second context features for the second stage. In the second step, the co-located hyper priors and second context features are processed to generate appropriate entropy parameters for reconstructing the second latent, which are then convolved to derive third context features. In the third step, both the first step hyper priors and the first and second step context features are used to derive entropy parameters for properly decoding the third latent. Similarly, the third latent is then convolved to derive fourth context features for the fourth step. Finally (in the fourth step), the fourth latent is reconstructed in a similar manner as in the previous step to obtain the full reconstructed latent representation.

[0068] It should be understood that the above descriptions and / or examples are provided for illustrative purposes only. The scope of the present disclosure is not limited in this respect. The embodiments of the present disclosure should be considered as examples for illustrating the general concept and should not be construed in a narrow sense. Moreover, these embodiments may be applied individually or in any combination.

[0069] FIG. 16 illustrates a flowchart of a method 1600 of data processing according to some embodiments of the present disclosure. The method 1600 may be embodied during conversion between data and a bitstream of data. As shown in FIG. 16, the method 1600 begins at 1602, where a first portion of a first sample of a reconstructed latent representation of the data may be determined. The first portion represents a prediction of the first sample. By way of example and not limitation, the first portion of the first sample may be determined based on a set of samples of the reconstructed latent representation. In one example, the first portion may be a prediction of the first sample. Alternatively, the first portion may be a predicted average value of the first sample. In some embodiments, the reconstructed latent representation may be a quantized latent representation of the data.

[0070] In some embodiments, the first sub-network may be used to generate intermediate information based on the set of samples. Furthermore, the first portion may be generated based on the intermediate information by the second sub-network. By way of example and not limitation, the first sub-network may be autoregressive and may be referred to as a context model sub-network, a context sub-network, a context model, and / or the like. Moreover, the second sub-network may be referred to as a predictive sub-network, a fusion sub-network, a predictive fusion sub-network, and / or the like.

[0071] In some alternative embodiments, the first portion may be generated based on the first quantized hyper-latent representation. For example, the generation of the first portion may include processing the first quantized hyper-latent representation using a lightweight hyper-decoder sub-network, which may also be referred to as a hyper-decoder sub-network. By way of example, an output of the processing of the first quantized hyper-latent representation may be determined as the first portion of the first samples. The generation of the first quantized hyper-latent representation is described in more detail below.

[0072] At 1604, a second portion of the first sample is determined. The second portion indicates a difference between the first sample and the first portion. In one example, the second portion may be a difference between the first sample and the first portion. As an example, the second portion may be obtained by subtracting the first portion from the first sample. The second portion may also be referred to as a residual or a quantized residual of the first sample.

[0073] At 1606, a transformation is performed based on the second portion. In one example, the transformation may include encoding the data into a bitstream. Alternatively or additionally, the transformation may include decoding the data from the bitstream. It should be understood that the above description is for illustrative purposes only. The scope of the present disclosure is not limited in this respect.

[0074] Considering the foregoing, the reconstructed latent samples are split into two parts, which allows for decoupling the sequential entropy coding process from the computationally complex neural network. Compared to conversion solutions, where the entropy coding process and the neural network operations are interleaved, the proposed method advantageously allows for the entropy coding process to be performed independently from the neural network, thus improving coding efficiency.

[0075] In some alternative embodiments, at 1602, the first sub-network may be used to generate intermediate information based on the set of samples. Additionally, the third sub-network may be used to determine the first hyper information based on the first quantized hyper latent representation. Furthermore, the first portion may be generated based on the intermediate information and the first hyper information using the second sub-network. By way of example and not limitation, the third sub-network may be a hyper-decoder sub-network.

[0076] In some embodiments, the first quantized hyper-latent representation may be determined based on the bitstream. For example, the first quantized hyper-latent representation may be decoded from the bitstream in a decoding process. Alternatively, the first quantized hyper-latent representation may be generated using a fourth sub-network based on the latent representation of the data. By way of example and not limitation, the fourth sub-network may be a hyper-encoder sub-network.

[0077] In some embodiments, the first hyper information may include first probability distribution information. In one example, the first probability distribution information may include a mean value. Additionally or alternatively, the first hyper information may include prediction information.

[0078] In some embodiments, at 1604, the second hyper information may be generated based on the second quantized hyper latent representation using a fifth sub-network. In one example, the fifth sub-network may be a hyperscale decoder sub-network. The second quantized hyper latent representation may be determined based on a first portion of the bitstream. For example, the second quantized hyper latent representation may be decoded from the first portion of the bitstream. Further, at 1604, the second portion may be obtained by performing an entropy decoding process on the second portion of the bitstream based on the second hyper information. The second portion may be different from the first portion. For example, the first portion and the second portion may be two sub-bitstreams of the bitstream.

[0079] In some embodiments, the second hyper information may include second probability distribution information. In one example, the second probability distribution information may include a variance. In another example, the second probability distribution information may include a standard deviation. It should be understood that the above examples are provided for illustrative purposes only. The scope of the present disclosure is not limited in this respect.

[0080] In some embodiments, the entropy decoding process may be an arithmetic decoding process. Additionally or alternatively, the entropy decoding process may be performed using a zero-mean probability distribution. In some further embodiments, the entropy decoding process may be performed using a variance.

[0081] In some embodiments, the second quantized hyper-latent representation may be the same as the first quantized hyper-latent representation. Alternatively, the second quantized hyper-latent representation may be different from the first quantized hyper-latent representation.

[0082] In some embodiments, a first sample may be determined based on the first portion and the second portion at 1606. By way of example and not limitation, the first sample may be determined based on a sum of the first portion and the second portion. Further, a transformation may be performed based on a composite transformation on the first sample.

[0083] In some embodiments, at 1604, a second portion may be determined based on the first portion and the second sample of the latent representation of the data. The second sample corresponds to the first sample, and the latent representation corresponds to a reconstructed latent representation. In other words, the first sample is the reconstructed second sample, i.e., a reconstructed version of the second sample. In some embodiments, the latent representation may be obtained by performing an analytical transformation on the data.

[0084] In some embodiments, a residual may be obtained based on a difference between the first portion and the second sample, and the second portion may be obtained by quantizing the residual. Alternatively, the residual may not be quantized to obtain the second portion.

[0085] In some embodiments, the first sample may be determined based on the first portion and the second portion. By way of example and not limitation, the first sample may be determined based on a sum of the first portion and the second portion.

[0086] In some embodiments, the second sample is quantized before being used to determine the second portion. Alternatively, the second sample is not quantized before being used to determine the second portion.

[0087] In some embodiments, the first portion is quantized before being used to determine the second portion and the first sample. Alternatively, the first portion is not quantized before being used to determine the second portion and the first sample.

[0088] In some embodiments, a fourth sub-network may be used to generate a second quantized hyper-latent representation based on the latent representation of the data at 1606. Additionally, a fifth sub-network may be used to generate second hyper-information based on the second quantized hyper-latent representation, and an entropy encoding process may be performed on the second portion based on the second hyper-information.

[0089] In some embodiments, the fourth sub-network may be a hyper-encoder sub-network, or the fifth sub-network may be a hyper-scale decoder sub-network. In some embodiments, the entropy encoding process may be an arithmetic encoding process. In one example, the entropy encoding process may be performed using a zero-mean probability distribution. In another example, the entropy encoding process may be performed using a variance.

[0090] In some embodiments, the second quantized hyper-latent representation may be the same as the first quantized hyper-latent representation, in which case the entropy encoding process may be performed on the first quantized hyper-latent representation.

[0091] In some embodiments, the second quantized hyper-latent representation may differ from the first quantized hyper-latent representation, in which case, at 1606, an entropy encoding process may be performed on the first quantized hyper-latent representation and the second quantized hyper-latent representation.

[0092] According to an embodiment of the present disclosure, a non-transitory computer readable recording medium is proposed. A bit stream of data is stored in the non-transitory computer readable recording medium. The bit stream may be generated by a method executed by a data processing device. According to the method, a first portion of a first sample of a reconstructed latent representation of data is determined. The first portion is indicative of a prediction of the first sample. Moreover, a second portion of the first sample is determined. The second portion is indicative of a difference between the first sample and the first portion. Furthermore, the bit stream is generated based on the second portion.

[0093] According to an embodiment of the present disclosure, a method for storing a bitstream of data is proposed, in which a first portion of a first sample of a reconstructed latent representation of data is determined, the first portion being indicative of a prediction of the first sample. Furthermore, a second portion of the first sample is determined, the second portion being indicative of a difference between the first sample and the first portion. Furthermore, the bitstream is generated based on the second portion, and the bitstream is stored in a non-transitory computer-readable recording medium.

[0094] Implementations of the present disclosure can be described in terms of the following clauses, the features of which can be combined in any reasonable manner.

[0095] Clause 1. A data processing method, comprising, during conversion between data and a bitstream of said data, determining a first portion of a first sample of a reconstructed latent representation of said data, said first portion indicative of a prediction of said first sample, determining a second portion of said first sample, said second portion indicative of a difference between said first sample and said first portion, and performing said conversion based on said second portion.

[0096] Clause 2. The method of clause 1, wherein determining the first portion comprises determining the first portion based on a set of samples of the reconstructed latent representation.

[0097] Clause 3. The method of clause 2, wherein determining the first portion based on the set of samples includes: generating intermediate information based on the set of samples using a first sub-network; and generating the first portion based on the intermediate information by a second sub-network.

[0098] Clause 4. The method of clause 3, wherein the first sub-network is autoregressive.

[0099] Clause 5. The method of any one of clauses 3 to 4, wherein the first sub-network is a context model sub-network or a context sub-network, or the second sub-network is a prediction sub-network or a fusion sub-network.

[0100] Clause 6. The method of any one of clauses 3 to 5, wherein the step of generating the first portion includes the steps of: using a third sub-network to generate first hyper information based on a first quantized hyper latent representation; and using the second sub-network to generate the first portion based on the intermediate information and the first hyper information.

[0101] Clause 7. The method of clause 1, wherein determining the first portion includes determining the first portion based on a first quantized hyper-latent representation.

[0102] Clause 8. The method of clause 7, wherein determining the first portion based on the first quantized hyper-latent representation includes processing the first quantized hyper-latent representation using a third sub-network.

[0103] Clause 9. The method of clause 6 or 8, wherein the third sub-network is a hyperdecoder sub-network.

[0104] Clause 10. A method according to any one of clauses 6 to 9, wherein the first quantized hyper-latent representation is determined based on the bitstream or the first quantized hyper-latent representation is generated using a fourth sub-network based on a latent representation of the data.

[0105] Clause 11. The method of clause 10, wherein the fourth sub-network is a hyperencoder sub-network.

[0106] Clause 12. The method of any one of clauses 6 and 9 to 11, wherein the first hyper information comprises first probability distribution information.

[0107] Clause 13. The method of clause 12, wherein the first probability distribution information includes a mean value.

[0108] Clause 14. The method of any one of clauses 6 and 9 to 11, wherein the first hyper information comprises predictive information.

[0109] Clause 15. The method of any one of clauses 1 to 14, wherein the step of determining the second portion includes the steps of: generating second hyper information based on a second quantized hyper latent representation using a fifth sub-network, wherein the second quantized hyper latent representation is determined based on a first portion of the bitstream; and obtaining the second portion by performing an entropy decoding process on the second portion of the bitstream based on the second hyper information, wherein the second portion is different from the first portion.

[0110] Clause 16. The method of clause 15, wherein the second hyper information includes second probability distribution information.

[0111] Clause 17. The method of clause 16, wherein the second probability distribution information includes a variance.

[0112] Clause 18. The method of any one of clauses 15 to 17, wherein the fifth sub-network is a hyperscale decoder sub-network.

[0113] Clause 19. A method according to any one of clauses 15 to 18, wherein the entropy decoding process is an arithmetic decoding process.

[0114] Clause 20. A method according to any one of clauses 15 to 19, wherein the entropy decoding process is performed using a zero-mean probability distribution.

[0115] Clause 21. The method of any one of clauses 15 to 20, wherein the entropy decoding process is performed using a distribution.

[0116] Clause 22. A method according to any one of clauses 15 to 21, wherein the second quantized hyper-latent representation is the same as the first quantized hyper-latent representation or the second quantized hyper-latent representation is different from the first quantized hyper-latent representation.

[0117] Clause 23. The step of performing the conversion comprises: determining the first sample based on the first portion and the second portion; 23. The method of any one of clauses 1 to 22, comprising: performing the conversion based on a synthetic transformation on the first sample.

[0118] Clause 24. The method of clause 23, wherein the first sample is determined based on a sum of the first portion and the second portion.

[0119] Clause 25. The method of any one of clauses 1 to 14, wherein determining the second portion comprises determining the second portion based on the first portion and a second sample of a latent representation of the data, the second sample corresponding to the first sample and the latent representation corresponding to the reconstructed latent representation.

[0120] Clause 26. The method of clause 25, wherein the latent representation is obtained by performing an analytical transformation on the data.

[0121] Clause 27. A method according to any one of clauses 25 to 26, wherein the step of determining the second part based on the first part and the second sample comprises the steps of obtaining a residual based on a difference between the first part and the second sample, and obtaining the second part by quantizing the residual.

[0122] Clause 28. The method of any one of clauses 25 to 26, wherein the first sample is determined based on the first portion and the second portion.

[0123] Clause 29. The method of clause 28, wherein the first sample is determined based on a sum of the first portion and the second portion.

[0124] Clause 30. A method according to any one of clauses 25 to 29, wherein the second samples are quantized before being used to determine the second portion.

[0125] Clause 31. A method according to any one of clauses 25 to 30, wherein the first portion is quantized before being used to determine the second portion and the first sample.

[0126] Clause 32. A method according to any one of clauses 1 to 14 or 25 to 31, wherein the step of performing the transformation comprises the steps of: generating a second quantized hyper latent representation based on the latent representation of the data using a fourth sub-network; generating second hyper information based on the second quantized hyper latent representation using a fifth sub-network; and performing an entropy encoding process on the second portion based on the second hyper information.

[0127] Clause 33. The method of clause 32, wherein the second hyper information includes second probability distribution information.

[0128] Clause 34. The method of clause 33, wherein the second probability distribution information includes a variance.

[0129] Clause 35. The method of any one of clauses 32 to 34, wherein the fourth sub-network is a hyper-encoder sub-network or the fifth sub-network is a hyper-scale decoder sub-network.

[0130] Clause 36. A method according to any one of clauses 32 to 35, wherein the entropy encoding process is an arithmetic encoding process.

[0131] Clause 37. A method according to any one of clauses 32 to 36, wherein the entropy encoding process is performed using a zero-mean probability distribution.

[0132] Clause 38. A method according to any one of clauses 32 to 37, wherein the entropy encoding process is performed using variance.

[0133] Clause 39. The method of any one of clauses 32 to 38, wherein the second quantized hyper-latent representation is the same as the first quantized hyper-latent representation.

[0134] Clause 40. The method of clause 39, wherein the step of performing the transformation further comprises the step of performing the entropy encoding process on the first quantized hyper-latent representation.

[0135] Clause 41. The method of any one of clauses 32 to 38, wherein the second quantized hyper-latent representation is different from the first quantized hyper-latent representation.

[0136] Clause 42. The method of clause 41, wherein the step of performing the transformation further comprises the step of performing the entropy encoding process on the first quantized hyper-latent representation and the second quantized hyper-latent representation.

[0137] Clause 43. The method of any one of clauses 1 to 42, wherein the first part is a prediction of the first sample or the second part is a quantized residual of the first sample.

[0138] Clause 44. A method according to any one of clauses 1 to 43, wherein the reconstructed latent representation is a quantized latent representation of the data.

[0139] Clause 45. A method according to any one of clauses 1 to 44, wherein the data comprises pictures or images of a video.

[0140] Clause 46. A method according to any one of clauses 1 to 45, wherein the converting comprises encoding the data into the bitstream.

[0141] Clause 47. A method according to any one of clauses 1 to 45, wherein the conversion comprises decoding the data from the bitstream.

[0142] Clause 48. An apparatus for processing data comprising a processor and a non-transitory memory comprising instructions which, when executed by the processor, cause the processor to perform a method according to any one of clauses 1 to 47.

[0143] Clause 49. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method of any one of clauses 1 to 47.

[0144] Clause 50. A non-transitory computer readable recording medium storing a bitstream of data generated by a method executed by a data processing apparatus, the method comprising the steps of: determining a first portion of a first sample of a reconstructed latent representation of the data, the first portion indicative of a prediction of the first sample; determining a second portion of the first sample, the second portion indicative of a difference between the first sample and the first portion; and generating the bitstream based on the second portion.

[0145] Clause 51. A method for storing a bitstream of data, the method comprising the steps of determining a first portion of a first sample of a reconstructed latent representation of the data, the first portion indicative of a prediction of the first sample; determining a second portion of the first sample, the second portion indicative of a difference between the first sample and the first portion; generating the bitstream based on the second portion; and storing the bitstream on a non-transitory computer-readable recording medium.

[0146] Exemplary Devices 17 illustrates a block diagram of a computing device 1700 capable of implementing various embodiments of the present disclosure. The computing device 1700 may be embodied as or included in the source device 110 (or the data encoder 114) or the destination device 120 (or the data decoder 124).

[0147] It will be appreciated that the computing device 1700 shown in FIG. 17 is for illustrative purposes only and is not intended to suggest any limitation on the functionality and scope of the embodiments of the present disclosure.

[0148] 17, the computing device 1700 includes a general-purpose computing device 1700. The computing device 1700 may include at least one or more processors or processing units 1710, a memory 1720, a storage unit 1730, one or more communication units 1740, one or more input devices 1750, and one or more output devices 1760.

[0149] In some embodiments, the computing device 1700 may be embodied as any user terminal or server terminal having computing capabilities. The server terminal may be a server provided by a service provider, a large-scale computing device, etc. The user terminal may be any type of mobile, fixed, or portable terminal, including, for example, a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device, or any combination thereof (including accessories and peripherals of these devices, or any combination thereof). It is contemplated that the computing device 1700 may support any type of interface to a user, such as "wearable" circuitry.

[0150] The processing unit 1710 may be a physical or virtual processor and may implement various processes based on programs stored in the memory 1720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of the computing device 1700. The processing unit 1710 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0151] Computing device 1700 typically includes a variety of computer storage media. Such media may be any media accessible by computing device 1700, including but not limited to volatile and nonvolatile media, or removable and non-removable media. Memory 1720 may be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory), or any combination thereof. Storage unit 1730 may be any removable or non-removable medium, including a machine-readable medium such as a memory, a flash memory drive, a magnetic disk, or another medium that can be used to store information and / or data and that can be accessed by computing device 1700.

[0152] The computing device 1700 may further include additional removable / non-removable, volatile / non-volatile memory media. Although not shown in FIG. 17, a magnetic disk drive that reads from and writes to a removable non-volatile magnetic disk and an optical disk drive that reads from and writes to a removable non-volatile optical disk may be provided. In such a case, each drive may be connected to a bus (not shown) via one or more data media interfaces.

[0153] The communication unit 1740 communicates with further computing devices via a communication medium. Moreover, the functionality of the components in the computing device 1700 may be embodied by a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device 1700 can operate in a networked environment using logical connections with one or more other servers, networked personal computers (PCs), or further general network nodes.

[0154] The input device(s) 1750 may be one or more of a variety of input devices such as a mouse, a keyboard, a tracking ball, a voice input device, etc. The output device(s) 1760 may be one or more of a variety of output devices such as a display, a loudspeaker, a printer, etc. The communication unit 1740 may enable the computing device 1700 to further communicate with one or more external devices (not shown), such as a storage device and a display device, one or more devices allowing a user to interact with the computing device 1700, or any device (such as a network card, a modem, etc.) allowing the computing device 1700 to communicate with one or more other computing devices, as needed. Such communication may be performed via an input / output (I / O) interface (not shown).

[0155] In some embodiments, instead of being integrated into a single device, some or all of the components of computing device 1700 may be located in a cloud computing architecture. In a cloud computing architecture, the components may be provided remotely and work together to perform the functions described in this disclosure. In some embodiments, cloud computing provides computing, software, data access, and storage services without the end user needing to be aware of the physical location or configuration of the systems or hardware providing these services. In various embodiments, cloud computing provides services over a wide area network (such as the Internet) using appropriate protocols. For example, a cloud computing provider provides applications over a wide area network that can be accessed through a web browser or other computing components. The software or components of the cloud computing architecture and corresponding data may be stored on a server located in a remote location. Computing resources in a cloud computing environment may be consolidated or distributed at remote data center locations. A cloud computing infrastructure may act as a single access point for users but provide services through a shared data center. Thus, a cloud computing architecture may be used to provide the components and functions described herein from a service provider located in a remote location. Alternatively, they may be provided from a traditional server or installed directly or otherwise on a client device.

[0156] The computing device 1700 may be used to implement data encoding / decoding in embodiments of the present disclosure. The memory 1720 may include one or more data coding modules 1725 having one or more program instructions. These modules are accessible and executable by the processing unit 1710 to perform the functions of various embodiments described herein.

[0157] In an example embodiment performing data encoding, the input device 1750 may receive data to be encoded as input 1770. The data may be processed, for example, by the data coding module 1725 to generate an encoded bitstream. The encoded bitstream may be provided as output 1780 via the output device 1760.

[0158] In an example embodiment performing data decoding, the input device 1750 may receive an encoded bitstream as input 1770. The encoded bitstream may be processed, for example, by the data coding module 1725 to generate decoded data. The decoded data may be provided as output 1780 via the output device 1760.

[0159] Although the present disclosure has been particularly shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application as defined by the appended claims. Such variations are intended to be within the scope of the present application. Accordingly, the foregoing description of the embodiments of the present application is not intended to be limiting.

Claims

1. 1. A data processing method comprising the steps of: determining a first portion of a first sample of a reconstructed latent representation of the data during conversion between the data and a bitstream of the data, the first portion representing a prediction of the first sample; determining a second portion of the first sample, the second portion being indicative of a difference between the first sample and the first portion; and performing the conversion based on the second portion.

2. The step of determining the first portion comprises: The method of claim 1 , comprising determining the first portion based on a set of samples of the reconstructed latent representation.

3. Determining the first portion based on the set of samples comprises: generating intermediate information based on the set of samples using a first sub-network; and generating, by a second sub-network, the first portion based on the intermediate information.

4. The method of claim 3 , wherein the first sub-network is autoregressive.

5. the first sub-network is a context model sub-network or a context sub-network; or The method according to claim 3 , wherein the second sub-network is a prediction sub-network or a fusion sub-network.

6. The step of generating the first portion comprises: generating a first hyper information based on the first quantized hyper latent representation using a third sub-network; and generating the first portion based on the intermediate information and the first hyper information using the second sub-network.

7. The step of determining the first portion comprises: The method of claim 1 , comprising determining the first portion based on a first quantized hyper-latent representation.

8. Determining the first portion based on the first quantized hyper-latent representation comprises: The method of claim 7 , further comprising processing the first quantized hyper-latent representation using a third sub-network.

9. 9. A method according to claim 6 or 8, wherein the third sub-network is a hyperdecoder sub-network.

10. the first quantized hyper-latent representation is determined based on the bitstream; or 10. The method of claim 6, wherein the first quantized hyper-latent representation is generated using a fourth sub-network based on a latent representation of the data.

11. The method of claim 10 , wherein the fourth sub-network is a hyperencoder sub-network.

12. 12. The method of claim 6, wherein the first hyper information comprises first probability distribution information.

13. The method of claim 12 , wherein the first probability distribution information comprises a mean value.

14. 12. The method of claim 6, wherein the first hyper information comprises prediction information.

15. The step of determining the second portion comprises: generating second hyper information based on a second quantized hyper-latent representation using a fifth sub-network, the second quantized hyper-latent representation being determined based on a first portion of the bitstream; 15. The method of claim 1, further comprising: obtaining a second portion of the bitstream by performing an entropy decoding process on the second portion of the bitstream based on the second hyper information, the second portion being different from the first portion.

16. The method of claim 15 , wherein the second hyper information comprises second probability distribution information.

17. The method of claim 16 , wherein the second probability distribution information comprises a variance.

18. 18. The method of claim 15, wherein the fifth sub-network is a hyperscale decoder sub-network.

19. 19. The method of any one of claims 15 to 18, wherein the entropy decoding process is an arithmetic decoding process.

20. 20. The method of any one of claims 15 to 19, wherein the entropy decoding process is performed using a zero-mean probability distribution.

21. The method according to any one of claims 15 to 20, wherein the entropy decoding process is performed using variance.

22. the second quantized hyper-latent representation is the same as the first quantized hyper-latent representation, or 22. The method of claim 15, wherein the second quantized hyper-latent representation is different from the first quantized hyper-latent representation.

23. The step of performing the conversion comprises: determining the first sample based on the first portion and the second portion; and performing the conversion based on a synthesis transformation on the first sample.

24. 24. The method of claim 23, wherein the first sample is determined based on a sum of the first portion and the second portion.

25. The step of determining the second portion comprises:

15. The method of claim 1, further comprising determining the second portion based on the first portion and a second sample of a latent representation of the data, the second sample corresponding to the first sample and the latent representation corresponding to the reconstructed latent representation.

26. The method of claim 25 , wherein the latent representations are obtained by performing an analytical transformation on the data.

27. Determining the second portion based on the first portion and a second sample comprises: obtaining a residual based on a difference between the first portion and the second sample; obtaining the second portion by quantizing the residual; 27. The method of any one of claims 25 to 26, comprising:

28. 27. The method of any one of claims 25 to 26, wherein the first sample is determined based on the first portion and the second portion.

29. 30. The method of claim 28, wherein the first sample is determined based on a sum of the first portion and the second portion.

30. 30. The method of claim 25, wherein the second samples are quantized before being used to determine the second portion.

31. 31. A method according to any one of claims 25 to 30, wherein the first portion is quantized before being used to determine the second portion and the first sample.

32. The step of performing the conversion comprises: generating a second quantized hyper-latent representation based on the latent representation of the data using a fourth sub-network; generating second hyper information based on the second quantized hyper latent representation using a fifth sub-network; and performing an entropy encoding process on the second portion based on the second hyper information.

33. 33. The method of claim 32, wherein the second hyper information comprises second probability distribution information.

34. The method of claim 33 , wherein the second probability distribution information comprises a variance.

35. 35. The method of any one of claims 32 to 34, wherein the fourth sub-network is a hyper-encoder sub-network or the fifth sub-network is a hyper-scale decoder sub-network.

36. 36. The method of any one of claims 32 to 35, wherein the entropy encoding process is an arithmetic encoding process.

37. 37. The method of any one of claims 32 to 36, wherein the entropy encoding process is performed using a zero-mean probability distribution.

38. 38. The method according to any one of claims 32 to 37, wherein the entropy encoding process is performed using variance.

39. 39. The method of any one of claims 32 to 38, wherein the second quantized hyper-latent representation is the same as the first quantized hyper-latent representation.

40. The step of performing the conversion comprises:

40. The method of claim 39, further comprising performing the entropy encoding process on the first quantized hyper-latent representation.

41. 39. The method of any one of claims 32 to 38, wherein the second quantized hyper-latent representation is different from the first quantized hyper-latent representation.

42. The step of performing the conversion comprises:

42. The method of claim 41, further comprising performing the entropy encoding process on the first quantized hyper-latent representation and the second quantized hyper-latent representation.

43. 43. The method of claim 1, wherein the first part is a prediction of the first sample or the second part is a quantized residual of the first sample.

44. 44. The method of claim 1, wherein the reconstructed latent representation is a quantized latent representation of the data.

45. 45. A method according to any preceding claim, wherein the data comprises pictures or images of a video.

46. 46. ​​The method of any one of claims 1 to 45, wherein the converting comprises encoding the data into the bitstream.

47. 46. ​​The method of claim 1, wherein the converting comprises decoding the data from the bitstream.

48. 48. An apparatus for processing data comprising a processor and a non-transitory memory comprising instructions which, when executed by the processor, cause the processor to perform a method according to any one of claims 1 to 47.

49. 48. A non-transitory computer readable storage medium storing instructions that cause a processor to perform the method of any one of claims 1 to 47.

50. A non-transitory computer-readable recording medium storing a bitstream of data produced by a method executed by a data processing apparatus, the method comprising: determining a first portion of a first sample of a reconstructed latent representation of the data, the first portion indicating a prediction of the first sample; determining a second portion of the first sample, the second portion being indicative of a difference between the first sample and the first portion; and generating the bitstream based on the second portion.

51. 1. A method for storing a bitstream of data, comprising the steps of: determining a first portion of a first sample of a reconstructed latent representation of the data, the first portion indicating a prediction of the first sample; determining a second portion of the first sample, the second portion being indicative of a difference between the first sample and the first portion; generating the bitstream based on the second portion; storing the bitstream on a non-transitory computer-readable recording medium.

Images