Data processing method and device and medium

By using the converter model of attention mechanism in image/video compression, combined with autoregression and hyper-priori model, the problem of difficult to capture long-range correlation between quantized hidden is solved, and the encoding and decoding efficiency and image/video compression are improved. performance.

CN118985129APending Publication Date: 2024-11-19DOUYIN VISION CO LTD +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202380022448.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-02-17
Filing Date
2023-02-16
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The prior art is difficult to effectively capture the long-range correlation between quantized hidden quantities in image/video compression, resulting in insufficient encoding and decoding efficiency.

Method used

A transformer model with attention mechanism is adopted, combined with autoregressive model and hyper-prior model, and the long-range and short-range correlation between quantized hidden factors is constructed through the attention mechanism to generate a more accurate entropy codec probability distribution.

Benefits of technology

Improves encoding and decoding efficiency, enhances the modeling ability of quantized hidden quantities, and achieves more efficient image/video compression.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118985129A_ABST
    Figure CN118985129A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a data processing scheme. A method for data processing is presented. The method comprises: during a transition between data and a bitstream of data, determining a probability distribution for entropy coding associated with the bitstream by using a first model having an attention mechanism; and performing a conversion based on the probability distribution.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to data processing techniques, and more particularly, to transformer-based probabilistic modeling for data encoding and decoding. Background Art

[0002] Image / video compression is an important technology to reduce the cost of image / video transmission and storage in a lossless or lossy manner. Image / video compression technology can be divided into two branches, classical video codec methods and neural network-based video compression methods. Classical video codec schemes adopt transform-based solutions, in which researchers exploit statistical dependencies in latent variables (e.g., wavelet coefficients) by carefully manually designing entropy codes to model dependencies in the quantization domain. There are two forms of neural network-based video compression: neural network-based codec tools and end-to-end neural network-based video compression. Neural network-based codec tools are embedded into existing classical video codecs as coding tools and are only used as part of the framework, while end-to-end neural network-based video compression is a separate framework developed based on neural networks and does not rely on classical video codecs. The coding and decoding efficiency of image / video codecs is generally expected to be further improved. Summary of the invention

[0003] Embodiments of the present disclosure provide a solution for data processing.

[0004] In a first aspect, a method for data processing is provided. The method comprises: during conversion between data and a bitstream of the data, determining a probability distribution for entropy coding associated with the bitstream by using a first model with an attention mechanism; and performing conversion based on the probability distribution.

[0005] According to the method of the first aspect of the present disclosure, a probability distribution for entropy encoding and decoding is obtained by using a first model with an attention mechanism. Compared with the traditional convolution-based solution, the first model can additionally capture the long-range correlation between the quantized latent quantities. By simultaneously capturing the ability of long-range and short-range correlations between the quantized latent quantities, the first model has a better receptive field than the convolution-based solution. Therefore, the proposed method can advantageously improve the encoding and decoding efficiency.

[0006] In a second aspect, a device for data processing is provided. The device comprises a processor and a non-volatile memory having instructions thereon. The instructions, when executed by the processor, cause the processor to perform the method according to the first aspect of the present disclosure.

[0007] In a third aspect, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores instructions, which enable a processor to execute the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a data bitstream generated by a method performed by an apparatus for data processing. The method includes: determining a probability distribution for entropy coding and decoding associated with a bitstream by using a first model with an attention mechanism; and generating a bitstream based on the probability distribution.

[0009] In a fifth aspect, a method for storing a bitstream of data is provided. The method includes: determining a probability distribution for entropy coding and decoding associated with the bitstream by using a first model with an attention mechanism; generating a bitstream based on the probability distribution; and storing the bitstream in a non-transitory computer-readable recording medium.

[0010] This Summary is intended to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become more apparent through the following detailed description with reference to the accompanying drawings. In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0012] Figure 1 A block diagram showing an example data encoding and decoding system according to some embodiments of the present disclosure is shown;

[0013] Figure 2 A typical transform coding scheme is shown;

[0014] Figure 3 Images from the Kodak dataset and different representations of the images are shown;

[0015] Figure 4 The network architecture of the autoencoder implementing the hyper-prior model is shown;

[0016] Figure 5 A block diagram of the combined model is shown;

[0017] Figure 6 The encoding process of the combined model is shown;

[0018] Figure 7 The decoding process of the combined model is shown;

[0019] Figure 8 An example operation diagram of the transformer attention mechanism is shown;

[0020] Fig. 9An example framework of a method for data processing according to an embodiment of the present disclosure is shown;

[0021] Fig.10 An example structure of a transformer context model according to an embodiment of the present disclosure is shown;

[0022] Fig.11 An example operation diagram of masked attention according to an embodiment of the present disclosure is shown;

[0023] Fig.12 A flowchart showing a method for data processing according to an embodiment of the present disclosure; and

[0024] Fig.13 A block diagram of a computing device is shown in which various embodiments of the present disclosure may be implemented.

[0025] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION

[0026] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described only for the purpose of illustrating and helping those skilled in the art to understand and implement the present disclosure, without implying any limitation on the scope of the present disclosure. In addition to the methods described below, the disclosure described herein can also be implemented in various ways.

[0027] In the following description and claims, unless defined otherwise, all scientific and technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.

[0028] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment must include the particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, whether or not explicitly described, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in relation to other embodiments.

[0029] It should be understood that although the terms "first" and "second" etc. can be used to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish one element from another element. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0030] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms "including", "comprising", and / or "having" when used herein indicate the presence of the features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof. Example Environment

[0031] Figure 1 1 is a block diagram illustrating an example data encoding and decoding system 100 that can utilize the techniques of the present disclosure. As shown, the data encoding and decoding system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a data encoding device, and the destination device 120 may also be referred to as a data decoding device. In operation, the source device 110 may be configured to generate encoded data, and the destination device 120 may be configured to decode the encoded data generated by the source device 110. The source device 110 may include a data source 112, a data encoder 114, and an input / output (I / O) interface 116.

[0032] Data sources 112 may include sources such as data capture devices. Examples of data capture devices include, but are not limited to, interfaces that receive data from data providers, computer graphics systems for generating data, and / or combinations thereof.

[0033] The data may include one or more images or one or more pictures of a video. The data encoder 114 encodes the data from the data source 112 to generate a bitstream. The bitstream may include a sequence of bits that form a coded representation of the data. The bitstream may include a coded picture and associated data. The coded picture is a coded representation of the picture. The associated data may include a sequence parameter set, a picture parameter set, and other grammatical structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The coded data may be directly transmitted to the destination device 120 via the network 130A via the I / O interface 116. The coded data may also be stored on a storage medium / server 130B for access by the destination device 120.

[0034] The destination device 120 may include an I / O interface 126, a data decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain encoded data from the source device 110 or the storage medium / server 130B. The data decoder 124 may decode the encoded data. The display device 122 may display the decoded data to the user. The display device 122 may be integrated with the destination device 120, or may be outside the destination device 120, which is configured to be connected to an external display device interface.

[0035] The data encoder 114 and the data decoder 124 may operate according to a data codec standard, such as a video codec standard or a still picture codec standard and other existing and / or further standards.

[0036] Some example embodiments of the present disclosure will be described in detail below. It should be noted that the section titles used in this document are for ease of understanding, and the embodiments disclosed in the section are not limited to that section. In addition, although some embodiments are described with reference to general video codecs or other specific data codecs, the disclosed technology is also applicable to other codec technologies. In addition, although some embodiments describe the coding and decoding steps in detail, it should be understood that the corresponding decoding steps of canceling the coding will be implemented by the decoder. In addition, the term data processing includes data coding or compression, data decoding or decompression, and data transcoding, in which data is represented from one compression format to another compression format or at different compression bit rates. 1. Overview The present disclosure relates to end-to-end image compression technology. Specifically, it is about probabilistic modeling of entropy coding in a learning image compression framework. These ideas can be applied alone or in various combinations. It can be directly applied to the probabilistic modeling of entropy in an end-to-end image compression framework. In addition, it can also be used as an intra-frame part of learning video compression to help the compression of residual information of the entropy part or intra-frame frames. In addition, it can also be applied to traditional video coding standards such as HEVC, or standard universal video coding as a context modeling tool to further remove redundancy in residual information. 2. Abbreviations ANN Artificial Neural Network E2E End to End JPEG Joint Photographic Experts Group MPEG Moving Picture Experts Group VCEG Video Codec Experts Group VTM VVC test model VVC Universal Video Codec 3. Background Image / video compression is an important technology to reduce the cost of image / video transmission and storage in a lossless or lossy manner. Image / video compression technology can be divided into two branches, classical video codec methods and neural network-based video compression methods. Classical video codec schemes adopt transform-based solutions, in which researchers exploit statistical dependencies in latent variables (e.g., wavelet coefficients) by carefully manually designing entropy codes to model dependencies in the quantization domain. There are two forms of neural network-based video compression: neural network-based codec tools and end-to-end neural network-based video compression. The former is embedded into existing classical video codecs as a coding tool and is used only as part of the framework, while the latter is a separate framework developed based on neural networks and does not rely on classical video codecs. 3.1. Traditional Image / Video Compression Traditional image compression standards such as JPEG, BPG and VVC use a hybrid framework to compress input images. In these standards, a combination of prediction, transform, quantization and entropy coding is used to reduce redundancy within the image. Under the development of the past few decades, traditional codecs have increased compression efficiency by nearly 50% through manual design on some modules. A series of classic image codec standards have been developed to adapt to the increased visual content. The international standards organization ISO / IEC has two expert groups, the Joint Photographic Experts Group (JPEG) and the Moving Picture Experts Group (MPEG), and the ITU-T also has its own Video Coding Experts Group (VCEG), which is used for the standardization of image coding and decoding technology. The influential image coding standards released by these organizations include JPEG, JPEG 2000, H.262, H.264 / AVC and H.265 / HEVC. After H.265 / HEVC, the Joint Video Experts Group (JVET) formed by MPEG and VCEG has been working on a new video codec standard, the Universal Video Codec (VVC). The first version of VVC was released in July 2020. Compared to HEVC, VVC is reported to have an average of 50% bitrate reduction at the same visual quality. 3.2. Compression methods based on neural networks In the past five years, thanks to the great progress made by artificial neural networks (ANN), learning-based media codec solutions have opened up a new branch in image compression. In this branch, ANN will replace some functions inside traditional codecs as modules, or completely replace traditional codecs in an end-to-end manner. One benefit of such deep networks is considered to be the ability to process data with multiple levels of abstraction and convert data into different kinds of representations. Note that these representations are not manually designed; instead, deep networks include processing layers that learn from massive data using general machine learning procedures. Deep learning eliminates the need for manual representations and is therefore considered useful, especially for processing native unstructured data such as acoustic and visual signals, while processing such data has been a long-standing difficulty in the field of artificial intelligence. After several years of development, ANN-based tools have achieved some performance improvements over traditional codecs. For end-to-end ANN-based image compression, better performance has also been achieved than the latest conventional codec VTM. However, end-to-end ANN-based image compression is still immature and there is still a lot of room for improvement in compression performance. 3.3. Learning Image Compression Existing neural network approaches for image compression can be divided into two categories, namely pixel probability modeling and autoencoders. The former belongs to predictive encoding and decoding strategies, while the latter is a transform-based solution. Sometimes, these two approaches are combined in the literature. 3.3.1 Pixel Probabilistic Modeling According to Shannon information theory, the optimal method for lossless encoding and decoding can achieve the minimum encoding and decoding ratio log 2 p(x), where p(x) is the probability of symbol x. Many lossless codecs have been developed in the literature, and among them arithmetic coding is considered the best. Given a probability distribution p(x), arithmetic coding ensures that the codec ratio is as close as possible to its theoretical limit log without considering round-off errors. 2 p(x). Therefore, the remaining question is how to determine the probability, which is very challenging for natural images / videos due to the dimensionality reduction. Following the predictive coding strategy, one way to model p(x) is to predict pixel-by-pixel probability in raster scan order based on previous observations, where x is the image. p(x)=p(x 1 )p(x 2 |x 1 )...p(x i |x 1 , ..., x i-1 )...p(x m×n |x 1 , ..., x m×n-1) (1) where m and n are the height and width of the image, respectively. The previous observation is also called the context of the current pixel. When the image is large, it may be difficult to estimate the conditional probability, so a simplified approach is to limit the scope of its context. p(x)=p(x 1 )p(x 2 |x 1 )...p(x i |x i-k , ..., x i-1 )...p(x m×n |x m×n-k , ..., x m×n-1 ) (2)where k is a predefined constant that controls the scope of the context. It should be noted that the condition may also consider sample values ​​of other color components. For example, when encoding and decoding RGB color components, the R sample depends on the previously encoded pixel (including R / G / B samples), the current G sample can be encoded and decoded based on the previously encoded pixel and the current R sample pair, while for encoding and decoding the current B sample, the previously encoded pixel and the current R sample and the current G sample may also be considered. Neural networks were initially introduced for computer vision tasks and have been shown to be effective in regression and classification problems. Therefore, it has been proposed to use neural networks to estimate a given its context x. 1 ,x 2 ,…,x i-1 The probability p(x i ). In existing designs, pixel probabilities are proposed for binary images, i.e., x i ∈{-1,+1}. A Neural Autoregressive Distribution Estimator (NADF) is designed for pixel probability modeling, where is a feed-forward network with a single hidden layer. Similar work is presented in another existing design, where the feed-forward network also has connections that skip the hidden layer and parameters are also shared. Both solutions are performed on the binarized MNIST dataset. In the existing design, NADF is extended to a real-valued model RNADE, where the probability p(x i |x 1 ,…,x i-1 ) is derived by a mixture of Gaussians. Their feed-forward networks also have a single hidden layer, but the hidden layer has rescaling to avoid saturation and uses rectified linear units (ReLU) instead of sigmoids. Among the existing designs, NADF and RNADE are improved by reorganizing the order of pixels and by deeper neural networks. Designing advanced neural networks plays an important role in improving pixel probability modeling. In existing designs, multidimensional long short-term memory (LSTM) is proposed, which works with a mixture of conditional Gaussian scale mixtures for probabilistic modeling. LSTM is a special kind of recurrent neural network (RNN), and it has been shown to be good at modeling sequence data. The spatial variant of LSTM was later used for images. Several different neural networks have been studied, including RNN and CNN, namely PixelRNN and PixelCNN. In PixelRNN, two variants of LSTM are proposed, called row LSTM and diagonal BiLSTM, the latter of which is specifically designed for images. PixelRNN synthesizes residual connections to help train deep neural networks with up to 12 layers. In PixelCNN, masked convolutions are used to adapt to the shape of the context. Compared with previous work, PixelRNN and PixelCNN are more specialized for natural images: they consider pixels as discrete values ​​(e.g., 0, 1, ..., 255) and predict multinomial distributions of discrete values; they process color images in RGB color space; they work well on large-scale image datasets ImageNet. In the existing designs, Gated PixelCNN is proposed to improve PixelCNN and achieve comparable performance to PixelRNN but with much lower complexity. In the existing designs, PixelCNN++ is proposed on PixelCNN with the following improvements: discretized logical mixture likelihood is used instead of 256-way multinomial distribution; downsampling is used to capture structures at multiple resolutions; additional shortcut connections are introduced to speed up training; dropout is adopted for regularization; RGB is combined for one pixel. In the existing designs, PixelSNAIL is proposed, in which random convolution is combined with self-attention. Most of the above methods directly model the probability distribution in the pixel domain. Some researchers have also tried to model the probability distribution as a condition of an explicit or implicit representation. That is, we can estimate: Where h is an additional condition, and p(x)=p(h)p(x|h) means that the modeling is split into an unconditional one and a conditional one. The additional condition can be image label information or a high-level representation. 3.3.2 Autoencoder Autoencoders are derived from the well-known work proposed by Hinton and Salakhutrinoven. The method is trained for dimensionality reduction and contains two parts: encoding and decoding. The encoding part converts the high-dimensional input signal into a low-dimensional representation, usually with reduced spatial size but with a larger number of channels. The decoding part tries to recover the high-dimensional input from the low-dimensional representation. Autoencoders enable automatic learning of representations and eliminate the need for hand-crafted features, which is also considered one of the most important advantages of neural networks. Figure 2 A typical transform coding scheme is shown. The original image x passes through the analysis network g a is transformed to achieve the implicit representation y. The implicit representation y is quantized and compressed into bits. The number of bits R is used to measure the encoding and decoding ratio. Then the quantized implicit representation By synthesizing the network g s is inversely transformed to obtain the reconstructed image The distortion is obtained by transforming x and transforming And the function g p is calculated in the perceptual space. It is intuitive to apply autoencoder networks to lossy compression. We just need to encode the learned latent representations from the trained neural network. However, adapting the autoencoder to images is not trivial because the original autoencoder is not optimized for compression and therefore it is not efficient by directly using the trained autoencoder. In addition, there are other major challenges: First, the low-dimensional representation should be quantized before being encoded, but quantization is not differentiable, which is required in backpropagation when training neural networks. Second, the goals in the compression scenario are different because both distortion and ratio need to be considered. Estimating the ratio is challenging. Third, practical image coding and decoding schemes need to support variable ratios, scalability, encoding speed / decoding speed, and interoperability. In response to these challenges, many researchers have actively contributed to this field. Prototype autoencoder for image compression Figure 2 It can be regarded as a transform coding strategy. The original image x passes through the analysis network y = g a (x) is transformed, where y is the latent representation to be quantized and encoded. The synthesis network will transform the quantized latent representation Perform inverse transform to obtain the reconstructed image The framework is trained using a rate-distortion loss function, i.e., where D is x and The distortion between them, R is expressed from the quantized is the ratio calculated or estimated, and λ is the Lagrange multiplier. It should be noted that D can be calculated in the pixel domain or the perceptual domain. All existing research works follow this prototype, and the difference may only be the network structure or the loss function. In terms of network structure, RNN and CNN are the most widely used architectures. In the RNN-related category, Toderci et al. proposed a general framework for variable-rate image compression using RNN. They use binary quantization to generate codes and do not consider the ratio during training. The framework actually provides scalable encoding and decoding functions, in which RNNs with convolutional and deconvolutional layers are reported to perform appropriately. Toderci et al. then proposed an improved version by upgrading the encoder with a neural network similar to PixelRNN to compress binary codes. It is reported that the performance is better than JPEG on the Kodak image dataset using the MS-SSIM evaluation metric. Johnston et al. further improved the RNN-based solution by introducing hidden state induction. In addition, the SSIM weighted loss function is also designed, and the spatial adaptive bitrate mechanism is enabled. They achieve better results than BPG on the Kodak image dataset using MS-SSIM as the evaluation metric. Coveell et al. support spatially adaptive bitrate by training a stopping-tolerant RNN. Ballé et al. proposed a general framework for optimizing image compression for rate-distortion. Multivariate quantization is used to generate integer codes and the ratio is considered during training, i.e., the loss is a joint rate-distortion cost, which can be MSE or other. They add random uniform noise during training to motivate quantization and use the differential entropy of the noise code as a proxy for the ratio. They use generalized dinormalization (GDN) as the network structure, which includes a linear mapping followed by a nonlinear parameter normalization. The effectiveness of GDN for image coding and decoding has been verified in existing designs. Ballié et al. then proposed an improved version in which they used 3 convolutional layers, each followed by a downsampling layer and a GDN layer as a forward transform. Therefore, they used 3 layers of inverse GDN, each followed by an upsampling layer and a convolutional layer to motivate the inverse transform. In addition, an arithmetic coding method is designed to compress integer codes. The performance is reported to be better than JPEG and JPEG 2000 on the Kodak dataset in terms of MSE. In addition, Balfé et al. improved the method by designing a scale hyper-prior in the autoencoder. They used a subnet h a Transform the implicit representation y to z = h a (y) and z will be quantized and transmitted as side information. Therefore, the inverse transform is performed by trying to Decoding to quantization The standard deviation of the subnet h s is implemented, which will be in the arithmetic codec was further used during the period. On the Kodak image set, their method is slightly worse than BPG in terms of PSNR. D.Minnen et al. further exploit the structure in the residual space by introducing an autoregressive model to estimate both the standard deviation and the mean. In the existing design, Z.Cheng et al. use a Gaussian mixture model to further remove the redundancy in the residual. The reported performance is comparable to VVC on the Kodak image set using PSNR as the evaluation metric. 3.3.3 Super Prior Model In the transform codec approach to image compression, the encoder subnetwork (Section 2.3.2) uses a parametric analysis transform The image vector x is transformed into a latent representation y, which is then quantized to form because is a discrete value that can be losslessly compressed using entropy coding techniques such as arithmetic coding and transmitted as a sequence of bits. Figure 3 Images from the Kodak dataset and different representations of the images are shown. Figure 3 The leftmost image in the figure shows an image from the Kodak dataset. Figure 3 The center-left image in shows a visualization of the latent representation y of the image. Figure 3 The center-right image in shows the standard deviation σ of the latent quantity. Figure 3 The rightmost image in the figure shows the latent y after introducing the super-prior (super-encoder and super-decoder) network. As from Figure 3 The middle left image and the middle right image are obvious. There is significant spatial dependence between the elements of . Notably, their scales (center right image) appear to be spatially coupled. Introducing a set of random variables is introduced to capture spatial dependencies and further reduce redundancy. In this case, the image compression network is Figure 4 Depicted in. exist Figure 4 In the figure, the left hand side of the model is the encoder g a and decoder g s (Explained in Section 2.3.2). On the right hand side is the additional super encoder h a Network and for obtaining Additional super decoder h s In this architecture, the encoder inputs an image x to g a , producing a response y with a spatially varying standard deviation. The response y is fed to h a , summarizing the standard deviation distribution in z. z is then quantized The encoder then uses the quantized vector To estimate σ, the spatial distribution of the standard deviation, and use it to compress and transmit the quantized image representation The decoder first recovers the compressed signal Then it uses h s to obtain σ, which provides the correct probability estimate for the successful recovery Then it Feed to g s to obtain the reconstructed image. When the super encoder and super decoder are added to the image compression network, the quantized latent The space redundancy is reduced. Figure 3 The rightmost image in FIG. 4 corresponds to the quantized latent when the super encoder / decoder is used. Compared to the middle right image, the spatial redundancy is significantly reduced because the samples of the quantized latent are less correlated. Figure 4 The network architecture of the autoencoder implementing the super-prior model is shown. The image autoencoder network is shown on the left, and the right corresponds to the super-prior subnetwork. The analysis and synthesis transformations are denoted as g a and g a Q represents quantization, AE and AD represent arithmetic encoder and arithmetic decoder respectively. The super prior model consists of two sub-networks, the super encoder (with h a denoted) and super decoder (with h s The super prior model generates quantized super implicit quantities. It includes the quantization of implicit Information about the probability distribution of samples. is included in the bitstream and are transmitted together to the receiver (decoder). 3.3.4 Context Model Although the hyper-prior model improves the quantization implicit Additional improvements can be obtained by modeling the probability distribution of , by exploiting an autoregressive model that predicts the quantized latent quantity from its causal context (context model). Figure 5 The block diagram of the combined model is shown. The combined model jointly optimizes the autoregressive component, which together with the hyperprior and the underlying autoencoder estimates the probability distribution of the latent from its causal context (context model). The real-valued latent representation is quantized (Q) to produce the quantized latent and quantized super-implicit Quantification Implicit and quantized super-implicit Compressed into a bitstream using an Arithmetic Encoder (AE) and decompressed by an Arithmetic Decoder (AD). The highlighted areas correspond to the components executed by the receiver (ie, decoder) to recover the image from the compressed bitstream. The term autoregressive means that the output of a process is later used as input to it. For example, the context model subnetwork generates one sample of the latent quantity, which is later used as input to get the next sample. In existing designs, a joint architecture is used where both the super-prior model sub-networks (super-encoder and super-decoder) and the context model sub-network are utilized. The super-prior and context models are combined to learn the quantized latent The probability model of is then used for entropy coding and decoding. Figure 5 As depicted in , the outputs of the context subnetwork and the super decoder subnetwork are combined through the subnetwork called entropy parameters, which generate mean μ and scale (or variance) σ parameters for the Gaussian probability model. The Gaussian probability model is then used to encode the quantized latent samples into the bitstream with the help of the Arithmetic Encoder (AE) module. In the decoder, the Gaussian probability model is utilized to obtain the quantized latent from the bitstream through the Arithmetic Decoder (AD) module Typically, latent samples are modeled as Gaussian distribution or Gaussian mixture model (not limited to). Figure 5 , the context model and the hyper-prior are jointly used to estimate the probability distribution of latent samples. Since a Gaussian distribution can be defined by a mean and a variance (also called sigma or scale), a joint model is used to estimate the mean and variance (denoted as μ and σ). 3.3.5 Encoding process using joint autoregressive hyper-prior model Figure 5 Corresponding to the state of the art compression methods. In this section and the next section, the encoding process and the decoding process will be described separately. Figure 6 Depicts the encoding process. The input image is first processed through the encoder subnetwork. The encoder transforms the input image into a transformed representation called the latent, denoted by y. y is then input to the quantizer block denoted by Q to obtain the quantized latent It is then converted into a bitstream (Bitstream 1) using an arithmetic coding block (denoted as AE). The arithmetic coding blocks are converted one by one in a sequential order. into a bitstream (bitstream 1). The modular super encoder, context, super decoder, and entropy parameter subnetworks are used to estimate the quantized latent The hidden quantity y is input to the super encoder, which outputs the super hidden quantity (denoted by z). The super hidden quantity is then quantized And a second bitstream (Bitstream2) is generated using the arithmetic coding (AE) module. The decomposition entropy module generates a probability distribution, which is used to encode the quantization super-hidden variable into the bitstream. The quantization super-hidden variable includes information about the quantization hidden variable. information about the probability distribution of . The entropy parameter subnetwork generates a probability distribution estimate that is used to encode the quantized latent The information generated by the entropy parameters usually includes mean μ and scale (or variance) σ parameters, which are used together to obtain a Gaussian probability distribution. The Gaussian distribution of a random variable x is defined as Where parameter μ is the mean or expectation of the distribution (as well as its median and mode), while parameter σ is its standard deviation (or variance or scale). In order to define a Gaussian distribution, the mean and variance need to be determined. In existing designs, the entropy parameter module is used to estimate the mean and variance values. The subnetwork superdecoder generates part of the information used by the entropy parameter subnetwork, and the other part of the information is generated by an autoregressive module called the context module. The context module generates information about the probability distribution of samples of the quantized latent quantity using samples that have been encoded by the arithmetic coding (AE) module. Usually a matrix consisting of many samples. Samples can be referred to using indices such as or Depends on the matrix Dimensions of . Sample are encoded one by one by the AE, usually using raster scan order. In raster scan order, the rows of the matrix are processed from top to bottom, where samples in a row are processed from left to right. In this scenario (where raster scan order is used by the AE to encode samples into the bitstream), the context module uses the coded samples before generating the same samples in raster scan order. The information generated by the context module and the super decoder is combined by the entropy parameter module to generate the implicit quantity used for encoding quantization to the probability distribution of the bitstream (bitstream 1). Finally, as a result of the encoding process, the first bitstream and the second bitstream are transmitted to a decoder. Note that other names can be used for the above modules. In the above description, Figure 6 All elements in are collectively called encoders. The analysis transformation that transforms the input image into a latent representation is also called an encoder (or autoencoder). 3.3.6 Decoding process using the joint autoregressive hyper-prior model Figure 7The decoding process is depicted separately. In the decoding process, the decoder first receives a first bitstream (bitstream 1) and a second bitstream (bitstream 2) generated by the corresponding encoder. Bitstream 2 is first decoded by the arithmetic decoding (AD) module by using the probability distribution generated by the decomposition entropy subnetwork. The decomposition entropy module usually generates the probability distribution using a predetermined template, such as a predetermined mean and variance value in the case of a Gaussian distribution. The output of the arithmetic decoding process of bitstream 2 is It is a quantized super-hidden quantity. The AD process is restored to the AE process applied in the encoder. The AE and AD processes are lossless, which means that the quantized super-hidden quantity generated by the encoder is can be reconstructed at the decoder without any changes. In obtaining After that, it is processed by the super decoder, and the output of the super decoder is fed to the entropy parameter module. The three sub-networks, context, super decoder and entropy parameter, employed in the decoder are the same as those in the encoder. Therefore, exactly the same probability distribution can be obtained in the decoder (as in the encoder), which is essential for reconstructing the quantized latent information without any loss. As a result, the quantization implicit The same version of can be obtained in the encoder. The arithmetic decoding module decodes the quantized latent samples one by one from the bitstream of bitstream 1 following the probability distribution (e.g., mean and variance parameters) obtained by the entropy parameter subnetwork. From a practical point of view, the autoregressive model (context model) is inherently serial and therefore cannot be accelerated using techniques such as parallelization. Finally, the fully reconstructed quantized implicit is input to the composite transform (in Figure 7 The decoder module is used to obtain the reconstructed image. In the above description, Figure 7 All elements in are collectively referred to as decoders. The synthetic transform that transforms the quantized latent into the reconstructed image is also called a decoder (or autodecoder). 3.4. Video Compression Based on Neural Networks Similar to conventional video codec techniques, neural images serve as the basis for intra-frame compression in neural network-based video compression, so the development of neural network-based video compression technology is later than that of neural network-based image compression, but more efforts are needed to solve the challenges due to its complexity. Since 2017, several researchers have been working on neural network-based video compression schemes. Compared with image compression, video compression requires effective methods to remove inter-picture redundancy. Inter-picture prediction is then a key step in these works. Motion estimation and compensation are widely adopted, but have only recently been implemented by trained neural networks. The research on neural network-based video compression can be divided into two categories according to the target scenario: random access and low latency. In the case of random access, the decoding can be started from any point in the sequence, usually the whole sequence is divided into multiple separate segments, and each segment can be decoded independently. In the case of low latency, it aims to reduce the decoding time, so that usually only the previous frame in time can be used as a reference frame to decode the subsequent frame. 3.4.1 Low Latency Chen et al. first proposed a video compression scheme with a trained neural network. They first divided the video sequence frames into blocks, and each block will choose one of the two available modes, intra codec or inter codec. If intra codec is selected, there is an associated autoencoder to compress the block. If inter codec is selected, motion estimation and compensation are performed by traditional methods, and the trained neural network will be used for residual compression. The output of the autoencoder is directly quantized and encoded by the Huffman method. Chen et al. proposed another neural network-based video codec scheme with PixelMotionCNN. Frames are compressed in temporal order and each frame is divided into blocks that are compressed in raster scan order. Each frame will first be extrapolated by the two previous reconstructed frames. When a block is to be compressed, the extrapolated frame is fed into PixelMotionCNN along with the context of the current block to derive the latent representation. The residual is then compressed using a variable ratio image scheme. The scheme is comparable to that implemented on H.264. Lu et al. proposed a video compression framework based on a truly end-to-end neural network, in which all modules are implemented through neural networks. The scheme accepts the current frame and the previously reconstructed frame as input, and the optical flow will be derived as motion information through a pre-trained neural network. The motion information will be warped together with the reference frame, followed by a neural network that generates motion compensated frames. The residual and motion information are compressed through two separate neural autoencoders. The entire framework is trained with a single rate-distortion loss function. It achieves better performance than H.264. Rppel et al. proposed a video compression scheme based on advanced neural networks. It inherits and extends the traditional video codec scheme of neural networks with the following main features: 1) only one autoencoder is used to compress motion information and residuals; 2) motion compensation with multiple frames and multiple optical flows; 3) online states are learned and propagated through the following frames over time. This scheme achieves better performance than the HEVC reference software in MS-SSIM. J.Lin et al. proposed a video compression framework based on an extended end-to-end neural network. In this solution, multiple frames are used as references. Therefore, it is possible to provide a more accurate prediction of the current frame by using multiple reference frames and associated motion information. In addition, motion field prediction is deployed to remove motion redundancy along the temporal channel. A post-processing network is also introduced in this work to remove reconstruction artifacts from the previous process. The performance is better than H.265 in terms of both PSNR and MS-SSIM. Erikur et al. proposed a scale-space flow that replaces the commonly used optical flow by adding a scale parameter based on an existing designed framework. It is reported that better performance is achieved than H.264. Z.Hu et al. proposed a multi-resolution representation for optical flow. Specifically, the motion estimation network generates multiple optical flows with different resolutions and lets the network learn which one to choose under the loss function. The performance is slightly improved and outperforms H.265. 3.4.2 Random Access Wu et al. proposed a neural network-based video compression scheme with frame interpolation. The key frame is first compressed by a neural image compressor, and the remaining frames are compressed in a hierarchical order. They perform motion compensation in the perceptual domain, i.e., feature maps are derived at multiple spatial scales of the original frame and the motion is used to warp the feature maps, which will be used in the image compressor. The method is reported to be comparable to H.264. Djuloah et al. proposed a method for interpolation-based video compression, where the interpolation model combines motion information compression and image synthesis, and the same autoencoder is used for both the image and the residual. Airdessein et al. proposed a neural network-based video compression method based on variational autoencoders and deterministic encoders. Specifically, the model consists of an autoencoder and an autoregressive prior. Unlike previous methods, this method accepts a group of pictures (GOP) as input and incorporates a 3D autoregressive prior by considering temporal correlation while encoding and decoding implicit representations. It provides performance comparable to H.265. 3.5 Transformers for Computer Vision Tasks Recently, Transformer models have shown excellent performance on a wide range of natural language tasks (e.g., text classification, machine translation, and question answering). Due to the success of Transformers in natural language processing, Transformer models and their variants have been transferred to computer vision tasks such as image recognition, object detection, segmentation, image super-resolution, and have shown impressive performance compared to convolutional neural networks. Unlike convolutional networks, Transformer features model long-range dependencies between elements of the input sequence, which has led to exciting progress on several vision tasks using Transformer networks. The key idea of ​​the Transformer is the attention mechanism, such as Figure 8 shown. Figure 8 An example operation diagram of the transformer attention mechanism is shown. Given an input sequence, triplets of key, query, and value are computed. An attention map is derived from these triplets and used to reweight the values. Given a sequence, the attention mechanism estimates the relationship between an element and other elements to find the most relevant information in the entire sequence. T The relationship weight matrix is ​​obtained. After the relationship calculation, normalization is performed to obtain the relative relationship result. The relative relationship is then used as a weight to calculate the weighted sum of all elements in the input to aggregate relevant information. Since the calculation of the relationship matrix is ​​a global calculation involving all elements, the transformer-based solution has a better receptive field than the convolutional solution, which brings the success of the current transformer-based method. 4. Question like Figure 4 As shown, in order to construct the quantized implicit The relationship between adjacent spatial elements within the frame. The state-of-the-art intra-frame compression frameworks include autoregressive models and hyper-prior models. Both are designed to reduce the redundancy between elements in different spatial locations. However, this may lead to the following problems: 1. Limited by the receptive field of the convolutional layer, the autoregressive model and the hyper-prior model can only model the correlation between the quantized latent quantities within a limited range, but lack the ability to capture long-range correlations. 2. The Transformer has the ability to model long-range relationships. However, it is not autoregressive, which is necessary for the decoding process. 5. Detailed solution In order to solve the above problems and some unmentioned problems, the methods outlined below are disclosed. The solutions should be considered as examples to explain the general concept and should not be interpreted in a narrow way. In addition, these solutions can be applied individually or in combination in any way. The technique described in this paper provides a probabilistic modeling approach for end-to-end image compression. Transformers are applied to the current end-to-end image compression framework to capture long-range relationships. Benefiting from the ability to capture both long-range and short-range correlations between quantized latent quantities, the proposed method achieves better compression efficiency. In summary, this scheme includes the following aspects: 1) To solve the first problem, a transformer is introduced into the entropy probability modeling part, which can be combined with the current model by one or more of the following methods: a. In one example, the transformer can be used as a plug-in module in probabilistic modeling and combined with existing autoregressive and hyper-prior models to improve encoding and decoding performance: i. In one example, the output of the encoder can be used as the input of the transformer. And the output of the transformer can be combined with the output of the autoregressive model and the hyper-prior model to obtain a probability distribution for entropy coding. 1. In one example, the output of the transformer is combined with the output of the autoregressive model and the hyper-prior model as the input of the sub-network. The probability distribution for entropy encoding and decoding is obtained from the output of the sub-network. ii. In one example, the latents from the encoder and the autoregressive / hyper-prior model can be used as input to the transformer. 1. In one example, the output of the transformer can be combined with a hyper-prior / autoregressive model through an additional sub-network. 2. Alternatively, the output of the transformer can be directly combined with the hyper-prior / autoregressive model. iii. In one example, the latents from the encoder and the additional sub-network can be used as input to the transformer. The output of the transformer can be combined with the outputs of the autoregressive model and the hyper-prior model to obtain a probability distribution for entropy encoding and decoding. 1. In one example, quantizing implicit is the input of the additional subnetwork. 2. In one example, the latent quantity y is the input of the additional subnetwork. 3. In one example, the original image x is the input to the additional subnetwork. iv. In one example, whether to use the transformer as a plug-in module can be signaled from the encoder to the decoder. v. In one example, whether to use the transformer as a plug-in module can be obtained by the decoder. b. In one example, the transformer can also be used to completely replace the existing entropy module, which can remove long-range and short-range redundancies between elements in different spatial levels: i. In one example, a transformer can replace the autoregressive model, and a hyper-prior model can cooperate with the transformer to obtain the desired probability distribution. 1. In one example, the output of the encoder and the hyper-prior model can be fed into a transformer to obtain a probability distribution for entropy coding and decoding. 2. In one example, the output of the encoder may be the only input to the transformer. a. In one example, the output of the transformer may not be used directly for entropy coding. The output of the transformer may be combined with the output super-prior through an additional sub-network to obtain a probability distribution for entropy coding and decoding. b. Alternatively, the output of the transformer is simply the probability distribution of the elements to be encoded or decoded. ii. In one example, the transformer can replace the hyper-prior model. The autoregressive model can cooperate with the transformer to obtain the desired probability distribution. 1. In one example, the output of the encoder and the autoregressive model can be fed into the transformer to directly obtain the probability distribution for entropy encoding and decoding. 2. In one example, the output of the encoder may be the only input to the transformer. a. In one example, the output of the transformer may not be used directly for entropy coding and decoding. It can be combined with the output of the autoregressive model through an additional sub-network to obtain a probability distribution for entropy coding and decoding. b. Alternatively, the output of the transformer is simply the probability distribution of the elements to be encoded or decoded. iii. In one example, the transformer can replace the hyper-prior model and the autoregressive model to obtain the desired probability distribution. 1. In one example, the output of the encoder can be considered as the input of the transformer. 2. In one example, the output of the encoder and the additional network can be considered as the input of the transformer. a. In one example, is the input of the additional subnetwork. b. In one example, the original image x is the input to the additional subnetwork. iv. In one example, whether to use a transformer instead of an autoregressive model and / or a hyper-prior model can be signaled from the encoder to the decoder. v. In one example, whether to replace the autoregressive model and / or the hyper-prior model with a transformer can be derived by the decoder. c. More than one converter model may be used. i. In one example, several transformers may be used to predict the probabilities of all elements separately, and their results may be combined to obtain the final probability result. 1. In one example, the combination process can be a weighted sum, and the weights can be derived through learnable parameters or latent information. 2. In one example, the combination process can be handled by multiple neural networks. ii. In one example, elements in different spatial locations may use different transformer models. iii. In one example, the transformer model to be used may be signaled from the encoder to the decoder. 1. In one example, the information transmitted by the signal can be obtained through RD loss estimation. 2. In one example, information transmitted through a signal can be obtained through hidden information. iv. In one example, the transformer model to be used may be obtained by the decoder. 1. In one example, the information transmitted by the signal can be obtained through super-prior information. 2. In one example, the information transmitted by the signal can be obtained through an autoregressive model. 3. In one example, the information transmitted by the signal can be obtained through a combination of hyper-prior information and an autoregressive model. 2) To solve the second problem, the transformer context model is designed to obtain global relationships in an autoregressive manner. One or more of the following methods are disclosed: a. In one example, enter is fed into the converter, the input Can be used to derive queries, keys, and values ​​to complete attention computations. One or more networks can process the output of attention computations to obtain context parameters for entropy probability modeling: i. In one example, input implicit It can be processed by three different sub-networks to obtain query, key and value information for global relation computation. ii. In one example, the input implicit Can be set directly to query, key and value. iii. In one example, to ensure the autoregressive property, a mask is used on the relationship matrix: 1. In one example, a strictly lower triangular matrix can be used: a. In one example, a strictly lower triangular matrix can be used as a mask. b. Alternatively, a subset of the strictly lower triangular matrix can be used as a mask. 2. In one example, a lower triangular matrix is ​​used. Since the elements on the diagonal of the matrix cannot be obtained by autoregression, these elements are considered as side information and transmitted in the bit stream. a. In one example, a lower triangular matrix can be used as a mask. b. Alternatively, a subset of the lower triangular matrix can be used as a mask. 3. In one example, a lower triangular matrix is ​​used, and the sum of the queries over the channel dimension is used as a replacement for the elements on the matrix diagonal. a. In one example, a lower triangular matrix can be used as a mask. b. Alternatively, a subset of the lower triangular matrix can be used as a mask. 4. In one example, a lower triangular matrix is ​​used. The additional information is used to replace the elements on the diagonal of the matrix. a. In one example, a lower triangular matrix can be used as a mask. b. Alternatively, a subset of the lower triangular matrix can be used as a mask. c. In one example, the replacement is derived from information from a hyper-prior / autoregressive model. 5. In one example, the full matrix is ​​used. The additional information is used to replace the elements on the upper triangular matrix. a. In one example, the replacement is derived from information from a hyper-prior / autoregressive model. iv. In one example, different numbers of attention modules can be utilized: 1. In one example, only one attention module is used. 2. Alternatively, the multiplicative attention modules will be used to obtain the results separately and then aggregated to obtain the result. a. In one example, the aggregation method can be in channel / Stitching in the spatial domain. b. Alternatively, the aggregation method can be a linear operation (addition / subtraction). b. In one example, enter and additional information is fed into the transformer. is processed as key and value information, while the additional information will be used as query information, and several networks can process the output of the attention calculation to obtain the context parameters required for entropy probability modeling: i. In one example, input implicit and additional information can be processed by three sub-networks to obtain query, key, and value information for global relation computation, respectively. ii. In one example, the input implicit Can be set directly as keys and values, and additional information can be set directly as queries. iii. In one example, additional information is derived from information from a hyper-prior / autoregressive model. iv. In one example, to ensure the autoregressive property, a mask is used on the relationship matrix. 1. In one example, a strictly triangular matrix can be used: a. In one example, a strictly lower triangular matrix can be used as a mask. b. Alternatively, a subset of the strictly lower triangular matrix can be used as a mask. 2. In one example, a triangular matrix is ​​used. Since the elements on the diagonal of the relationship matrix cannot be obtained in an autoregressive manner, the autocorrelation of the query is used as a substitute for the diagonal elements. a. In one example, a lower triangular matrix can be used as a mask. b. Alternatively, a subset of the lower triangular matrix can be used as a mask. v. In one example, the entire matrix is ​​used. Since the upper triangular matrix elements cannot be obtained at the decoding end, additional information is used as a query and key to obtain the results in the corresponding area. 1. In one example, the entire matrix is ​​used. 2. In one example, a subset of the entire matrix is ​​used. vi. In one example, different numbers of attention modules can be utilized: 1. In one example, only one attention module is used. 2. Alternatively, the multiplicative attention modules will be used to obtain the results separately and then aggregated to obtain the result. a. In one example, the aggregation method can be in channel / Stitching in the spatial domain. b. Alternatively, the aggregation method can be a linear operation (add / reduce). 3) The transformer model can be used for image encoding and decoding or video encoding and decoding. a. In one example, a transformer model can be used with the prediction results. i. In one example, the intra prediction result can be applied in the transformer. ii. Alternatively, inter-frame prediction results can be used. b. In one example, a transformer model can be used together with motion information. i. In one example, the transformer can be combined with the motion vector. ii. In one example, the transformer can be combined with optical flow. iii. In one example, the transformer can be combined with implicit motion information. c. In one example, previously encoded frames can be used in the converter. 6. Examples This embodiment describes an example of how the designed transformer module solves the problem of modeling long-range relationships. 6.1 Overall Framework Fig. 9A possible framework of the proposed method is shown in Figure 1. In this embodiment, the transformer is used as a replacement for the autoregressive model. Specifically, for an input image x, with parameters φ ga The encoder network g a Transform it into latent information y: y=g a (x;φ ga ). (4) Then, the latent information y is quantized by the quantization function Q(.) to reduce the information, which can be expressed as: The rounding operation is used in the test phase in the quantization part. Since the rounding operation is non-differential, the uniform noise u(-0.5, 0.5) is used as a replacement in the training phase, which can be formulated as: In the entropy coding part, we first use the parameter φ ha The super prior encoder h a , super prior encoder h a Use the hidden quantity y as input and the super prior information z as output. Similar to y, the super prior information z is quantized by the quantization function Q(.), and then the quantized information is estimated by the decomposition model F The probability distribution of The above process can be formulated as: After that, arithmetic coding can be performed according to Will Convert to a bit stream. In order to decode The corresponding inverse operation is performed through the arithmetic decoding process. Then the decoded are fed to the reference synthesis network R f and super decoder h s Reference synthesis network R f is used to obtain the reference information r for the transformer context model, while the super decoder h s Generates information for entropy estimation module e. In addition to the hyper-prior model, the transformer context model t is also used to help model the probability distribution. The additional information p is the input of t, and the output of t is combined with the super decoder h through the entropy estimation module e s In the estimation module e, The probability distribution model of is modeled as a Gaussian distribution, and the output of e is P y The estimated mean μ and standard deviation σ of : Among them, φ e ,φ t , φ hs Represent e, t, R respectively f ,h s , N represents the Gaussian distribution function. Probability distribution based on estimates The encoder can convert the quantized implicit Convert it into a bit stream and convert it back to quantization implicit through inverse operation. After that, with parameter φ gs The decoder network g s Will as input and obtain the final reconstruction 6.2 Transformer Context Model and Reference Synthesis Network In order to obtain the quantized The global relationships within the framework leverage the transformer context model, and Fig.10 A possible structure of a transformer context model is shown. For the input reference information r and the quantized implicit Different embedding layers are used to eliminate the reference information r and the quantization implicit The domain gap between . Specifically, a single convolutional layer with a kernel size of 1x1 is used in the embedding layer. After the embedding layer, positional encoding is performed on the output of the embedding layer. The positional encoding contains a learnable tensor of shape (16,16,C). It marks the spatial position of the latent variable and is combined with the output of the embedding layer through an addition operation. To implement the autoregressive process, masked attention is performed in the Transformer context model. Fig.11 A possible solution for masking attention is shown: Specifically, set r to the query information, For key and value information. Both are flattened into 2D matrices by raster scan order in the spatial domain. To obtain the association matrix between the query and the key, matrix multiplication is used after the flattening operation. Since the autocorrelation of the current element to be decoded and the correlation between the current element and the undecoded elements are not available at the decoder stage, a strict lower triangular mask is utilized on the correlation matrix. As an alternative to autocorrelation, the sum of squares of the query in the channel dimension is used, which is diagonalized and then combined with the mask matrix. After that, SoftMax is used to normalize the relationship matrix. The output of SoftMax is masked with a strict lower triangular matrix to ensure autoregression when calculated with value information. Finally, based on the relative relationship matrix, most of the relevant elements before the element to be encoded / decoded are combined through multiple mask relationship matrices and values. In order to improve the fitting ability of the model, a multi-masked attention model is adopted. The multi-masked attention model processes the input information separately and combines them through splicing operations. The combination of multiple masked attention modules is called a masked multi-head attention module. Specifically, the number of multi-head attention used in this scheme is 4. After the masked multi-head attention module, the latent information is further processed through residual connections, normalization, and linear layers. The final output is combined with the output of the super-prior decoder to obtain the final probability distribution

[0037] More details of embodiments of the present disclosure related to transformer-based probabilistic modeling for data encoding and decoding are described below.As used herein, the term "data" may refer to an image, a picture in a video, a video, or any other data suitable for being encoded and decoded.

[0038] As mentioned above, in order to construct a quantitative implicit representation of the data The relationship between adjacent spatial elements in the existing encoding and decoding framework includes autoregressive models (such as Figure 6 and Figure 7 contextual models as shown) and hyper-prior models such as Figure 6 and Figure 7 The super encoder and / or super decoder shown in FIG. 1 ). Both the autoregressive model and the super prior model are designed to reduce the redundancy between elements in different spatial positions. However, due to the limited receptive field of the convolutional layer, the autoregressive model and the super prior model can only model the correlation between the quantized latent quantities within a limited range, but lack the ability to capture long-range correlations.

[0039] In order to solve the above problems and some unmentioned problems, a data processing solution as described below is disclosed. The embodiments of the present disclosure should be considered as examples to explain general concepts and should not be interpreted in a narrow manner. In addition, these embodiments can be applied individually or in combination in any way.

[0040] Fig.12 1 shows a flow chart of a method 1200 for data processing according to some embodiments of the present disclosure. The method 1200 may be implemented during conversion between data and a bit stream of data. Fig.12As shown, method 1200 starts at 1202, where a probability distribution for entropy encoding and decoding associated with a bitstream is determined by using a first model with an attention mechanism. As used herein, the term "attention mechanism" may refer to an existing mechanism for modeling long-range relationships in a neural network, such as across text in the context of natural language processing (NLP). With the help of an attention mechanism, shortcuts can be built between a context vector and an input to allow the model to attend to different parts.

[0041] As an example and not limitation, the first model can be used as a plug-in module in probability modeling. In other words, the first model can be used in conjunction with the autoregressive model and the hyper-prior model to determine the probability distribution. In one example, the intermediate information can be determined based on the quantized implicit representation of the data using the first model. In addition, the probability distribution can be generated based on the intermediate information and the output of the autoregressive model and the output of the hyper-prior model. For example, the probability distribution is generated by using another model different from the first model. In some embodiments, the first model can be designed based on a transformer architecture and can be referred to as a transformer, a transformer model, a transformer context model, etc. For illustration purposes, Fig.10 An example structure of a transformer context model according to an embodiment of the present disclosure is shown. It should be understood that the above examples are described for illustrative purposes only. The scope of the present disclosure is not limited in this regard.

[0042] At 1204, a conversion is performed based on the probability distribution. In one example, the conversion may include encoding the data into a bitstream. Entropy coding (such as arithmetic coding or Huffman coding) may be performed on the quantized information based on the probability distribution to generate a bitstream. Alternatively or additionally, the conversion may include decoding the data from the bitstream. Entropy decoding (such as arithmetic decoding or Huffman decoding) may be performed on the bitstream based on the probability distribution to obtain quantized information that can be further processed to reconstruct the data. It should be understood that the above description is described for descriptive purposes only. The scope of the present disclosure is not limited in this regard.

[0043] Based on the above, the first model using the attention mechanism obtains the probability distribution for entropy encoding and decoding. Compared with the traditional convolution-based solution, the first model can additionally capture the long-range correlation between the quantization latent quantities. By capturing both the long-range and short-range correlations between the quantization latent quantities, the first model has a better receptive field than the convolution-based solution. Therefore, the proposed method can advantageously improve the encoding and decoding efficiency.

[0044] In some embodiments, at 1202, intermediate information may be generated by using a first model based on a quantized implicit representation of the data and a first output of the autoregressive model. In addition, a probability distribution may be generated based on the intermediate information and a second output of the hyper-prior model. In one example, the probability distribution may be generated by another model based on the intermediate information and the second output. Alternatively, the probability distribution may be generated by directly combining the intermediate information and the second output.

[0045] In some alternative embodiments, at 1202, intermediate information can be generated by using the first model based on the quantized implicit representation of the data and the second output of the hyper-prior model. In addition, a probability distribution can be generated based on the intermediate information and the first output of the autoregressive model. In one example, the probability distribution can be generated by another model based on the intermediate information and the first output. Alternatively, the probability distribution can be generated by directly combining the intermediate information and the first output.

[0046] In some embodiments, at 1202, intermediate information may be generated based on a quantized implicit representation of the data and an output of another model using a first model. In addition, a probability distribution may be generated based on the intermediate information, a first output of the autoregressive model, and a second output of the hyper-prior model. As an example and not limitation, the input of the other model may include at least one of the following: data, an implicit representation of the data, or a quantized implicit representation of the data.

[0047] In some embodiments, information about whether the probability distribution is determined by using a combination of the first model with the autoregressive model and the hyper-prior model may be indicated in the bitstream. In other words, whether the first model is used as a plug-in module may be signaled in the bitstream. Alternatively, information about whether the probability distribution is determined by using a combination of the first model with the autoregressive model and the hyper-prior model may be determined by a decoder.

[0048] In some embodiments, the probability distribution can be determined by using a combination of a first model and a hyper-prior model. For example, the first model can be used to replace the autoregressive model and cooperate with the hyper-prior model to generate a probability distribution. In one example, the probability distribution can be determined by using the first model based on a quantized implicit representation of the data and the output of the hyper-prior model. Alternatively, the input of the first model can include a quantized implicit representation of the data. In some embodiments, the probability distribution can be determined by using another model based on the output of the first model and the output of the hyper-prior model. Alternatively, the probability distribution can be determined as the output of the first model.

[0049] In some alternative embodiments, the probability distribution can be determined by using a combination of the first model and the autoregressive model. For example, the first model can be used to replace the hyper-prior model and cooperate with the autoregressive model to generate a probability distribution. In one example, the probability distribution can be determined by using the first model based on the quantized implicit representation of the data and the output of the autoregressive model. Alternatively, the input of the first model may include the quantized implicit representation of the data. In some embodiments, the probability distribution can be determined by using another model based on the output of the first model and the output of the autoregressive model. Alternatively, the probability distribution can be determined as the output of the first model.

[0050] In some embodiments, the probability distribution can be determined by using a first model. For example, the first model can be used to replace an autoregressive model and a hyper-prior model. In one example, the input of the first model can include a quantized implicit representation of the data. Alternatively, the input can also include the output of another model. For example, the input of another model can include at least one of the following: data, or a quantized implicit representation of the data.

[0051] In some embodiments, information about whether the first model is used to replace the autoregressive model may be indicated in the bitstream. Information about whether the first model is used to replace the hyper-prior model may be indicated in the bitstream. Alternatively or additionally, information about whether the first model is used to replace the autoregressive model and the hyper-prior model may be indicated in the bitstream.

[0052] In some alternative embodiments, information about whether the first model is used to replace the autoregressive model may be determined by the decoder. Information about whether the first model is used to replace the hyper-prior model may be determined by the decoder. Alternatively or additionally, information about whether the first model is used to replace the autoregressive model and the hyper-prior model may be determined by the decoder.

[0053] In some embodiments, the probability distribution can be determined by using multiple first models. For example, more than one transformer model can be used. As an example and not limitation, at 1202, a first candidate probability distribution for entropy coding and decoding can be determined by using one of the multiple first models. A second candidate probability distribution for entropy coding and decoding can be determined by using another one of the multiple first models. Additionally, a probability distribution can be generated based on the first candidate probability distribution and the second candidate probability distribution. In one example, a probability distribution can be generated based on a weighted sum of the first candidate probability distribution and the second candidate probability distribution. The weights for the first candidate probability distribution and the second candidate probability distribution can be determined based on a learnable parameter or implicit information. In another example, the probability distribution is generated by at least one other model (such as a neural network).

[0054] In some embodiments, one of the plurality of first models may be used for a first element at a first spatial position of a quantized implicit representation of the data. Another of the plurality of first models may be used for a second element at a second spatial position of the quantized implicit representation. The second element spatial position is different from the first element spatial position. For example, different transformer models may be used for elements in different spatial positions.

[0055] In some embodiments, information about the first model to be used can be indicated in the bitstream. For example, which first model to use can be sent from the encoder to the decoder with a signal. In one example, the information can be determined based on a rate-distortion (RD) loss estimate. Alternatively, the information can be determined based on implicit information.

[0056] In some alternative embodiments, information about the first model to be used can be determined by the decoder. For example, the decoder can determine which first model to use. In one example, the information can be determined based on super prior information. In another example, the information can be determined by an autoregressive model. In another example, the information can be determined by a combination of super prior information and an autoregressive model. It should be understood that the above description is described for descriptive purposes only. The scope of the present disclosure is not limited in this regard.

[0057] In some embodiments, at 1202, context parameters may be determined based on a quantized implicit representation of the data using a first model. In addition, a probability distribution may be generated based on the context parameters. In one example, to determine the context parameters, an attention calculation may be performed based on the quantized implicit representation. In addition, the context parameters may be generated based on the result of the attention calculation using at least one sub-network.

[0058] In some embodiments, in order to perform attention calculation, a relationship between the encoded and decoded elements of the quantized implicit representation of the data and the current element may be generated, and a result of the attention calculation may be generated based on the relationship. The relationship may be determined in an autoregressive manner.

[0059] In some embodiments, a query, key, and value for performing attention computation may be determined based on a quantized implicit representation. In one example, the query may be determined based on the quantized implicit representation using a first subnetwork, the key may be determined based on the quantized implicit representation using a second subnetwork, and the value may be determined based on the quantized implicit representation using a third subnetwork. The first subnetwork, the second subnetwork, and the third subnetwork may be different from each other. In another example, each of the query, key, and value may be a quantized implicit representation.

[0060] In some embodiments, the relationship can be represented by a relationship matrix, and a mask can be applied to the relationship matrix to generate a result. For example, a strict lower triangular matrix can be used to generate the result. As used herein, the term "lower triangular matrix" (also referred to as "lower triangle matrix") may refer to a matrix in which all elements above the main diagonal are zero, i.e., matrix A = [a ij , such that for i < j, a ij = 0. The term "strict lower triangular matrix" (also referred to as "strict lower triangle matrix") may refer to a lower triangular matrix with zeros on the diagonal and above the diagonal, i.e., matrix A = [a ij , such that for i <= j, a ij = 0.

[0061] In some embodiments, the mask can be a strict lower triangular matrix. Alternatively, the mask can be a subset of the strict lower triangular matrix. That is, some elements of the strict lower triangular matrix can be used as the mask.

[0062] In some embodiments, a lower triangular matrix can be used to generate the result, and the elements on the diagonal of the relationship matrix can be indicated in the bitstream. In one example, the mask can be a lower triangular matrix. Alternatively, the mask can be a subset of the lower triangular matrix. That is, some elements of the lower triangular matrix can be used as the mask.

[0063] In some embodiments, a lower triangular matrix can be used to generate the result, and the elements on the diagonal of the relationship matrix can be set to the sum of the queries of the channel dimensions. In one example, the mask can be a lower triangular matrix. Alternatively, the mask can be a subset of the lower triangular matrix. That is, some elements of the lower triangular matrix can be used as the mask.

[0064] In some embodiments, a lower triangular matrix can be used to generate the result, and the elements on the diagonal of the relationship matrix can be determined based on additional information. As an example, the additional information can include at least one of the output of an autoregressive model or the output of a hyperprior model. In one example, the mask can be a lower triangular matrix. Alternatively, the mask can be a subset of the lower triangular matrix. That is, some elements of the lower triangular matrix can be used as the mask.

[0065] In some embodiments, the full relationship matrix can be used to generate the result, and the elements on the upper triangular part of the relationship matrix can be determined based on additional information. As an example, the additional information can include at least one of the output of an autoregressive model or the output of a hyperprior model.

[0066] In some embodiments, multiple attention modules can be used to perform attention calculations. As an example, the output of one of the multiple attention modules can be determined as the result of the attention calculation. Alternatively, the result of the attention calculation can be generated by performing an aggregation process on the outputs of multiple attention modules. In one example, the aggregation process can include splicing in the channel or spatial domain. In another example, the aggregation process can include linear operations such as addition and / or subtraction.

[0067] In some embodiments, a key and a value for performing an attention computation may be determined based on a quantized implicit representation. A query for performing an attention computation may be determined based on additional information. As an example, the additional information may be determined based on at least one of an output of a hyper-prior model or an output of an autoregressive model. In one example, the query may be determined based on the additional information using a first subnetwork, the key may be determined based on the quantized implicit representation using a second subnetwork, and the value may be determined based on the quantized implicit representation using a third subnetwork. In another example, the query may be the additional information, and each of the key and the value may be a quantized implicit representation.

[0068] In some embodiments, the relationship can be represented by a relationship matrix, and a mask can be applied on the relationship matrix to generate a result. In some embodiments, a strict triangular matrix can be used to generate a result. As an example, the strict triangular matrix can be a strict lower triangular matrix, and the mask can be a strict lower triangular matrix or a subset of a strict lower triangular matrix.

[0069] In some alternative embodiments, a triangular matrix may be used to generate the result, and the elements on the diagonal of the relationship matrix may be determined based on the autocorrelation of the query. As an example, the triangular matrix may be a lower triangular matrix, and the mask may be a lower triangular matrix or a subset of the lower triangular matrix.

[0070] In some embodiments, the entire relationship matrix can be used to generate the results, and the elements on the upper triangular portion of the relationship matrix can be determined based on using the additional information as a query and a key. Alternatively, a subset of the relationship matrix can be used to generate the results, and the elements on the upper triangular portion of the relationship matrix can be determined based on using the additional information as a query and a key.

[0071] In some embodiments, multiple attention modules can be used to perform attention calculations. As an example, the output of one of the multiple attention modules can be determined as the result of the attention calculation. Alternatively, the result of the attention calculation can be generated by performing an aggregation process on the outputs of multiple attention modules. In one example, the aggregation process can include splicing in the channel or spatial domain. In another example, the aggregation process can include linear operations such as addition or subtraction.

[0072] In some embodiments, the conversion can be performed by using a second model with an attention mechanism. As an example, the second model can be designed based on a transformer architecture and can be referred to as a transformer model, etc. In one example, a prediction associated with the data for performing the conversion can be determined by using a second model. For example, the prediction can include at least one of an intra-frame prediction or an inter-frame prediction. In another example, motion information associated with the data for performing the conversion can be determined by using a second model. For example, the motion information can include at least one of a motion vector, an optical flow, or implicit motion information. In some embodiments, the data can include multiple frames, and the input of the second model can include a decoded frame of the multiple frames.

[0073] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a data bitstream generated by a method performed by an apparatus for data processing. According to the method, a probability distribution of entropy encoding and decoding associated with a bitstream is determined by using a first model with an attention mechanism. In addition, a bitstream is generated based on the probability distribution.

[0074] According to some other embodiments of the present disclosure, a method for storing a bitstream of a video is provided. In the method, a probability distribution of entropy encoding and decoding associated with the bitstream is determined by using a first model with an attention mechanism. In addition, the bitstream is generated based on the probability distribution, and the bitstream is stored in a non-transitory computer-readable recording medium.

[0075] Implementations of the present disclosure may be described in view of the following clauses, and features of the following clauses may be combined in any reasonable way.

[0076] Item 1. A data processing method, comprising: during conversion between data and a bit stream of the data, determining a probability distribution for entropy encoding and decoding associated with the bit stream by using a first model with an attention mechanism; and performing the conversion based on the probability distribution.

[0077] Clause 2. The method of clause 1, wherein the first model comprises a transformer model or a transformer context model.

[0078] Clause 3. A method according to any one of clauses 1-2, wherein the probability distribution is determined by using the first model in combination with an autoregressive model and a hyper-prior model.

[0079] Clause 4. A method according to clause 3, wherein determining the probability distribution comprises: generating intermediate information based on a quantized implicit representation of the data by using the first model; and generating the probability distribution based on the intermediate information, the output of the autoregressive model and the output of the hyper-prior model.

[0080] Clause 5. The method of clause 4, wherein the probability distribution is generated by using another model different from the first model.

[0081] Clause 6. A method according to Clause 3, wherein determining the probability distribution comprises: generating intermediate information based on a quantized latent representation of the data and a first output of the autoregressive model by using the first model; and generating the probability distribution based on the intermediate information and a second output of the hyper-prior model.

[0082] Clause 7. The method of clause 6, wherein the probability distribution is generated by another model based on the intermediate information and the second output.

[0083] Clause 8. The method of clause 6, wherein the probability distribution is generated by directly combining the intermediate information and the second output.

[0084] Clause 9. A method according to Clause 3, wherein determining the probability distribution comprises: generating intermediate information based on a quantized latent representation of the data and a second output of the hyper-prior model by using the first model; and generating the probability distribution based on the intermediate information and a first output of the autoregressive model.

[0085] Clause 10. The method of clause 9, wherein the probability distribution is generated by another model based on the intermediate information and the first output.

[0086] Clause 11. The method of clause 10, wherein the probability distribution is generated by directly combining the intermediate information and the first output.

[0087] Clause 12. A method according to clause 3, wherein determining the probability distribution comprises: generating intermediate information based on a quantized latent representation of the data and an output of another model by using the first model; and generating the probability distribution based on the intermediate information, a first output of the autoregressive model and a second output of the hyper-prior model.

[0088] Clause 13. The method of clause 12, wherein the input to the other model comprises at least one of: the data, an implicit representation of the data, or a quantized implicit representation of the data.

[0089] Clause 14. The method of any of clauses 1-13, wherein information about whether the probability distribution is determined by using the first model in combination with an autoregressive model and a hyper-prior model is indicated in the bitstream.

[0090] Clause 15. A method according to any of clauses 1-13, wherein the information about whether the probability distribution is determined by using the first model in combination with an autoregressive model and a hyper-prior model is determined by a decoder.

[0091] Clause 16. The method of any of clauses 1-2, wherein the probability distribution is determined by using the first model in combination with a hyper-prior model.

[0092] Clause 17. The method of clause 16, wherein the probability distribution is determined by using the first model based on a quantized latent representation of the data and an output of the hyper-prior model.

[0093] Clause 18. The method of clause 16, wherein an input to the first model comprises a quantized latent representation of the data.

[0094] Clause 19. The method of clause 18, wherein the probability distribution is determined by using another model based on an output of the first model and an output of the hyper-prior model.

[0095] Clause 20. The method of clause 18, wherein the probability distribution is determined as an output of the first model.

[0096] Clause 21. The method of any of clauses 1-2, wherein the probability distribution is determined by using the first model in combination with an autoregressive model.

[0097] Clause 22. The method of clause 21, wherein the probability distribution is determined by using the first model based on a quantized latent representation of the data and an output of the autoregressive model.

[0098] Clause 23. A method according to clause 21, wherein the input to the first model comprises a quantized latent representation of the data.

[0099] Clause 24. The method of clause 23, wherein the probability distribution is determined by using another model based on an output of the first model and an output of the autoregressive model.

[0100] Clause 25. The method of clause 23, wherein the probability distribution is determined as an output of the first model.

[0101] Clause 26. A method according to any of clauses 1-2, wherein the input to the first model comprises a quantized latent representation of the data.

[0102] Clause 27. The method of clause 26, wherein the input further comprises an output of another model.

[0103] Clause 28. The method of clause 27, wherein the input to the other model comprises at least one of: the data, or a quantized implicit representation of the data.

[0104] Clause 29. A method according to any one of clauses 1-2, wherein information about whether the first model is used to replace an autoregressive model is indicated in the bitstream, information about whether the first model is used to replace a hyper-prior model is indicated in the bitstream, or information about whether the first model is used to replace both the autoregressive model and the hyper-prior model is indicated in the bitstream.

[0105] Clause 30. A method according to any one of clauses 1-2, wherein information about whether the first model is used to replace an autoregressive model is determined by a decoder, information about whether the first model is used to replace a hyper-prior model is determined by a decoder, or information about whether the first model is used to replace both the autoregressive model and the hyper-prior model is determined by a decoder.

[0106] Clause 31. The method of any one of clauses 1-2, wherein the probability distribution is determined by using a plurality of the first models.

[0107] Clause 32. According to the method described in Clause 31, determining the probability distribution includes: determining a first candidate probability distribution for the entropy coding and decoding by using one of the multiple first models; determining a second candidate probability distribution for the entropy coding and decoding by using another first model among the multiple first models; and generating the probability distribution based on the first candidate probability distribution and the second candidate probability distribution.

[0108] Clause 33. A method according to clause 32, wherein the probability distribution is generated based on a weighted sum of the first candidate probability distribution and the second candidate probability distribution, and the weights for the first candidate probability distribution and the second candidate probability distribution are determined based on learnable parameters or latent information.

[0109] Clause 34. The method of clause 32, wherein the probability distribution is generated by at least one additional model.

[0110] Clause 35. A method according to Clause 31, wherein one of the plurality of first models is used for a first element at a first spatial position of a quantized implicit representation of the data, and another first model of the plurality of first models is used for a second element at a second spatial position of the quantized implicit representation, the second element spatial position being different from the first element spatial position.

[0111] Clause 36. The method of clause 31, wherein information about the first model to be used is indicated in the bitstream.

[0112] Clause 37. The method of clause 36, wherein the information is determined based on a rate-distortion (RD) loss estimate.

[0113] Clause 38. A method according to clause 36, wherein the information is determined based on implicit information.

[0114] Clause 39. The method of clause 31, wherein information about the first model to be used is determined by a decoder.

[0115] Clause 40. The method of clause 39, wherein the information is determined based on super-prior information.

[0116] Clause 41. The method of clause 39, wherein the information is determined by an autoregressive model.

[0117] Clause 42. The method of clause 39, wherein the information is determined by a combination of hyper-prior information and an autoregressive model.

[0118] Clause 43. A method according to any one of clauses 1-2, wherein determining the probability distribution comprises: determining context parameters based on a quantized implicit representation of the data by using the first model; and generating the probability distribution based on the context parameters.

[0119] Clause 44. A method according to clause 43, wherein determining the context parameters comprises: performing attention calculation based on the quantized latent representation; and generating the context parameters based on the result of the attention calculation by using at least one sub-network.

[0120] Clause 45. A method according to clause 44, wherein performing the attention calculation includes: generating a relationship between a decoded element and a current element of a quantized implicit representation of the data; and generating the result based on the relationship.

[0121] Clause 46. The method of clause 45, wherein the relationship is determined in an autoregressive manner.

[0122] Clause 47. A method according to any of clauses 44-46, wherein the query, key and value used to perform the attention calculation are determined based on the quantized latent representation.

[0123] Clause 48. A method according to clause 47, wherein the query is determined based on the quantized latent representation by using a first sub-network, the key is determined based on the quantized latent representation by using a second sub-network, and the value is determined based on the quantized latent representation by using a third sub-network, and the first sub-network, the second sub-network and the third sub-network are different from each other.

[0124] Clause 49. The method of clause 47, wherein each of the query, the key, and the value is the quantized implicit representation.

[0125] Clause 50. The method of any one of clauses 45-49, wherein the relationship is represented by a relationship matrix, and a mask is applied to the relationship matrix for generating the result.

[0126] Clause 51. The method of clause 50, wherein a strictly lower triangular matrix is ​​used to generate the result.

[0127] Clause 52. The method of clause 51, wherein the mask is the strictly lower triangular matrix.

[0128] Clause 53. The method of clause 51, wherein the mask is a subset of the strictly lower triangular matrix.

[0129] Clause 54. The method of clause 50, wherein a lower triangular matrix is ​​used to generate the result, and elements on the diagonal of the relation matrix are indicated in the bitstream.

[0130] Clause 55. The method of clause 54, wherein the mask is the lower triangular matrix.

[0131] Clause 56. The method of Clause 54, wherein the mask is a subset of the lower triangular matrix.

[0132] Clause 57. The method of clause 50, wherein a lower triangular matrix is ​​used to generate the result and the elements on the diagonal of the relationship matrix are set to the sum of the queries over the channel dimension.

[0133] Clause 58. The method of clause 57, wherein the mask is the lower triangular matrix.

[0134] Clause 59. The method of Clause 57, wherein the mask is a subset of the lower triangular matrix.

[0135] Clause 60. The method of clause 50, wherein a lower triangular matrix is ​​used to generate the result and the elements on the diagonal of the relationship matrix are determined based on additional information.

[0136] Clause 61. The method of clause 60, wherein the mask is the lower triangular matrix.

[0137] Clause 62. The method of clause 60, wherein the mask is a subset of the lower triangular matrix.

[0138] Clause 63. A method according to any of clauses 60-62, wherein the additional information comprises at least one of an output of an autoregressive model or an output of a hyper-prior model.

[0139] Clause 64. The method of clause 50, wherein the entire relationship matrix is ​​used to generate the result, and elements of an upper triangular portion of the relationship matrix are determined based on additional information.

[0140] Clause 65. The method of clause 64, wherein the additional information comprises at least one of an output of an autoregressive model or an output of a hyper-prior model.

[0141] Clause 66. A method according to any of clauses 44-65, wherein multiple attention modules are used to perform the attention calculation.

[0142] Clause 67. A method according to clause 66, wherein an output of an attention module among the plurality of attention modules is determined as the result of the attention calculation.

[0143] Clause 68. A method according to Clause 66, wherein the result of the attention calculation is generated by performing aggregation processing on the outputs of the multiple attention modules.

[0144] Clause 69. The method of clause 68, wherein the aggregation process comprises stitching in the channel domain or the spatial domain.

[0145] Clause 70. The method of clause 68, wherein the aggregation process comprises a linear operation.

[0146] Clause 71. The method of clause 70, wherein the linear operation comprises addition or subtraction.

[0147] Clause 72. A method according to any of clauses 45-46, wherein the keys and values ​​used to perform the attention calculation are determined based on the quantized implicit representation, and the query used to perform the attention calculation is determined based on additional information.

[0148] Clause 73. The method of clause 72, wherein the additional information is determined based on at least one of an output of a hyper-prior model or an output of an autoregressive model.

[0149] Clause 74. A method according to any one of clauses 72-73, wherein the query is determined based on the additional information by using a first subnetwork, the key is determined based on the quantized latent representation by using a second subnetwork, and the value is determined based on the quantized latent representation by using a third subnetwork.

[0150] Clause 75. The method of any of Clauses 72-73, wherein the query is the additional information, and each of the key and the value is the quantized implicit representation.

[0151] Clause 76. A method according to any of clauses 72-75, wherein the relationship is represented by a relationship matrix and a mask is applied to the relationship matrix for generating the result.

[0152] Clause 77. The method of clause 76, wherein a strictly triangular matrix is ​​used to generate the result.

[0153] Clause 78. The method of Clause 77, wherein the strictly triangular matrix is ​​a strictly lower triangular matrix and the mask is the strictly lower triangular matrix.

[0154] Clause 79. The method of Clause 77, wherein the strictly triangular matrix is ​​a strictly lower triangular matrix, and the mask is a subset of the strictly lower triangular matrix.

[0155] Clause 80. The method of clause 76, wherein a triangular matrix is ​​used to generate the result, and the elements on the diagonal of the relationship matrix are determined based on the autocorrelation of the query.

[0156] Clause 81. The method of Clause 80, wherein the triangular matrix is ​​a lower triangular matrix and the mask is the lower triangular matrix.

[0157] Clause 82. The method of Clause 80, wherein the triangular matrix is ​​a lower triangular matrix and the mask is a subset of the lower triangular matrix.

[0158] Clause 83. The method of clause 76, wherein the entire relationship matrix is ​​used to generate the result, and elements of an upper triangular portion of the relationship matrix are determined based on using additional information as the query and the key.

[0159] Clause 84. The method of clause 76, wherein a subset of the relationship matrix is ​​used to generate the result, and the elements of the upper triangular portion of the relationship matrix are determined based on using additional information as the query and the key.

[0160] Clause 85. A method according to any of clauses 72-84, wherein multiple attention modules are used to perform the attention calculation.

[0161] Clause 86. A method according to clause 85, wherein an output of an attention module among the plurality of attention modules is determined as the result of the attention calculation.

[0162] Clause 87. A method according to clause 85, wherein the result of the attention calculation is generated by performing aggregation processing on the outputs of the multiple attention modules.

[0163] Clause 88. The method of clause 87, wherein the aggregation process comprises stitching in the channel domain or the spatial domain.

[0164] Clause 89. The method of clause 87, wherein the aggregation process comprises a linear operation.

[0165] Clause 90. The method of clause 89, wherein the linear operation comprises addition or subtraction.

[0166] Clause 91. A method according to any of clauses 1-90, wherein the conversion is performed by using a second model having the attention mechanism.

[0167] Clause 92. The method of clause 91, wherein the second model comprises a transformer model.

[0168] Clause 93. The method of clause 91, wherein a prediction associated with the data used to perform the transformation is determined using the second model.

[0169] Clause 94. The method of clause 93, wherein the prediction comprises at least one of intra prediction or inter prediction.

[0170] Clause 95. The method of any one of clauses 91-94, wherein motion information associated with the data used to perform the conversion is determined using the second model.

[0171] Clause 96. The method of clause 95, wherein the motion information comprises at least one of motion vectors, optical flow, or latent motion information.

[0172] Clause 97. A method according to any of clauses 91-96, wherein the data comprises a plurality of frames and the input to the second model comprises encoded frames of the plurality of frames.

[0173] Clause 98. The method of any of clauses 1-97, wherein the data comprises at least one of: an image, a picture of a video, or a video.

[0174] Clause 99. The method of any one of clauses 1-98, wherein the converting comprises encoding the data into the bitstream.

[0175] Clause 100. The method of any one of clauses 1-98. wherein the converting comprises decoding the data from the bitstream.

[0176] Clause 101. An apparatus for data processing, comprising a processor and a non-transitory memory having instructions thereon, the instructions, when executed by the processor, causing the processor to perform a method according to any one of clauses 1-100.

[0177] Clause 102. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method of any of clauses 1-100.

[0178] Item 103. A non-transitory computer-readable recording medium storing a bit stream of data, the bit stream being generated by a method performed by a device for data processing, wherein the method comprises: determining a probability distribution for entropy encoding and decoding associated with the bit stream by using a first model having an attention mechanism; and generating the bit stream based on the probability distribution.

[0179] Item 104. A method for storing a bitstream of data, comprising: determining a probability distribution of entropy encoding and decoding associated with the bitstream by using a first model with an attention mechanism; generating the bitstream based on the probability distribution; and storing the bitstream in a non-transitory computer-readable recording medium. Example Device

[0180] Fig.13 A block diagram of a computing device 1300 is shown in which various embodiments of the present disclosure may be implemented. The computing device 1300 may be implemented as a source device 110 (or data encoder 114) or a destination device 120 (or data decoder 124).

[0181] It should be understood that Fig.13 The computing device 1300 shown in FIG. 1 is for illustrative purposes only and is not intended to in any way imply any limitation on the functionality and scope of the embodiments of the present disclosure.

[0182] like Fig.13As shown, computing device 1300 includes a general computing device 1300. Computing device 1300 may include at least one or more processors or processing units 1310, memory 1320, storage unit 1330, one or more communication units 1340, one or more input devices 1350, and one or more output devices 1360.

[0183] In some embodiments, the computing device 1300 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server provided by a service provider, a large computing device, etc. The user terminal can be, for example, any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet computer, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device, or any combination thereof, and includes accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 1300 can support any type of interface to the user (such as a "wearable" circuit device, etc.).

[0184] The processing unit 1310 may be a physical processor or a virtual processor and may implement various processes based on a program stored in the memory 1320. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the computing device 1300. The processing unit 1310 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0185] The computing device 1300 typically includes various computer storage media. Such media can be any media accessible by the computing device 1300, including but not limited to volatile media and non-volatile media, or removable media and non-removable media. The memory 1320 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM) or flash memory), or any combination thereof. The storage unit 1330 can be any removable or non-removable medium, and can include machine-readable media, such as a memory, a flash drive, a disk, or other media that can be used to store information and / or data and can be accessed in the computing device 1300.

[0186] The computing device 1300 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Fig.13 Although not shown in the figure, a magnetic disk drive for reading from and / or writing to a removable nonvolatile magnetic disk and an optical disk drive for reading from and / or writing to a removable nonvolatile optical disk may be provided. In this case, each drive may be connected to the bus (not shown) via one or more data medium interfaces.

[0187] The communication unit 1340 communicates with another computing device via a communication medium. In addition, the functionality of the components in the computing device 1300 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 1300 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general network nodes.

[0188] Input device 1350 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 1360 may be one or more of various output devices, such as a display, speaker, printer, etc. With the aid of communication unit 1340, computing device 1300 may also communicate with one or more external devices (not shown), such as storage devices and display devices, and computing device 1300 may also communicate with one or more devices that enable a user to interact with computing device 1300, or any device that enables computing device 1300 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.), if necessary. Such communication may be performed via an input / output (I / O) interface (not shown).

[0189] In some embodiments, some or all components of the computing device 1300 may also be arranged in a cloud computing architecture rather than being integrated in a single device. In a cloud computing architecture, components may be provided remotely and work together to implement the functions described in the present disclosure. In some embodiments, cloud computing provides computing, software, data access and storage services, which will not require the end user to know the physical location or configuration of the system or hardware that provides these services. In various embodiments, cloud computing provides services via a wide area network (e.g., the Internet) using a suitable protocol. For example, a cloud computing provider provides applications via a wide area network, which can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data may be stored on a remote server. The computing resources in a cloud computing environment may be merged or distributed at the location of a remote data center. Cloud computing infrastructures may provide services through a shared data center, although they appear as a single access point for users. Therefore, cloud computing architectures may be used to provide components and functions described herein from a service provider at a remote location. Alternatively, they may be provided by a conventional server, or installed directly or otherwise on a client device.

[0190] In an embodiment of the present disclosure, the computing device 1300 may be used to implement data encoding / decoding. The memory 1320 may include one or more data encoding / decoding modules 1325 having one or more program instructions. These modules can be accessed and executed by the processing unit 1310 to perform the functions of the various embodiments described herein.

[0191] In an example embodiment where data encoding is performed, input device 1350 may receive data as input 1370 to be encoded. The data may be processed by, for example, data encoding / decoding module 1325 to generate an encoded bitstream. The encoded bitstream may be provided as output 1380 via output device 1360.

[0192] In an example embodiment where data decoding is performed, input device 1350 may receive an encoded bitstream as input 1370. The encoded bitstream may be processed by, for example, data encoding and decoding module 1325 to generate decoded data. The decoded data may be provided as output 1380 via output device 1360.

[0193] Although the present disclosure has been specifically shown and described with reference to the preferred embodiments of the present disclosure, it will be understood by those skilled in the art that various changes can be made in form and detail without departing from the spirit and scope of the present application as defined by the appended claims. These changes are intended to be encompassed by the scope of the present application. Therefore, the foregoing description of the embodiments of the present application is not intended to be limiting.

Claims

1. A data processing method, comprising: During conversion between data and a bitstream of the data, determining a probability distribution for an entropy codec associated with the bitstream using a first model with an attention mechanism; as well as The converting is performed based on the probability distribution. 2 . The method of claim 1 , wherein the first model comprises a transformer model or a transformer context model.

3. The method according to any one of claims 1-2, wherein the probability distribution is determined by using the first model in combination with an autoregressive model and a hyper-prior model.

4. The method of claim 3, wherein determining the probability distribution comprises: generating intermediate information based on a quantized latent representation of the data by using the first model; as well as The probability distribution is generated based on the intermediate information, an output of the autoregressive model, and an output of the hyper-prior model. The method of claim 4 , wherein the probability distribution is generated by using another model different from the first model.

6. The method of claim 3, wherein determining the probability distribution comprises: generating intermediate information based on the quantized latent representation of the data and a first output of the autoregressive model by using the first model; as well as The probability distribution is generated based on the intermediate information and a second output of the super-prior model. The method of claim 6 , wherein the probability distribution is generated by another model based on the intermediate information and the second output.

8. The method of claim 6, wherein the probability distribution is generated by directly combining the intermediate information and the second output.

9. The method of claim 3, wherein determining the probability distribution comprises: generating intermediate information based on the quantized latent representation of the data and a second output of the hyper-prior model by using the first model; as well as The probability distribution is generated based on the intermediate information and a first output of the autoregressive model.

10. The method of claim 9, wherein the probability distribution is generated by another model based on the intermediate information and the first output. The method of claim 10 , wherein the probability distribution is generated by directly combining the intermediate information and the first output.

12. The method of claim 3, wherein determining the probability distribution comprises: generating intermediate information based on a quantized latent representation of the data and an output of another model by using the first model; as well as The probability distribution is generated based on the intermediate information, a first output of the autoregressive model, and a second output of the hyper-prior model.

13. The method of claim 12, wherein the input to the other model comprises at least one of: said data, an implicit representation of the data, or A quantitative implicit representation of the data.

14. The method according to any one of claims 1 to 13, wherein information about whether the probability distribution is determined by using the first model in combination with an autoregressive model and a hyper-prior model is indicated in the bitstream.

15. The method according to any one of claims 1-13, wherein the information about whether the probability distribution is determined by using the first model in combination with an autoregressive model and a hyper-prior model is determined by a decoder.

16. The method according to any one of claims 1-2, wherein the probability distribution is determined by using the first model in combination with a hyper-prior model.

17. The method of claim 16, wherein the probability distribution is determined by using the first model based on a quantized latent representation of the data and an output of the hyper-prior model.

18. The method of claim 16, wherein an input to the first model comprises a quantized latent representation of the data.

19. The method of claim 18, wherein the probability distribution is determined based on an output of the first model and an output of the super-prior model by using another model.

20. The method of claim 18, wherein the probability distribution is determined as an output of the first model.

21. The method of any one of claims 1-2, wherein the probability distribution is determined by using the first model in combination with an autoregressive model.

22. The method of claim 21, wherein the probability distribution is determined by using the first model based on a quantized latent representation of the data and an output of the autoregressive model.

23. The method of claim 21, wherein an input to the first model comprises a quantized latent representation of the data.

24. The method of claim 23, wherein the probability distribution is determined based on an output of the first model and an output of the autoregressive model using another model.

25. The method of claim 23, wherein the probability distribution is determined as an output of the first model.

26. A method according to any one of claims 1-2, wherein the input to the first model includes a quantized latent representation of the data.

27. The method of claim 26, wherein the input further comprises an output of another model.

28. The method of claim 27, wherein the input to the another model comprises at least one of: the data, or A quantitative implicit representation of the data.

29. The method according to any one of claims 1-2, wherein information about whether the first model is used to replace an autoregressive model is indicated in the bitstream, Information as to whether the first model is used to replace the super-prior model is indicated in the bitstream, or Information about whether the first model is used to replace the autoregressive model and the hyper-prior model is indicated in the bitstream.

30. The method according to any one of claims 1-2, wherein the information about whether the first model is used to replace the autoregressive model is determined by a decoder, The information about whether the first model is used to replace the super-prior model is determined by the decoder, or Information about whether the first model is used to replace the autoregressive model and the hyper-prior model is determined by a decoder.

31. The method of any one of claims 1-2, wherein the probability distribution is determined by using a plurality of the first models.

32. The method of claim 31 , determining the probability distribution comprises: Determining a first candidate probability distribution for the entropy coding by using a first model among the plurality of the first models; Determining a second candidate probability distribution for the entropy coding by using another first model among the plurality of first models; as well as The probability distribution is generated based on the first candidate probability distribution and the second candidate probability distribution.

33. A method according to claim 32, wherein the probability distribution is generated based on a weighted sum of the first candidate probability distribution and the second candidate probability distribution, and the weights for the first candidate probability distribution and the second candidate probability distribution are determined based on learnable parameters or latent information.

34. The method of claim 32, wherein the probability distribution is generated by at least one additional model.

35. A method according to claim 31, wherein one of the plurality of first models is used for a first element at a first spatial position of a quantized latent representation of the data, and another first model of the plurality of first models is used for a second element at a second spatial position of the quantized latent representation, the second element spatial position being different from the first element spatial position.

36. The method of claim 31 , wherein information about the first model to be used is indicated in the bitstream.

37. The method of claim 36, wherein the information is determined based on a rate-distortion (RD) loss estimate.

38. The method of claim 36, wherein the information is determined based on latent information.

39. The method of claim 31 , wherein information about the first model to be used is determined by a decoder.

40. The method of claim 39, wherein the information is determined based on super-prior information.

41. The method of claim 39, wherein the information is determined by an autoregressive model.

42. The method of claim 39, wherein the information is determined by a combination of hyper-prior information and an autoregressive model.

43. The method of any one of claims 1-2, wherein determining the probability distribution comprises: determining a context parameter based on a quantized latent representation of the data using the first model; as well as Based on the context parameters, the probability distribution is generated.

44. The method of claim 43, wherein determining the context parameter comprises: Performing attention calculation based on the quantized latent representation; as well as The context parameter is generated based on a result of the attention calculation by using at least one sub-network.

45. The method of claim 44, wherein performing the attention calculation comprises: generating a relationship between a coded element and a current element of a quantized implicit representation of the data; as well as The result is generated based on the relationship.

46. ​​The method of claim 45, wherein the relationship is determined in an autoregressive manner.

47. A method according to any one of claims 44-46, wherein the query, key and value used to perform the attention calculation are determined based on the quantized latent representation.

48. The method of claim 47, wherein the query is determined based on the quantized latent representation using a first subnetwork, The key is determined based on the quantized latent representation using a second sub-network, and The value is determined based on the quantized latent representation by using a third sub-network, the first sub-network, the second sub-network, and the third sub-network being different from each other.

49. The method of claim 47, wherein each of the query, the key, and the value is the quantized latent representation.

50. The method of any one of claims 45-49, wherein the relationships are represented by a relationship matrix and a mask is applied to the relationship matrix for generating the result.

51. The method of claim 50, wherein a strictly lower triangular matrix is ​​used to generate the result.

52. The method of claim 51, wherein the mask is the strictly lower triangular matrix.

53. The method of claim 51, wherein the mask is a subset of the strictly lower triangular matrix.

54. The method of claim 50, wherein a lower triangular matrix is ​​used to generate the result and the elements on the diagonal of the relationship matrix are indicated in the bitstream.

55. The method of claim 54, wherein the mask is the lower triangular matrix.

56. The method of claim 54, wherein the mask is a subset of the lower triangular matrix.

57. The method of claim 50, wherein a lower triangular matrix is ​​used to generate the result and the elements on the diagonal of the relationship matrix are set to the sum of the queries over the channel dimension.

58. The method of claim 57, wherein the mask is the lower triangular matrix.

59. The method of claim 57, wherein the mask is a subset of the lower triangular matrix.

60. The method of claim 50, wherein a lower triangular matrix is ​​used to generate the result and the elements on the diagonal of the relationship matrix are determined based on additional information.

61. The method of claim 60, wherein the mask is the lower triangular matrix.

62. The method of claim 60, wherein the mask is a subset of the lower triangular matrix.

63. A method according to any one of claims 60-62, wherein the additional information includes at least one of an output of an autoregressive model or an output of a hyper-prior model.

64. The method of claim 50, wherein the entire relationship matrix is ​​used to generate the result and elements of an upper triangular portion of the relationship matrix are determined based on additional information.

65. The method of claim 64, wherein the additional information comprises at least one of an output of an autoregressive model or an output of a hyper-prior model.

66. A method according to any one of claims 44-65, wherein multiple attention modules are used to perform the attention calculation.

67. A method according to claim 66, wherein the output of an attention module among the plurality of attention modules is determined as the result of the attention calculation.

68. A method according to claim 66, wherein the result of the attention calculation is generated by performing aggregation processing on the outputs of the multiple attention modules.

69. The method of claim 68, wherein the polymerization process comprises stitching in the channel domain or the spatial domain.

70. The method of claim 68, wherein the aggregation process comprises a linear operation.

71. The method of claim 70, wherein the linear operation comprises addition or subtraction.

72. A method according to any one of claims 45-46, wherein the keys and values ​​used to perform the attention calculation are determined based on the quantized implicit representation, and the query used to perform the attention calculation is determined based on additional information.

73. The method of claim 72, wherein the additional information is determined based on at least one of an output of a hyper-prior model or an output of an autoregressive model.

74. The method of any one of claims 72-73, wherein the query is determined based on the additional information by using the first sub-network, The key is determined based on the quantized latent representation using a second sub-network, and The value is determined based on the quantized latent representation using a third sub-network.

75. The method of any one of claims 72-73, wherein the query is the additional information, and each of the key and the value is the quantized latent representation.

76. A method according to any one of claims 72-75, wherein the relationship is represented by a relationship matrix and a mask is applied to the relationship matrix to generate the result.

77. The method of claim 76, wherein a strictly triangular matrix is ​​used to generate the result.

78. The method of claim 77, wherein the strictly triangular matrix is ​​a strictly lower triangular matrix and the mask is the strictly lower triangular matrix.

79. The method of claim 77, wherein the strictly triangular matrix is ​​a strictly lower triangular matrix and the mask is a subset of the strictly lower triangular matrix.

80. The method of claim 76, wherein a triangular matrix is ​​used to generate the results and the elements on the diagonal of the relationship matrix are determined based on the autocorrelation of the query.

81. The method of claim 80, wherein the triangular matrix is ​​a lower triangular matrix and the mask is the lower triangular matrix.

82. The method of claim 80, wherein the triangular matrix is ​​a lower triangular matrix and the mask is a subset of the lower triangular matrix.

83. The method of claim 76, wherein the entire relationship matrix is ​​used to generate the result, and elements of an upper triangular portion of the relationship matrix are determined based on using additional information as the query and the key.

84. The method of claim 76, wherein a subset of the relationship matrix is ​​used to generate the result, and elements of an upper triangular portion of the relationship matrix are determined based on using additional information as the query and the key.

85. A method according to any one of claims 72-84, wherein multiple attention modules are used to perform the attention calculation.

86. A method according to claim 85, wherein the output of an attention module among the multiple attention modules is determined as the result of the attention calculation.

87. A method according to claim 85, wherein the result of the attention calculation is generated by performing aggregation processing on the outputs of the multiple attention modules.

88. The method of claim 87, wherein the polymerization process comprises stitching in the channel domain or the spatial domain.

89. The method of claim 87, wherein the aggregation process comprises a linear operation.

90. The method of claim 89, wherein the linear operation comprises addition or subtraction.

91. A method according to any one of claims 1-90, wherein the conversion is performed by using a second model having the attention mechanism.

92. The method of claim 91, wherein the second model comprises a transformer model.

93. The method of claim 91, wherein a prediction associated with the data used to perform the transforming is determined using the second model.

94. The method of claim 93, wherein the prediction comprises at least one of intra-frame prediction or inter-frame prediction.

95. The method of any one of claims 91-94, wherein motion information associated with the data used to perform the conversion is determined by using the second model.

96. The method of claim 95, wherein the motion information comprises at least one of motion vectors, optical flow, or latent motion information.

97. A method according to any one of claims 91-96, wherein the data includes a plurality of frames and the input to the second model includes encoded frames of the plurality of frames.

98. The method of any one of claims 1-97, wherein the data comprises at least one of: image, A picture of the video, or video.

99. The method of any one of claims 1-98, wherein the converting comprises encoding the data into the bitstream.

100. The method of any one of claims 1-98. wherein said converting comprises decoding said data from said bitstream.

101. An apparatus for data processing, comprising a processor and a non-volatile memory having instructions thereon, the instructions, when executed by the processor, causing the processor to perform a method according to any one of claims 1-100.

102. A non-transitory computer-readable storage medium storing instructions, wherein the instructions cause a processor to execute the method according to any one of claims 1-100.

103. A non-transitory computer-readable recording medium storing a bit stream of data, the bit stream being generated by a method performed by an apparatus for data processing, wherein the method comprises: Determining a probability distribution for entropy coding associated with the bitstream using a first model with an attention mechanism; as well as The bitstream is generated based on the probability distribution.

104. A method for storing a bit stream of data, comprising: Determining a probability distribution of entropy encoding and decoding associated with the bitstream using a first model with an attention mechanism; generating the bitstream based on the probability distribution; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

Citation Information

Cited By

  • Learning image compression method based on side information autoregression

    CN119603465A

  • A learning image compression method based on side information autoregression

    CN119603465B