Method and system for video compression tranformer

Abstract

Apparatus and methods involve encoding, by an encoder of a transmission computing device, a plurality of consecutive input video frames into corresponding quantized representation sequences; predicting, by a Transformer of the transmitting computing device, a probability quality function (PMF) as a conditional distribution of a given quantized representation in the sequence of quantized representations, wherein the conditional distribution is based on at least one dependency between one or more quantized representations in the sequence of quantized representations that occur prior to the given quantized representation; generating, by the transmission computing device, a plurality of compressed video frames by applying entropy coding to each quantized representation based on the predicted PMF, wherein entropy coding includes assigning a smaller number of bits to values having a higher frequency of occurrence; and transmitting, by the transmission computing device, the plurality of compressed video frames.

Description

[0001] Cross-references to related publications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 365,882, filed on June 6, 2022, which is incorporated herein by reference in its entirety. Background Art

[0003] Video compression methods often rely on an ever-increasing number of architectural biases and priors, including motion prediction and warping operations, resulting in complex models. Neural network-based video compression techniques have recently emerged that rival non-neural network techniques in rate-distortion performance. Existing models exploit complex connections between various subcomponents. The resulting methods are complex, challenging to implement, and inherently restrict themselves to working only on data that matches the architectural biases. Specifically, several techniques rely on some form of motion prediction followed by a warping operation. Summary of the invention

[0004] As described in this paper, flow prediction, warping, and residual compensation can be replaced with a Transformer-based temporal entropy model. Experiments show that the resulting Video Compression Transformer (VCT) can outperform the state-of-the-art for standard video compression datasets while being immune to architectural biases and priors.

[0005] The techniques described in this paper can be used as the basis for a new generation of video codecs. Such techniques can reduce the bandwidth required for video conferencing and video streaming and improve the utilization of storage space, thereby increasing the ability to preserve knowledge, thus having a net positive impact on society.

[0006] In one aspect, a computer-implemented method is provided. An encoder of a transmission computing device encodes a plurality of consecutive input video frames into a corresponding sequence of quantized representations. A Transformer of the transmission computing device predicts a probability mass function (PMF) as a conditional distribution of a given quantized representation, given at least one dependency between one or more quantized representations that appear before the given quantized representation in the sequence of quantized representations. The transmission computing device generates a plurality of compressed video frames by applying entropy coding to each quantized representation based on the predicted PMF, wherein the entropy coding includes assigning a smaller number of bits to values ​​with a higher frequency of occurrence. The transmission computing device transmits the plurality of compressed video frames.

[0007] In another aspect, a computing device is provided. The computing device includes one or more processors and a data storage device. The data storage device stores computer executable instructions, which, when executed by one or more processors, cause the computing device to perform functions. These functions include: encoding a plurality of consecutive input video frames into corresponding quantized representation sequences by an encoder of a transmission computing device; predicting a probability mass function (PMF) as a conditional distribution of a given quantized representation in the quantized representation sequence by a Transformer of the transmission computing device, wherein the conditional distribution is based on at least one dependency between one or more quantized representations that appear before the given quantized representation in the quantized representation sequence; generating a plurality of compressed video frames by the transmission computing device by applying entropy coding and decoding to each quantized representation based on the predicted PMF, wherein the entropy coding and decoding includes assigning a smaller number of bits to values ​​with a higher frequency of occurrence; and transmitting the plurality of compressed video frames by the transmission computing device.

[0008] In another aspect, a computer program is provided. The computer program includes instructions that cause the computer to perform functions when executed by a computer. These functions include: encoding a plurality of consecutive input video frames into corresponding quantized representation sequences by an encoder of a transmission computing device; predicting a probability mass function (PMF) as a conditional distribution of a given quantized representation in the quantized representation sequence by a Transformer of the transmission computing device, wherein the conditional distribution is based on at least one dependency between one or more quantized representations that appear before the given quantized representation in the quantized representation sequence; generating a plurality of compressed video frames by the transmission computing device by applying entropy coding and decoding to each quantized representation based on the predicted PMF, wherein the entropy coding and decoding includes assigning a smaller number of bits to values ​​with a higher frequency of occurrence; and transmitting the plurality of compressed video frames by the transmission computing device.

[0009] In another aspect, a product is provided. The product includes one or more computer-readable media on which computer-readable instructions are stored, and when executed by one or more processors of a computing device, the computer-readable instructions cause the computing device to perform functions. These functions include: encoding a plurality of consecutive input video frames into corresponding quantized representation sequences by an encoder of a transmission computing device; predicting a probability mass function (PMF) as a conditional distribution of a given quantized representation in the quantized representation sequence by a Transformer of the transmission computing device, wherein the conditional distribution is based on at least one dependency between one or more quantized representations that appear before the given quantized representation in the quantized representation sequence; generating a plurality of compressed video frames by the transmission computing device by applying entropy coding and decoding to each quantized representation based on the predicted PMF, wherein the entropy coding and decoding includes assigning a smaller number of bits to values ​​with a higher frequency of occurrence; and transmitting the plurality of compressed video frames by the transmission computing device.

[0010] In another aspect, a system is provided. The system includes: a device for encoding a plurality of consecutive input video frames into corresponding quantized representation sequences by an encoder of a transmission computing device; a device for predicting a probability mass function (PMF) as a conditional distribution of a given quantized representation in the quantized representation sequence by a Transformer of the transmission computing device, wherein the conditional distribution is based on at least one dependency between one or more quantized representations that appear before the given quantized representation in the quantized representation sequence; a device for generating a plurality of compressed video frames by applying entropy coding to each quantized representation based on the predicted PMF by the transmission computing device, wherein the entropy coding includes assigning a smaller number of bits to values ​​with a higher frequency of occurrence; and a device for transmitting the plurality of compressed video frames by the transmission computing device.

[0011] In another aspect, a decoding device is provided. The decoding device includes one or more processors and a data storage device. The data storage device stores computer executable instructions, which, when executed by one or more processors, cause the decoding device to perform functions. These functions include receiving a plurality of compressed video frames as corresponding quantized representation sequences by a decoder of the decoding device. These functions also include predicting a probability mass function (PMF) by a Transformer of the decoding device as a conditional distribution of a given quantized representation in the quantized representation sequence, wherein the conditional distribution is based on at least one dependency between one or more quantized representations that appear before the given quantized representation in the quantized representation sequence. These functions also include generating a plurality of decompressed video frames by the decoding device by applying entropy decoding to each quantized representation based on the predicted PMF, wherein entropy decoding includes inverting entropy coding, and entropy coding assigns a smaller number of bits to values ​​with a higher frequency of occurrence. These functions further include providing the plurality of decompressed video frames by the decoding device.

[0012] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a diagram illustrating an example Transformer architecture for video compression according to an example embodiment.

[0014] Figure 2 is a diagram illustrating an example transformation from a representation to a corresponding word-gram according to an example embodiment.

[0015] Figure 3 An example operation of a Transformer according to an example embodiment is shown.

[0016] Figure 4A Example rate distortion according to an example embodiment is shown.

[0017] Figure 4B Example rate distortion according to an example embodiment is shown.

[0018] Figure 4C Example rate distortion according to an example embodiment is shown.

[0019] Figure 4D Example rate distortion according to an example embodiment is shown.

[0020] Figure 5 An example comparison with other models according to example embodiments is shown.

[0021] Figure 6 An example visualization showing sample means from a block autoregressive distribution predicted by a Transformer according to an example embodiment.

[0022] Fig. 7A A table showing three phases of training according to an example embodiment.

[0023] Figure 7B A table showing experimental results for the number of previous frames to be input to a Transformer according to example embodiments.

[0024] Figure 7C A table showing experimental runtimes of components according to an example embodiment.

[0025] Figure 8 is a diagram illustrating the training phase and the inference phase of a machine learning model according to an example embodiment.

[0026] Fig. 9 A distributed computing architecture according to an example embodiment is depicted.

[0027] Fig.10 is a block diagram of a computing device according to an example embodiment.

[0028] Fig.11 Depicted is a network of computing clusters arranged as a cloud-based server system according to an example embodiment.

[0029] Fig.12 is a flowchart of a method according to an example embodiment. DETAILED DESCRIPTION

[0030] The present application relates to the use of neural networks (such as Transformers) for video compression (e.g., neural video compression). The prior art typically relies on a growing number of architectural biases and priors, including motion prediction and warping operations, resulting in complex models. As described herein, input frames can be independently mapped to representations, and Transformers can be used to model dependencies between such representations. Specifically, Transformers can be trained to predict the distribution of future representations based on past representations. It can be shown that the resulting video compression Transformer outperforms existing methods based on standard video compression datasets. Experiments on synthetic data show that Transformer-based models can learn to handle complex motion patterns such as translations, blurs, and fades purely from the data.

[0031] Overview

[0032] Neural network-based video compression techniques have recently emerged that rival non-neural techniques in rate-distortion performance. These methods tend to combine various architectural biases and priors inspired by classical non-neural methods. Like the “hand-crafted” classical codecs, neural methods have become increasingly “hand-crafted”, with complex connections between many subcomponents. The resulting methods can be complex, can be challenging to implement, and are often limited to working only on data that matches the architectural biases. Specifically, many methods rely on some form of motion prediction followed by a warping operation. These methods can use the predicted flow to warp a previous reconstruction and compute the residual.

[0033] As described in the paper, flow prediction, warping, and residual compensation can be replaced with a Transformer-based temporal entropy model. It can be shown that the resulting Video Compression Transformer (VCT) outperforms existing methods for a standard video compression dataset while being unaffected by its architectural biases and priors. Furthermore, as described in the paper, synthetic data can be created to explore the impact of architectural biases. Specifically, despite the fact that the Transformer does not have any of these components, the described techniques still perform well for operations such as translation or blurring on static frames. More importantly, for videos that do not have obvious matching architectural components (e.g., sharpening, fades between scenes), the described model outperforms existing models. This highlights the benefits of removing hand-crafted elements and letting the Transformer learn from the data.

[0034] In some embodiments, a Transformer may be used to compress video in two steps: first, a lossy transform codec may be used to convert frame x to i Mapping from image space to quantized representation y i . Then, you can i Recovery and reconstruction Second, the Transformer can be configured to exploit temporal redundancy to model the distribution of representations. This predicted distribution can then be used to entropy encode and decode the quantized y i Perform lossless compression. The better the Transformer predicts the distribution, the fewer bits are likely to be needed to store the representation.

[0035] This type of video compression method avoids complex state transitions or distortion operations by letting the Transformer learn to exploit arbitrary relationships between frames. In addition, for example, due to the reconstruction does not depend on the previous reconstruction, so the temporal error propagation can be reduced by the construction of Transformer. In the warp-based method, the reconstruction It is distorted Therefore, Visual errors in are usually propagated forward and require extra bits to correct them with the residual.

[0036] In some aspects, the model VCT described in this article can be viewed from the perspective of a language translation transformer. For example, two previous representations y i-2 ,y i-1 will be converted to y i However, there are various challenges in directly applying NLP formulas.

[0037] Consider an example 1080p video frame; using a typical neural image compression encoder downscaled by a factor of 16 and with 192 output channels, the (1080, 1920, 3)-dimensional input frame can be mapped to a (68, 120, 192)-dimensional feature representation, resulting in about 1.6 million symbols. Simply correlating all of these symbols with all the symbols in the previous representation will result in a 1.6M×1.6M-dimensional attention matrix. To address this computational impracticality, independence assumptions can be added to shrink the attention matrix and enable parallel execution on subsets of the symbols.

[0038] Figure 1 1 is a diagram illustrating an example Transformer architecture 100 for video compression according to an example embodiment. Some embodiments may involve encoding, by an encoder of a transmission computing device, a plurality of consecutive input video frames into a corresponding sequence of quantized representations. For example, the representation x (e.g., x j-2 , x j-1 , x j ) is independently and lossily mapped to an input frame 105 represented as y (e.g., y i-2 ,y i-1 , y) is a quantized representation 110. The representation can be recovered from the quantized representation 110 as Some embodiments may involve predicting, by a Transformer of a transmission computing device, a probability mass function (PMF) as a conditional distribution of a given quantized representation in a sequence of quantized representations, wherein the conditional distribution is based on at least one dependency between one or more quantized representations that occur before the given quantized representation in the sequence of quantized representations. For example, in order to store y with a small number of bits i , one or more Transformers 120 and one or more Mask Transformers 125 may be used to model the dependency and, given the previously transmitted representation, for y i, predicting a probability mass function (PMF) 130 (denoted as P). In some embodiments, the dependency may be a temporal dependency. Some embodiments may involve generating, by a transmission computing device, a plurality of compressed video frames by applying entropy coding to each quantized representation based on the predicted PMF, wherein the entropy coding includes assigning a smaller number of bits to values ​​with a higher frequency of occurrence. For example, P may be used to apply entropy coding to the quantized y i Generally speaking, better predictions of conditional probability P130 by Transformer 120, 125 can be translated into requiring fewer bits to store y i The model as described need not include hard-coded components for motion prediction and / or warping. These aspects are described in more detail.

[0039] Frame encoding and decoding

[0040] In some embodiments, the video encoding and decoding process may include two parts: first, a neural network encoder E (eg, an image encoder based on a convolutional neural network (CNN)) may be used to convert each frame x into i Independently encoded into quantized representation In some embodiments, encoding of each frame may include quantizing the quantized representation to an integer grid. In some embodiments, the encoder may perform spatial downscaling and increase the channel dimension. For example, encoder E may be configured to downscale spatially and increase the channel dimension. In some embodiments, this may result in (H, W, d c )-dimensional feature map y i , where the parameters H and W can be 1 / 16 of the input image resolution. The decoder D can be used to get y i Recovery and reconstruction Some embodiments may involve applying neural image compression to train one or more of the encoder or decoder to a corresponding lossy transform, where the target distortion variable is based on the range of each quantized representation. For example, the encoder E and / or the decoder D may be trained using neural image compression techniques. For example, by changing y i The size of each element in the range can train E and D to achieve almost any desired distortion. For the purpose of illustration and to maintain a clearer exposition, it can be assumed that the encoder-decoder pair E, D achieves a fixed distortion.

[0041] Generally speaking, in the input frame sequence x i Lossy conversion to represent sequence Afterwards, y i Losslessly store to disk. However, in some embodiments, such an approach may be suboptimal. For example, let y i Each element yi，j express Assume that all Symbols appear with equal probability, that is, You can use Ones bit to transmit y i Using the parameter L = 32, this would mean that 9 Mbytes or ≈ 2 Gbps at 30 fps might be required to encode a single HD frame (where H·W·d c ≈1.6M). While this is an effective compression scheme that can produce the desired distortion (e.g., fixed distortion as previously described), it can be inefficient. Therefore, there is a need to improve such compression techniques, as described herein.

[0042] Example Codec Scheme

[0043] In some embodiments, given an estimate y i The probability mass function (PMF) P of the distribution Q of the symbols in can be expressed using entropy coding (EC) The number of bits transmitted given by y i By using EC, more frequently occurring values ​​can be encoded using fewer bits, thereby improving compression efficiency. In some embodiments, the average number of bits corresponds to the cross entropy of the conditional distribution relative to the predicted PMF. In general, the expected term (in the above expression) can represent the average bit count corresponding to the cross entropy of Q with respect to P. Therefore, P can be estimated as a conditional distribution using the Transformer model. In some embodiments, the prediction of PMF can involve maintaining the codec efficiency of the entropy codec by adjusting the cross entropy. For example, the cross entropy can be minimized to maximize the codec efficiency. Such details are now provided.

[0044] Temporal entropy model based on Transformer

[0045] To transmit F frames x1,...,x F For a video, the encoder E can map each frame to produce a quantized representation y1, ..., y F In y i , ..., y i-1 If it has been transmitted, in order to transmit y i , the Transformer can be configured to predict P(y i |y i-2 ,y i-1 ). Using this distribution, we can i Entropy encoding is performed to create a compressed binary representation that can then be transmitted.

[0046] In order to compress the video, we use Transformer to predict P(y j |y j-2 ,y j-1 ) (for j∈{1, ..., F}) and fill with zeros when predicting the distribution for y1, y2, the above procedure can be applied iteratively. The receiver or receiving device or decoding device can follow the same procedure to recover y j Some embodiments may involve receiving, by a decoder of a decoding device, a plurality of compressed video frames as a corresponding sequence of quantized representations. Figure 1 , represented by y (for example, y i-2 ,y i-1 , y) may be received by a decoder D. Some embodiments may involve predicting, by a Transformer of a decoding device, a probability mass function (PMF) as a conditional distribution of a given quantized representation in a sequence of quantized representations, wherein the conditional distribution is based on at least one dependency between one or more quantized representations that occur before the given quantized representation in the sequence of quantized representations. For example, the decoder iteratively determines P(y j |y j-2 ,y j-1 ). Some embodiments may involve generating, by a decoding device, a plurality of decompressed video frames by applying entropy decoding to each quantized representation based on the predicted PMF, wherein the entropy decoding comprises inverting the entropy encoding, and the entropy encoding assigns a smaller number of bits to values ​​with a higher frequency of occurrence. For example, the decoder iteratively uses P(y j |y j-2 ,y j-1 ) for each y j Perform entropy decoding. After determining each representation y1, y2, ..., y F Thereafter, the receiver may be configured to generate a reconstruction and provide a plurality of decompressed video frames.

[0047] Word

[0048] Figure 2 is a diagram illustrating an example transformation 200 from a representation 205 to a corresponding word-gram 215 according to an example embodiment. Some embodiments involve spatially partitioning a given quantized representation into non-overlapping blocks of size N×N. For example, when processing the current representation y i When , it can be divided into two parts of size w in space. C × C Non-overlapping blocks 210, such as Figure 2 As shown. One or more quantized representations that appear before a given quantized representation can be configured as overlapping blocks of size M×M, where M>N. For example, the previous representation y i-2 ,y i-1(as indicated by transmission 220) becomes the corresponding overlapping w P × P Block (where w P >w C ) to provide the information for predicting P(y i |y i-2 ,y i-1 ) and spatial context. In general, a larger spatial range can provide useful context to predict the distribution of the current block. In some embodiments, due to the downscaling performed by the convolutional encoder E, the block 210 may span a relatively large spatial region in the image space. In some embodiments, each block can be spatially flattened to generate one or more tokens for the Transformer. In addition, as shown in the figure, each block 210 can be spatially flattened (see Figure 2 ) to obtain word unit 215 for Transformer.

[0049] As shown in the figure, a given representation 205 (denoted as y) can be split into non-overlapping w using a sliding window C × C Block 210. In addition, for example, the previous representation (represented by the transmitted 220 and represented by y i-1 ,y i-2 ) can be split into overlapping w P × P Block 210, stride w C (w P >w C Block 210 can be spatially flattened (e.g., raster scan order, see left arrow) to obtain tokens 215 for the Transformer, which are still d C dimensional, because they are y i Another view of. Figure 2 It shows that w C =3, w P =5, d C =5, but other values ​​may also be used, such as, for example, w C =4, w P =8, d C = 192. In some embodiments, the Transformer can be configured to run independently on the corresponding block 210 and / or word unit 215, that is, Figure 2 Tokens of the same hue in can be processed together, at the expense of less spatial context for parallel execution. A side benefit is that the number of tokens used for the Transformer does not have to be a function of image resolution.

[0050] In general, the independence assumption enables the process to focus on a single group of blocks 210. As shown, given i-2 、b i-1 (in transmitted block 220) words, can be in block b i (in the block to be transmitted 225) words t1, t2, ..., t 16 distribution.

[0051] Figure 3 2 shows an example operation of the Transformer 300 according to an example embodiment. The Transformer 300 may be configured to operate on a set of chunks and / or tokens, such as represented by b i-1 、b i-2 The transmitted block 310 and denoted as b i The block to be transmitted (or given block) 315 (such as Figure 2 The Transformer layer (e.g., Figure 1 120 ), and shows a mask Transformer layer (e.g., Figure 1 A second magnified view 320 of the mask Transformer 125 of FIG. 310 can be extracted from the transmitted block 310, denoted as z joint As shown in the figure, the mask Transformer 315A T cur Prediction probability mass function (PMF) 315B, P(t3|t S , t1, t2, z joint ), where t S is the learned starting token. Therefore, the prediction of PMF can be based on the spatial context and temporal context derived from overlapping blocks. These steps are described in more detail below.

[0052] Step 1: Time Mixer

[0053] Some embodiments involve extracting, by a first Transformer, temporal information corresponding to one or more quantized representations that occurred before a given quantized representation from each of the overlapping blocks separately. For example, two Transformers may be used to extract temporal information corresponding to one or more quantized representations that occurred before a given quantized representation. i-2 、b i-1 Extract time information from the first Transformer 310A T sepEach previous block (transmitted 310) may be operated on separately. The outputs in the word-unit dimension may then be concatenated. Some embodiments involve temporal information extracted by the second Transformer blend. For example, a second Transformer 310 may be applied to the result joint To mix information across time. The output is time information 310C z joint , the time information may include features that indicate the model’s past knowledge.

[0054] Step 2: Intra-block autoregression

[0055] The second part of the method may involve the Mask Transformer 315A T cur , the masked Transformer can be configured to predict one or more PMFs 315B for each word using intra-block autoregression. Masked Transformer 315A T cur It can be based on time information 310Cz joint and the previously transmitted tokens within the block.

[0056] For entropy coding and decoding, both the sender device and the receiver device may be configured to obtain the same PMF 315B, ie, the mask Transformer 315A T cur can be configured to be causal and start from a known initialization point. Thus, the starting token t S can be used as a known initialization point for learning.

[0057] In some embodiments, in order to transmit a word, time information 310C may be determined. joint . Then, [t S ] Input to the mask Transformer 315A T cur , thus generating the first PMF P(t1|t S ;z joint ). Subsequently, entropy coding and decoding can be used to use the first PMFP (t1|t S ;z joint ) replace d in word t1 C symbols are stored in the bitstream. Then, you can enter [t S , t1], thereby generating the second PMFP (t2|t1, t S ;z joint ), t2 can be stored in the bit stream, and the process can be continued iteratively.

[0058] A receiver or decoding device may receive the resulting bit stream and may receive the same distribution and thus word-units by first converting [t S ] Input to the mask Transformer 315AT cur , determine the first PMFP (t1|t S ; z joint ), entropy decode t1 from the bitstream, and then input [t S , y1] to determine the second PMFP (t2|t1, t s ; z joint ), the process can be continued iteratively. Figure 3 This process is shown for P(t3|...).

[0059] In some embodiments, the process can be run on the blocks in parallel. Thus, by running times to transmit and / or receive y i Each run can usually generate In order to ensure that T cur The causal relationship between can mask the self-attention block.

[0060] Independence

[0061] In addition to assuming that y i In addition to the fact that the blocks in T are independent (because each word is a vector), we can also assume that the symbols within each word are conditionally independent given the previous word, that is, T cur It can predict the d required for a word C In some embodiments, all possible The joint distribution of the implementation can be used, and channel autoregression and / or vector quantization can be used for the word unit. In contrast to the autoregressive image compression entropy model, the Transformer-based VCT described in this paper does not rely on additional auxiliary information.

[0062] Example Architecture

[0063] Transformer

[0064] like Figure 3 As shown, Transformer can be based on various architectures. In some embodiments, a fully connected layer can be used and the learned position embedding can be added to d C dimensional word is projected to d T dimensional space (for example, in an example model d T =768). Although T sep 310A and T joint310B Both may include one or more stacks of multi-head self-attention (MHSA) layers, but T cur The 315A can be configured to use masked "conditional" Transformer layers. For example, these layers can be used in conjunction with masked MHSA layers using z joint As the key (K) and value (V) MHSA layer alternates between, Figure 3 In some embodiments, it can be T sep 310A uses six (6) Transformer layers, for T joint 310B uses four (4) Transformer layers and is T cur 315A uses five (5) masked Transformer layers. Additionally, for example, sixteen (16) attention heads may be used. Additionally, in some embodiments, separate temporal position embeddings may be learned to add to the T joint 310B input.

[0065] Image encoders and decoders

[0066] In some embodiments, the image encoder and / or decoder E, D may include one or more architectures based on standard image compression methods. For example, the image encoder and / or decoder E, D may be a CNN-based image encoder and / or decoder. For example, the encoder E may include four (4) strided convolution layers, with a total downscaling to a factor of 16. For the decoder D, transposed convolutions may be used, and low-resolution residual blocks may be optionally added. In some embodiments, d ED =192 filters.

[0067] One or more variations of the architecture may also be used. For example, the following architecture may be used for an image encoder and / or decoder E, D. Indicates that it has d ED = 192 filters and a 5×5 convolution with a stride of two (2), followed by a leaky ReLu activation (e.g., α = 0.2). In some embodiments, the encoder E may include four such convolutions Let T denote an arrangement with d ED A 5×5 transposed convolution with 2 filters and stride 2 is also followed by a leaky ReLu, and let R denote the residual block (i.e., R may include two convolutions with jump connections around it). The decoder D may include an arrangement such as RRRRTRRTRRTT. Thus, the increase in resolution may result in fewer residual blocks. In some examples, the shorthand 4220 may be used to count the residual blocks R between each transposed convolution T in the arrangement RRRRTRRTRRTT.

[0068] Example losses and training process

[0069] The above modeling choices can enable an efficient training procedure. In some embodiments, training can include three stages, which enables rapid experimentation.

[0070] In some embodiments, the training of the encoder E and / or decoder D can be based on a rate-distortion tradeoff loss. For example, in stage I, each frame encoder E and decoder D can be trained by minimizing the rate-distortion tradeoff. Let represents a uniform distribution in the interval [-0.5, 0.5]. The following loss function can be minimized:

[0071]

[0072] The expression represents the bit rate r, and the expression represents the distortion d, and where Generally speaking, refers to the unquantized representation, and the term x~p x represents a frame drawn from the training set. In general, Equation 1 expresses the minimization of the reconstruction error under the constraint that the encoder output can be efficiently quantized, where the parameter λ maintains the trade-off. For stage I, a mean-scaled hyper-prior method can be used to estimate p. In some embodiments, the hyper-prior uses a variational autoencoder (VAE) to estimate the PMF of y by predicting p(y|z), where z represents the originally transmitted side information. To enable end-to-end training, y can be added to the PMF of y when determining r. Add independent and identically distributed (iid) uniform noise u, and can be The rounding is done to feed it to the decoder D using direct estimation of gradients (STE).

[0073] For stage II, the Transformer can be trained to obtain p, and the bitrate can be minimized based on the following relationship:

[0074]

[0075]

[0076] in And among them ( represents a triplet of adjacent video frames. In some embodiments, it can be assumed that the d in each word C One or more of the unquantized elements follows a Gaussian distribution And the Transformer can be configured to predict d per token C The mean and d C Finally, the fine-tuning process can be jointly applied in stage III, adding the distortion loss d from Eq. 1 to Eq. 2.

[0077] In some embodiments, the model can be trained from scratch and some performance improvements can be achieved. For example, the training in Phase III can be performed directly from scratch using 1E -4 The learning rate (LR) is used for 750k steps. In cases where the main encoder E and decoder D exhibit unstable training when trained from scratch, a lightweight architecture such as Efficient Learning Image Compression (ELIC) (e.g., spatial channel context-adaptive coding and decoding with uneven grouping) can be utilized. In addition, for example, training a hyper-prior may not be required in this setting.

[0078] In some embodiments, the discrete PMFP of the quantized symbols may be determined by convolving p with a unit width box (e.g., for entropy coding) and evaluated at discrete points as follows:

[0079]

[0080] In some embodiments, the F , 256,256,3) pixels to perform training, where B represents the batch size and N F Indicates the number of frames. A linear decay LR can be used to schedule with warm-up, where the warm-up can be performed for 10k steps and a linear decay is applied from this LR to 1E -5 In some embodiments, the training in stage I may be based on a parameter λ=0.01. In addition, for example, to navigate the rate-distortion tradeoff and obtain results for multiple code rates, a parameter λ=0.01·2 may be used in stage III. i Nine (9) models are fine-tuned (i∈{-3, ..., 5}). In some embodiments, one or more of these models can be trained on four (4) Google Cloud TPUv4 chips.

[0081] Latent Residual Predictor (LRP)

[0082] In some embodiments, to further enhance the representations learned by the Transformer, image compression techniques such as latent residual predictors (LRPs) can be used. cur The output feature z cur Including with y i The same spatial dimensions, and at a given time, the output feature z cur It can represent the knowledge of the Transformer about the current representation and the previous representation related to a given time. Output feature z cur can be used to determine P, and these output features can be considered to constitute features that may help reconstruct Therefore, we can i =y i +f LRP (z cur ) is input to D to use z cur (This can be achieved in Phase III, for example), where f LRP Can include from d T to d ED A 1×1 convolutional map is formed, followed by a residual block. Therefore, may depend indirectly on y i-2 ,y i-1 ,y i Since this is a bounded window back into the past and y i 'Does not depend on Therefore, the process remains unaffected by the propagation of time errors, which is a significant technological advance.

[0083] experiment

[0084] Dataset

[0085] In some embodiments, training can be performed on one million Internet video clips, each of which may include nine (9) frames. For example, a high-resolution video can be downscaled by a random factor (removing previous compression artifacts), from which a center 256 cropped dataset can be obtained. In some embodiments, the training batches can include randomly selected triplets of adjacent frames. In addition, for example, evaluation can be performed on common benchmark datasets, such as (1) the MCL-JCV dataset, which includes thirty 1080p videos captured at 25FPS or 30FPS and an average of 137 frames per video, and (2) the UVG dataset, which includes twelve 1080p 120FPS videos, each of which has 300 or 600 frames.

[0086] Composite Video

[0087] In some embodiments, a synthetic dataset can be generated. For example, three parameterized synthetic datasets (such as Figure 5 As a specific example, the data set may include a variable parameter x, and for each value of x, multiple videos (e.g., 100 videos) may be generated. In addition, for example, each video may include 12 frames of 512×512 pixels (px). Several image operations may be used to generate these data sets, such as (i) a shift operation, in which a translation may be performed from the center of the image toward the lower right of the image, with each step of x pixels; (ii) a sharpen or blur operation, in which when x≥0, a Gaussian blur may be applied using σx·t at time step t, and when x<0, a video that becomes clearer over time may be generated by playing the video blurred with |x| in reverse; and (iii) a fade operation, in which a linear transition between two unrelated images (e.g., as in a scene cut) may be performed based on alpha blending.

[0088] Model

[0089] Several comparisons can be performed to evaluate the VCT model described in this article. For example, non-neural, standard codec High Efficiency Video Codec (HEVC) using FFmpeg ×265 codec can be run in "medium" and "very slow" settings, and H.264 video format using ×264 can be run in "medium" settings. For a fair comparison with the VCT model, B-frames can be disabled without otherwise constraining the codec. Public DVC code can be run, and additional figures can be obtained from the literature where available, introducing, for example, scale-space flow; architectural components that support warping and blurring; neural methods run on MCL-JCV (e.g., Efficient Learning Flexible Rate Video Codec (ELF-VC)), which use more motion priors to extend motion compensation of scale-space flow (SSF); models based on warping plus residual codecs in representation space (e.g., Deep Video Compression (FVC) and Deep Contextual Video Compression (DCVC) in feature space), using convolutional long short-term memory (ConvLSTM) as a sequence model (e.g., Recurrent Learning Video Compression (RLVC) using Recurrent Autoencoders and Recurrent Probabilistic Models); and / or additional models for studying lossless transmission of representations using CNNs for temporal entropy modeling. The performance of architectural deviations on synthetic data can be observed by reproducing SSF based on the same training data as VCT. In some embodiments, common peak signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM) can be evaluated with RGB. As used herein, the term "PSNR" generally refers to the ratio between the maximum possible value (power) of a signal and the power of the distortion noise that affects the quality of its representation. As used herein, the term "MS-SSIM" generally refers to SSIM performed at multiple scales (MS) through a multi-stage subsampling process. In general, values ​​closer to 1 indicate higher image quality, and values ​​closer to 0 indicate lower image quality. In addition, for example, the model can be trained using MSE as the distortion, and the expression 200·(1-MS-SSIM(x, x)) can be used as the training target in stage III (see Fig. 7A ) to obtain the MS-SSIM model.

[0090] result

[0091] Model comparison

[0092] 4A to 4D An example rate distortion according to an example embodiment is shown. Figure 4A Figure 405 and Figure 4C The rate-distortion comparison for the MCL-JCV dataset (≈27 FPS) is shown in Figure 415 of Figure 4B Figure 410 and Figure 4DA rate-distortion comparison for the UVG dataset (≈120 frames per second (FPS)) is shown in Figure 420 of . The MCL-JCV dataset may include twenty-four (24) source videos with a resolution of 1920×1080, and each source sequence has 51 H.264 / AVC encoded segments. The UVG dataset may include a set of 4K 120FPS test sequences that can be used for download in raw format and playback in the form of basic HEVC, Moving Picture Experts Group 4 (MP4), transport stream (TS), and dynamic adaptive streaming over HTTP (DASH) streams. As shown in the figure, the number of bits per pixel (bpp) and megabits per second (mbps) are reported. Figure 405 shows the PSNR of the MCL-JCV dataset, and Figure 410 shows the PSNR of the UVG dataset. Figure 415 shows the MS-SSIM of the MCL-JCV dataset, and Figure 420 shows the MS-SSIM of the UVG dataset. As indicated in the figure, despite its simplicity and not exploiting motion or warping components, VCT generally outperforms other methods in terms of PSNR and MS-SSIM.

[0093] Synthetic Data

[0094] Figure 5 An example comparison with other models according to an example embodiment is shown. The synthetic datasets introduced in this article can be used to demonstrate how Transformer learns to exploit various types of temporal patterns. In addition, for example, the expression-based To determine the HEVC Through relationships To determine the quality factor q that matches the parameter λ, so that for λ=0.01, q=25 is obtained. To understand the types of temporal patterns that the Transformer has learned to exploit, videos representing common patterns can be synthesized. Example comparisons with HEVC, which has built-in support for shifted motion, and SSF, which has built-in support for shifted motion and blur, are shown. The VCT model described in this article can learn analytical patterns purely from the data. For each dataset, different values ​​of the parameter x can be used (as described above under "Experiments"), and the points in the figure represent the average evaluation loss of 100 videos created with x.

[0095] The learning of the Transformer can be analyzed in different ways. For example, for videos with shift-based motion, VCT reduces the RD loss by ≈45% compared to SSF, with SSF saturating around x=10, which may be due to the shallow CNN used for flow estimation. Since HEVC supports motion compensation by arbitrary shifts of the previous frame, it seems to perform well for this type of video. For shifts that are multiples of 16, the representation shifts by one (1) symbol per step, and VCT matches HEVC. This is likely due to the fact that the encoder in VCT is CNN-based; therefore, it is shift-equivariant for shifts that are multiples of the stride (16). However, a shift of [1, 15] pixels may cause the representation to change in complex ways.

[0096] Additionally, for example, for blurring and / or sharpening, VCT generally outperforms both HEVC and SSF, even though the latter explicitly supports blurring. Note that the curve for SSF is asymmetric: since it has built-in support for blurring, blurring reduces the RD loss by ≈20% compared to sharpening.

[0097] As another example, VCT can also learn to manage fades, exhibiting a near-constant RD loss as the variable x increases, in contrast to baselines that do not have explicit support for fades. For example, SSF is ≈20% better than HEVC, likely due to its blurring capabilities. Overall, synthetic data seems to provide better insight into the advantages of VCT over other methods.

[0098] Figure 505 shows a shift, where the translation is done from the center of the image toward the lower right, with each step being x pixels. Figure 510 shows sharpening or blurring, where if x≥0, a Gaussian blur is applied using σx.t at time step t. If x<0, a video that becomes sharper over time can be created by playing the video blurred with |x| in reverse. Figure 515 shows a fade, where a linear transition between two unrelated images (such as in a scene cut) is performed using alpha blending. The green curve represents data for the VCT model described in this article.

[0099] Visualization of certainty during decoding

[0100] As described in the paper, after observing k tokens in each block, the Transformer can predict the PMF P(t k+1 |t≤k,z joint ). In general, this allows the Transformer to determine the joint distribution P(t ＞k|...). In general, as the Transformer gets more accurate at predicting the future, the distribution will become concentrated on the actual future tokens that the Transformer will decode.

[0101] Figure 6 An example visualization of sample means from a block autoregressive distribution predicted by a Transformer is shown in accordance with an example embodiment. Figure 6 In , the sample mean of the distribution can be visualized by taking the sample mean of the distribution input through the decoder D. For example, one can sample N realizations of unseen tokens in each block, conditioned on k decoded tokens, k∈{0,2,13}. Figure 6 The middle image 615 in shows what the Transformer expects at the current frame, before decoding any information (k=0, ie, 0 bits).

[0102] In some embodiments, the model appears to be able to implicitly learn second-order motion. For example, the kilobytes (kB) required to transmit a decoded (grey) word-gram is shown. Two previous reconstructions, labeled 605, are shown. and the one marked 610 The middle image at 615 shows what the Transformer expects at the current frame before any information is decoded (0 kB). The next two images at 620 and 625 show that as more tokens are decoded, the model's predictive power increases and the image obtained from the sample mean appears to become clearer. Note that for actual video codecs, sampling from the model does not have to be performed.

[0103] Fig. 7A Table 700A shows three stages of training according to an example embodiment. For example, to improve training efficiency (note the step / s column), training can be divided into three stages (e.g., stages I, II, and III as previously described). As previously described, the parameter λ manages the rate-distortion tradeoff, r represents the bit rate, d represents the distortion, B represents the batch size, and NF represents the number of frames.

[0104] Ablation

[0105] Figure 7BTable 700B shows experimental results for the number of previous frames input to the Transformer according to an example embodiment. Table 700B shows ablations based on how many previous frames are input to the Transformer ("context") and whether LRP is used. The effects of temporal context from previous frames and LRP can be observed on MCL-JCV. For example, the process can be started from a baseline that does not use any previous frames (i.e., an image model that encodes and decodes each frame independently). Conditioning on one previous frame reduces the bitrate by -58%. Using two previous frames gives an additional improvement of -6%. In one configuration of VCT with LRP added, an increase of 0.7dB in PSNR can be observed at the same bitrate. Additional previous frames may not generally further enhance learning, making the model rather simple.

[0106] Runtime

[0107] Figure 7C Table 700C shows experimental runtimes of components according to example embodiments. Table 700C shows example runtimes of VCT components. For example, the Transformer and decoder D can be run using a Google Cloud Tensor Processing Unit (TPU) v4. Entropy encoding and decoding (EC) can be run on a CPU. To obtain the Transformer (T sep , T joint , T cur ) and the runtime of the decoder D, one can deploy a Google Cloud TPU v4 (single core) using Flax, which has an efficient implementation for the autoregressive Transformer. Alternatively, for example, one can use Tensorflow Compression to measure the time it takes to do entropy encoding and decoding (EC) on an Intel Skylake CPU core. Figure 7C In Figure 1, data observed for 1280×720px, 852×480px, and 480×360px are shown. In some embodiments, the FPS estimate can be determined based on the expression 1000 / (sum of the individual runtimes in ms). In some embodiments, running T at 720p cur This may take ≈2.8ms; and this may take time to run times to decode the frame. In order to run T joint , T sep It may need to be run once per representation, since running T on the previous representation sep The output can be reused to run T joint .

[0108] Several applications are possible.In one aspect, a plurality of consecutive input video frames may be transformed into a plurality of compressed video frames.

[0109] In some example embodiments, a user may capture a video using a mobile device, and successive input video frames may be transformed into a plurality of compressed video frames.

[0110] In some example embodiments, the user can save the compressed video frame in a video library. In other example embodiments, the user can transfer the compressed video frame to another server or device.

[0111] In some aspects, a user may receive multiple compressed video frames at another device, decompress them, and view them using a video player.

[0112] These and other example applications are within the scope of the present disclosure.

[0113] Train machine learning models to generate inferences / predictions

[0114] Figure 8 A diagram 800 is shown illustrating a training phase 802 and an inference phase 804 of a trained machine learning model 832, according to an example embodiment. Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions about (patterns in) the training data. The resulting trained machine learning algorithms may be referred to as trained machine learning models. For example, Figure 8 A training phase 802 is shown, wherein one or more machine learning algorithms 820 are being trained on training data 810 to become a trained machine learning model 832. Then, during an inference phase 804, the trained machine learning model 832 may receive input data 830 and one or more inference / prediction requests 840 (perhaps as part of the input data 830), and responsively provide one or more inferences and / or predictions 850 as output.

[0115] Therefore, the trained machine learning model 832 may include one or more models of one or more machine learning algorithms 820. The machine learning algorithm 820 may include, but is not limited to: an artificial neural network (e.g., a convolutional neural network, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system as described herein). The machine learning algorithm 820 may be supervised or unsupervised, and may implement any suitable combination of online learning and offline learning.

[0116] In some examples, the machine learning algorithm 820 and / or the trained machine learning model 832 can be accelerated using an on-device coprocessor, such as a graphics processing unit (GPU), a tensor processing unit (TPU), a digital signal processor (DSP), and / or an application-specific integrated circuit (ASIC). Such an on-device coprocessor can be used to accelerate the machine learning algorithm 820 and / or the trained machine learning model 832. In some examples, the trained machine learning model 832 can be trained, resident, and executed on a particular computing device to provide inferences, and / or can otherwise make inferences for a particular computing device.

[0117] In some aspects, the training data 810 may include one million Internet video clips, each of which has nine frames. A high-resolution video may be obtained and downscaled by a random factor (removing previous compression artifacts). Thus, a center 256 crop may be generated. A training batch may consist of randomly selected triplets of adjacent frames. In some implementations, more than three adjacent frames may be used.

[0118] During the training phase 802, the machine learning algorithm 820 may be trained using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques by providing at least training data 810 as training input. Unsupervised learning involves providing a portion (or all) of the training data 810 to the machine learning algorithm 820, and the machine learning algorithm 820 determining one or more output inferences based on the provided portion (or all) of the training data 810. Supervised learning involves providing a portion of the training data 810 to the machine learning algorithm 820, wherein the machine learning algorithm 820 determines one or more output inferences based on the provided portion of the training data 810, and accepting or correcting the output inferences based on correct results associated with the training data 810. In some examples, supervised learning of the machine learning algorithm 820 may be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or the set of labels may be used to correct the inferences of the machine learning algorithm 820.

[0119] Semi-supervised learning involves having the correct result for a portion (but not all) of the training data 810. During semi-supervised learning, supervised learning is used for a portion of the training data 810 that has the correct result, and unsupervised learning is used for a portion of the training data 810 that does not have the correct result. Reinforcement learning involves the machine learning algorithm 820 receiving a reward signal about a previous reasoning, where the reward signal can be a numerical value. During reinforcement learning, the machine learning algorithm 820 can output an inference and receive a reward signal in response, where the machine learning algorithm 820 is configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing the expected sum of the numerical values ​​provided by the reward signal over time. In some examples, the machine learning algorithm 820 and / or the trained machine learning model 832 can be trained using other machine learning techniques, including but not limited to incremental learning and curriculum learning.

[0120] In some examples, the machine learning algorithm 820 and / or the trained machine learning model 832 may use transfer learning techniques. For example, the transfer learning technique may involve pre-training the trained machine learning model 832 on a set of data and additionally training it using the training data 810. More specifically, the machine learning algorithm 820 may be pre-trained on data from one or more computing devices, and the resulting trained machine learning model may be provided to the computing device CD1, where CD1 is intended to execute the trained machine learning model during the inference phase 804. Then, during the training phase 802, the pre-trained machine learning model may be additionally trained using the training data 810, where the training data 810 may be derived from the kernel data and non-kernel data of the computing device CD1. This further training of the machine learning algorithm 820 and / or the pre-trained machine learning model using the training data 810 of the CD1 data may be performed using supervised learning or unsupervised learning. Once the machine learning algorithm 820 and / or the pre-trained machine learning model has been trained at least on the training data 810, the training phase 802 may be completed. The resulting trained machine learning model can be used as at least one of the trained machine learning models 832.

[0121] Specifically, once the training phase 802 has been completed, the trained machine learning model 832 can be provided to the computing device (if not already on the computing device). After the trained machine learning model 832 is provided to the computing device CD1, the reasoning phase 804 can begin.

[0122] During the inference phase 804, the trained machine learning model 832 may receive input data 830 and generate and output one or more corresponding inferences and / or predictions 850 about the input data 830. Thus, the input data 830 may be used as an input to the trained machine learning model 832 for providing corresponding inferences and / or predictions 850 to kernel components and non-kernel components. For example, the trained machine learning model 832 may generate inferences and / or predictions 850 in response to one or more inference / prediction requests 840. In some examples, the trained machine learning model 832 may be executed by part of other software. For example, the trained machine learning model 832 may be executed by an inference or prediction daemon so that it is available at any time to provide inferences and / or predictions upon request. The input data 830 may include data from the computing device CD1 executing the trained machine learning model 832 and / or input data from one or more computing devices other than CD1.

[0123] Input data 830 may include a video set provided by one or more sources. The image set may include video frames, videos resident on computing device CD1, and / or other videos.

[0124] Inferences and / or predictions 850 may include output images, output intermediate images, numerical values, and / or other output data produced by a trained machine learning model 832 operating on input data 830 (and training data 810). In some examples, the trained machine learning model 832 may use the output inferences and / or predictions 850 as input feedback 860. The trained machine learning model 832 may also rely on past inferences as input to generate new inferences.

[0125] A neural network based on an encoder, transformer, or decoder may be an example of a machine learning algorithm 820. After training, the trained version of the neural network may be an example of a trained machine learning model 832. In this approach, an example of an inference / prediction request 840 may be a request to compress a video, and a corresponding example of an inference and / or prediction 850 may be an output compressed video.

[0126] In some examples, a computing device CD_SOLO may include a trained version of the neural network (possibly after training).The computing device CD_SOLO may then receive a request for a compressed video and generate the compressed video using the trained version of the neural network.

[0127] In some examples, two or more computing devices CD_CLI and CD_SRV may be used to provide compressed video; for example, a first computing device CD_CLI may generate and send a request for compressed video to a second computing device CD_SRV. CD_SRV may then generate compressed video using a trained version of a neural network and respond to CD_CLI's request for compressed video. CD_CLI may then provide the requested compressed video (e.g., using a user interface and / or display, a video player, etc.) upon receiving a response to the request.

[0128] Sample Data Network

[0129] Fig. 9 A distributed computing architecture 900 is depicted in accordance with an example embodiment. The distributed computing architecture 900 includes server devices 908, 910 configured to communicate with programmable devices 904a, 904b, 904c, 904d, 904e via a network 906. The network 906 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, an intranet, the public Internet, or any other type of network configured to provide a communication path between networked computing devices. The network 906 may also correspond to a combination of one or more LANs, WANs, intranets, and / or the public Internet.

[0130] although Fig. 9 Only five programmable devices are shown, but the distributed application architecture can serve tens, hundreds, or thousands of programmable devices. In addition, programmable devices 904a, 904b, 904c, 904d, 904e (or any additional programmable devices) can be any kind of computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head-mounted device (HMD), a network terminal, a mobile computing device, and the like. In some examples, such as shown in programmable devices 904a, 904b, 904c, 904e, the programmable device can be directly connected to the network 906. In other examples, such as shown in programmable device 904d, the programmable device can be indirectly connected to the network 906 via an associated computing device (such as programmable device 904c). In this example, programmable device 904c can act as an associated computing device for transmitting electronic communications between programmable device 904d and network 906. In other examples, such as shown in programmable device 904e, the computing device can be part of a vehicle (such as a car, truck, bus, boat or ship, airplane, etc.) and / or located inside the vehicle. In Fig. 9 In other examples not shown, the programmable device can be connected to the network 906 either directly or indirectly.

[0131] The server devices 908, 910 may be configured to perform one or more services as requested by the programmable devices 904a-904e. For example, the server devices 908 and / or 910 may provide content to the programmable devices 904a-904e. The content may include, but is not limited to, web pages, hypertext, scripts, binary data (such as compiled software), images, audio, and / or video. The content may include compressed and / or uncompressed content. The content may be encrypted and / or unencrypted. Other types of content are also possible.

[0132] As another example, server devices 908 and / or 910 may enable programmable devices 904a-904e to access software for database, search, computing, graphics, audio, video, web / internet utilization, and / or other functions. Many other examples of server devices are possible.

[0133] Computing device architecture

[0134] Fig.10 is a block diagram of an example computing device 1000 according to an example embodiment. Specifically, Fig.10 The computing device 1000 shown may be configured to perform at least one function of and / or related to the Transformer-based neural network 100 and / or method 1200. In some embodiments, the computing device 1000 may include an encoder and / or a decoder to perform encoding and / or decoding, as described herein.

[0135] The computing device 1000 may include a user interface module 1001, a network communication module 1002, one or more processors 1003, a data storage device 1004, one or more cameras 1018, one or more sensors 1020 and a power system 1022, all of which may be linked together via a system bus, network or other connection mechanism 1005.

[0136] The user interface module 1001 may be operable to send data to and / or receive data from an external user input / output device. For example, the user interface module 1001 may be configured to send data to and / or receive data from a user input device (such as a touch screen, a computer mouse, a keyboard, a keypad, a touch pad, a trackball, a joystick, a voice recognition module, and / or other similar devices). The user interface module 1001 may also be configured to provide output to a user display device, such as one or more cathode ray tubes (CRTs), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices, whether known now or developed later. The user interface module 1001 may also be configured to generate sound output using devices such as speakers, speaker jacks, audio output ports, audio output devices, headphones, and / or other similar devices. The user interface module 1001 may also be configured with one or more tactile devices, which may generate tactile outputs, such as vibrations and / or other outputs that may be detected by touch and / or physical contact with the computing device 1000. In some examples, user interface module 1001 may be used to provide a graphical user interface (GUI) for utilizing computing device 1000 , such as a graphical user interface of a mobile telephone device.

[0137] The network communication module 1002 may include one or more devices that provide one or more wireless interfaces 1007 and / or one or more wired interfaces 1008 that may be configured to communicate via a network. The wireless interface 1007 may include one or more wireless transmitters, receivers, and / or transceivers, such as Bluetooth. TM Transceiver, Transceiver, Wi-Fi TM Transceiver, WiMAX TM Transceiver, LTE TM The wired interface 1008 may include one or more wired transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or a similar transceiver that can be configured to communicate via a twisted pair, a coaxial cable, a fiber optic link, or a similar physical connection to a wired network.

[0138] In some examples, the network communication module 1002 can be configured to provide reliable, protected and / or authenticated communications. For each communication described herein, information for facilitating reliable communications (e.g., ensuring message delivery) can be provided, which may be part of a message header and / or message trailer (e.g., packet / message ordering information, encapsulation header and / or encapsulation trailer, size / time information, and transmission verification information (such as a cyclic redundancy check (CRC) and / or parity value)). One or more cryptographic protocols and / or algorithms can be used to secure (e.g., be encoded or encrypted) and / or decrypt / decode communications, such as, but not limited to, the Data Encryption Standard (DES), the Advanced Encryption Standard (AES), the Rivest-Shamir-Adelman (RSA) algorithm, the Diffie-Hellman algorithm, a secure socket protocol such as the Secure Sockets Layer (SSL) or the Transport Layer Security (TLS), and / or a Digital Signature Algorithm (DSA). Other cryptographic protocols and / or algorithms besides those listed herein may be used to protect (and then decrypt / decode) communications.

[0139] The one or more processors 1003 may include one or more general purpose processors and / or one or more special purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application specific integrated circuits, etc.). The one or more processors 1003 may be configured to execute computer readable instructions 1006 contained in the data storage device 1004 and / or other instructions as described herein.

[0140] The data storage device 1004 may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of the one or more processors 1003. The one or more computer-readable storage media may include volatile and / or non-volatile storage components that may be fully or partially integrated with at least one of the one or more processors 1003, such as optical, magnetic, organic or other memory or disk storage devices. In some examples, the data storage device 1004 may be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disk storage unit), while in other examples, the data storage device 1004 may be implemented using two or more physical devices.

[0141] Data storage 1004 may include computer-readable instructions 1006 and possibly additional data. In some examples, data storage 1004 may include storage required to perform at least a portion of the methods, scenarios, and techniques described herein and / or at least a portion of the functionality of the devices and networks described herein. In some examples, data storage 1004 may include storage for trained neural network models 1012 (e.g., models of trained neural networks such as encoders, transformers, decoders). Specifically, in these examples, computer-readable instructions 1006 may include instructions that, when executed by processor 1003, enable computing device 1000 to provide some or all of the functionality of trained neural network model 1012.

[0142] In some embodiments, the computing device 1000 may be a decoding device. The computer-readable instructions 1006 may include instructions that enable the computing device 1000 to perform functions when executed by the processor 1003. These functions may include receiving a plurality of compressed video frames as corresponding quantized representation sequences by a decoder of a decoding device. These functions may also include predicting a probability mass function (PMF) by a Transformer of a decoding device as a conditional distribution of a given quantized representation in a quantized representation sequence, wherein the conditional distribution is based on at least one dependency between one or more quantized representations that appear before a given quantized representation in a quantized representation sequence. These functions may also include generating a plurality of decompressed video frames by a decoding device by applying entropy decoding to each quantized representation based on the predicted PMF, wherein entropy decoding includes inverting entropy coding, and entropy coding assigns a smaller number of bits to values ​​with a higher frequency of occurrence. These functions may additionally include providing the plurality of decompressed video frames by a decoding device.

[0143] In some embodiments, the mean quantile corresponds to the cross entropy of the conditional distribution with respect to the predicted PMF.

[0144] Some embodiments are directed to maintaining decoding efficiency of entropy decoding by adjusting cross-entropy.

[0145] In some embodiments, the decoder may be a convolutional neural network (CNN) based image decoder.

[0146] Some embodiments involve applying neural image decompression to train a decoder as a lossy transform, where a target distortion variable is based on a range of each quantized representation.

[0147] In some embodiments, the training of the decoder may be based on a rate-distortion trade-off loss.

[0148] In some embodiments, at least one dependency may be a time dependency.

[0149] In some embodiments, the decoding device comprises a video player, and the functionality for providing the plurality of decompressed video frames involves outputting, by the video player, the plurality of decompressed video frames.

[0150] Some embodiments involve obtaining a trained version of a decoder and a transformer at a decoding device. Decoding may be performed by the trained version of the decoder. Prediction may be performed by the trained version of the transformer.

[0151] In some embodiments, the decoder may be trained at the decoding device.

[0152] Some embodiments involve storing a plurality of decompressed video frames at a decoding device.

[0153] In some embodiments, each frame of the plurality of compressed video frames may be independently decoded.

[0154] In some embodiments, the decoding device may be a mobile phone.

[0155] In some examples, computing device 1000 may include one or more cameras 1018. Camera 1018 may include one or more image capture devices, such as still cameras and / or video cameras, that are equipped to capture light and record the captured light in one or more images; that is, camera 1018 may generate images of the captured light. The one or more images may be one or more still images and / or one or more images used in video imagery. Camera 1018 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as light of one or more other frequencies. Some embodiments may involve capturing multiple input video frames using a camera. For example, a video camera may capture a video of a scene. Such embodiments may also involve receiving multiple input video frames from a camera by an encoder. The captured frames may be sent to an encoder for compression based on the methods described herein.

[0156] In some examples, computing device 1000 may include one or more sensors 1020. Sensors 1020 may be configured to measure conditions within computing device 1000 and / or conditions in the environment of computing device 1000 and provide data regarding these conditions. For example, sensors 1020 may include one or more of the following: (i) sensors for obtaining data about computing device 1000, such as, but not limited to, a thermometer for measuring the temperature of computing device 1000, a battery sensor for measuring the charge of one or more batteries of power system 1022, and / or other sensors measuring conditions of computing device 1000; (ii) identification sensors for identifying other objects and / or devices, such as, but not limited to, radio frequency identification (RFID) readers, proximity sensors, one-dimensional barcode readers, two-dimensional barcode (e.g., quick response (QR) code) readers, and laser trackers, where identification sensors may be configured to read identifiers (such as RFID tags, barcodes, QR codes) and / or other devices and / or objects that are configured to be read and provide at least identification information; (iii) sensors for measuring the location of computing device 1000; and (iv) sensors for measuring the location of computing device 1000. (iv) an environmental sensor for obtaining data indicative of the environment of the computing device 1000, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a motion sensor, a microphone, a sound sensor, an ultrasonic sensor, and / or a smoke sensor; and / or (v) a force sensor for measuring one or more forces (e.g., inertial forces and / or gravity) acting on the computing device 1000, such as, but not limited to, one or more sensors measuring force, torque, ground force, friction in one or more dimensions, and / or a ZMP sensor that identifies a zero moment point (ZMP) and / or the location of the ZMP. Many other examples of sensors 1020 are also possible.

[0157] The power system 1022 may include one or more batteries 1024 and / or one or more external power interfaces 1026 for providing power to the computing device 1000. Each of the one or more batteries 1024 may act as a storage power source for the computing device 1000 when electrically coupled to the computing device 1000. The one or more batteries 1024 of the power system 1022 may be configured to be portable. Some or all of the one or more batteries 1024 may be able to be easily removed from the computing device 1000. In other examples, some or all of the one or more batteries 1024 may be located inside the computing device 1000 and may therefore not be easily removed from the computing device 1000. Some or all of the one or more batteries 1024 may be rechargeable. For example, a rechargeable battery may be charged via a wired connection between the battery and another power supply device (such as by one or more power supply devices located outside the computing device 1000 and connected to the computing device 1000 via one or more external power interfaces). In other examples, some or all of the one or more batteries 1024 may be non-rechargeable batteries.

[0158] The one or more external power interfaces 1026 of the power system 1022 may include one or more wired power interfaces, such as USB cables and / or power cords, that implement a wired power connection to one or more power supply devices located outside the computing device 1000. The one or more external power interfaces 1026 may include one or more wireless power interfaces, such as Qi wireless chargers, that implement wireless power connections to one or more external power supply devices. Once a power connection to an external power source is established using the one or more external power interfaces 1026, the computing device 1000 can draw power from the external power source through the established power connection. In some examples, the power system 1022 may include associated sensors, such as battery sensors or other types of power sensors associated with one or more batteries.

[0159] Cloud-based servers

[0160] Fig.11 A network 906 of computing clusters 1109a, 1109b, 1109c arranged as a cloud-based server system is depicted according to an example embodiment. The computing clusters 1109a, 1109b, and 1109c can be cloud-based devices that store program logic and / or data for cloud-based applications and / or services; for example, performing at least one function of and / or related to a neural network, encoder, transformer, decoder, and / or method 2400.

[0161] In some embodiments, computing clusters 1109a, 1109b, and 1109c may be a single computing device residing in a single computing center. In other embodiments, computing clusters 1109a, 1109b, and 1109c may include multiple computing devices in a single computing center, or even multiple computing devices located in multiple computing centers in different geographical locations. For example, Fig.11 Each of the computing clusters 1109a, 1109b, and 1109c are depicted as residing in a different physical location.

[0162] In some embodiments, the data and services at the computing clusters 1109a, 1109b, 1109c may be encoded as computer-readable information stored in a non-transitory tangible computer-readable medium (or computer-readable storage medium) and accessible by other computing devices. In some embodiments, the computing clusters 1109a, 1109b, 1109c may be stored on a single disk drive or other tangible storage medium, or may be implemented on multiple disk drives or other tangible storage media located at one or more distinct geographic locations.

[0163] Fig.11 A cloud-based server system is depicted according to an example embodiment. Fig.11 , the functionality of encoders, transformers, decoders, and / or computing devices may be distributed among computing clusters 1109a, 1109b, and 1109c. Computing cluster 1109a may include one or more computing devices 1100a, cluster storage arrays 1110a, and cluster routers 1111a connected via a local cluster network 1112a. Similarly, computing cluster 1109b may include one or more computing devices 1100b, cluster storage arrays 1110b, and cluster routers 1111b connected via a local cluster network 1112b. Likewise, computing cluster 1109c may include one or more computing devices 1100c, cluster storage arrays 1110c, and cluster routers 1111c connected via a local cluster network 1112c.

[0164] In some embodiments, each of computing clusters 1109a, 1109b, and 1109c may have an equal number of computing devices, an equal number of cluster storage arrays, and an equal number of cluster routers. However, in other embodiments, each computing cluster may have a different number of computing devices, a different number of cluster storage arrays, and a different number of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster may depend on one or more computing tasks assigned to each computing cluster.

[0165] For example, in computing cluster 1109a, computing device 1100a may be configured to perform various computing tasks of an encoder, a transformer, a decoder, and / or a computing device. In one embodiment, various functionalities of a neural network and / or a computing device may be distributed between one or more of computing devices 1100a, 1100b, and 1100c. The computing devices 1100b and 1100c in respective computing clusters 1109b and 1109c may be configured similarly to computing device 1100a in computing cluster 1109a. On the other hand, in some embodiments, computing devices 1100a, 1100b, and 1100c may be configured to perform different functions.

[0166] In some embodiments, computing tasks and stored data associated with the neural network and / or computing devices may be distributed across computing devices 1100a, 1100b, and 1100c based at least in part on processing requirements of the neural network and / or computing devices, processing capabilities of computing devices 1100a, 1100b, 1100c, latency of network links between computing devices in each computing cluster and between the computing clusters themselves, and / or other factors that may impact cost, speed, fault tolerance, resilience, efficiency, and / or other design goals of the overall system architecture.

[0167] The cluster storage arrays 1110a, 1110b, 1110c of the computing clusters 1109a, 1109b, and 1109c may be data storage arrays that include disk array controllers configured to manage read and write access to groups of hard disk drives. The disk array controllers, alone or in conjunction with their respective computing devices, may also be configured to manage backup or redundant copies of data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.

[0168] Similar to the way that the functionality of the encoder, transformer, decoder, and / or computing device may be distributed across the computing devices 1100a, 1100b, 1100c of the computing clusters 1109a, 1109b, 1109c, the various active and / or backup portions of these components may be distributed across the cluster storage arrays 1110a, 1110b, 1110c. For example, some cluster storage arrays may be configured to store a portion of the data of a first layer of a neural network and / or computing device, while other cluster storage arrays may store other portions of the data of a second layer of the neural network and / or computing device. Additionally, for example, some cluster storage arrays may be configured to store data of an encoder of a neural network, while other cluster storage arrays may store data of a decoder of a neural network. Additionally, some cluster storage arrays may be configured to store backup versions of data stored in other cluster storage arrays.

[0169] The cluster routers 1111a, 1111b, 1111c in the computing clusters 1109a, 1109b, and 1109c may include networking devices configured to provide internal and external communications for the computing clusters. For example, the cluster router 1111a in the computing cluster 1109a may include one or more Internet switching and routing devices configured to (i) provide local area network communications between the computing device 1100a and the cluster storage array 1110a via a local cluster network 1112a, and (ii) provide wide area network communications between the computing cluster 1109a and the computing clusters 1109b and 1109c via a wide area network link 1113a to the network 906. Cluster routers 1111b and 1111c may include similar network devices as cluster router 1111a, and cluster routers 1111b and 1111c may perform similar networking functions for computing clusters 1109b and 1109b as cluster router 1111a performs for computing cluster 1109a.

[0170] In some embodiments, the configuration of the cluster routers 1111a, 1111b, 1111c may be based at least in part on the data communication requirements of the computing devices and the cluster storage arrays, the data communication capabilities of the network devices in the cluster routers 1111a, 1111b, 1111c, the latency and throughput of the local cluster network 1112a, 1112b, 1112c, the latency, throughput, and cost of the wide area network links 1113a, 1113b, 1113c, and / or other factors that may affect the cost, speed, fault tolerance, resilience, efficiency, and / or other design criteria of the adjusted system architecture.

[0171] Example How-to

[0172] Fig.12is a flow chart of a method 1200 according to an example embodiment. The method 1200 may be performed by a computing device, such as computing device 1000.

[0173] At block 1210 , the method involves encoding, by an encoder of a transmitting computing device, a plurality of consecutive input video frames into a corresponding sequence of quantized representations.

[0174] At box 1220, the method involves predicting, by a Transformer of a transmission computing device, a probability mass function (PMF) as a conditional distribution of a given quantized representation in a sequence of quantized representations, wherein the conditional distribution is based on at least one dependency between one or more quantized representations that appear before the given quantized representation in the sequence of quantized representations.

[0175] At block 1230, the method involves generating, by a transmission computing device, a plurality of compressed video frames by applying entropy coding to each quantized representation based on the predicted PMF, wherein the entropy coding includes assigning a smaller number of bits to values ​​having a higher frequency of occurrence.

[0176] At block 1240, the method involves transmitting, by a transmitting computing device, a plurality of compressed video frames.

[0177] Some embodiments involve receiving the plurality of compressed video frames by a receiving computing device. Such embodiments also involve generating, by a decoder of the receiving computing device, a plurality of decompressed video frames based on a probability mass function.

[0178] In some embodiments, the mean number of quantiles may correspond to the cross entropy of the conditional distribution with respect to the predicted PMF.

[0179] In some embodiments, the prediction of the PMF involves maintaining the codec efficiency of the entropy codec by adjusting the cross entropy.

[0180] In some embodiments, the encoder may perform spatial downscaling and increase the channel dimensionality.

[0181] In some embodiments, the encoder may be a convolutional neural network (CNN) based image encoder.

[0182] In some embodiments, the decoder may be a convolutional neural network (CNN) based image decoder.

[0183] In some embodiments, encoding of each frame involves quantizing the quantized representation to a grid of integers.

[0184] Some embodiments involve applying neural image compression to train one or more of an encoder or a decoder to a corresponding lossy transform, where a target distortion variable is based on a range of each quantized representation.

[0185] In some embodiments, training of one or more of the encoder or decoder may be based on a rate-distortion tradeoff loss.

[0186] In some embodiments, at least one dependency may be a time dependency.

[0187] Some embodiments involve spatially partitioning a given quantized representation into non-overlapping blocks of size N×N. One or more quantized representations that precede a given quantized representation may be configured as overlapping blocks of size M×M, where M>N. Each block may be spatially flattened to generate one or more tokens for a Transformer. The prediction of the PMF may be based on the spatial context and temporal context derived from the overlapping blocks.

[0188] In some embodiments, Transformer prediction of PMF involves extracting, by a first Transformer, temporal information corresponding to one or more quantized representations that occurred before a given quantized representation from each of the overlapping blocks separately. Such embodiments also involve mixing the extracted temporal information by a second Transformer.

[0189] In some embodiments, the transmission computing device may include a camera. Such embodiments involve capturing multiple input video frames using the camera. Such embodiments also involve receiving multiple input video frames from the camera by the encoder.

[0190] In some embodiments, the receiving computing device may include a video player. Such embodiments involve outputting, by the video player, a plurality of decompressed video frames.

[0191] Some embodiments involve obtaining a trained version of an encoder and a transformer at a transmission computing device. Encoding can be performed by a trained version of the encoder. Prediction can be performed by a trained version of the transformer.

[0192] Some embodiments involve obtaining a trained version of a decoder at a receiving computing device.The generation of the plurality of decompressed video frames may be performed by the trained version of the decoder.

[0193] In some embodiments, the encoder may be trained at the transmitting computing device.

[0194] In some embodiments, the decoder may be trained at the receiving computing device.

[0195] Some embodiments involve storing a plurality of compressed video frames at a transmitting computing device.

[0196] Some embodiments involve storing a plurality of decompressed video frames at a receiving computing device.

[0197] In some embodiments, the transmitting computing device may be the same as the receiving computing device.

[0198] In some embodiments, each frame of the plurality of input video frames may be independently encoded.

[0199] In some embodiments, one or more of the transmitting computing device or the receiving computing device may be a mobile phone.

[0200] The present disclosure is not limited in terms of the specific embodiments described in this application, which are intended to be illustrative of various aspects. As will be appreciated by those skilled in the art, many modifications and variations can be made without departing from the spirit and scope of the present disclosure. In addition to those methods and devices listed herein, those skilled in the art will also understand from the foregoing description that the functionally equivalent methods and devices within the scope of the present disclosure. Such modifications and variations are intended to fall within the scope of the appended claims.

[0201] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the accompanying drawings, similar symbols generally identify similar components unless the context otherwise dictates. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the various aspects of the present disclosure, as generally described herein and as shown in the accompanying drawings, may be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are expressly contemplated herein.

[0202] With respect to any one or all of the ladder diagrams, scenes and flow charts in the accompanying drawings and as discussed herein, each frame and / or communication can represent information processing and / or information transmission according to an example embodiment. Alternative embodiments are included in the scope of these example embodiments. In these alternative embodiments, for example, the functions described as frames, transmissions, communications, requests, responses and / or messages may not be performed in the order shown or discussed, including substantially simultaneously or in reverse order, depending on the functionality involved. In addition, more or less frames and / or functions can be used together with any one of the ladder diagrams, scenes and flow charts discussed herein, and these ladder diagrams, scenes and flow charts can be combined in part or in whole with each other.

[0203] The box representing information processing may correspond to a circuit that can be configured to perform a specific logical function of the method or technology described herein. Alternatively or additionally, the box representing information processing may correspond to a module, a fragment, or a portion of a program code (including related data). The program code may include one or more instructions that can be executed by a processor to implement a specific logical function or action in a method or technology. The program code and / or related data may be stored on any type of computer-readable medium (such as a storage device including a disk or hard drive or other storage medium).

[0204] Computer readable media may also include non-transitory computer readable media, such as non-transitory computer readable media that store data in a short period of time, such as register memory, processor cache, and random access memory (RAM). Computer readable media may also include non-transitory computer readable media that store program code and / or data in a longer period of time, such as auxiliary or persistent long-term storage devices, such as, for example, read-only memory (ROM), optical disks or disks, compact disc read-only memory (CD-ROM). Computer readable media may also be any other volatile or non-volatile storage system. Computer readable media may be considered to be, for example, a computer readable storage medium or a tangible storage device.

[0205] In addition, the boxes representing one or more information transfers may correspond to information transfers between software modules and / or hardware modules in the same physical device. However, other information transfers may be between software modules and / or hardware modules in different physical devices.

[0206] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are provided for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the appended claims.

Claims

1. A computer-implemented method comprising: The encoder of the transmission computing device encodes a plurality of consecutive input video frames into corresponding quantized representation sequences; predicting, by a Transformer of the transmission computing device, a probability mass function (PMF) as a conditional distribution of a given quantized representation in the sequence of quantized representations, wherein the conditional distribution is based on at least one dependency between one or more quantized representations that occur before the given quantized representation in the sequence of quantized representations; generating, by the transmission computing device, a plurality of compressed video frames by applying entropy coding to each quantized representation based on the predicted PMF, wherein the entropy coding comprises assigning a smaller number of bits to values ​​having a higher frequency of occurrence; and The plurality of compressed video frames are transmitted by the transmission computing device.

2. The computer-implemented method of claim 1 , further comprising: Receiving the plurality of compressed video frames by a receiving computing device; as well as A decoder of the receiving computing device generates a plurality of decompressed video frames based on the predicted PMF.

3. A computer-implemented method as described in claim 1 or 2, wherein the mean quantile corresponds to the cross entropy of the conditional distribution with respect to the predicted PMF.

4. The computer-implemented method of claim 3, wherein the prediction of the PMF further comprises: The encoding and decoding efficiency of the entropy encoding and decoding is maintained by adjusting the cross entropy.

5. The computer-implemented method of claim 1, wherein the encoder performs spatial downscaling and increases channel dimensionality.

6. The computer-implemented method of claim 1, wherein the encoder is a convolutional neural network (CNN) based image encoder.

7. The computer-implemented method of claim 2, wherein the decoder is a convolutional neural network (CNN) based image decoder.

8. The computer-implemented method of claim 1, wherein the encoding of each frame further comprises quantizing the quantized representation to an integer grid.

9. The computer-implemented method of any one of claims 1 to 8, further comprising: Applying neural image compression trains one or more of the encoder or the decoder to a corresponding lossy transform, wherein a target distortion variable is based on a range of each quantized representation.

10. The computer-implemented method of claim 9, wherein the training of the one or more of the encoder or the decoder is based on a rate-distortion tradeoff loss.

11. The computer-implemented method of claim 1, wherein the at least one dependency is a time dependency.

12. The computer-implemented method of claim 1 , further comprising: The given quantized representation is spatially partitioned into non-overlapping blocks of size N×N, and wherein the one or more quantized representations that occur before the given quantized representation are arranged as overlapping blocks of size M×M, where M>N, where each block is spatially flattened to generate one or more tokens for the Transformer, and Wherein the prediction of the PMF is based on a spatial context and a temporal context derived from the overlapping blocks.

13. The computer-implemented method of claim 12, wherein the prediction of the PMF by the Transformer comprises: extracting, by the first Transformer, from each of the overlapping blocks, separately, temporal information corresponding to the one or more quantized representations that occurred before the given quantized representation; as well as The extracted temporal information is mixed by the second Transformer.

14. The computer-implemented method of claim 1, wherein the transmitting computing device comprises a camera, and the method further comprises: capturing the plurality of input video frames using the camera; as well as The plurality of input video frames are received by the encoder from the camera.

15. The computer-implemented method of claim 2, wherein the receiving computing device comprises a video player, and the method further comprises: The multiple decompressed video frames are output by the video player.

16. The computer-implemented method of claim 1, further comprising: obtaining, at the transport computing device, trained versions of the encoder and the Transformer, wherein the encoding is performed by the trained version of the encoder, and Wherein the prediction is performed by the trained version of the Transformer.

17. The computer-implemented method of claim 2, further comprising: obtaining a trained version of the decoder at the receiving computing device, and Wherein the generating of the plurality of decompressed video frames is performed by the trained version of the decoder.

18. The computer-implemented method of claim 9, wherein the encoder is trained at the transmitting computing device.

19. The computer-implemented method of claim 9, wherein the decoder is trained at a receiving computing device.

20. The computer-implemented method of claim 1, further comprising: The plurality of compressed video frames are stored at the transmission computing device.

21. The computer-implemented method of claim 2, further comprising: The plurality of decompressed video frames are stored at the receiving computing device.

22. The computer-implemented method of any one of claims 2 to 21, wherein the transmitting computing device is the same as the receiving computing device.

23. The computer-implemented method of claim 1, wherein each frame of the plurality of input video frames is independently encoded.

24. A computing device comprising: one or more processors; as well as A data storage device having computer executable instructions stored thereon, which when executed by the one or more processors cause the computing device to perform functions including the computer-implemented method of any one of claims 1 to 23.

25. The computing device of claim 24, wherein the computing device is a mobile phone.

26. A computer program comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 23.

27. An article, comprising: One or more non-transitory computer-readable media having computer-readable instructions stored thereon, which, when executed by one or more processors of a computing device, cause the computing device to perform functions including the computer-implemented method of any one of claims 1 to 23.

28. A computing device comprising: Apparatus for performing the computer-implemented method of any one of claims 1 to 23.

29. A decoding device, comprising: one or more processors; as well as A data storage device, wherein the data storage device has computer executable instructions stored thereon, the computer executable instructions, when executed by the one or more processors, causing the decoding device to perform functions, the functions comprising: Receiving, by a decoder of the decoding device, a plurality of compressed video frames as a corresponding sequence of quantized representations; predicting, by a Transformer of the decoding device, a probability mass function (PMF) as a conditional distribution of a given quantized representation in the sequence of quantized representations, wherein the conditional distribution is based on at least one dependency between one or more quantized representations that occur before the given quantized representation in the sequence of quantized representations; generating, by the decoding device, a plurality of decompressed video frames by applying entropy decoding to each quantized representation based on the predicted PMF, wherein the entropy decoding comprises inverting an entropy encoding and the entropy encoding assigns a smaller number of bits to values ​​having a higher frequency of occurrence; and The plurality of decompressed video frames are provided by the decoding device.

30. The decoding apparatus of claim 30, wherein an average number of bits corresponds to a cross entropy of the conditional distribution with respect to the predicted PMF.

31. The decoding apparatus of claim 30, wherein the function for the prediction of the PMF further comprises: The decoding efficiency of the entropy decoding is maintained by adjusting the cross entropy.

32. The decoding apparatus of claim 30, wherein the decoder is an image decoder based on a convolutional neural network (CNN).

33. The decoding device according to any one of claims 30 to 32, wherein the function further comprises: Neural image decompression is applied to train the decoder as a lossy transform, where the target distortion variable is based on the range of each quantized representation.

34. The decoding apparatus of claim 33, wherein the training of the decoder is based on a rate-distortion trade-off loss.

35. The decoding apparatus of claim 30, wherein the at least one dependency is a time dependency.

36. The decoding device of claim 30, further comprising a video player, and the function for providing the plurality of decompressed video frames further comprises: The multiple decompressed video frames are output by the video player.

37. The decoding device of claim 30, wherein the functions further comprise: obtaining, at the decoding device, trained versions of the decoder and the Transformer, wherein the decoding is performed by the trained version of the decoder, and Wherein the prediction is performed by the trained version of the Transformer.

38. The decoding device of claim 30, wherein the decoder is trained at the decoding device.

39. The decoding device of claim 30, wherein the functions further comprise: The plurality of decompressed video frames are stored at the decoding device.

40. The decoding apparatus of claim 30, wherein each of the plurality of compressed video frames is independently decoded.

41. The decoding device of claim 30, wherein the decoding device is a mobile phone.

Images