Attention-Based Context Modeling for Image and Video Compression

Abstract

A method and apparatus for entropy encoding and decoding of a latent tensor are described. The method includes dividing a latent tensor into segments in space-time, each segment including at least one latent tensor element. The arrangement of the segments is processed by a neural network, the neural network including at least one attention layer. Based on the processed segments, a probabilistic model is obtained for entropy encoding or decoding of the latent tensor elements.

Description

[Technical Field]

[0001] Embodiments of the present invention relate to the field of artificial intelligence (AI)-based video or picture compression techniques, and in particular to context models that use attention layers in neural networks to process elements of a latent tensor. [Background technology]

[0002] Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as digital TV broadcasting, video distribution over the Internet and mobile networks, real-time conversation applications such as video chat and video conferencing, DVD and Blue-ray discs, video content acquisition and editing systems, and camcorders for security applications.

[0003] The amount of video data required to represent even a relatively short video is significant, resulting in difficulties when the data is to be streamed or otherwise communicated over communication networks with limited bandwidth capacity. Therefore, video data is typically compressed before being communicated over modern telecommunications networks. Video size can also be an issue when the video is stored on a storage medium because memory resources may be limited. Video compression devices often use software and / or hardware at the source side to code video data before transmission or storage, thereby reducing the amount of data required to represent a digital video picture. The compressed data is then received at the destination side by a video decompression device, which decodes the video data. Due to limited network resources and the growing demand for higher video quality, improved compression and decompression techniques are desirable that increase compression ratios with little or no sacrifice in image quality.

[0004] In recent years, deep learning has become increasingly popular in the field of picture and video encoding and decoding. Summary of the Invention

[0005] Embodiments of the present disclosure provide apparatus and methods for entropy encoding and decoding of a latent tensor, including dividing the latent tensor into segments and obtaining a probabilistic model for entropy encoding of a current element of the latent tensor by processing a set of elements through one or more layers of a neural network, including an attention layer.

[0006] According to an embodiment, there is provided a method for entropy encoding of a latent tensor, the method including: dividing the latent tensor into a plurality of segments in a spatial dimension, each segment including at least one latent tensor element; processing an arrangement of the plurality of segments by one or more layers of a neural network including at least one attention layer; and obtaining a probabilistic model for entropy encoding of a current element of the latent tensor based on the processed plurality of segments.

[0007] The method considers spatial correlation within the latent tensor and spatial adaptation for implicit entropy estimation. An attention mechanism adaptively weights the importance of previously coded latent segments. The contribution of the segments to the entropy modeling of the current element corresponds to their respective importance. Thus, the performance of entropy estimation is improved.

[0008] In an exemplary implementation, splitting the latent tensor includes splitting the latent tensor into two or more segments in the channel dimension.

[0009] Segmenting the latent tensor in the channel dimension improves the performance of entropy estimation by allowing cross-channel correlation to be used for context modeling.

[0010] For example, processing the placement may include placing multiple segments in a predefined order, with segments having the same spatial coordinates being grouped together.

[0011] Such an arrangement can improve the performance of entropy estimation by focusing on cross-channel correlations through the associated processing order.

[0012] In an exemplary implementation, processing the arrangement includes arranging the multiple segments such that segments with different spatial coordinates are arranged consecutively in a predefined order.

[0013] Such an arrangement can improve the performance of entropy estimation by focusing on spatial correlations due to the associated processing order.

[0014] For example, processing with a neural network may include applying a first neural sub-network to extract features of the plurality of segments and providing an output of the first neural sub-network as an input to a subsequent layer in the neural network.

[0015] Processing the neural network input to extract features from multiple segments allows the attention layer to focus on independent deep features of the input.

[0016] In an exemplary implementation, processing with a neural network further includes providing position information of the plurality of segments as input to at least one attention layer.

[0017] Positional coding allows the attention layer to exploit the order of the input sequence.

[0018] In an exemplary implementation, processing the arrangement of the plurality of segments includes selecting a subset of segments from the plurality of segments, the subset being provided as input to a subsequent layer in the neural network.

[0019] Selecting a subset of segments allows support for larger sized latent tensors with reduced memory size and / or reduced processing requirements.

[0020] For example, processing by at least one attention layer in the neural network further includes applying a mask that masks elements in the attention layer that follow the current element in the processing order of the latent tensor.

[0021] Applying the mask ensures that only previously coded elements can be processed, so the coding order is preserved. The mask reflects the availability of information at the coding side to the decoding side.

[0022] In an exemplary implementation, the neural network includes a second neural sub-network, which processes the output of the attention layer.

[0023] The neural sub-network may process the features output by the attention layer to provide probabilities for the symbols used for encoding, allowing for efficient encoding and / or decoding.

[0024] For example, at least one of the first neural sub-network and the second neural sub-network is a multi-layer perceptron.

[0025] Multi-layer perceptrons can provide an efficient implementation of neural networks.

[0026] In an exemplary implementation, at least one attention layer in the neural network is a multi-head attention layer.

[0027] A multi-head attention layer can improve probability estimation by processing different representations of the input in parallel, providing more projection and attention computations corresponding to different viewpoints of the same input.

[0028] For example, at least one attention layer in a neural network is included in a transformer sub-network.

[0029] The transformer sub-network can provide an efficient implementation of the attention mechanism.

[0030] In an exemplary implementation, the method further includes padding the beginning of the arrangement of the plurality of segments with a zero segment prior to processing by the neural network. do.

[0031] The padding with zeros at the beginning of the constellation reflects the availability of information at the decoding side, so that the causality of the coding order is preserved.

[0032] For example, the method may further include entropy encoding the current element into a first bitstream using the obtained probability model. do.

[0033] The size of the bitstream can be reduced by using a probabilistic model obtained by processing multiple segments through a neural network with an attention layer.

[0034] In an exemplary implementation, the method further includes quantizing the latent tensor before dividing it into segments. do.

[0035] Quantized latent tensors allow for a more efficient encoding process because they result in simplified probability models, and such latent tensors can be compressed and processed with reduced complexity and more efficiently represented in the bitstream.

[0036] For example, the method may further comprise selecting a probability model for entropy coding according to a computational complexity and / or a characteristic of the first bitstream. do.

[0037] Allowing for a choice of context modeling strategies can allow for better performance during the encoding process and provide flexibility in adapting the encoded bitstream to a desired application.

[0038] In an exemplary implementation, the method further includes hyper-encoding the latent tensor to obtain a hyper-latent tensor, entropy encoding the hyper-latent tensor into a second bitstream, entropy decoding the second bitstream, and hyper-decoding the hyper-latent tensor to obtain a hyper-decoder output. do.

[0039] Introducing a hyperprior model can further improve the probability model and thus the coding efficiency by determining additional redundancies in the latent tensor.

[0040] For example, the method may further include dividing the hyperdecoder output into a plurality of hyperdecoder output segments, each hyperdecoder output including one or more hyperdecoder output elements, and for each segment in the plurality of segments, concatenating the segment with a set of hyperdecoder output segments in the plurality of hyperdecoder output segments before obtaining the probability model. do.

[0041] The probability model can be further improved by concatenating the hyperdecoder output with each segment in the plurality of segments.

[0042] In an exemplary implementation, the set of hyperdecoder output segments associated with each segment includes one or more of the following: a hyperdecoder output segment corresponding to the respective segment, or multiple hyperdecoder output segments corresponding to the same channel as the respective segment, or multiple hyperdecoder output segments spatially adjacent to the respective segment, or multiple hyperdecoder output segments including a neighboring segment spatially adjacent to the respective segment and a segment corresponding to the same channel as the neighboring segment.

[0043] The probability model can be further improved by including each set of hyperdecoder output segments. The performance and complexity behavior may depend on the set of hyperdecoder output segments and the content being coded.

[0044] The method further comprises adaptively selecting the set of hyperdecoder output segments according to computational complexity and / or characteristics of the first bitstream. do.

[0045] Allowing the selection of additional hyperprior modeling strategies can allow for better performance during the encoding process and provide flexibility in adapting the encoded bitstream to the desired application.

[0046] In an exemplary implementation, one or more of the steps of processing with a neural network and entropy encoding the current element are performed in parallel for each segment in the plurality of segments.

[0047] Parallel processing of segments can result in faster encoding into a bitstream.

[0048] According to an embodiment, there is provided a method of encoding image data, the method comprising: obtaining a latent tensor by processing the image data with a self-encoding convolutional neural network; and entropy encoding the latent tensor into a bitstream using a probability model generated according to any of the above methods.

[0049] Since the latent tensors for image reconstruction can still have a significant size, entropy coding can be easily and advantageously applied to image coding to effectively reduce data rates, for example, when transmission or storage of pictures or videos is desired.

[0050] According to an embodiment, a method for entropy decoding of a latent tensor is provided, comprising: initializing the latent tensor with zeros; dividing the latent tensor into a plurality of segments in spatial dimensions, each segment including at least one latent tensor element; processing an arrangement of the plurality of segments by one or more layers of a neural network including at least one attention layer; and obtaining a probabilistic model for entropy decoding of a current element of the latent tensor based on the processed plurality of segments.

[0051] The method considers spatial correlation within the latent tensor and spatial adaptation for implicit entropy estimation. An attention mechanism adaptively weights the importance of previously coded latent segments. The contribution of the segments to the entropy modeling of the current element corresponds to their respective importance. Thus, the performance of entropy estimation is improved.

[0052] In an exemplary implementation, splitting the latent tensor includes splitting the latent tensor into two or more segments in the channel dimension.

[0053] Segmenting the latent tensor in the channel dimension improves the performance of entropy estimation by allowing cross-channel correlation to be used for context modeling.

[0054] For example, processing the placement may include placing multiple segments in a predefined order, with segments having the same spatial coordinates being grouped together.

[0055] Such an arrangement can improve the performance of entropy estimation by focusing on cross-channel correlations through the associated processing order.

[0056] In an exemplary implementation, processing the arrangement includes arranging the multiple segments such that segments with different spatial coordinates are arranged consecutively in a predefined order.

[0057] Such an arrangement can improve the performance of entropy estimation by focusing on spatial correlations due to the associated processing order.

[0058] For example, processing with a neural network may include applying a first neural sub-network to extract features of the plurality of segments and providing an output of the first neural sub-network as an input to a subsequent layer in the neural network.

[0059] Processing the neural network input to extract features from multiple segments allows the attention layer to focus on independent deep features of the input.

[0060] In an exemplary implementation, processing with a neural network further includes providing position information of the plurality of segments as input to at least one attention layer.

[0061] Positional coding allows the attention layer to exploit the order of the input sequence.

[0062] In an exemplary implementation, processing the arrangement of the plurality of segments includes selecting a subset of segments from the plurality of segments, the subset being provided as input to a subsequent layer in the neural network.

[0063] Selecting a subset of segments allows support for larger sized latent tensors with reduced memory size and / or reduced processing requirements.

[0064] In an exemplary implementation, the neural network includes a second neural sub-network, which processes the output of the attention layer.

[0065] The neural sub-network may process the features output by the attention layer to provide probabilities for the symbols used for encoding, allowing for efficient encoding and / or decoding.

[0066] For example, at least one of the first neural sub-network and the second neural sub-network is a multi-layer perceptron.

[0067] Multi-layer perceptrons can provide an efficient implementation of neural networks.

[0068] In an exemplary implementation, at least one attention layer in the neural network is a multi-head attention layer.

[0069] A multi-head attention layer can improve probability estimation by processing different representations of the input in parallel, providing more projection and attention computations corresponding to different viewpoints of the same input.

[0070] For example, at least one attention layer in a neural network is included in a transformer sub-network.

[0071] The transformer sub-network can provide an efficient implementation of the attention mechanism.

[0072] In an exemplary implementation, the method further includes padding the beginning of the arrangement of the plurality of segments with a zero segment prior to processing by the neural network. do.

[0073] The padding with zeros at the beginning of the constellation reflects the availability of information at the decoding side, so that the causality of the coding order is preserved.

[0074] For example, the method may further include entropy decoding the current element into a first bitstream using the obtained probability model. do.

[0075] The size of the bitstream can be reduced by using a probabilistic model obtained by processing multiple segments through a neural network with an attention layer.

[0076] For example, the method may further comprise selecting a probability model for entropy coding according to a computational complexity and / or a characteristic of the first bitstream. do.

[0077] Enabling a choice of context modeling strategies Decryption Allows for better performance during the process, Decryption This allows for flexibility in adapting the resulting bitstream to the desired application.

[0078] In an exemplary implementation, the method further includes entropy decoding the hyper-latent tensor from the second bitstream and hyper-decoding the hyper-latent tensor to obtain a hyper-decoder output. do.

[0079] Introducing a hyperprior model can further improve the probability model and thus the coding efficiency by determining additional redundancies in the latent tensor.

[0080] For example, the method may further include dividing the hyperdecoder output into a plurality of hyperdecoder output segments, each hyperdecoder output including one or more hyperdecoder output elements, and for each segment in the plurality of segments, concatenating the segment with a set of hyperdecoder output segments in the plurality of hyperdecoder output segments before obtaining the probability model. do.

[0081] The probability model can be further improved by concatenating the hyperdecoder output with each segment in the plurality of segments.

[0082] In an exemplary implementation, the set of hyperdecoder output segments associated with each segment includes one or more of the following: a hyperdecoder output segment corresponding to the respective segment, or multiple hyperdecoder output segments corresponding to the same channel as the respective segment, or multiple hyperdecoder output segments spatially adjacent to the respective segment, or multiple hyperdecoder output segments including a neighboring segment spatially adjacent to the respective segment and a segment corresponding to the same channel as the neighboring segment.

[0083] The probability model can be further improved by including each set of hyperdecoder output segments. The performance and complexity behavior may depend on the set of hyperdecoder output segments and the content being coded.

[0084] The method further comprises adaptively selecting the set of hyperdecoder output segments according to computational complexity and / or characteristics of the first bitstream. do.

[0085] Allowing for the selection of additional hyperprior modeling strategies Decryption Allows for better performance during the process, Decryption This allows for flexibility in adapting the resulting bitstream to the desired application.

[0086] According to an embodiment, there is provided a method of decoding image data, the method comprising entropy decoding a latent tensor (4020) from a bitstream according to any of the methods described above, and obtaining the image data by processing the latent tensor with an auto-encoding convolutional neural network.

[0087] Since the latent tensors for image reconstruction may still have a significant size, entropy decoding can be easily and advantageously applied to image decoding to effectively reduce data rates, for example, when transmission or storage of pictures or videos is desired.

[0088] In an exemplary implementation, a computer program is stored on a non-transitory medium and includes code instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of a method according to any of the methods described above.

[0089] According to an embodiment, an apparatus for entropy encoding of a latent tensor is provided, the apparatus having a processing circuit configured to divide the latent tensor into a plurality of segments in a spatial dimension, each segment including at least one latent tensor element, process the arrangement of the plurality of segments by one or more layers of a neural network including at least one attention layer, and obtain a probabilistic model for entropy encoding of a current element of the latent tensor based on the processed plurality of segments.

[0090] According to an embodiment, an apparatus for entropy decoding of a latent tensor is provided, the apparatus having a processing circuit configured to initialize a latent tensor with zeros, divide the latent tensor into a plurality of segments in spatial dimensions, each segment including at least one latent tensor element, process the arrangement of the plurality of segments by one or more layers of a neural network including at least one attention layer, and obtain a probabilistic model for entropy decoding of a current element of the latent tensor based on the processed plurality of segments.

[0091] The device provides the advantages of the above method.

[0092] The present invention may be implemented in hardware (HW) and / or software (SW), or any combination thereof. Furthermore, HW-based implementations may be combined with SW-based implementations.

[0093] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0094] Embodiments of the invention will now be described in more detail with reference to the accompanying figures and drawings.

[0095] [Figure 1]FIG. 1 is a schematic diagram illustrating channels processed by layers of a neural network. [Figure 2] FIG. 1 is a schematic diagram showing a neural network autoencoder. [Figure 3a] FIG. 1 is a schematic diagram illustrating an exemplary network architecture on the encoder and decoder side including a hyperprior model. [Figure 3b] FIG. 1 is a schematic diagram illustrating a general network architecture on the encoder side including a hyperprior model. [Figure 3c] FIG. 1 is a schematic diagram showing a typical decoder-side network architecture including a hyperprior model. [Figure 4] FIG. 1 is a schematic diagram of a latent tensor obtained from an input image. [Figure 5a] 1 shows a first embodiment of a transformer network. [Figure 5b] 1 shows a second embodiment of a transformer network. [Figure 6a] FIG. 1 is a schematic diagram showing an attention network. [Figure 6b] FIG. 1 is a schematic diagram illustrating a multi-head attention network. [Figure 7a] 1 illustrates an example of context modeling using attention and placement according to a first embodiment. [Figure 7b] 10 shows an example of context modeling using attention and placement according to the second or third embodiment. [Figure 8] 1 illustrates an exemplary separation of latent tensors into segments and an exemplary arrangement of the segments. [Figure 9] 10 illustrates an exemplary separation of a latent tensor into segments, including a separation in the channel dimension, and an exemplary arrangement of the segments. [Figure 10a] Represents the padding for the segment alignment. [Figure 10b] Represents padding for placement in the first coding order of segments that include separation in the channel dimension. [Figure 10c] represents padding for placement in the second coding order of segments that include separation in the channel dimension. [Figure 11] 10 shows an example of concatenating a set of processed segments of a latent tensor and a set of hyperprior output segments. [Figure 12] A further embodiment is shown that concatenates the processed segments of the latent tensor and the set of hyperprior output segments. [Figure 13] a represents an exemplary processing order of segments of a latent tensor, b represents an exemplary processing order of segments of a latent tensor including separation in the channel dimension, where segments with the same spatial coordinate are processed consecutively, and c represents an exemplary processing order of segments of a latent tensor including separation in the channel dimension, where segments with the same channel segment index are processed consecutively. [Figure 14] a represents an exemplary processing order of segments of a latent tensor, where a subset of the segments are used for context modeling; b represents an exemplary processing order of segments of a latent tensor including separation in the channel dimension, where segments with the same spatial coordinate are processed consecutively and a subset of the segments are used for context modeling; and c represents an exemplary processing order of segments of a latent tensor including separation in the channel dimension, where segments with the same channel segment index are processed consecutively and a subset of the segments are used for context modeling. [Figure 15] 1 is a block diagram illustrating an example of a video coding system configured to implement embodiments of the present invention. [Figure 16] FIG. 2 is a block diagram illustrating another example of a video coding system configured to implement embodiments of the present invention. [Figure 17] FIG. 1 is a block diagram illustrating an example of an encoding device or a decoding device. [Figure 18] FIG. 10 is a block diagram illustrating another example of an encoding device or a decoding device. DETAILED DESCRIPTION OF THE INVENTION

[0096] In the following description, reference is made to the accompanying figures, which form part of this disclosure and which show, by way of illustration, specific aspects of embodiments of the present invention or in which embodiments of the present invention may be used. It is understood that embodiments of the present invention may be used in other aspects and may involve structural or logical changes not depicted in the figures. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

[0097] For example, in connection with a described method, the disclosure also applies to a corresponding device or system configured to perform the method, and vice versa. For example, when one or more specific method steps are described, a corresponding device may include one or more units, e.g., functional units, for performing the described step or steps (e.g., one unit that performs one or more steps, or multiple units that each perform one or more of the steps), even if such one or more units are not explicitly described or shown. On the other hand, for example, when a specific apparatus is described in terms of one or more units, e.g., functional units, a corresponding method may include one or more units, e.g., functional units, for performing the function or functions of the one or more units. or multiple (e.g., one step of performing the functions of one or more units, or multiple steps each performing the functions of one or more of multiple units) even if such one or more steps are not explicitly described or shown. Furthermore, it is understood that the features of various exemplary embodiments and / or aspects described herein may be combined with each other unless specifically stated otherwise.

[0098] In image / video compression algorithms, entropy estimation is one of the components that offers significant benefits. Examples of entropy estimation include explicit entropy estimation and / or implicit entropy estimation. Explicit entropy estimation can be achieved by a hyperprior that compresses entropy estimation parameters and transmits side information via a second bitstream. Implicit entropy estimation can use already decoded elements of a first bitstream and include them in the entropy estimation of the primary bitstream, taking into account the causal relationship of the coding order. Implicit entropy estimation is usually referred to as an autoregressive context model and can typically be a two-dimensional (2D) masked convolution. However, 2D masked convolution provides only finite, small support. This limits the performance of implicit entropy estimation because long-range dependencies are not considered.

[0099] Furthermore, once trained, convolution kernels inherently cannot adapt to the characteristics of the bitstream, i.e., the latent tensor elements. Because the same kernel is applied to all positions in the compressed bitstream, convolution kernels are position-independent. This limits the performance of implicit models, as they can only learn position-specific dependencies. Even as the kernel size of masked convolutions increases, the performance of implicit models barely improves, as their non-adaptivity allows them to exploit a fixed set of position-specific internal relationships between previously coded elements.

[0100] Furthermore, the implicit model with 2D masked convolution encodes / decodes all channels of the latent tensor at once and does not exploit any cross-channel correlation. Due to the lack of per-channel autoregression, channel elements of the currently coded latent element do not have access to information of other spatially co-located elements with different channel indices. The lack of per-channel autoregression also leads to performance degradation.

[0101] Autoencoders and Unsupervised Learning An autoencoder is a type of executive neural network used to learn efficient data coding in an unsupervised manner. Its schematic is shown in Figure 2. The goal of an autoencoder is to learn a representation (encoding) of a data set, usually for dimensionality reduction, by training the network to ignore signal "noise". Along with the reduction side, a reconstruction side is also learned, where the autoencoder tries to generate a representation from the reduced encoding that is as close as possible to the original input, hence the name. In the simplest case, given one hidden layer, the encoder stage of an autoencoder takes an input x and maps it to h: h=σ(Wx+b) This image h is usually called a code, latent variable, or latent representation. Here, σ is an element-wise activation function, such as the sigmoid function or the rectified linear unit. W is a weight matrix, and b is a bias vector. The weights and biases are usually initialized randomly and then iteratively updated during training via backpropagation. The decoder stage of the autoencoder then maps h to a reconstruction x' of the same shape as x: x'=σ'(W'h'+b') Here, σ′, W′, and b′ for the decoder can be independent of the corresponding σ, W, and b for the encoder.

[0102] Variational autoencoder models make strong assumptions about the distribution of latent variables. They use a variational approach to latent representation learning, which results in an additional loss component and a specific estimator for the training algorithm called the Stochastic Gradient Variational Bayes (SGVB) estimator. The data is represented by a directed graphical model p θ (x|h), and the encoder uses the posterior distribution p θ Approximation q to (h|x) φSuppose we are learning (h|x), where φ and θ represent the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vectors in a VAE typically matches the probability distribution of the training data much more closely than a standard autoencoder.

[0103] Recent advances in the field of artificial neural networks, especially convolutional neural networks, have sparked researchers' interest in applying neural network-based techniques to image and video compression tasks. For example, end-to-end optimized image compression using networks based on variational autoencoders has been proposed.

[0104] Data compression is therefore considered a fundamental and well-studied problem in engineering, and is commonly formulated with the goal of designing a code for a particular discrete data set with minimal entropy. The problem is closely related to probabilistic source modeling, as the solution relies heavily on knowledge of the probabilistic structure of the data. However, because all practical codes must have finite entropy, continuous-valued data (such as a vector of image pixel intensities) must be quantized into a finite set of discrete values, which introduces error.

[0105] This situation, known as the lossy compression problem, requires a tradeoff between two competing costs: the entropy of the discretized representation (rate) and the error (distortion) resulting from quantization. Different compression applications, such as data storage or transmission over limited-capacity channels, require different rate-distortion tradeoffs.

[0106] Joint optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in a high-dimensional space is intractable. For this reason, most existing image compression methods work by linearly transforming a data vector into an appropriate continuous-valued representation, quantizing its elements separately, and encoding the resulting discrete representation using a reversible entropy code. This approach is called transform coding, due to the central role of the transform.

[0107] For example, JPEG uses a discrete cosine transform on blocks of pixels, and JPEG 2000 uses a multi-scale orthogonal wavelet decomposition. Typically, the three components of a transform coding method—the transform, the quantizer, and the entropy code—are individually optimized (often by manual parameter tuning). Modern video compression standards such as HEVC, VVC, and EVC also use transformed representations to code the residual signal after prediction. Several transforms are used for this purpose, such as the discrete cosine transform (DCT) and discrete sine transform (DST), as well as the low-frequency non-separable manually optimized transform (LFNST).

[0108] Variational Image Compression The variational autoencoder (VAE) framework can be viewed as a nonlinear transform coding model. The transformation process can be divided into four main parts. Figure 3a illustrates the VAE framework. In Figure 3a, an encoder 310 maps an input image x 311 to a latent representation (denoted as y) via a function y = f(x). This latent representation may also be referred to below as a part of the "latent space" or a point in the "latent space". The function f() is a transformation function that transforms the input signal 311 into a more compressible representation y.

[0109] The input image 311 to be compressed is represented as a 3D tensor with size H×W×C, where H and W are the height and width of the image, and C is the number of color channels. In a first step, the input image passes through an encoder 310, which downsamples the input image 311 by applying multiple convolutions and nonlinear transformations to generate a latent space feature tensor (hereafter referred to as latent tensor) y. (This is not resampling in the classical sense, but in deep learning, down- and upsampling are common terms for changing the height and width size of a tensor.) The latent tensor y 4020 corresponding to the input image 4010, shown exemplarily in FIG. 4, is expressed as (H / D e )×(W / D e )×C e where De is the downsampling factor of the encoder and Ce is the number of channels.

[0110] The difference between pixels in the input / output images and the latent tensor is shown in Figure 4. The latent tensor 4020 is a multidimensional array of elements that typically does not represent picture information. Two of the dimensions relate to the height and width of the image, while the information and content relate to lower-resolution representations of the image. The third dimension, the channel dimension, relates to different representations of the same image in latent space.

[0111] A latent space can be understood as a compressed data representation where similar data points are close to each other in the latent space. Latent spaces are useful for learning features of data or finding simpler representations of data for analysis. Quantizer 320 is the latent representation y,

number

number

[0112] The entropy estimate of the latent tensor y can be improved by further applying an optional hyperprior model.

[0113] The first step to obtain the hyperprior model is the hyperencoder 330 is applied to the latent tensor y, and the hyperencoder 330 downsamples the latent tensor to a hyper-latent tensor z by convolution and nonlinear transformation. The latent tensor x is then (H / D h )×(W / D h )×C h It has a size of

[0114] In the next step, a quantizer 331 can be run on the latent tensor z. A factorization entropy model 342 is then run on the quantized hyperlatent tensor

number

number

number

[0115] The factorized entropy model 342 serves as a codebook with parameters available at the decoder side. The entropy decoder 343 recovers the quantized hyper-latent tensor from the bitstream 341 by using the factorized entropy model 342. The recovered quantized hyper-latent tensor is upsampled in the hyper-decoder 350 by applying multiple convolution operations and nonlinear transformations. The hyper-decoder output tensor 430 is denoted by ψ.

[0116] The HyperEncoder / Decoder (also known as HyperPrior) 330-350 encodes the quantized latent representation to obtain the rate achievable by lossless entropy source coding.

number

number

number

number

number

[0117] In FIG. 3a, component AE370 is an arithmetic coding module, which:

number

number

[0118] Arithmetic decoding (AD) 372 is the process that reverses the binarization process, where binary digits are converted back to sample values. Arithmetic decoding is provided by arithmetic decoding module 372.

[0119] In Figure 3a, there are two interconnected sub-networks. A sub-network in this context is a logical division between parts of an overall network. For example, in Figure 3a, modules 310, 320, 370, 372, and 380 are called the "encoder / decoder" sub-network. The "encoder / decoder" sub-network is responsible for encoding (generating) and decoding (parsing) the first bitstream, "y bitstream." The second bitstream in Figure 3a is Sub The network has modules 330, 331, 340, 343, 350 and 360 and is called the "hyper-encoder / decoder" sub-network. The second sub-network is responsible for generating a second bitstream, the "z-bitstream". The two sub-networks have different purposes.

[0120] The first subnetwork is responsible for: Transforming 310 the input image 311 into its latent representation y (it is easier to compress x), Latent expression y

number

number

[0121] The purpose of the second sub-network is to obtain statistical properties of the samples of the "y bit-stream" (e.g., mean, variance, and correlation between samples of the y bit-stream) so that the compression of the y bit-stream by the first sub-network is more efficient. The second sub-network generates a second bit-stream, the "z bit-stream," which includes said information (e.g., mean, variance, and correlation between samples of the y bit-stream).

[0122] No. 2 Sub The network

number

number

number

number

number

number

number

number

number

number

number

number

[0123] 3a describes an example of a VAE (Variational Autoencoder), the details of which may vary in different implementations. For example, in a specific implementation, additional components may be present to more efficiently obtain statistical characteristics of samples of the first bitstream. In one such implementation, a context modeler may be present, which is directed to extracting cross-correlation information of the y bitstream. The statistical information provided by the second sub-network may be used by the AE (Arithmetic Encoder) 370 and AD (Arithmetic Decoder) 372 components.

[0124] Figure 3a shows the encoder and decoder in a single diagram. As will be apparent to those skilled in the art, the encoder and decoder may be, and very often are, embedded in different devices.

[0125] 3b and 3c separately represent the encoder and decoder components corresponding to the VAE framework. As input, the encoder receives a picture according to some embodiments. The input picture may include one or more channels, such as color channels or other types of channels, e.g., depth channels or motion information channels. The outputs of the encoder (shown in FIG. 3b) are a y bitstream and a z bitstream. The y bitstream is the output of the first sub-network of the encoder, and the z bitstream is the output of the second sub-network of the encoder.

[0126] Similarly, in FIG. 3c, two bitstreams, y bitstream and z bitstream, are received as input and the reconstructed (decoded) image is

number

[0127] Specifically, as can be seen in Figure 3b, the encoder comprises an encoder 310 which converts an input x into a signal y, which is then fed to a quantizer 320. The quantizer 320 feeds information to an arithmetic coding module 370 and to a hyper-encoder 330, which may receive the signal y rather than a quantized version. The hyper-encoder 330 feeds the z bitstream already discussed above to a hyper-decoder 350, which then feeds information to an arithmetic coding module 370. The sub-steps discussed above with reference to Figure 3a may also be part of this encoder.

[0128] The output of the arithmetic coding module is the y bitstream. The y and z bitstreams are the output of the signal coding, which are then fed (transmitted) to the decoding process. Unit 310 is called an "encoder," but the complete subnetwork depicted in FIG. 3b can also be called an "encoder." The encoding process generally refers to a unit (module) that converts an input into a coded (e.g., compressed) output. From FIG. 3b, it can be seen that unit 310 can actually be considered the core of the entire subnetwork, since it performs the conversion of input x to y, which is a compressed version of x. Compression in encoder 310 can be achieved, for example, by applying a neural network, or in general, any processing network including one or more layers. In such a network, compression can be performed by cascaded processes, including downsampling, to reduce the size and / or number of input channels. Thus, the encoder may be referred to, for example, as a neural network (NN)-based encoder.

[0129] The remaining parts in the figure (quantization unit, hyperencoder, hyperdecoder, arithmetic encoder / decoder) are all parts that either improve the efficiency of the encoding process or are involved in converting the compressed output y into a series of bits (bitstream). Quantization may be provided to further compress the output of the NN encoder 310 by lossy compression. The AE 370, in combination with the hyperencoder 330 and hyperdecoder 350 used to configure the AE 370, can perform binarization, which allows the quantized signal to be further compressed by lossless compression. Therefore, the entire sub-network in Figure 3b can also be referred to as an "encoder."

[0130] Most deep learning (DL)-based image / video compression systems reduce the dimensionality of a signal before converting it into binary digits (bits). For example, in a VAE framework, an encoder, which is a nonlinear transform, maps an input image x to y, where y has a smaller width and height than x. Because y has a smaller width and height, and thus a smaller size, the dimensionality (size) of the signal is reduced, and therefore it is easier to compress signal y. In general, an encoder does not necessarily need to reduce the size in both (or generally, all) dimensions. Rather, some exemplary implementations may provide an encoder that reduces the size in only one dimension (or generally, a subset of dimensions).

[0131] The arithmetic encoder and decoder are specific implementations of entropy coding. The AE and AD can be replaced by any other entropy coding means. Also, the quantization operation and the corresponding quantization unit do not necessarily exist and / or can be replaced by other units.

[0132] Artificial Neural Networks Artificial neural networks (ANNs), or connectionist systems, are computing systems loosely inspired by the biological neural networks that make up animal brains. Such systems typically "learn" to perform tasks by examining examples, without being programmed with task-specific rules. For example, in image recognition, a system can learn to identify images that contain cats by analyzing sample images that have been manually labeled as "cat" or "no cat" and then using the results to identify cats in other images. They do this without any prior knowledge of cats—for example, that cats have fur, tails, whiskers, and cat-like faces. Instead, they automatically generate discriminative characteristics from the examples they process.

[0133] ANNs are based on a collection of connected units or nodes called artificial neurons, which are loosely modeled after neurons in a biological brain. Similar to synapses in a biological brain, each connection can send a signal to other neurons. When an artificial neuron receives a signal, it can process it and send a signal to the neurons connected to it.

[0134] In an ANN implementation, the "signals" in the connections are real numbers, and the output of each neuron is calculated by some nonlinear function of the sum of the inputs. These connections are called edges. Neurons and edges usually have weights that are adjusted as training progresses. The weights increase or decrease the strength of the signal in the connection. Neurons may have a threshold such that a signal is sent only if the aggregate signal exceeds that threshold. Neurons are usually aggregated into layers. Different layers may perform different transformations on their inputs. Signals propagate from the first layer (input layer) to the last layer (output layer), possibly after passing through the layers multiple times.

[0135] The original goal of the ANN approach was to solve problems in the same way as the human brain. Over time, attention shifted to performing specific tasks, diverging from biology. ANNs are used for a variety of tasks, including computer vision, speech recognition, machine translation, social network filtering, playing board and video games, medical diagnosis, and even activities traditionally considered exclusive to humans, such as painting.

[0136] The name "convolutional neural network" (CNN) refers to the network's use of a mathematical operation called convolution, which is a special kind of linear operation. A convolutional network is a neural network that uses convolution instead of the more common matrix multiplication in at least one layer.

[0137] Figure 1 illustrates the general concept of processing by a neural network such as a CNN. A convolutional neural network consists of an input layer, an output layer, and multiple hidden layers. The input layer is the layer to which the input (e.g., a portion of the image shown in Figure 1) is supplied for processing. The hidden layers of a CNN typically consist of a series of convolutional layers that convolve via multiplication or other dot products. The result of a layer is one or more feature maps (f.maps in Figure 1), sometimes called channels. Some or all layers may include subsampling. As a result, the feature maps may be smaller, as shown in Figure 1. The activation function of a CNN is typically a ReLU (Renormalized Logical Unit) layer, followed by additional convolutions, such as pooling layers, fully connected layers, and normalization layers, called hidden layers because their inputs and outputs are masked by the activation function and the final convolution. Although the layers are colloquially referred to as convolutions, this is by convention. Mathematically, it is technically a sliding dot product or cross-correlation. This is important for the index into the matrix as it affects how the weight at a particular index point is determined.

[0138] As shown in Figure 1, when programming a CNN to process images, the input is a tensor with the shape (number of images) × (image width) × (image height) × (image depth). Then, after passing through the convolutional layers, the image is abstracted into a feature map with the shape (number of images) × (feature map width) × (feature map height) × (feature map channels). A convolutional layer in a neural network should have the following attributes: a convolution kernel (hyperparameters) defined by width and height, and the number of input and output channels (hyperparameters). The depth of the convolutional filter (input channels) should be equal to the number of channels (depth) of the input feature map.

[0139] Traditional multi-layer perceptron (MLP) models have been used for image recognition. However, full connectivity between nodes has led to problems with high dimensionality and has not scaled well for high-resolution images. A 1000x1000 pixel image with RGB color channels has 3 million weights, far too many to be efficiently processed at scale with full connectivity. Furthermore, such network architectures do not consider the spatial structure of the data, treating distant input pixels the same as nearby pixels. This ignores the locality of reference within the image data, both computationally and semantically. Therefore, full connectivity of neurons is wasteful for purposes such as image recognition, which are dominated by spatially local input patterns.

[0140] Convolutional neural networks are biologically inspired variants of multilayer perceptrons specifically designed to emulate the behavior of the visual cortex. These models mitigate the challenges posed by MLP architectures by exploiting the strong spatial local correlations present in natural images. Convolutional layers are the core building blocks of CNNs. Their parameters consist of a set of learnable filters (kernels, as described above) whose receptive fields are small but extend across the depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, and a dot product is calculated between the filter's entries and the input, producing a two-dimensional activation map for that filter. As a result, the network learns filters that are activated when it detects a particular type of feature at a certain spatial location in the input.

[0141] Stacking the activation maps of all filters along the depth dimension forms the complete output volume of the convolutional layer. Therefore, every entry in the output volume can also be interpreted as the output of a neuron that examines a small region in the input and shares parameters with neurons in the same activation map. A feature map or activation map is the output activation of a given filter. Feature map and activation are synonymous. In some papers, it is called an activation map because it is a mapping corresponding to the activation of different parts of the image, or it is also called a feature map because it is a mapping of where certain types of features are found in the image. High activation means that a particular feature is found.

[0142] Another important concept in CNNs is pooling, a form of nonlinear downsampling. There are several nonlinear functions that implement pooling, but the most common is max pooling, which divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such subregion.

[0143] Intuitively, the exact location of a feature is less important than its rough location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. Pooling layers serve to progressively reduce the spatial size of the representation, reducing the number of parameters, memory footprint, and computational effort in the network, thereby also controlling overfitting. In CNN architectures, it is common to periodically insert pooling layers between successive convolutional layers. The pooling operation provides another form of translation invariance.

[0144] A pooling layer operates independently on all depth slices of the input, varying its size spatially. The most common form is a pooling layer with a filter of size 2x2, applied by 2 along both the width and height with a stride of 2 for each depth slice of the input, discarding 75% of the activations. In this case, all max operations exceed 4 numbers. The depth dimension remains unchanged. In addition to max pooling, pooling units can use other functions such as average pooling and L2-norm pooling. While average pooling was historically popular, it has recently become less commonly used compared to max pooling, which often outperforms it in practice. Due to aggressive reductions in representation size, the trend these days is to use smaller filters or discard pooling layers entirely. "Region of interest" pooling (also known as ROI pooling) is a variant of max pooling, where the output size is fixed and the input rectangle is a parameter. Pooling is a key component of convolutional neural networks for object detection based on the Fast R-CNN architecture.

[0145] The term ReLU, abbreviated as Rectified Linear Unit, applies a non-saturating activation function. By setting negative values ​​to zero, they are effectively removed from the activation map. This enhances the nonlinearity of the decision function and the entire network without affecting the receptive fields of the convolutional layers. Other functions, such as saturated hyperbolic tangent and sigmoid functions, are also used to enhance nonlinearity. ReLU is often preferred over other functions because it can train neural networks several times faster without significantly affecting generalization accuracy.

[0146] After several convolutional and max-pooling layers, the higher level inference of a neural network is performed via fully connected layers. As in regular (non-convolutional) artificial neural networks, neurons in fully connected layers connect with all activations of the previous layer. Therefore, their activations can be computed as an affine transformation, which is a matrix multiplication followed by a bias offset (vector addition of a learned or fixed bias term).

[0147] The "loss layer" (containing the computation of the loss function) specifies how training penalizes deviations between predicted (output) labels and true labels, and is usually the final layer of a neural network. Various loss functions suitable for different tasks can be used: softmax loss is used to predict one class out of K mutually exclusive classes; sigmoid cross entropy loss is used to predict K independent probability values ​​in [0,1]; and Euclidean loss is used for regression to real-valued labels.

[0148] In summary, FIG. 1 illustrates the data flow in a typical convolutional neural network. First, an input image passes through a convolutional layer and is abstracted into a feature map containing several channels, corresponding to the number of filters in the set of learnable filters in this example. The feature map is then subsampled, e.g., using a pooling layer, thereby reducing the dimensionality of each channel of the feature map. The data is then sent to another convolutional layer, which may have a different number of output channels. As mentioned above, the number of input and output channels is a hyperparameter of the layer. To establish network connectivity, these parameters must be synchronized between two connected layers, so that the number of input channels in the current layer should be equal to the number of output channels in the previous layer. For the first layer, which processes input data, e.g., an image, the number of input channels is typically equal to the number of channels in the data representation, e.g., three channels for an RGB or YUV representation of an image or video, or one channel for a grayscale image or video representation.

[0149] Attention Mechanism in Deep Learning Attention mechanisms are deep learning techniques that allow neural networks to enhance or focus on important parts of the input while fading out irrelevant parts. This simple yet powerful concept can be applied in areas such as natural language processing, recommendation, healthcare analytics, image processing, and speech recognition, for example.

[0150] Traditionally, attention is computed over the entire input sequence (global attention). Despite its simplicity, such an approach can be computationally expensive. Local attention can be a solution.

[0151] One implementation of an attention mechanism is the so-called Transformer model, which applies an attention layer followed by a feedforward neural network. Two examples of the Transformer block are shown in Figures 5a and 5b. In the Transformer model, the input tensor x is first fed into a neural network layer to extract features of the input tensor. This results in a so-called embedding tensor e 5010, which contains the latent space elements that are used as input to the Transformer. The size of the input tensor x and the embedding tensor e is S × d input and S × d e where S is the number of consecutive elements and d is the dimension of each consecutive element. Positional encodings 5020 may be added to the embedding tensor. Positional encodings 5020 allow the transformer to take into account the order of the input sequence. Such positional encodings provide a representation of the position of an element within the arrangement of elements of the input tensor. These encodings may be learned, or alternatively, a predefined tensor may represent the order of the sequence.

[0152] Once the positional encoding 5020 is calculated, it is piecewise added to the embedding vector 5010. The input vector is then prepared to enter the transformer block. The example transformer block of Figure 5a consists of two consequent steps: multi-head attention 5030 and a linear transformation 5032 with a nonlinear activation function applied separately and identically to each position. Two summation and normalization blocks 5031 and 5033 combine the outputs of the attention layer 5030 and MLP 5032, respectively, with residual connections 5034 and 5035. The example transformer block of Figure 5b also consists of two consequent steps: attention 5051 and a linear transformation 5053 with a nonlinear activation function applied separately and identically to each position. The example of Figure 5b shows a different arrangement of normalization blocks 5050 and 5052 and residual connections 5054 and 5055. The role of the attention block here is similar to that of the regression cell, but with fewer computational requirements. The transformer block does not necessarily need to use a multi-head attention layer, or L multi-head attention layers. The multi-head attention layers in these examples may be replaced by any other type of attention mechanism.

[0153] In the self-attention module, all three vectors are taken from the same sequence and represent vectors with embedded positional encoding.

[0154] A typical attention mechanism consisting of a query Q 620, a key K 621, and a value V 622 is exemplarily shown in Figure 6. The origin of this name comes from search engines, where a user's query is matched against keys in an internal engine and the results are represented by some values.

[0155] The embedding vector is expressed as positional encoding p e After combining with S, three different representations, namely, queues Q, keys K, and values ​​V, are obtained by the feedforward neural network layer. The queues, keys, and values ​​are S×d q , S × d k and S × dv Usually, the cue, key, and value may have the same dimension d. To calculate self-attention, we first calculate the scaled dot product (QK T / d k ) can be calculated and a softmax function can be applied to obtain attention scores. These scores are then multiplied by a value to achieve self-attention. The self-attention mechanism is:

number

[0156] The computed attention is then added to the embedding vector by forming a residual connection and normalized by a normalization layer. Finally, a multi-layer feedforward neural network (also known as a multi-layer perceptron) containing the residual connections is applied, and the final output is normalized. All of the above steps (after generating the embedding tensor) describe one layer of a transformer, which can be repeated L times to generate a transformer network containing L layers.

[0157] In other words, the attention layer obtains multiple representations of the input sequence, e.g., keys, cues, and values. To obtain a representation from the multiple representations, the input sequence is processed by a respective set of weights. The weight sets may be obtained in a training phase. These weight sets may be jointly learned along with the rest of the neural network, including such an attention layer. During inference, the output is calculated as a weighted sum of the processed input sequence.

[0158] One extension to the above attention mechanism is multi-head attention. In this version, the final dimensions of query, key and value are divided into h sub-representations, and for each sub-representation, attention is calculated separately. The final attention is calculated by concatenating each sub-attention and feeding it into a feed-forward neural network (FFN). The formulation of multi-head attention is given as follows:

number

[0159] Multi-head attention allows for parallelization and for each embedding tensor to have multiple representations.

[0160] A single attention function is depicted in Figure 6a, and its parallel application in multi-head attention is shown in Figure 6b. By performing more projection and attention computations, the model is able to take different perspectives on the same input sequence. It jointly handles information from different angles, mathematically represented via different linear subspaces.

[0161] The exemplary single-attention function of Figure 6a performs alignment 620 between key 610 and query 611, as described above, and obtains output 640 by applying weighted sum 630 to attention scores and values ​​612. The exemplary multi-head attention function of Figure 6b performs alignment 660 for each pair of key 650 and query 651, which may belong to a different linear subspace. For each obtained attention score and each value 652, a weighted sum 670 is calculated. The results are concatenated 680 to obtain output 690.

[0162] The next step after multi-head attention in the transformer block is a simple positionally fully connected feedforward network. There are residual connections around each block, followed by layer normalization. The residual connections help the network keep track of the data it is observing. The layer normalization serves to reduce the variance of the features.

[0163] There are several different architectures of Transformers in the literature, and the order and type of their components can vary. However, the basic logic is similar: some kind of attention mechanism followed by another neural network encapsulates the layers of the Transformer layer, and multiple layers of this architecture form the Transformer network. As explained above, two examples are given in Figures 5a and 5b. The present invention is not limited to the above-mentioned exemplary implementation of the attention mechanism.

[0164] Attention-Based Context Modeling The process of obtaining a context model by applying a neural network including an attention layer is exemplarily shown in Figures 7a and 7b.

[0165] Image data to be compressed can be represented as a three-dimensional tensor 311 with size H×W×C, where H and W are the height and width of the image, and C is the number of color channels. The input image can be processed by an autoencoder 310, as described above with reference to FIG. 3a. Such an autoencoder 310 downsamples the input image by applying a number of convolutions and nonlinear regressions to generate a latent tensor y. The latent tensor is expressed as (H / D e )×(W / D e )×C ewhere De is the downsampling factor of the encoder 310 and Ce is the number of channels. The obtained latent tensor can be encoded into a bitstream 371 using the probability model generated by attention-based context modeling.

[0166] The latent tensor y may be quantized, which may be performed by a quantization block 320.

[0167] The context model for entropy coding of the latent tensor can be determined by applying an attention layer 732. The latent spatial feature tensor contains one or more elements and is divided 700 into multiple segments 820 in the spatial dimension as shown in Figures 7a and 8. Each segment contains at least one latent tensor element. A segment has a dimension (p H ×p W ×C e ) and C e is the number of pixels in the channel dimension of the latent tensor, and the spatial dimension of the latent tensor is divided into patches, each of which is (p H ×p W ) elements. Separation 700 is exemplarily shown in FIG. 8, which shows an exemplary 4×4×C e The dimensional latent tensor 810 is divided into 16 elements 820 in the spatial dimension, with each segment having dimensions (1 × 1 × C e )

[0168] The latent tensors to be entropy encoded may be generated from image data processed by an autoencoding neural network as described above, although the present invention is not limited to image data from an autoencoder. The latent tensors to be entropy encoded may also be generated while processing any other type of input data, such as multidimensional point clouds, audio data, video data, etc.

[0169] The arrangement 830 of the multiple segments is processed by one or more layers of a neural network. Such arrangement can be predefined, i.e., the scan order in the spatial and / or channel directions can be specified. The arrangement in a first embodiment can include reshaping the latent tensor into a sequential form 830.

number

number

[0170] The neural network includes at least one attention layer. The attention mechanism was described above in the section "Attention Mechanism in Deep Learning" with reference to Figures 5 and 6. The attention layer may be a multi-head attention layer, as exemplarily described with reference to Figure 6b. The attention layer may also be included in a transformer sub-network, as exemplarily described with reference to Figures 5a and 5b. The attention layer to be included in the transformer sub-network may be any type of attention layer or a multi-head attention layer.

[0171] A probability model for entropy coding of the current element of the latent tensor is obtained based on the processed multiple segments. The current element may be entropy coded into a first bitstream, e.g., the y bitstream 371 in FIG. 3, using the obtained probability model for entropy coding. A specific implementation of the entropy coding may be, for example, the arithmetic coding discussed in the "Variational Image Compression" section. The present invention is not limited to such an exemplary arithmetic coding. Any entropy coding that can code based on an estimated entropy per element can use the probabilities obtained by the present invention.

[0172] Separation of the latent tensor may include separating the latent tensor into two or more segments 701 in the channel dimension, as shown, for example, in FIG. CS Such separation of channel segments is

number

[0173] Maximum number of channel segments N CS teeth,

number

[0174] In a second embodiment, the segments may be arranged in a predefined order, with segments 931, 932, and 933 having the same spatial coordinates being grouped together. CS Segments at a first spatial coordinate 930 having different channel segment indices within [[(x,y,y) -1]] may be grouped together. Subsequently, segments at a second spatial coordinate 931 having different channel indices may be grouped together. The arrangement 930 may be

number

[0175] In the third embodiment, segments 941, 942, and 943 having different spatial coordinates may be arranged 701 consecutively in a predefined order. In other words, the segments corresponding to a first channel segment index 940 may be grouped together. Subsequently, the segments corresponding to a second channel segment index 941 may be grouped together. The arrangement 940 may be

number

[0176] For simplicity's sake,

number

number

[0177] The beginning of the multiple segment arrangement of any of the above exemplary embodiments may be padded 710, 711 with a zero segment 1000 before processing by the neural network. The zero segment may have the same dimensions as each segment in the multiple segments. Each element in the zero segment may be zero. Figure 10a illustrates such padding 710 for a first example, while Figures 10b and 10c illustrate padding 711 for second and third examples corresponding to different arrangements of the latent tensor. Figure 10a exemplarily illustrates the segments of the latent tensor for the first example, which does not perform separation in the channel dimension. C in the channel dimension e The zero segment 1000 containing elements is padded at the beginning of the constellation to obtain a padded constellation 1030. Figure 10b shows the padding of the constellation of the second embodiment with a coding order that encodes segments with the same spatial coordinates first. The zero segment 1001 in this example is zero, C e / N CSSimilarly, FIG. 10c exemplarily illustrates the padding arrangement of the third embodiment with a coding order that encodes segments of the same channel segment index first. The zero segment 1002 in this example is C e / N CS Contains elements. The padding ensures that the causality of the coding sequence is not disturbed, i.e., a decoder can decode the data from the bitstream without any additional prior knowledge.

[0178] Multiple segments of the latent tensor may be processed by a first neural sub-network 720. Such a first neural sub-network may extract features from the multiple segments. The features may be independent deep features (also called embeddings). Thus, the first neural sub-network 720 is a so-called embedding layer that extracts context embeddings in a high-dimensional real-valued vector space. The first neural sub-network 720 may be a fully connected neural network, such as the multi-layer perceptron described above. For example, a convolutional neural network (CNN) or a recurrent neural network (RNN) may be used. The output of the first neural sub-network 720, the so-called context embeddings, may be provided as input to subsequent layers of the neural network.

[0179] Positional information 721 of multiple segments may be provided as input to the attention layer. Such positional information 721 may be combined with the output of the first neural sub-network 720, e.g., by concatenation, addition, etc. Context embeddings may be combined with the positional information 721 and normalized 731. Positional encodings include positional information, e.g., coordinates in a linear space. Positional encodings allow the attention layer to understand the sequential order of the input sequence. For example, these encodings can be learned, or predefined tensors representing the order of the sequence can be used.

[0180] In processing by the attention layer 732, a mask may be applied, which masks subsequent elements in the attention tensor that follow the current element in the processing order of the latent tensor. The mask prevents subsequent elements from being used in the calculation of the attention tensor. In other words, the attention mechanism can be adapted to autoregressive tasks to ensure causality at the decoder side. Such a masked attention mechanism is an attention mechanism that is masked so as not to process any data that is not in a position prior to the current position in the attention layer input order. Masking is exemplarily shown in Figures 13a-c, which show the segment processing order for the current segment. The segments yet to be coded are shown.

[0181] The attention mechanism is applied by default to the entire sequence S. It is applied to each successive element s in S. i means that it applies attention to itself and all other elements. This behavior is undesirable for autoregressive tasks, as the network cannot use any elements that have not yet been processed. To combat this issue, the attention mechanism can be constrained by masking the scaled dot products within the attention mechanism. The mask can be written as an S × S matrix, whose lower triangle (including the diagonal) contains ones and whose upper triangle (excluding the diagonal) consists of minus infinity (softmax(-∞) = 0). Masked attention can be formulated as follows:

number

[0182] The masking of the present invention is not limited to the application of this exemplary matrix M. Any other masking technique may be applied.

[0183] The output of the attention layer 732 may be processed by a second neural sub-network 735, which may be a multi-layer perceptron. The output of the attention layer 732 may be normalized 734 before processing by the second neural sub-network 735. The output of the attention layer 732 may be combined with a context embedding or with a combined representation of the context embedding and location information 721 by residual connections 737.

[0184] The output of the attention-based context model is denoted by φ.

[0185] The probability model 770 for entropy coding may be selected depending on the computational complexity and / or characteristics of the first bitstream 731. The characteristics of the first bitstream 731 may include a predefined target rate or frame size. Rules may also be predefined for which option to use. In this case, the rules may be known by the decoder, so no additional signaling is required.

[0186] The selection may include selecting whether the separation of the latent tensors is performed in the channel dimension, and the selection may include selecting between various ways how the alignment is performed, e.g., in the first spatial dimension or in the first channel dimension.

[0187] For example, if separation in the channel dimension is not performed, the performance of the context model may be limited because cross-channel correlation is not considered for entropy modeling, but this may result in faster encoding and decoding since fewer autocorrelation steps are required.

[0188] For example, N CSIf N is >1, cross-channel correlation is taken into account, which may improve the performance of the context model. CS but

number

[0189] The first step to obtain an arbitrary hyperprior model is the hyperencoder shown in Figure 3a. 330 is applied to the latent tensor to obtain a hyper-latent tensor. The hyper-latent tensor may be encoded into a second bitstream, e.g., z-stream 341. The second bitstream may be entropy decoded, and the hyper-decoder output is obtained by hyper-decoding the hyper-latent tensor. The hyper-prior model may be obtained as described in the "Variational Image Compression" section. However, this disclosure is not limited to this example implementation.

[0190] Similar to the latent tensor, the output ψ of any hyperdecoder may be divided into multiple hyperdecoder output segments 740. Each hyperdecoder output segment may include one or more hyperdecoder output elements. For each segment in the multiple segments, the segment and the set of hyperdecoder output segments in the multiple hyperdecoder output segments may be concatenated before obtaining the probabilistic model 770. In other words, the tensors φ and ψ may be concatenated in the channel dimension (the last dimension), resulting in a concatenated two-dimensional tensor.

[0191] The hyperdecoder output segments may be arranged corresponding to the arrangement of the multiple segments. The output of the hyperdecoder ψ is

number

[0192] An example set of hyper-encoder output segments is shown in Figure 11. The arrangement of the segments of the latent tensor may be performed according to the second or third embodiment. In a fourth embodiment, the set of hyper-decoder output segments to be concatenated with each segment 1100 may include a hyper-decoder output segment 1110 corresponding to each segment. The hyper-decoder output segment 1110 may have the same spatial coordinates and the same channel segment index as each segment 1100. That is, the hyper-decoder output segment 1110 may be co-located with each segment 1100. Using co-located hyper-decoder output segments 1110 in the concatenation may reduce computational complexity and therefore result in faster processing.

[0193] In a fifth embodiment, the set of hyperdecoder output segments to be concatenated with each segment 1100 may include multiple hyperdecoder output segments corresponding to the same channel as the respective segment. In other words, the multiple hyperdecoder output segments may include hyperdecoder output segments that have the same spatial coordinates as the respective segment, i.e., belong to the same co-located channel. In the example of FIG. 11 , there are three co-located hyperdecoder output segments 1120, 1121, and 1122, with the first hyperdecoder output segment 1120 having the same channel segment index and spatial coordinates as each exemplary segment of the latent tensor y. The remaining two hyperdecoder output segments 1121 and 1122 may have the same spatial coordinates as each exemplary segment 1100. The multiple hyperdecoder output segments 1120, 1121, and 1122 may belong to the same co-located channel as each segment 1100. This set of hyperdecoder output segments may improve the performance of probability estimation because additional cross-channel correlation of the hyperdecoder outputs is taken into account.

[0194] In a sixth embodiment, the set of hyperdecoder output segments to be concatenated with each segment 1100 may include multiple hyperdecoder output segments 1130 that are spatially proximate to the respective segment 1100. The multiple hyperdecoder output segments 1130 that are spatially proximate to each segment 1100 are exemplarily shown in FIG. 11 and may have the same channel segment index as each segment 1100. The multiple hyperdecoder output segments 1130 may belong to the same spatial neighborhood of each segment 1100. This set of hyperdecoder output segments may improve the performance of probability estimation because more spatial correlation of the hyperdecoder outputs is taken into account.

[0195] In the seventh embodiment, the set of hyperdecoder output segments to be concatenated with each segment may include multiple hyperdecoder output segments, including adjacent segment 1140 that is spatially proximate to each segment, and segments 1141 and 1142 that correspond to the same channel as the adjacent segment 1140. In other words, the set of hyperdecoder output segments may include hyperdecoder output segments 1140 that are spatially proximate to each segment 1100 and may have the same channel segment index as each segment 1100, as exemplarily shown in Figure 11. Furthermore, the set of hyperdecoder output segments may include hyperdecoder output segments 1141 and 1142 that may have the same spatial coordinates as the spatially proximate hyperdecoder output segment 1140 and a channel segment index that is different from the channel segment index of the spatially proximate hyperdecoder output segment 1140. The hyperdecoder output segments to be concatenated may belong to the same local neighborhood as each segment 1100. This set of hyperdecoder output segments may improve the performance of the probability estimation since more spatial and cross-channel correlation of the hyperdecoder output is taken into account.

[0196] The set of hyperdecoder output segments to be concatenated with each segment is not limited to the above examples. Any other set of hyperdecoder output segments may be concatenated with each segment of the latent tensor. For example, any combination of the fourth through seventh embodiments above may be used. Any of the fourth through seventh embodiments above and any combination thereof may be combined with either the arrangement of the second or third embodiment.

[0197] 12 shows an example of concatenation 750 when separation into segments in the channel dimension is not performed. Arrangement of the segments of the latent tensor may be performed according to the first embodiment. For example, the set of hyperdecoder output segments to be concatenated with each segment 1200 may include a hyperdecoder output segment 1210 corresponding to that segment. The hyperdecoder output segment 1210 may have the same spatial coordinates and the same channel segment index as each segment 1200. For example, the set of hyperdecoder output segments to be concatenated with each segment 1200 may include multiple hyperdecoder output segments 1230 that are spatially proximate to that segment 1200. Each segment 1200 A plurality of such hyperdecoder output segments 1230 in close spatial proximity are exemplarily shown in FIG.

[0198] The concatenated tensor is S×(C φ +C ψ’ ) and C φ and C ψ’ are the number of channels in the tensor φ and the number of channels from the tensor ψ, respectively. The result of the concatenation may be processed by an aggregation process 760. For example, aggregation may be performed by a set of fully connected neural networks and nonlinear transformations for the last dimension. For example, aggregation may be implemented by one or more layers of convolutions and nonlinear transformations with a kernel size of 1×1. The entropy model 770 is

number

number

[0199] Similar to the selection of a probability model for entropy coding, the set of hyperdecoder output segments may be adaptively selected depending on the computational complexity and / or characteristics of the first bitstream. The characteristics of the first bitstream may include a predefined target rate or frame size. A set of rules may be predefined which options to use. In this case, the rules may be known by the decoder, so no additional signaling is required.

[0200] During encoding, all elements of the latent tensor are available, so that processing by the neural network and / or entropy coding of the current element can be performed in parallel for each segment in the multiple segments.

[0201] Processing the placement by the neural network may include selecting a subset of segments. Such a subset of segments may be selected from the plurality of segments. The subset may be fed to a subsequent layer of the neural network. For example, the subset may be selected before applying the first neural sub-network. Such a subset of segments may include segments within a local neighborhood in the spatial dimension. This is exemplarily shown in FIGS. 14a-c. In the example of FIGS. 14a-c, segments spatially close to the current segment are represented, which may be used by context modeling. FIG. 14a illustrates a case where separation in the channel dimension cannot be performed. FIG. 14b illustrates an exemplary case where a segment of a first channel is processed before a segment of a second channel. FIG. 14c illustrates an exemplary case where a segment with a first channel segment index is processed before a segment with a second channel segment index. The size of the attention mechanism in the context model may be determined by:

number

number

[0202] For decoding the latent space feature tensor from the first bitstream, the latent tensor is initialized with zeros because the decoder is agnostic to the latent tensor and its statistical properties. The latent space feature tensor contains one or more elements and is divided 700 into multiple segments 820 in the spatial dimension, as shown in Figures 7a and 8. Each segment contains at least one latent tensor element. Similar to the encoding side, the arrangement of the multiple segments is handled by one or more layers of a neural network. The arrangement corresponding to the first embodiment of the encoding side may include reshaping the latent tensor into a sequential form 830.

number

number

[0203] The neural network includes at least one attention layer. The attention mechanism is described above in the section "Attention Mechanism in Deep Learning" with reference to Figures 5 and 6. Corresponding to the encoding, the attention layer may be a multi-head attention layer as exemplarily described with reference to Figure 6b. The attention layer may be included in a transformer sub-network, as exemplarily described with reference to Figures 5a and 5b.

[0204] A probability model for entropy decoding of the current element of the latent tensor is obtained based on the processed multiple segments. The current element can be decoded from a first bitstream, e.g., y bitstream 371 in FIG. 3, using the obtained probability model for entropy decoding. A specific implementation for entropy decoding can be, for example, arithmetic decoding as discussed in the "Variational Image Compression" section. The present invention is not limited to such exemplary arithmetic decoding. Any entropy decoding that can base its decoding on an estimated entropy per element can use the probabilities obtained by the present invention.

[0205] Separation of the latent tensor may include separating the latent tensor into two or more segments in the channel dimension 701, as shown, for example, in Figure 9. Such separation is described in detail on the encoding side.

[0206] The segments may be arranged in a predefined order, with segments 931, 932 and 933 having the same spatial coordinates being grouped together. This arrangement 930 corresponds to the second embodiment whose encoding is described in detail above.

[0207] The segments with different spatial coordinates 941, 942 and 943 may be arranged consecutively in a predefined order, such an arrangement 940 being similar to the third embodiment described above in detail for encoding.

[0208] The beginning of any of the above example multi-segment arrangements may be padded 710, 711 with a zero segment 1000 before processing by the neural network. The zero segment may have the same dimensions as each segment in the multi-segment arrangement, as exemplarily shown in Figures 10a-c. Padding is described in detail above with reference to Figures 10a-c.

[0209] According to the encoding side, multiple segments of the latent tensor may be processed by a first neural sub-network 720. Such a first neural sub-network may extract features from the multiple segments. The features may be independent deep features. The first neural sub-network 720 may be a multi-layer perceptron. Position information 721 of the multiple segments may be provided as input to an attention layer. Such position information 721 may be combined with the output of the first neural sub-network 720, for example, by concatenation, addition, etc.

[0210] The output of the attention layer 732 can be processed by a second neural sub-network 735, which can be a multi-layer perceptron, similar to the encoding side. The output of the attention layer 732 can be combined with the context embedding or with a combined representation of the context embedding and the location information 721 by the residual connection 737.

[0211] Similar to the encoding, the probability model 770 for entropy coding may be selected depending on the computational complexity and / or characteristics of the first bitstream 731. The characteristics of the first bitstream 731 may include a predefined target rate or frame size. A set of rules may be predefined which options to use. In this case, the rules may be known by the decoder.

[0212] The hyper-latent tensors may be entropy decoded from the second bitstream 341. The obtained hyper-latent tensors may be hyper-decoded into a hyper-decoder output ψ.

[0213] Similar to the latent tensor, the output of any hyperdecoder ψ may be split into multiple hyperdecoder output segments 740. Each hyperdecoder output segment may contain one or more hyperdecoder output elements. For each segment in the multiple segments, that segment and a set of hyperdecoder output segments in the multiple hyperdecoder output segments may be concatenated 750 before a probabilistic model 770 is obtained.

[0214] An example set of hyperencoder output segments is shown in FIG. 11 and is described in detail on the encoding side in the fourth, fifth, sixth and seventh embodiments. decoder The set of output segments may include one or more of the following: a hyperdecoder output segment corresponding to each of the segments, or multiple hyperdecoder output segments corresponding to the same channel as each of the segments, or multiple hyperdecoder output segments spatially adjacent to each of the segments, or multiple hyperdecoder output segments including an adjacent segment spatially adjacent to each of the segments and a segment corresponding to the same channel as the adjacent segment. Any of the above fourth to seventh embodiments and any combination thereof may be combined with any of the arrangements of the second or third embodiment.

[0215] Similar to the selection of a probability model for entropy coding, the set of hyperdecoder output segments may be adaptively selected depending on the computational complexity and / or characteristics of the first bitstream, including, for example, a predefined target rate or frame size. A set of rules may predefine the options to be used, in which case the rules may be known by the decoder.

[0216] Processing of the alignment by the neural network may include selecting a subset of segments. Such a subset of segments may be selected from the plurality of segments. The subset may be fed to a subsequent layer of the neural network. Examples are described above with reference to Figures 14a-c.

[0217] A probabilistic model using attention layers may be applied to entropy decoding of latent tensors, which may be processed by a self-decoding convolutional neural network to obtain image data as discussed above.

[0218] Implementation in Picture Coding The encoder 20 may be configured to receive a picture 17 (or picture data 17), e.g., a picture in a sequence of pictures forming a video or a video sequence. The received picture or picture data may be a preprocessed picture 19 (or preprocessed picture data 19). For simplicity, the following description refers to the picture 17. The picture 17 may also be referred to as the current picture or the picture to be coded (particularly in video coding, to distinguish the current picture from other pictures, e.g., pictures that have been coded and / or decoded previously in the same video sequence, i.e., the video sequence that also includes the current picture).

[0219] A (digital) picture is or can be viewed as a two-dimensional array or matrix of intensity-valued samples. The samples in the array are sometimes called pixels (short for picture element) or pels. The number of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and / or resolution of the picture. For color representation, three color components are used; that is, a picture can be represented by or contain three sample arrays. In an RGB format or color space, a picture has corresponding red, green, and blue sample arrays. However, in video coding, each pixel is typically represented in a luminance and chrominance format or color space, e.g., YCbCr, which includes a luminance component denoted by Y (sometimes L is used instead) and two chrominance components denoted by Cb and Cr. The luminance (or luma for short) component Y represents brightness or gray-level intensity (e.g., similar to a grayscale picture), while the two chrominance (or chroma for short) components Cb and Cr represent chromaticity or color information components. Thus, a picture in YCbCr format includes a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). A picture in RGB format may be converted or translated to YCbCr format, and vice versa, a process also known as color conversion or translation. If a picture is monochrome, the picture may only have a luminance sample array. Thus, a picture may be, for example, an array of luma samples in monochrome format, or an array of luma samples and two corresponding arrays of chroma samples in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0220] Hardware and software implementation Some further implementations in hardware and software are described below.

[0221] Any of the encoding devices described with reference to Figures 15-18 may provide means for performing entropy encoding of a latent tensor. The processing circuitry in any of these example devices is configured to: divide the latent tensor into multiple segments in spatial dimensions, each segment including at least one latent tensor element; process the arrangement of the multiple segments through one or more layers of a neural network including at least one attention layer; and obtain a probabilistic model for entropy encoding of a current element of the latent tensor based on the processed multiple segments.

[0222] 15-18 may include a processing circuit configured to perform a decoding method, including initializing a latent tensor with zeros, dividing the latent tensor into a plurality of segments in a spatial dimension, each segment including at least one latent tensor element, processing the arrangement of the plurality of segments through one or more layers of a neural network including at least one attention layer, and obtaining a probabilistic model for entropy decoding of a current element of the latent tensor based on the processed plurality of segments.

[0223] In summary, a method and apparatus are described for entropy encoding and decoding of a latent tensor, including dividing the latent tensor into segments in a spatial dimension, each segment including at least one latent tensor element. The arrangement of the segments is processed by a neural network, which includes at least one attention layer. Based on the processed segments, a probabilistic model is obtained for entropy encoding or decoding of the latent tensor elements.

[0224] In the following embodiment of the video coding system 10, the video encoder 20 and the video decoder 30 are described based on FIGS. 15 and 16. FIG.

[0225] 15 is a schematic block diagram of an example coding system 10, e.g., video coding system 10 (or coding system 10 for short), that may employ the techniques of the present application. Video encoder 20 (or encoder 20 for short) and video decoder 30 (or decoder 30 for short) of video coding system 10 represent examples of devices that may be configured to perform techniques in accordance with various examples described herein.

[0226] figure 15 As shown in FIG. 1, coding system 10 may generate coded picture data 21, e.g., coded picture data 21 to a destination device 14 for decoding.

[0227] The source device 12 comprises an encoder 20 and may further, that is, optionally, comprise a picture source 16 , a pre-processor (or pre-processing unit) 18 , for example a picture pre-processor 18 , and a communication interface or unit 22 .

[0228] Picture source 16 may comprise or be any kind of picture capture device, e.g., a camera that captures pictures of the real world, and / or any kind of picture generation device, e.g., a computer graphics processor that generates computer-animated pictures, or any kind of other device that acquires and / or provides real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). Picture source may also be any kind of memory or storage that stores any of the above pictures.

[0229] To distinguish from the processing performed by the preprocessor 18 or preprocessing unit 18, the pictures or picture data 17 may also be referred to as raw pictures or raw picture data 17.

[0230] The pre-processor 18 is configured to receive (raw) picture data 17 and perform pre-processing on the picture data 17 to obtain a pre-processed picture 19 or pre-processed picture data 19. The pre-processing performed by the pre-processor 18 may include, for example, cropping, color format conversion (e.g., RGB to YCbCr), color correction, or noise removal. It may be understood that the pre-processing unit 18 may be any component.

[0231] Video encoder 20 is configured to receive pre-processed picture data 19 and to provide encoded picture data 21 .

[0232] The communications interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and to transmit the encoded picture data 21 (or any further processed version thereof) via the communications channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0233] The destination device 14 comprises a decoder 30 (eg a video decoder 30 ) and may further, that is, optionally comprise a communications interface or unit 28 , a post-processor 32 (or post-processing unit 32 ), and a display device 34 .

[0234] The communications interface 28 of the destination device 14 is configured to receive the encoded picture data 21 (or any further processed version thereof), for example directly from the source device 12 or from any other source, for example a storage device, for example a coded picture data storage device, and to provide the encoded picture data 21 to the decoder 30.

[0235] The communication interface 22 and the communication interface 28 are connected to the encoded picture data 21 or the encoded data 21may be configured to transmit or receive over a direct communication link between the source device 12 and the destination device 14, e.g., a direct wired or wireless connection, or over any type of network, e.g., a wired or wireless network or any combination thereof, or any type of private and public network, or any type of combination thereof.

[0236] The communications interface 22 may be configured, for example, to package the encoded picture data 21 into a suitable format, for example packets, and / or to process the encoded picture data using any type of transmission coding or processing for transmission over a communications link or network.

[0237] The communications interface 28 forms the counterpart to the communications interface 22 and may be configured, for example, to receive transmitted data and process the transmitted data using any kind of corresponding transmission decoding or processing and / or unpackaging to obtain the encoded picture data 21.

[0238] Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrow for communication channel 13 in FIG. 15 pointing from source device 12 to destination device 14, or as bidirectional communication interfaces, and may be configured, for example, to send and receive messages, for example, to set up connections, and to confirm and exchange any other information related to the communication link and / or data transmission, for example, coded picture data transmission.

[0239] The decoder 30 is arranged to receive the coded picture data 21 and to provide decoded picture data 31 or decoded pictures 31 .

[0240] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), e.g., the decoded picture 31, to obtain post-processed picture data 33, e.g., the post-processed picture 33. The post-processing performed by the post-processing unit 32 may include, e.g., color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling, or any other processing, e.g., to prepare the decoded picture data 31 for display, e.g., by a display device 34.

[0241] A display device 34 of the destination device 14 is configured to receive the post-processed picture data 33 for displaying the picture, e.g., to a user or viewer. The display device 34 may be or include any type of display for presenting the reconstructed picture, such as an internal or external display or monitor, for example, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0242] Coding system 10 further includes a training engine 25. Training engine 25 is configured to train encoder 20 (or a module within encoder 20) or decoder 30 (or a module within decoder 30) to process input pictures or generate probability models for entropy coding, as discussed above.

[0243] 15 depicts source device 12 and destination device 14 as separate devices, an embodiment of the device may have both or both functions, i.e., source device 12 or corresponding functions and destination device 14 or corresponding functions. In such an embodiment, source device 12 or corresponding functions and destination device 14 or corresponding functions may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0244] As will be clear to those skilled in the art based on the description, the presence and (exact) separation of different units or functions in the source device 12 and / or destination device 14 shown in FIG. 15 may vary depending on the actual device and application.

[0245] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30), or both encoder 20 and decoder 30, may be implemented by processing circuitry shown in FIG. 16, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, dedicated to video coding, or any combination thereof. Encoder 20 may be implemented by processing circuitry 46 to implement the various modules discussed with respect to the encoder of FIG. 3b and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented by processing circuitry 46 to implement the various modules discussed with respect to the decoder of FIG. 3c and / or any other decoder system or subsystem described herein. The processing circuitry may be configured to perform various operations discussed below. If the techniques are implemented partially in software, as shown in FIG. 18, a device may store software-like instructions on a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Either video encoder 20 or video decoder 30 may be incorporated into a single device as part of a combined encoder / decoder (CODEC), for example as shown in FIG.

[0246] Source device 12 and destination device 14 may comprise any of a wide range of devices, including any type of handheld or stationary device, e.g., a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a video game console, a video streaming device (e.g., a content service server or content distribution server), a broadcast receiver device, a broadcast transmitter device, etc., and may use no operating system or any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0247] In some cases, the video coding system 10 depicted in FIG. 15 is merely an example, and the techniques herein may be applied to video coding settings (e.g., video encoding or video decoding) that do not necessarily involve any data communication between the encoding device and the decoding device. In other examples, data may be received from local memory, streamed over a network, etc. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve and decode data from memory. In some examples, the encoding and decoding do not communicate with each other but simply transmit and receive data. Encode it To memory memory The method is performed by a device that reads and decodes data from memory.

[0248] For convenience of description, embodiments of the present invention are described herein with reference to reference software, for example, High-Efficiency Video Coding (HEVC), or Versatile Video Coding (VVC), a next-generation video coding standard developed by the Joint Collaboration Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). Those skilled in the art will appreciate that embodiments of the present invention are not limited to HEVC or VVC.

[0249] 17 is a schematic diagram of a video coding device 400 according to an embodiment of the present disclosure. The schematic diagram of the video coding device 400 is suitable for implementing the disclosed embodiments described herein. In an embodiment, the video coding device 400 may be a decoder, such as the video decoder 30 of FIG. 15, or an encoder, such as the video encoder 20 of FIG. 15.

[0250] Video coding device 400 has an ingress port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data, a processor, logic unit, or central processing unit (CPU) 430 for processing the data, a transmitter unit (Tx) 440 and an egress port 450 (or output port 450) for transmitting the data, and a memory 460 for storing the data. Video coding device 400 may also have optical-electrical (OE) and electro-optical (EO) components coupled to ingress port 410, receiver unit 420, transmitter unit 440, and egress port 450 for inputting and outputting optical or electrical signals.

[0251] The processor 430 is implemented in hardware and software. The processor 430 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), FPGA, ASIC, and DSP. The processor 430 communicates with the ingress port 410, the receiver unit 420, the transmitter unit 440, the egress port 450, and the memory 460. The processor 430 includes a coding module 470. The coding module 470 implements the disclosed embodiments described above. For example, the coding module 470 implements, processes, prepares, or provides various coding operations. Thus, the inclusion of the coding module 470 significantly enhances the functionality of the video coding device 400 and achieves the transition of the video coding device 400 to different states. Alternatively, the coding module 470 is implemented as instructions stored in the memory 460 and executed by the processor 430.

[0252] Memory 460 may include one or more disks, tape drives, and solid-state drives, and may be used as an overflow data storage device for storing programs when such programs are selected for execution and for storing instructions and data read during program execution. Memory 460 may be, for example, volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and / or static random access memory (SRAM).

[0253] FIG. 18 is a simplified block diagram of an apparatus 500 that may be used as either or both of source device 12 and destination device 14 of FIG. 15, according to an example embodiment.

[0254] Processor 502 of device 500 can be a central processing unit. Alternatively, processor 502 can be any type of device, or multiple devices, now existing or later developed, that can manipulate or process information. While the disclosed implementations can be performed by a single processor, such as processor 502, as shown, advantages in speed and efficiency can be achieved using more than one processor.

[0255] The memory 504 of the apparatus 500 may be a read-only memory (ROM) device or a random-access memory (RAM) device in implementation. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 accessed by the processor 502 using a bus 512. The memory 504 may further include an operating system 508 and application programs 510, which include at least one program that enables the processor 502 to perform the methods described herein. For example, the application programs 510 may include Applications 1 through N, which may further include a video coding application that performs the methods described herein, including encoding and decoding using a neural network with a subset of partially updatable layers.

[0256] The apparatus 500 may also include one or more output devices, such as a display 518. The display 518, in one example, may be a touch-sensitive display that combines a display with touch-sensitive elements operative to detect touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0257] Although depicted here as a single bus, bus 512 of device 500 may be comprised of multiple buses. Additionally, secondary storage 514 may be directly coupled to other components of device 500 or may be accessed over a network, and may comprise a single integrated unit such as a memory card or multiple units such as multiple memory cards. Device 500 may thus be implemented in a wide variety of configurations.

[0258] Although embodiments of the present invention have been described primarily in terms of video coding, it should be noted that embodiments of coding system 10, encoder 20 and decoder (and correspondingly system 10), and other components described herein, may also be configured for still picture processing or coding, i.e., processing or coding of individual pictures independent of any preceding or consecutive pictures as found in video coding. In general, only inter prediction units 244 (encoder) and 344 (decoder) may not be available when picture processing coding is limited to a single picture 17. All other functionality (also called tools or techniques) of the video encoder 20 and the video decoder 30, such as residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, partitioning 262 / 362, intra prediction 254 / 354, and / or loop filtering 220, 320, as well as entropy encoding 270 and entropy decoding 304, may be used for still picture processing as well.

[0259] For example, embodiments of the encoder 20 and decoder 30, and the functions described herein with reference to the encoder 20 and decoder 30, may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on a computer-readable medium or transmitted over a communication medium as one or more instructions or code and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to tangible media such as data storage media, or communication media, including any medium that facilitates transfer of a computer program from one place to another, for example, according to a communication protocol. In this manner, computer-readable media may generally correspond to (1) tangible computer-readable storage media that are non-transitory, or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0260] By way of example, and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of data or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio waves, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio waves, and microwaves are included within the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, and instead cover non-transitory, tangible storage media. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where a disk typically reproduces data magnetically, while a disc reproduces data optically with a laser. Combinations of the above should also be included within the scope of computer-readable media.

[0261] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor," as used herein, may refer to any of the above structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided in dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Alternatively, the techniques may be implemented entirely in one or more circuit or logic elements.

[0262] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, the various units, in combination with appropriate software and / or firmware, may be combined into a codec hardware unit or provided by a collection of interoperable hardware units including one or more processors as described above.

Claims

1. 1. A method for entropy coding of a latent tensor, comprising: dividing the latent tensor into a plurality of segments in a spatial dimension, each segment including at least one latent tensor element; processing the alignment of the plurality of segments through one or more layers of a neural network, including at least one attention layer; obtaining a probabilistic model for entropy coding of the current element of the latent tensor based on the processed segments; and The processing by the neural network comprises: providing position information of the plurality of segments as input to the at least one attention layer. method.

2. dividing the latent tensor into two or more segments in a channel dimension; The method of claim 1.

3. Processing the arrangement includes arranging the plurality of segments in a predefined order, with segments having the same spatial coordinates being grouped together. The method of claim 2.

4. processing the arrangement includes arranging the plurality of segments such that segments having different spatial coordinates are arranged consecutively in a predefined order; The method of claim 2.

5. processing with the neural network includes applying a first neural sub-network to extract features of the plurality of segments and providing an output of the first neural sub-network as an input to a subsequent layer within the neural network; 5. The method according to any one of claims 1 to 4.

6. the neural network includes a second neural sub-network, the second neural sub-network processing the output of the attention layer; The method of claim 5.

7. at least one of the first neural sub-network and the second neural sub-network is a multi-layer perceptron; The method of claim 6.

8. processing the arrangement of the plurality of segments includes selecting a subset of segments from the plurality of segments, the subset being provided as input to a subsequent layer within the neural network; 8. The method according to any one of claims 1 to 7.

9. The processing by the at least one attention layer in the neural network comprises: applying a mask to mask elements in an attention layer that follow the current element in the processing order of the latent tensor.

9. The method according to any one of claims 1 to 8.

10. the at least one attention layer in the neural network is a multi-head attention layer; 10. The method according to any one of claims 1 to 9.

11. the at least one attention layer in the neural network is included in a transformer sub-network; 11. The method according to any one of claims 1 to 10.

12. padding the beginning of the arrangement of the plurality of segments with a zero segment prior to processing by the neural network.

12. The method according to any one of claims 1 to 11.

13. and entropy encoding the current element into a first bitstream using the obtained probability model.

13. The method according to any one of claims 1 to 12.

14. quantizing the latent tensor before dividing it into segments.

14. The method according to any one of claims 1 to 13.

15. The computational complexity of a first bitstream in which the current element is entropy coded using the obtained probability model, and / or Characteristics of the first bitstream selecting the probability model for the entropy coding according to 15. The method according to any one of claims 1 to 14.

16. hyper-encoding the latent tensor to obtain a hyper-latent tensor; entropy encoding the hyper-latent tensor into a second bitstream; entropy decoding the second bitstream; and obtaining a hyper-decoder output by hyper-decoding the hyper-latent tensor; 16. The method of any one of claims 1 to 15, further comprising:

17. dividing said hyperdecoder output into a plurality of hyperdecoder output segments, each hyperdecoder output comprising one or more hyperdecoder output elements; for each segment among said plurality of segments, concatenating said segment with a set of hyperdecoder output segments among said plurality of hyperdecoder output segments before obtaining said probability model; 17. The method of claim 16, further comprising:

18. The set of hyperdecoder output segments concatenated with each segment is a hyperdecoder output segment corresponding to each such segment; or a plurality of hyperdecoder output segments corresponding to the same channel as each of said segments; or a plurality of hyperdecoder output segments spatially adjacent to each of said segments; or a plurality of hyperdecoder output segments including neighboring segments spatially adjacent to each of the segments and segments corresponding to the same channel as the neighboring segments; including one or more of:

18. The method of claim 17.

19. The computational complexity of a first bitstream in which the current element is entropy coded using the obtained probability model, and / or Characteristics of the first bitstream and adaptively selecting the set of hyperdecoder output segments according to 19. The method of claim 17 or 18.

20. one or more steps of processing with the neural network and entropy encoding the current element are performed in parallel for each segment in the plurality of segments.

20. The method of any one of claims 1 to 19.

21. 1. A method of encoding image data, comprising: obtaining a latent tensor by processing the image data with a self-encoding convolutional neural network; entropy coding the latent tensor into a bitstream using a probability model obtained by implementing the method of any one of claims 1 to 20; A method having the following.

22. 1. A method for entropy decoding of a latent tensor, comprising: initializing the latent tensor with zero; dividing the latent tensor into a plurality of segments in a spatial dimension, each segment including at least one latent tensor element; processing the alignment of the plurality of segments through one or more layers of a neural network, including at least one attention layer; obtaining a probability model for entropy decoding of the current element of the latent tensor based on the processed segments; and The processing by the neural network comprises: providing position information of the plurality of segments as input to the at least one attention layer. method.

23. dividing the latent tensor into two or more segments in a channel dimension; 23. The method of claim 22.

24. Processing the arrangement includes arranging the plurality of segments in a predefined order, with segments having the same spatial coordinates being grouped together.

24. The method of claim 23.

25. processing the arrangement includes arranging the plurality of segments such that segments having different spatial coordinates are arranged consecutively in a predefined order; 24. The method of claim 23.

26. processing with the neural network includes applying a first neural sub-network to extract features of the plurality of segments and providing an output of the first neural sub-network as an input to a subsequent layer within the neural network; 26. The method of any one of claims 22 to 25.

27. the neural network includes a second neural sub-network, the second neural sub-network processing the output of the attention layer; 27. The method of claim 26.

28. at least one of the first neural sub-network and the second neural sub-network is a multi-layer perceptron; 28. The method of claim 27.

29. processing the arrangement of the plurality of segments includes selecting a subset of segments from the plurality of segments, the subset being provided as input to a subsequent layer within the neural network; 29. The method of any one of claims 22 to 28.

30. the at least one attention layer in the neural network is a multi-head attention layer; 30. The method of any one of claims 22 to 29.

31. the at least one attention layer in the neural network is included in a transformer sub-network; 31. The method of any one of claims 22 to 30.

32. padding the beginning of the arrangement of the plurality of segments with a zero segment prior to processing by the neural network.

32. The method of claim 31 .

33. and entropy decoding the current element into a first bitstream using the obtained probability model.

33. A method according to any one of claims 22 to 32.

34. The computational complexity of a first bitstream in which the current element is entropy decoded using the obtained probability model, and / or Characteristics of the first bitstream selecting the probability model for the entropy decoding according to 34. A method according to any one of claims 22 to 33.

35. entropy decoding the hyper-latent tensor from the second bitstream; and obtaining a hyper-decoder output by hyper-decoding the hyper-latent tensor; 35. The method of any one of claims 22 to 34, further comprising:

36. dividing said hyperdecoder output into a plurality of hyperdecoder output segments, each hyperdecoder output comprising one or more hyperdecoder output elements; for each segment among said plurality of segments, concatenating said segment with a set of hyperdecoder output segments among said plurality of hyperdecoder output segments before obtaining said probability model; 36. The method of claim 35, further comprising:

37. The set of hyperdecoder output segments concatenated with each segment is a hyperdecoder output segment corresponding to each such segment; or a plurality of hyperdecoder output segments corresponding to the same channel as each of said segments; or a plurality of hyperdecoder output segments spatially adjacent to each of said segments; or a plurality of hyperdecoder output segments including neighboring segments spatially adjacent to each of the segments and segments corresponding to the same channel as the neighboring segments; including one or more of:

37. The method of claim 36.

38. The computational complexity of a first bitstream in which the current element is entropy decoded using the obtained probability model, and / or Characteristics of the first bitstream and adaptively selecting the set of hyperdecoder output segments according to 38. The method of claim 36 or 37.

39. 1. A method for decoding image data, comprising: entropy decoding latent tensors from the bitstream using the probability model obtained by implementing the method of any one of claims 22 to 38; and obtaining the image data by processing the latent tensor with a self-encoding convolutional neural network; A method having the following.

40. stored on a non-transitory medium and including code instructions; The code instructions, when executed by one or more processors, cause the one or more processors to perform the steps of the method of any one of claims 1 to 21. Computer program.

41. stored on a non-transitory medium and including code instructions; The code instructions, when executed by one or more processors, cause the one or more processors to perform the steps of the method of any one of claims 22 to 39. Computer program.

42. 1. An apparatus for entropy coding of a latent tensor, comprising: dividing the latent tensor into a plurality of segments in a spatial dimension, each segment containing at least one latent tensor element; processing the alignment of the plurality of segments through one or more layers of a neural network, including at least one attention layer; Obtain a probabilistic model for entropy coding of the current element of the latent tensor based on the processed segments. a processing circuit configured to The processing by the neural network comprises: providing position information of the plurality of segments as input to the at least one attention layer. Device.

43. 1. An apparatus for entropy decoding of a latent tensor, comprising: initializing the latent tensor with zero; dividing the latent tensor into a plurality of segments in a spatial dimension, each segment containing at least one latent tensor element; processing the alignment of the plurality of segments through one or more layers of a neural network, including at least one attention layer; Obtain a probability model for entropy decoding of the current element of the latent tensor based on the processed segments. a processing circuit configured to The processing by the neural network comprises: providing position information of the plurality of segments as input to the at least one attention layer. Device.

Images