Methods for encoding, transmitting, and decoding lossy images or videos, and data processing systems.

Abstract

1. A method for lossy image and video encoding, transmission, and decoding, comprising: receiving an input image at a first computer system; encoding the input image with a first trained neural network to generate a latent representation; performing a quantization process on the latent representation to generate a quantized latent representation, where a bin size used in the quantization process is based on the input image; and transmitting the quantized latent to a second computing system and decoding the quantized latent with a second trained neural network to generate an output image, where the output image is an approximation of the input image.

Description

[Technical Field]

[0001] The present invention relates to methods and systems for encoding, transmitting and decoding lossy images or video, methods, apparatus, computer programs and computer-readable storage media for encoding and transmitting lossy images or video, and methods, apparatus, computer programs and computer-readable storage media for receiving and decoding lossy images or video.

[0002] Demand for image and video content from users of communication networks is increasing. This includes not only the number of images displayed and the length of videos played, but also the demand for higher resolution content. As a result, the load on communication networks is increasing, and the amount of data transmitted is growing, leading to increased energy consumption of communication networks.

[0003] To mitigate the effects of these problems, image and video content is compressed before being transmitted over a network. There are two types of image and video content compression: lossless and lossy. Lossless compression compresses images and videos in a way that allows for the recovery of all the original information contained in the content. However, there are limits to the amount of data reduction that can be achieved using lossless compression. Lossy compression causes information to be lost from images and videos during the compression process. Known compression techniques attempt to minimize apparent information loss by removing information that results in changes to the decompressed image and video that are not particularly noticeable to the human visual system.

[0004] Artificial intelligence (AI)-based compression technologies achieve image and video compression and decompression by using trained neural networks in the compression and decompression processes. Generally, during the training of the neural network, the differences between the original image and video and the compressed and decompressed images and videos are analyzed, and the neural network's parameters are modified to reduce these differences while minimizing the data required for content transmission. However, AI-based compression methods can sometimes result in poorer compression in terms of the appearance of the compressed image and video and the amount of information required for transmission.

[0005] The present invention provides a method for encoding, transmitting and decoding lossy images and videos, the method comprising: receiving an input image in a first computer system; encoding the input image using a first trained neural network to generate a latent image representation; performing a quantization process on the latent representation to generate a quantized latent, wherein the bin sizes used in the quantization process are based on the input image; transmitting the quantized latent to a second computer system; and decoding the quantized latent using a second trained neural network to generate an output image, wherein the output image is an approximation of the input image.

[0006] The bin size may differ between at least two pixels of the latent representation.

[0007] The bin sizes may differ between at least two channels of the latent representation.

[0008] The bin size may be assigned to each pixel of the latent representation.

[0009] The quantization process may include performing an operation on the value of each pixel in the latent representation that corresponds to the bin size assigned to that pixel.

[0010] Quantization may involve subtracting the mean value of the latent representation from each pixel in the latent representation.

[0011] The quantization process may include rounding functions.

[0012] The size of the bins used to decode the quantized latent may be based on the previously decoded pixels of the quantized latent.

[0013] The quantization process may be constructed from a third trained neural network.

[0014] A third trained neural network may receive at least one previously decoded pixel of a quantized latent as input.

[0015] The method may further include the steps of: encoding a latent representation using a fourth trained neural network to generate a hyperlatent representation; performing a quantization process on the hyperlatent representation to generate a quantized hyperlatent; transmitting the quantized hyperlatent to a second computer system; and decoding the quantized hyperlatent using a fifth trained neural network to obtain the bin sizes, wherein the decoding of the quantized latent uses the obtained bin sizes.

[0016] The output of the fifth trained neural network may be processed by a further function to obtain the bin size.

[0017] A further function may be a sixth trained neural network.

[0018] The bin sizes used in the quantization process of the hyperlatent representation may be based on the input image.

[0019] The method may further include the steps of identifying at least one region of interest in an input image, and reducing the size of the bins used in the quantization process for at least one corresponding pixel in the latent representation within the identified region of interest.

[0020] The method may further include the steps of identifying at least one region of interest in an input image, and applying different quantization processes to at least one corresponding pixel of a latent representation within the identified region of interest.

[0021] At least one region of interest may be identified by a seventh trained neural network.

[0022] The locations of one or more regions of interest may be stored in a binary mask, which may be used to obtain the bin size.

[0023] The present invention provides a method for training one or more neural networks, the one or more neural networks for use in encoding, transmitting, and decoding lossy images or videos, the method comprising: receiving an input image on a first computer system; encoding the input image using a first neural network to generate a latent representation; performing a quantization process on the latent representation to generate a quantized latent representation, wherein the size of the bins used in the quantization process is based on the input image; decoding the quantized latent representation using a second neural network to generate an output image, wherein the output image is an approximation of the input image; determining a quantity based on the difference between the output image and the input image; updating the parameters of the first and second neural networks based on the determined quantity; and repeating the above steps using a first set of input images to generate a first trained neural network and a second trained neural network.

[0024] The method may further include the steps of: encoding a latent representation using a third neural network to generate a hyperlatent representation; performing a quantization process on the hyperlatent representation to generate a quantized hyperlatent representation; transmitting the quantized hyperlatent representation to a second computer system; decoding the quantized hyperlatent representation using a fourth neural network to obtain the bin size, wherein the decoding of the quantized latent uses the obtained bin size; and additionally updating the parameters of the third and fourth neural networks based on determined quantities to obtain the third and fourth trained neural networks.

[0025] The quantization process may include a first quantization approximation.

[0026] The determined quantity may also be based on a rate associated with the quantized potential, and a second quantization approximation may be used to determine the rate associated with the quantized potential, and the second quantization approximation may be different from the first quantization approximation.

[0027] The determined quantity may include a loss function, and the step of updating the neural network parameters may include evaluating the gradient of the loss function and backpropagating the gradient of the loss function through the neural network, wherein a third quantization approximation is used during the backpropagation of the gradient of the loss function, and the third quantization approximation is the same as the first quantization approximation.

[0028] The parameters of the neural network may be further updated based on the distribution of bin sizes.

[0029] At least one parameter of the distribution may be learned.

[0030] The distribution may also be an inverse gamma distribution.

[0031] The distribution may be determined by a fifth neural network.

[0032] The present invention provides a method for lossy image or video coding and transmission, the method comprising: receiving an input image in a first computer system; coding the input image using a first trained neural network to generate a latent representation; performing a quantization process on the latent representation to generate a quantized latent, wherein the bin sizes used in the quantization process are based on the input image; and transmitting the quantized latent.

[0033] The present invention provides a method for receiving and decoding a lossy image or video, the method comprising: receiving a quantized latent transmitted in accordance with the above method in a second computer system; and decoding the quantized latent using a second trained neural network to generate an output image, wherein the output image is an approximation of the input image.

[0034] The present invention provides a method for training one or more neural networks, the one or more neural networks for use in encoding, transmitting, and decoding lossy images or videos, the method comprising: receiving an input image on a first computer system; encoding the input image using a first neural network to generate a latent representation; performing a quantization process on the latent representation to generate a quantized latent; decoding the quantized latent using a second neural network to generate an output image, the output image being an approximation of the input image; determining a quantity based on the difference between the output image and the input image; updating the parameters of the first and second neural networks based on the determined quantity; and repeating the above steps using a plurality of sets of input images to generate one trained neural network and a second trained neural network, wherein at least one of the plurality of sets of input images comprises a first proportion of images containing a particular feature, and at least one other set of the plurality of sets of input images comprises a second proportion of images containing a particular feature, the second proportion being different from the first proportion.

[0035] The first proportion may be all of the images in the set of input images.

[0036] The specific features may be a human face, an animal face, letters, eyes, lips, a logo, a car, a flower, or a pattern.

[0037] Each of the multiple sets of input images may be used the same number of times during the repetition of the method step.

[0038] The difference between the output image and the input image may be determined, at least partially, by a neural network acting as a classifier.

[0039] A separate neural network, acting as a classifier, may be used for each of multiple sets of input images.

[0040] One or more parameters of the neural network acting as a classifier may be updated after a first number of training steps, and one or more other parameters of the neural network acting as a classifier may be updated after a second number of training steps, the second number being smaller than the first number.

[0041] The determined quantity may also be based on a rate associated with the quantized latent, and the parameter update for at least one of several sets of input images may use a first weighting of the rate associated with the quantized latent, and the parameter update for at least one other set of several sets of input images may use a second weighting of the rate associated with the quantized latent, the second weighting being different from the first weighting.

[0042] The difference between the output image and the input image may be determined at least partially using multiple perceptual metrics, the parameter update for at least one of the multiple sets of input images may be done using a first weighting set for the multiple perceptual metrics, and the parameter update for at least one other set of the multiple sets of input images may be done using a second weighting set for the multiple perceptual metrics, the second weighting set being different from the first weighting set.

[0043] The input image may be a modified image in which one or more regions of interest are identified by a third trained neural network, and other areas of the image are masked.

[0044] The area of ​​interest may include one or more features such as human faces, animal faces, letters, eyes, lips, logos, cars, flowers, and patterns.

[0045] The locations of one or more areas of interest may be stored in a binary mask.

[0046] The binary mask may also be an additional input to the first neural network.

[0047] The present invention provides a method for encoding, transmitting and decoding lossy images and videos, the method comprising: receiving an input image in a first computer system; encoding the input image using a first trained neural network to generate a latent representation; performing a quantization process on the latent representation to generate a quantized latent; transmitting the quantized latent to a second computer system; and decoding the quantized latent using a second trained neural network to generate an output image, the output image being an approximation of the input image, the first and second trained neural networks being trained according to the method described above.

[0048] The present invention provides a method for lossy image or video coding and transmission, the method comprising the steps of: receiving an input image in a first computer system; coding the input image using a first trained neural network to generate a latent representation; performing a quantization process on the latent representation to generate a quantized latent; and transmitting the quantized latent, the first trained neural network being trained according to the method.

[0049] The present invention provides a method for receiving and decoding a lossy image or video, the method comprising: receiving a quantized latent in accordance with the method of claim 48 in a second computer system; decoding the quantized latent using a second trained neural network to generate an output image, the output image being an approximation of the input image, the second trained neural network being trained in accordance with the method described above.

[0050] The present invention provides a method for training one or more neural networks, the one or more neural networks for use in encoding, transmitting and decoding lossy images or videos, the method comprising: receiving an input image on a first computer system; encoding the input image using a first neural network to generate a latent representation; performing a quantization process on the latent representation to generate a quantized latent representation; decoding the quantized latent representation using a second neural network to generate an output image, the output image being an approximation of the input image; determining a quantity based on a rate associated with the quantized latent, the evaluation of the rate including the step of interpolating a discrete probability rate-mass function; updating the parameters of the first and second neural networks based on the determined quantity; and repeating the above steps using a set of multiple input images to generate a first and second trained neural network.

[0051] At least one parameter of the discrete probability rate mass function may be additionally updated based on the evaluated rate.

[0052] The method may further include the steps of: encoding a latent representation using a third neural network to generate a hyperlatent representation; performing a quantization operation on the hyperlatent representation to generate a quantized hyperlatent; and decoding the quantized hyperlatent using a fourth neural network to obtain at least one parameter of a discrete probability mass function, wherein the parameters of the third and fourth neural networks are additionally updated based on the determined quantities, and a third trained neural network and a fourth trained neural network are obtained.

[0053] The interpolation may include at least one of the following: piecewise constant interpolation, nearest neighbor interpolation, linear interpolation, polynomial interpolation, spline interpolation, piecewise cubic interpolation, Gaussian processes, and kriging.

[0054] The discrete probability rate mass function can also be a categorical distribution.

[0055] The categorical distribution may be parameterized by at least one vector.

[0056] The categorical distribution may also be obtained by softmax projection of the vector.

[0057] The discrete probability mass function may be parameterized by at least a mean parameter and a scale parameter.

[0058] The discrete probability mass function may be multivariate.

[0059] The discrete probability mass function may be constructed from multiple points, where the first set of adjacent points may have a first interval, and the second set of adjacent points may have a second interval, which is different from the first interval.

[0060] The discrete probability mass function may be constructed from multiple points, where the first set of adjacent points may have a first interval, and the second set of adjacent points may have a second interval, the second of which is equal to the first interval.

[0061] At least one of the first interval and the second interval may be obtained using a fourth neural network.

[0062] At least one of the first interval and the second interval may be obtained based on the value of at least one pixel in the latent representation.

[0063] The present invention provides a method for encoding, transmitting and decoding lossy images and videos, the method comprising: receiving an input image in a first computer system; encoding the input image using a first trained neural network to generate a latent representation; performing a quantization process on the latent representation to generate a quantized latent; transmitting the quantized latent to a second computer system; and decoding the quantized latent using a second trained neural network to generate an output image, the output image being an approximation of the input image, the first and second trained neural networks being trained according to the method described above.

[0064] The present invention provides a method for encoding and transmitting lossy images or videos, the method comprising the steps of: receiving an input image in a first computer system; encoding the input image using a first trained neural network to generate a latent representation; performing a quantization process on the latent representation to generate a quantized latent; and transmitting the quantized latent, the first trained neural network being trained according to the method.

[0065] The present invention provides a method for receiving and decoding a lossy image or video, the method comprising: receiving a latent quantized according to the above method in a second computer system; decoding the quantized latent using a second trained neural network to generate an output image, the output image being an approximation of the input image, the second trained neural network being trained according to the above method.

[0066] The present invention provides a method for encoding, transmitting and decoding lossy images and videos, the method comprising: receiving an input image in a first computer system; encoding the input image using a first trained neural network to generate a latent representation; performing a first operation on the latent representation to obtain a residual latent; transmitting the residual latent to a second computer system; performing a second operation on the residual latent to obtain an acquired latent representation, the second operation comprising performing an operation on previously obtained pixels of the acquired latent; and decoding the acquired latent representation using a second trained neural network to generate an output image, the output image being an approximation of the input image.

[0067] The calculations on the acquired latent previously acquired pixels can be performed for each pixel for which the acquired latent previously acquired pixels are available.

[0068] At least one of the first and second operations may include solving an implicit system of equations.

[0069] The first operation may include a quantization operation.

[0070] The operations performed on the acquired latent previously obtained pixels may include matrix operations.

[0071] The matrix used to define a matrix operation may be sparse.

[0072] The matrix defining the matrix operation may have zero values ​​corresponding to acquired latent pixels that were not acquired when the matrix operation was performed.

[0073] The matrix used to define matrix operations may be a lower triangular matrix.

[0074] The second operation may be constructed using standard forward substitution.

[0075] The operations performed on the acquired latent previously obtained pixels may be constructed from a third trained neural network.

[0076] The method may further include the steps of: encoding a latent representation using a fourth trained neural network to generate a hyperlatent representation; transmitting the quantized hyperlatent to a second computer system; and decoding the quantized hyperlatent using a fifth trained neural network, wherein the operations performed on the previously obtained pixels of the acquired latent are based on the output of the fifth trained neural network.

[0077] Decoding the quantized hyperlatency using a fifth trained neural network may further generate mean parameters, and the implicit equation system may further include mean parameters.

[0078] The present invention provides a method for training one or more neural networks, the one or more neural networks for use in encoding, transmitting, and decoding lossy images or videos, the method comprising: receiving an input image on a first computer system; encoding the input image using a first neural network to generate a latent representation; performing a first operation on the latent representation to obtain a latent residual; performing a second operation on the latent residual to obtain the obtained latent representation, the second operation comprising performing an operation on previously obtained pixels of the obtained latent; decoding the quantized latent using a second neural network to generate an image, the output image being an approximation of the input image; determining a quantity based on the difference between the output image and the input image; updating the parameters of the first and second neural networks based on the determined quantity; and repeating the above steps with a first set of input images to generate a first trained neural network and a second trained neural network.

[0079] The operations performed on the acquired latent previously obtained pixels may include matrix operations.

[0080] The parameters of the matrix that define the matrix operation may be additionally updated based on the determined quantity.

[0081] The operations performed on the acquired latent previously obtained pixels may include a third neural network, the parameters of which may be additionally updated based on determined amounts to generate a third trained neural network.

[0082] The method further includes the steps of: encoding a latent representation using a fourth neural network to generate a hyperlatent representation; performing a quantization process on the hyperlatent representation to generate a quantized hyperlatent representation; transmitting the quantized hyperlatent representation to a second computer system; and decoding the quantized hyperlatent using a fifth neural network, wherein the operations performed on the previously obtained pixels of the acquired latent are based on the output of a fifth trained neural network, and the parameters of the fourth and fifth neural networks are additionally updated based on the determined amount to generate a fourth and fifth trained neural network.

[0083] The present invention provides a method for lossy image or video coding and transmission, the method comprising the steps of: receiving an input image in a first computer system; coding the input image using a first trained neural network to generate a latent representation; performing a first operation on the latent representation to obtain a residual latent; and transmitting the residual latent.

[0084] The present invention provides a method for receiving and decoding a lossy image or video, the method comprising: receiving a residual latent transmitted in accordance with the above method in a second computer system; performing a second operation on the residual latent to obtain an acquired latent representation, the second operation comprising performing an operation on previously obtained pixels of the acquired latent; and decoding the acquired latent representation using a second trained neural network to generate an output image, the output image being an approximation of the input image.

[0085] The present invention provides a method for training one or more neural networks, which are for use in encoding, transmitting, and decoding lossy images or videos, the method comprising the steps of: receiving an input image in a first computer system; encoding the input image using a first neural network to generate a latent representation; entropy encoding the latent representation; transmitting the entropy-encoded latent representation to a second computer system; entropy decoding the entropy-encoded latent representation; and using a second neural network to decode the latent representation The method includes the steps of: decoding the input to generate an output image, the output image being an approximation of the input image; determining a quantity based on the difference between the output image and the input image; updating the parameters of a first neural network and a second neural network based on the determined quantity; and repeating the above steps with a first set of input images to generate a first trained neural network and a second trained neural network, wherein the entropy decoding of the entropy-encoded latent representation is performed pixel by pixel, and the order of pixel by pixel decoding is additionally updated based on the determined quantity.

[0086] The order of decoding each pixel may be based on the latent representation.

[0087] Entropy-encoded latent entropy decoding may involve operations based on previously decoded pixels.

[0088] Determining the order of pixel-by-pixel decoding may involve ordering multiple pixels of the latent representation in a directed aperiodic graph.

[0089] The determination of the order of pixel-by-pixel decoding may involve operating on a latent representation using multiple adjacency matrices.

[0090] Determining the order of pixel-by-pixel decoding may involve dividing the latent representation into multiple subimages.

[0091] Multiple subimages may be obtained by convolving the latent representation with multiple binary mask kernels.

[0092] Determining the order of pixel-by-pixel decoding may involve ranking multiple pixels in the latent representation based on the magnitude of the quantities associated with each pixel.

[0093] The quantities associated with each pixel may be position parameters or scale parameters associated with that pixel.

[0094] The amount associated with each pixel may be additionally updated based on the evaluated difference.

[0095] Determining the order of pixel-by-pixel decoding may involve wavelet decomposition of multiple pixels in the latent representation.

[0096] The order of decoding each pixel may be based on the frequency components of the wavelet decomposition related to multiple pixels.

[0097] The method may further include the steps of: encoding a latent representation using a fourth trained neural network to generate a hyperlatent representation; transmitting the hyperlatent to a second computer system; and decoding the hyperlatent using a fifth trained neural network, wherein the order of pixel-by-pixel decoding is based on the output of the fifth trained neural network.

[0098] The present invention provides a method for encoding, transmitting and decoding lossy images and videos, the method comprising: receiving an input image in a first computer system; encoding the input image using a first trained neural network to generate a latent; entropy encoding the latent; transmitting the entropy-encoded latent to a second computer system; entropy decoding the entropy-encoded latent; and decoding the latent using a second trained neural network to generate an output image, wherein the output image is an approximation of the input image, the first and second trained neural networks being trained according to the method described above.

[0099] The present invention provides a method for lossy image or video coding and transmission, the method comprising the steps of: receiving an input image in a first computer system; coding the input image using a first trained neural network to generate a latent representation; entropy coding the latent representation; and transmitting the entropy coded latent representation, the first trained neural network being trained according to the method.

[0100] The present invention provides a method for receiving and decoding a lossy image or video, the method comprising: receiving an entropy-encoded latent representation transmitted in accordance with the above method in a second computer system; decoding the latent representation using a second trained neural network to generate an output image, the output image being an approximation of the input image, the second trained neural network being trained in accordance with the above method.

[0101] The present invention provides a method for training one or more neural networks, the one or more neural networks for use in encoding, transmitting and decoding lossy images or videos, the method comprising the steps of: receiving an input image on a first computer system; encoding the input image using a first neural network to generate a latent representation; decoding the latent representation using a second neural network to generate an output image, the output image being an approximation of the input image; and determining a quantity based on the difference between the output image and the input image and a rate associated with the latent representation, wherein a first weighting is used when determining the quantity. The method includes the steps of: applying a second weight to the difference between a force image and an input image, and applying a rate associated with the latent representation; updating the parameters of a first neural network and a second neural network based on a determined amount; and repeating the above steps with a first set of input images to generate a first trained neural network and a second trained neural network, wherein after at least one iteration of the above steps, at least one of the first weight and the second weight is additionally updated based on a further amount, the further amount based on at least one of the difference between an output image and an input image and a rate associated with the latent representation.

[0102] At least one of the difference between the output image and the input image and the rate associated with the latent representation may be recorded for each iteration of the step, and further amounts may be based on at least one of multiple previously recorded differences between the output image and the input image and multiple previously recorded rates associated with the latent representation.

[0103] Further quantities may be based on the average of multiple previously recorded differences or rates.

[0104] The mean may be at least one of the following: arithmetic mean, median, geometric mean, harmonic mean, exponential moving average, smoothed moving average, or linear weighted moving average.

[0105] Outliers may be removed from previously recorded differences or rates before determining further amounts.

[0106] Outliers may be removed only for the first predetermined number of repetitions of the step.

[0107] The rate associated with the latent expression may be calculated using the first method when determining the quantity, and the second method when determining further quantities, the first method being different from the second method.

[0108] At least one iteration of the step may be performed using input images from a second set of input images, in which case the parameters of the first and second neural networks do not need to be updated further.

[0109] The determined quantity may also be based on the output of a neural network that acts as a classifier.

[0110] The present invention provides a method for training one or more neural networks, the one or more neural networks for use in encoding, transmitting, and decoding lossy video, the method comprising the steps of: receiving an input video on a first computer system; encoding multiple frames of the input video using a first neural network to generate multiple latent representations; decoding the multiple latent representations using a second neural network to generate multiple frames of output video, the output video being an approximation of the input video; and determining a quantity based on the difference between the output image and the input image and a rate associated with the multiple latent representations, wherein the first weighting is the output image The method includes the steps of: applying a second weight to the difference between an image and an input image, where the second weight is applied to a rate associated with multiple latent representations; further updating the parameters of a first neural network and a second neural network based on a determined amount; and repeating the above steps with multiple input images to generate a first trained neural network and a second trained neural network, wherein after at least one of the repetitions of the above steps, at least one of the first weight and the second weight is further updated based on a further amount, where the further amount is based on the difference between an output image and an input image and at least one of a rate associated with multiple latent representations.

[0111] The input video contains at least one I-frame and multiple P-frames.

[0112] The quantity may be based on multiple first or second weights, each of which corresponds to one of multiple frames of the input video.

[0113] After at least one iteration of the step, at least one of the weights may be additionally updated based on an additional amount associated with each weight.

[0114] Each additional amount may be based on a predetermined target value of the difference between the output frame and the input frame, or on a rate associated with the latent representation.

[0115] An additional amount associated with an I-frame may have a first target value, and at least one additional amount associated with a P-frame may have a second target value, the second target value being different from the first target value.

[0116] Each additional quantity associated with the P-frame may have the same target value.

[0117] Multiple first or second weights may be initially set to zero.

[0118] According to the present invention, a method for encoding and transmitting and decoding lossy images and videos is provided, the method comprising the steps of: receiving an input image in a first computer system; encoding the input image using a first trained neural network to generate a latent representation; performing a quantization process on the latent representation to generate a quantized latent; transmitting the quantized latent to a second computer system; and decoding the quantized latent using a second trained neural network to generate an output image, wherein the output image is an approximation of the input image, the first and second trained neural networks are trained according to the above method.

[0119] The present invention provides a method for lossy image or video coding and transmission, the method comprising the steps of: receiving an input image or video in a first computer system; coding the input image or video using a first trained neural network to generate a latent representation; and transmitting the latent representation, the first trained neural network being trained according to the method.

[0120] The present invention provides a method for receiving and decoding a lossy image or video, the method comprising: receiving a latent representation in a second computer system according to the above method; decoding the latent representation using a second trained neural network to generate an output image or video, wherein the output image or video is an approximation of the input image or video, the second trained neural network is trained according to the above method.

[0121] The present invention provides a method for encoding, transmitting and decoding lossy images or videos, the method comprising: receiving an input image in a first computer system; encoding the input image using a first trained neural network to generate a latent representation; performing a quantization process on the latent representation to generate a quantized latent; entropy encoding the quantized latent using a probability distribution, the probability distribution being defined using a tensor network; transmitting the entropy-encoded quantized latent to a second computer system; entropy decoding the entropy-encoded quantized latent using a probability distribution to obtain a quantized latent; and decoding the quantized latent using a second trained neural network to generate an output image, the output image being an approximation of the input image.

[0122] The probability distribution may also be defined by Hermitian operators acting on a quantized latent, and these Hermitian operators are defined by a tensor network.

[0123] A tensor network may include a non-normal core tensor and one or more orthonormal tensors.

[0124] The method may further include the steps of: encoding a latent representation using a third trained neural network to generate a hyperlatent representation; performing a quantization process on the hyperlatent representation to generate a quantized hyperlatent representation; transmitting the quantized hyperlatent representation to a second computer system; and decoding the quantized hyperlatent representation using a fourth trained neural network, wherein the output of the fourth trained neural network is one or more parameters of a tensor network.

[0125] The tensor network may consist of a non-normalized core tensor and one or more orthonormal tensors, and the output of the fourth trained neural network may be one or more parameters of the non-normalized core tensor.

[0126] One or more parameters of the tensor network may be computed using one or more pixels of the latent representation.

[0127] The probability distribution may also relate to a subset of pixels in the latent representation.

[0128] The probability distribution may also be related to the channel of the latent representation.

[0129] A tensor network may be at least one of a tensor tree, locally purified states, a bone machine, matrix product states, and projective entangled pair state factorization.

[0130] The present invention provides a method for training one or more networks, which are used for encoding, transmitting, and decoding lossy images or videos, and includes the steps of: receiving a first input image; encoding the first input image using a first neural network to generate a latent representation; performing a quantization process on the latent representation to generate a quantized latent; entropy coding the quantized latent using a probability distribution, wherein the probability distribution is defined using a tensor network; entropy decoding the entropy coded quantized latent using the probability distribution to obtain a quantized latent; decoding the quantized latent amount using a second neural network to generate an output image, wherein the output image is an approximation of the input image; determining a quantity based on the difference between the output image and the input image; updating the parameters of the first and second neural networks based on the determined quantity; and repeating the above steps using a plurality of input images to generate a first trained neural network and a second trained neural network.

[0131] One or more parameters of the tensor network may be additionally updated based on the determined quantity.

[0132] A tensor network may include a non-normal core tensor and one or more orthonormal tensors, and the parameters of all tensors in the tensor network except the non-normal core tensor may be updated based on a determined quantity.

[0133] Tensor networks may be computed using latent representations.

[0134] The tensor network may be computed based on linear interpolation of the latent representation.

[0135] The determined quantity may also be based on the entropy of the tensor network.

[0136] The present invention provides a method for lossy image or video coding and transmission, the method comprising: receiving an input image in a first computer system; coding the input image using a first trained neural network to generate a latent representation; performing a quantization process on the latent representation to generate a quantized latent; entropy coding the quantized latent using a probability distribution, the probability distribution being defined using a tensor network; and transmitting the entropy-coded quantized latent.

[0137] The present invention provides a method for receiving and decoding a lossy image or video, the method comprising: receiving an entropy-coded quantization latent transmitted in accordance with the above method in a second computer system; entropy-decoding the entropy-coded quantization latent using a probability distribution to obtain a quantization latent; and decoding the quantization latent using a second trained neural network to generate an output image, wherein the output image is an approximation of the input image.

[0138] The present invention provides a method for encoding, transmitting and decoding lossy images and videos, the method comprising: receiving an input image on a first computer system; encoding the input image using a first trained neural network to generate a latent representation; encoding the latent representation using a second trained neural network to generate a hyper-latent representation; encoding the hyper-latent representation using a third trained neural network to generate a hyper-hyper-latent representation; transmitting the latent representation, hyper-latent representation and hyper-hyper-latent representation to a second computer system; decoding the hyper-hyper-latent representation using a fourth trained neural network; decoding the hyper-hyper-latent representation using the output of the fourth trained neural network and a fifth trained neural network; and decoding the latent representation using the output of the fifth trained neural network and a sixth trained neural network to generate an output image, wherein the output image is an approximation of the input image.

[0139] This method may further include a step of determining the rate of the input image, and if the determined rate satisfies predetermined conditions, the steps of encoding the hyperlatent representation and decoding the hyperlatent representation are not performed.

[0140] If the determined rate satisfies the predetermined conditions, the steps of encoding the hyperflat representation and decoding the hyperflat representation are not performed.

[0141] The present invention provides a method for training one or more networks, the one or more networks for use in encoding, transmitting, and decoding lossy images or videos, the method comprising: receiving an input image on a first computer system; encoding the input image using a first neural network to generate a latent representation; encoding the latent representation using a second neural network to generate a hyper-latent representation; encoding the hyper-latent representation using a third neural network to generate a hyper-hyper-latent representation; decoding the hyper-hyper-latent representation using a fourth neural network; decoding the hyper-latent representation using the output of the fourth neural network and a fifth neural network; decoding the latent representation using the output of the fifth neural network and a sixth neural network to generate an output image, wherein the output image is an approximation of the input image; determining a quantity based on the difference between the output image and the input image; updating the parameters of the third and fourth neural networks based on the determined quantity; and repeating the above steps using a plurality of input images to generate third and fourth trained neural networks.

[0142] The parameters of the first, second, fifth, and sixth neural networks do not need to be updated in at least one of the step iterations.

[0143] The method may further include a step of determining the rate of the input image, and if the determined rate satisfies a predetermined condition, the parameters of the first, second, fifth, and sixth neural networks are not updated in that iteration of the step.

[0144] A predetermined condition may be that the rate is below a predetermined value.

[0145] The parameters of the first, second, fifth, and sixth neural networks do not need to be updated after a predetermined number of steps have been repeated.

[0146] The parameters of the first, second, fifth, and sixth neural networks may be additionally updated based on the determined quantities to generate the first, second, fifth, and sixth trained neural networks.

[0147] Before performing other steps, at least one of the multiple input images may be subjected to at least one of the following processes: upsampling, smoothing filter, and random cropping.

[0148] The present invention provides a method for image or video encoding and transmission, the method comprising the steps of: receiving an input image on a first computer system; encoding the input image using a first trained neural network to generate a latent representation; encoding the latent representation using a second trained neural network to generate a hyper-latent representation; encoding the hyper-latent representation using a third trained neural network to generate a hyper-hyper-latent representation; and transmitting the latent representation, the hyper-latent representation, and the hyper-hyper-latent representation.

[0149] The present invention provides a method for receiving and decoding a lossy image or video, the method comprising: receiving a latent representation, a hyper-latent representation and a hyper-hyper-latent representation transmitted according to the above method in a second computer system; decoding the hyper-hyper-latent representation using a fourth trained neural network; decoding the hyper-latent representation using the outputs of the fourth and fifth trained neural networks; and decoding the latent representation using the outputs of the fifth and sixth trained neural networks to generate an output image, wherein the output image is an approximation of the input image.

[0150] According to the present invention, a data processing system is provided which is constructed to perform any of the above methods.

[0151] According to the present invention, a data processing device is provided which is constructed to perform any of the above methods.

[0152] According to the present invention, a computer program is provided that includes instructions causing a computer to perform one of the above methods when the program is executed by the computer.

[0153] According to the present invention, a computer-readable storage medium is provided which, when executed by a computer, includes instructions that cause the computer to perform any of the above methods. [Brief explanation of the drawing]

[0154] Herein, embodiments of the present invention will be described with reference to the following figures and examples. [Figure 1] This shows an example of an image or video compression, transmission, and decompression pipeline. [Figure 2] Further examples of image or video compression, transmission, and decompression pipelines, including hypernetworks, are presented. [Figure 3] This diagram shows a schematic representation of the coding phase of an example Al-based compression algorithm for video and image compression. [Figure 4] This diagram shows a schematic representation of the decoding phase of an example of an AI-based compression algorithm for video and image compression. [Figure 5] This shows an example of the distribution of quantization bin size values ​​that can be learned during training of an Al-based compression pipeline. [Figure 6] A heatmap is shown illustrating how the size of the learned quantization bins changes across latent channels for a given image. [Figure 7]A schematic diagram illustrating the coding phase of an Al-based compression algorithm utilizing hyperpliers and learned quantization bin sizes is shown. [Figure 8] A schematic diagram illustrating the decryption phase of an Al-based compression algorithm utilizing hyperpliers and learned quantization bin sizes is shown. [Figure 9] Here are some examples of inverse gamma distributions. [Figure 10] This section provides an example overview of a GAN architecture. [Figure 11] An example of a standard generative adversarial compression pipeline is shown. [Figure 12] This shows the failure modes of an architecture combining a GAN and an autoencoder. [Figure 13] This example demonstrates a compressed pipeline using a multi-classifier cGAN training with dataset bias. [Figure 14] This shows a comparison of rebuilding the same generative model trained at the same bitrate, with and without using a multi-discrimination dataset biasing scheme. [Figure 15] Here is an example of the results of bitrate-adjusted dataset bias. [Figure 16] An example of an Al-based compression pipeline is shown. [Figure 17] Further examples of Al-based compression pipelines are shown. [Figure 18] This example shows a compression pipeline with quantization using a quantization map. [Figure 19] This example shows the result of implementing a pipeline with a face detector used for identifying regions of interest. [Figure 20] The following shows different quantization functions Qm for the regions identified by the ROI detection network H(x). [Figure 21] Here are three examples of typical one-dimensional distributions that can be used in training an Al-based compression pipeline. [Figure 22] This section compares piecewise linear interpolation with piecewise cubic Hermitian interpolation. [Figure 23]This example demonstrates how an autoregressive structure defined by a sparse context matrix L can be used to parallelize components of a serial decoding path. [Figure 24] This example demonstrates the encoding process using the prediction context matrix Ly in an Al-based compression pipeline. [Figure 25] This example demonstrates the decoding process using the prediction context matrix Ly in an AI-based compression pipeline. [Figure 26] An example of a raster scan order for a single-channel image is shown. [Figure 27] Instead of all preceding variables, the next pixel shows a 3x3 receptive field conditioned on the local variable. [Figure 28] Here is an example of a DAG that describes the joint distribution of four variables {y1, y2, y3, y4}. [Figure 29] Here is an example of a two-step analytical algorithm (AO) where all current variables at each step are conditionally independent and can be evaluated in parallel. [Figure 30] The directed graph corresponding to AO in Figure 29 is shown. [Figure 31] This example shows how a 2x2 binary mask kernel that conforms to constraints (61) and (62) generates four subimages. [Figure 32] An example of an adjacency matrix A that determines the graph connectivity of an AO as defined by the binary mask kernel framework is shown. [Figure 33] This shows the index for the Adam7 interlacing method. [Figure 34] An example of visualizing the scale parameter σ, where AO is defined as y1, y2, ..., y16, is shown. [Figure 35] An example of permutation matrix representation is shown. [Figure 36] This example illustrates the concept of a ranking table as applied to the binary mask kernel framework. [Figure 37] This example demonstrates hierarchical autoregressive ordering based on wavelet transforms. [Figure 38]This diagram illustrates various tensors and their products. [Figure 39] Examples of 3-tensor decomposition and matrix product states are shown graphically. [Figure 40] An example of a Locally Purified State is shown graphically. [Figure 41] An example of a tensor tree is shown graphically. [Figure 42] An example of a 2x2 Projected Entangled Pair State is shown graphically. [Figure 43] An example of the procedure for converting a matrix product state into a canonical form is shown graphically. [Figure 44] This example shows an image or video compression pipeline with a tensor network predicted by a hyperencoder / hyperdecoder. [Figure 45] This shows an image or video decompression pipeline with a tensor network predicted by a hyperdecoder.

[0155] Compression can be applied to any form of information to reduce the amount of data required to store the information, i.e., the file size. Image and video information are examples of information that can be compressed. The file size required to store the information is sometimes called the rate during the compression process, specifically referring to the compressed file. Generally, there are lossless and lossy compression methods. Both compression methods reduce the file size. However, with lossless compression, no information is lost when the information is compressed and then decompressed. In other words, during decompression, the original file that stored the information is completely reconstructed. In contrast, with lossy compression, information is lost during the compression and decompression process, and the reconstructed file may differ from the original file. Image and video files containing image and video data are common targets for compression. JPEG, JPEG2000, AVC, HEVC, and AVI are examples of image and video file compression methods.

[0156] In image compression, the input image is sometimes represented as x. Image data can be stored in a tensor of dimension H×W×C, where H represents the image height, W represents the image width, and C represents the number of channels. Each H×W data point in the image represents the pixel value of the image at its corresponding location. Each channel C in the image represents a different component of the image for each pixel, which is combined when the image file is displayed by a device. For example, an image file might have three channels, each representing the red, green, and blue components of the image. In this case, the image information is stored in the RGB color space, sometimes called a model or format. Other examples of color spaces and formats include the CMKY and YCbCr color models. However, the channels in an image file are not limited to storing color information; other information can also be represented by channels. Since video can be considered a sequence of images, compression processes applied to images may also apply to video. Each image that makes up a video is sometimes called a video frame.

[0157] Video frames may be labeled according to their nature. For example, video frames may be labeled as I-frames and P-frames. An I-frame may be the first frame of a new section of video. For example, the first frame after a scene transition may be labeled an I-frame. A P-frame may be a subsequent frame after an I-frame. For example, the background or objects present in a P-frame may not change until the video progresses from an I-frame to a P-frame. Changes in a P-frame compared to an I-frame progressing through it may be described by the movement of objects within the frame or by the perspective movement of the frame.

[0158] The output image may differ from the input image. The image may be represented as JPEG0007860563000001.jpg75 (hereinafter also referred to as "x (hat)"; hereinafter, a character with a ^ symbol above it may also be referred to as a character (hat)). The difference between the input image and the output image is sometimes called distortion or quality difference. Distortion can be measured using any distortion function that takes the input image and the output image as inputs and provides an output that numerically represents the difference between the input image and the output image. An example of such a method is to use the mean squared error (MSE) between pixels of the input and output images, but as is well known to those skilled in the art, there are many other ways to measure distortion. The distortion function may include a trained neural network.

[0159] In general, the compression rate and distortion in lossy compression are related. An increase in rate leads to a decrease in distortion, and a decrease in rate leads to an increase in distortion. Changes in distortion can, in turn, affect the rate. The relationship between these quantities for a given compression technique can be defined by the rate-distortion equation.

[0160] Al-based compression processes may involve the use of neural networks. A neural network is a set of operations that can be performed on an input and produce an output. A neural network can be constructed with multiple layers. The first layer of the network receives the input. Each layer performs one or more operations on the input, producing the output of the first layer. The output of the first layer is then passed to the next layer of the network, where one or more operations are performed in a similar manner. The output of the final layer becomes the output of the neural network.

[0161] Each layer of a neural network may be divided into nodes. Each node may receive at least a portion of the input from the previous layer and provide an output to one or more nodes in a subsequent layer. Each node in a layer may perform one or more operations on at least a portion of the input to the layer. For example, a node may receive input from one or more nodes in the previous layer. One or more operations may include convolution, weights, biases, and activation functions. Convolution operations are used in convolutional neural networks. If convolution operations are present, the convolution may be performed on the entire input to the layer. Alternatively, convolution may be performed on at least a portion of the input to the layer.

[0162] Each of the one or more operations is defined by one or more parameters associated with each operation. For example, a weight operation may be defined by a weight matrix that defines the weights applied to each input from each node of the previous layer to each node of the current layer. In this example, each value in the weight matrix is ​​a parameter of the neural network. A convolution can be defined by a convolution matrix, also known as a kernel. In this example, one or more values ​​in the convolution matrix may be parameters of the neural network. The activation function may also be defined by values ​​that can be parameters of the neural network. The network parameters may be changed during training of the network.

[0163] Other features of a neural network may be predetermined and therefore do not change during network training. For example, the number of layers in the network, the number of nodes in the network, the one or more operations performed in each layer, and the connections between layers may be predetermined and therefore fixed before the training process takes place. These predetermined features are sometimes called the network's hyperparameters. These features are sometimes called the network's architecture.

[0164] To train a neural network, a training set of inputs with known expected outputs (sometimes called ground truth) can be used. The initial parameters of the neural network are randomized, and the first training input is provided to the network. The network's output is compared to the expected output, and based on the difference between the output and the expected output, the network's parameters are modified so that the difference between the network's output and the expected output is small. This process is repeated for multiple training inputs to train the network. The difference between the network's output and the expected output can be defined by a loss function. The result of the loss function is calculated using the difference between the network's output and the expected output, and the gradient of the loss function can be determined. Gradient descent backpropagation of the loss function may be used to update the neural network's parameters using the gradient dL / dy of the loss function. Multiple neural networks in a system may be trained simultaneously by backpropagating the gradient of the loss function to each network.

[0165] For Al-based image or video compression, the loss function can be defined by the rate-distortion equation. The rate-distortion equation can be expressed as Loss = D + λ*R, where D is the distortion function, λ is the weighting coefficient, and R is the rate loss. The Lagrange multiplier is provided as the weight of a particular term in the loss function relative to each of the other terms and can be used to control which terms of the loss function are favored when training the network.

[0166] For AI-based image or video compression, a training set of input images can be used. Examples of training sets of input images include the Kodak image set (e.g., www.cs.albany.edu / xypan / research / snr / Kodak.html), the IMAX image set, the Imagenet dataset (e.g., www.image-net.org / download), and the CLIC Training Dataset P ("professional") and M ("mobile") (e.g., http: / / challenge.compression.cc / tasks / ).

[0167] Figure 1 shows an example of Al-based compression processing 100. As the first step of Al-based compression processing, an input image 5 is provided. The input image 5 is then subjected to a function f acting as an encoder. θ This is provided to a trained neural network 110 characterized by the following. The encoder neural network 110 generates an output based on the input image. This output is called the latent representation of the input image 5. In the second step, the latent representation is quantized in a quantization process 140 characterized by operation Q, resulting in a quantized latent representation. The quantization process converts the continuous latent representation into a discrete quantized latent representation. An example of a quantization process is a rounding function.

[0168] In the third step, the quantized latent quantity is entropically coded using an entropy coding process 150 to generate a bitstream 130. The entropy coding process may be, for example, range coding or arithmetic coding. In the fourth step, the bitstream 130 may be transmitted over a communication network.

[0169] In the fifth step, the bitstream is entropy-decoded in the entropy decoding process 160. The quantized latent quantity is then decoded by a function g which acts as a decoder for decoding the quantized latent quantity.θ This is provided to another trained neural network 120 characterized by the following. The trained neural network 120 generates an output based on the quantized latent. The output may be the output image of an Al-based compression process 100. The encoder-decoder system may also be called an autoencoder.

[0170] The system described above may be distributed across multiple locations and / or devices. For example, the encoder 110 may be located on a device such as a laptop computer, desktop computer, smartphone, or server. The decoder 120 may be located on another device called a receiver device. The system used to encode, transmit, and decode the input image 5 in order to obtain the output image 6 may be called a compression pipeline.

[0171] The Al-based compression process may further include a hypernetwork 105 for transmitting metadata to improve the compression process. The hypernetwork 105 is a hyperencoder. A trained neural network 115 that functions as JPEG0007860563000002.jpg78 and a hyperdecoder It is constructed from a trained neural network 125 that functions as JPEG0007860563000003.jpg76. An example of such a system is shown in Figure 2. The building elements of the system that are not described further can be assumed to be the same as those described above. The neural network 115, which functions as a hyperdecoder, receives the latent, which is the output of the encoder 110. The hyperencoder 115 generates an output based on the latent representation, which is sometimes called the hyperlatent representation. The hyperlatent is then Q h In the quantization process 145 characterized by Q, a quantized hyperlatency is generated. h The quantization process 145 characterized by the above may be the same as the quantization process 140 characterized by Q.

[0172] In a similar manner to that described above for the quantized latent, the quantized hyperlatent is then entropically coded in the entropy coding process 155 to produce a bitstream 135. The bitstream 135 is entropically decoded in the entropy decoding process 165 to extract the quantized hyperlatent. The quantized hyperlatent is then used as input to a trained neural network 125 that acts as a hyperdecoder. However, in contrast to the compression pipeline 100, the output of the hyperdecoder may not be an approximation of the input to the hyperdecoder 115. Instead, the output of the hyperdecoder is used to provide parameters for use in the entropy coding process 150 and entropy decoding process 160 of the main compression process 100. For example, the output of the hyperdecoder 125 may include the mean, standard deviation, variance, or one or more of any other parameters used to describe the probabilistic model of the latent 10 representation in the entropy coding process 150 and entropy decoding process 160. In the example shown in Figure 2, for simplicity, only a single entropy decoding process 165 and hyperdecoder 125 are shown. However, in reality, decompression is usually performed on a separate device, so copies of these processes would exist on the device used for encoding to provide the parameters used in the entropy coding process 150.

[0173] At any stage of the Al-based compression process 100, further transformations can be applied to at least one of the latent and hyperlatent values. For example, at least one of the latent and hyperlatent values ​​may be transformed into residual values ​​before the entropy coding processes 150, 155 are performed. The residual values ​​may be determined by subtracting the mean of the distribution of the latent or hyperlatent values ​​from each latent or hyperlatent. Alternatively, the residual values ​​20 may be normalized.

[0174] To train the AI-based compression process described above, a training set of input images may be used as described above. During the training process, the parameters of both the encoder 110 and the decoder 120 may be updated simultaneously at each training step. If a hypernetwork 105 is also present, the parameters of both the hyperencoder 115 and the hyperdecoder 125 are additionally updated simultaneously at each training step.

[0175] The training process can also include a Generative Adversarial Network (GAN). When applied to AI-based compression, in addition to the compression pipeline described above, an additional neutral network acting as a discriminator is included in the system. The discriminator receives an input and outputs a score based on the input, providing an indicator of whether the discriminator considers the input to be ground truth or fake. For example, this indicator is a score, where a high score is associated with true inputs and a low score is associated with fake inputs. The discriminator is trained using a loss function that maximizes the difference in output representation between the input ground truth and the input fake.

[0176] If a GAN is incorporated into the training of the compression process, the output image 6 may be provided to a classifier. The output of the classifier may be used in the compression loss function as a measure of the distortion of the compression process. Alternatively, the classifier may receive both the input image 5 and the output image 6, and the difference in the output representations may be used in the compression loss function as a measure of the distortion of the compression process. The training of the neural network acting as the classifier and the other neural networks in the compression process can be performed simultaneously. During the use of the trained compression pipeline for image or video compression and transmission, the classifier neural network is removed from the system, and the output of the compression pipeline is the output image 6.

[0177] When a GAN is incorporated into the training process, the decoder 120 may perform hallucinations. Hallucination is the process of adding information to the output image 6 that was not present in the input image 5. For example, fine details that were not present in the input image 5 or were not received by the decoder 120 may be added to the output image 6. The hallucinations performed may be based on the quantized latent information received by the decoder 120.

[0178] As described above, the video is constructed from a series of sequentially arranged images. The video may be compressed, transmitted, and decompressed by applying the Al-based compression process 100 described above multiple times. For example, each frame of the video may be compressed, transmitted, and decompressed individually. The received frames may then be grouped to obtain the original video.

[0179] Here, we will explain many of the concepts related to the Al compression process described above. Each concept will be explained individually, but one or more of the following concepts may be applied in the Al-based compression process described above.

[0180] Learned quantization Quantization is a crucial step in AI-based compression pipelines. Generally, quantization is achieved by rounding data to the nearest integer, because some areas of images and videos can tolerate higher information loss, while others require finer detail. Below, we discuss how the size of the quantum quantization bins can be learned rather than fixed to rounding to the nearest integer. Several architectures to achieve this are detailed, including hypernetworks, context modules, and predicting bin sizes from additional neural networks. We also document the necessary changes to the loss function and quantization procedure required to train an AI-based compression pipeline with the learned quantization bin size, and show how to introduce Bayesian pliers to control the distribution of bin sizes learned during training. We show that the learned quantization bins can be used with or without split quantization. This innovation also allows strain gradients to flow through the decoder to the hypernetwork. Finally, we discuss generalized quantization functions that improve performance and execution time. Specifically, this innovation allows the inclusion of a context model in the decoder of the compression pipeline without incurring runtime penalties from repeatedly executing arithmetic (or other lossless) decoding algorithms. Our method for learning quantization bins is compatible with any method of transmitting metadata, including hyperpliers, autoregressive models, and implicit models.

[0181] The following description outlines, but is not limited to, the capabilities, scope, and future prospects of trained quantization bins and generalized quantization functions used in AI-based image and video compression.

[0182] A compression algorithm can be divided into two phases: an encoding phase and a decoding phase. In the encoding phase, the input data is transformed into latent variables that have a smaller representation (in bits) than the original input variables. In the decoding phase, the inverse transformation is applied to the latent variables to restore the original data (or an approximation of the original data).

[0183] Algorithmic compression systems must also be trained. This is the process of selecting parameters for an Algorithmic compression system that achieve good compression results (small file size and minimal distortion). During training, parts of the coding and decoding algorithms are executed to determine how to tune the parameters of the Algorithmic compression system.

[0184] More precisely, in Al-based compression, encoding generally takes the following form:

number

[0185] Here, x is the data to be compressed (image or video), and f enc An encoder is typically a neural network with trained parameters θ. The encoder transforms the input data x into a latent representation y in a lower-dimensional and more compressed form.

[0186] To further compress y and transmit it as a bit stream, established lossless coding algorithms such as arithmetic coding can be used. Such lossless coding algorithms may require that y be discrete rather than continuous, and may also require knowledge of the probability distribution of the latent representation. To achieve this, a quantification function Q (usually rounded to the nearest integer) is used to convert continuous data into discrete values ​​y (hats).

[0187] The required probability distribution p(y(hat)) is obtained by fitting the probability distribution to the latent space. While the probability distribution can be learned directly, it is often a parameterized distribution with parameters determined by a hypernetwork consisting of a hyperencoder and a hyperdecoder. When using a hypernetwork, an additional bitstream z(hat) (also called "side information") may be encoded, transmitted, and decoded:

Number

[0188] The encoding process (using the hypernetwork) is shown in Fig. 3. Fig. 3 is a schematic illustration of an example of the encoding phase of an Al-based compression algorithm for video and image compression.

[0189] Decoding is performed as follows:

Number

[0190] Summary: In an arithmetic decoder (or other lossless decoding algorithm), the latent distribution p(y(hat)) is used and the bitstream is converted to the quantized latent y(hat). Then, the function f dec converts the quantized latent data into a loss reconstruction of the input data denoted by x(hat). In Al-based compression, f dec is usually a neural network depending on the learned parameter θ.

[0191] When using a hypernetwork, first decode the side information bitstream, and then use it to obtain the parameters necessary to construct the required p(y(hat)) for decoding the main bitstream. An example of the decoding process using a hypernetwork is shown in Fig. 4. Fig. 4 is an example of the decoding phase of an Al-based compression algorithm for video and image compression.

[0192] Al-based compression relies on learning the parameters of encoding and decoding neural networks using typical optimization techniques with a “loss function.” The loss function is chosen to balance the goal of compressing images or videos to a small file size while maximizing reconstruction quality. Therefore, the loss function consists of two terms:

number

[0193] Here, R determines the cost of encoding the quantified latent according to the distribution p(y(hat)), D measures the quality of the reconstructed image, and λ is a parameter that determines the trade-off between low file size and reconstruction quality. A typical choice for R is cross-entropy.

number

[0194] The selection of JPEG0007860563000009.jpg712 is for quantization, and since the latent is rounded to the nearest integer, the probability distribution of p(y(hat)) is given by the integral of the (unquantized) latent distribution p(y) from y(hat)-1 / 2 to y(hat)+1 / 2, which is given by the cumulative distribution function, and this is given by the term of the cumulative distribution function p(y(hat)).

[0195] Function D can be chosen as mean squared error, but it can also be a combination of other metrics of perceived quality such as MS-SSIM, LPIPS, and / or (if using adversarial neural networks to enforce image quality) adversarial loss.

[0196] When using hypernetworks, an additional term can be added to R to represent the cost of transmitting additional side information:

number

[0197] In summary, note that the loss function explicitly depends on the choice of quantization scheme through the R term, and y (hat) also depends on the choice of quantization scheme, thus implicitly depending on the choice.

[0198] Here, we discuss how learned quantization bins can be used in AI-based image and video compression. The steps described are as follows: • Architecture required for learning and predicting the size of quantization bins • Modification of standard quantization functions and encoding / decoding processes to incorporate learned quantization bins. • A method for training a neural network using learned quantization bins.

[0199] A key step in a typical AI-based image and video compression pipeline is "quantization," where pixels in the latent representation are typically rounded to the nearest integer. This is necessary for algorithms that encode bitstreams losslessly. However, the quantization step itself introduces information loss, affecting reconstruction quality.

[0200] It is possible to improve the quantization function by training a neural network to predict the size of the quantization bin that should be used for each latent pixel. Typically, the latent y is rounded to the nearest integer, which corresponds to a "bin size" of 1. That is, all possible values ​​of y in an interval of length 1 map to the same y (hat).

number

[0201] However, this may not be the optimal choice for information loss. For some latent pixels, more information can be ignored without significantly impacting reconstruction quality (equivalently: using a bin larger than 1). Also, for other latent pixels, the optimal bin size is smaller than 1.

[0202] This problem can be solved by predicting the quantization bin size for each image and each pixel. This is done for the tensor JPEG0007860563000012.jpg636, and the quantization function is modified as follows.

Number

[0203] This is referred to as the "quantization latent residue" and is called JPEG0007860563000014.jpg87. Therefore, Equation 7 becomes as follows.

Number

[0204] Figure 6 is a heatmap showing how the learned quantization bin sizes vary across the latent channels for a given image. Different pixels are predicted to receive the benefit of larger or smaller quantization bins, corresponding to larger or smaller information losses.

[0205] Note that since the learned quantization bin sizes are incorporated into the modification of the quantization function Q, any data that we want to encode and transmit can utilize the learned quantization bin sizes. For example, instead of encoding the latent y, if we want to encode the mean-subtracted latent y - μ y this can be achieved.

Number

[0206] Similarly, the hyper latent, hyper-hyper latent, and other objects that we want to quantize can all be modified quantization functions Q for appropriately learned ΔΔ You can use it.

[0207] Here, we discuss several architectures for predicting the size of the quantization bin. A possible architecture is to use a hypernetwork to predict the quantization bin size Δ. The bitstream is encoded as follows:

number

[0208] Figure 7 is an example of a modified encoding process using a hypernetwork, and is a schematic diagram showing an example of the encoding phase of an Al-based compression algorithm using a hyperplier and learned quantization bin size for video and image compression.

[0209] During decoding, the bitstream is decoded reversibly as usual. Then, it is rescaled by multiplying two elements using Δ ξ. y This process is performed. The result of this conversion is denoted as y (hat) and passed to the decoder network as usual.

number

[0210] Figure 8 shows an example of a modified decoding process using a hypernetwork. Figure 8 is a schematic diagram illustrating an example of the decoding phase of an Al-based compression algorithm using a hyperplier and learned quantization bin size, for video and image compression.

[0211] Applying the above techniques can potentially improve the rate of Al-based compression pipelines by 1.5% and reduce strain as measured by MSE by 1.9%. Therefore, the performance of Al-based compression is improved.

[0212] Several variations of the above architecture will be explained in detail. The size of the hyperlatency quantization bins can be the learned parameters, and can also include a hyper-hypernetwork that predicts these variables. • Predictions of quantization bin sizes can be augmented using a so-called "context module" feature, which uses neighboring pixels to improve the prediction for a particular pixel. • Quantization bin size Δ from hyperdecoder y After obtaining the tensor, this tensor can be further processed with a nonlinear function. This nonlinear function is generally a neural network (but is not limited to this choice).

[0213] Furthermore, we emphasize that our method for learning quantization bins is compatible with all methods of conveying metainformation, including hyperpliers, hyperhyperpliers, autoregressive models, and implicit models.

[0214] To train a neural network with quantization bins learned for Al-based compression, we can modify the loss function. Specifically, the cost of encoding the data described in equation 5a can be modified as follows:

number

[0215] Instead of integrating over an interval of length 1, From JPEG0007860563000022.jpg616 This means that we need to integrate the probability distribution of the materials up to JPEG0007860563000023.jpg516.

[0216] Similarly, when using a hypernetwork, terms corresponding to the hyper-latent encoding cost JPEG0007860563000024.jpg537 is modified in exactly the same way as the latent encoding cost in order to incorporate the learned quantization bin size.

[0217] Neural networks are typically trained by a variant of gradient descent that uses backpropagation to update the parameters of the learned network. For this, it is necessary to calculate the gradients of all layers within the network, and for this, the layers of the network need to be constructed with differentiable functions. However, the quantization function Q and its learned bin modification Q Δ are not differentiable because of the existence of the rounding function. In AI-based compression, one of two differentiable approximations to quantization is used during network training to replace Q(y) (the approximation is not used when the network is trained and used for inference).

Number

[0218] When using the learned quantization bins, the quantization approximation during training is as follows.

Number

[0219] Instead of selecting any of these differentiable approximations during training, the AI-based compression pipeline uses JPEG0007860563000027.jpg613 when calculating the rate loss R, but it can also be trained using "split quantization" where the decoder is sent JPEG0007860563000028.jpg612 during training. The AI-based compression network can be trained with both split quantization and learned quantization bins.

[0220] First, note that there are two types of partitioned quantization. • Hard partition quantization: Decoder Receive JPEG0007860563000029.jpg612 • Soft partition quantization: The decoder is in the forward pass. I receive JPEG0007860563000030.jpg612, When calculating JPEG0007860563000031.jpg77, in the back pass of backpropagation Use JPEG0007860563000032.jpg613.

[0221] Note that in integer rounding quantization, hard partitioning and soft partitioning quantization are equivalent.

number

[0222] However, when using learned quantization bins, the quantization of hard split and soft split is not equivalent in the back pass, as follows.

number

[0223] In all quantization schemes, the rate gradient wrtΔ is negative.

number

[0224] In any quantization scheme, the rate term is Since it receives JPEG0007860563000036.jpg613, The filename becomes JPEG0007860563000037.jpg712. Since the filename is JPEG0007860563000038.jpg518, we obtain the following:

number

[0225] The strain gradient wrtΔ differs depending on the quantization scheme. The gradients are as follows:

number

[0226] In soft partitioning quantization, JPEG0007860563000041.jpg812 and This results in JPEG0007860563000042.jpg631. This is because the random noise ε in the back pass is in the front pass, This is because it does not rely on STE rounding when performing JPEG0007860563000043.jpg66. This means the following:

number

[0227] Therefore, in soft partition quantization, the rate gradient increases Δ, and the distortion gradient is 0 on average, so overall Δ→∞, and network training is not possible.

[0228] Conversely, in hard partitioning quantization... JPEG0007860563000045.jpg66 is Since it does not depend on JPEG0007860563000046.jpg612, it will be as follows.

number

[0229] In summary, when using partition quantization with learned quantization bins, use hard partition quantization, not soft partition quantization.

[0230] The non-trivial strain gradient, achievable with or without partition quantization, means that the strain gradient flows through the decoder into the hypernetwork. This is usually not possible in models with hypernetworks, but it is a feature introduced by our method of learning the size of the quantization bins.

[0231] In some compression pipelines, it is important to control the distribution of values ​​learned with respect to the quantization bin size (this is not always necessary). If necessary, this can be achieved by introducing an additional term into the loss function.

number

[0232] F Δ Distribution p Δ It is characterized by the choice of (Δ), which, following the terminology used in Bayesian statistics, is called the "plier" with respect to Δ. There are several options for the plier. • Any parametric probability distribution on positive numbers. Specifically, the inverse gamma distribution, some examples of which are shown in Figure 9. The distribution is shown for several parameter values. The parameters of the distribution can be fitted from other models, selected a priori, or learned during training. • A neural network that learns a prior distribution during training.

[0233] In the previous section, we provided a detailed explanation of the simplified quantization function Q, which utilizes a tensor of bin size Δ.

number

[0234] All of these methods can be extended to more generalized quantization functions. In the general case, Q is some invertible function of y and Δ. The coding is then given by the following equation:

number

number

[0235] This quantization function is more flexible than Equation 24 and consequently improves performance. The generalized quantization function can also be made context-aware by incorporating quantization parameters that use, for example, an autoregressive context model.

[0236] All methods in the previous section are compatible with the framework of generalized quantization functions: • Predict necessary parameters from hypernetworks, context modules, and implicit equations if necessary. • Appropriately adjust the loss function. • When using split quantization, hard split quantization should be employed. If necessary, introduce Bayesian pliers into the loss function to control its behavior.

[0237] More flexible generalized quantization functions improve performance. In addition, generalized quantization functions can rely on parameters determined autoregressively, which means that quantization depends on pixels that have already been encoded / decoded.

number

number

[0238] Generally, using autoregressive context models improves the performance of AI-based compression.

[0239] Autoregressive generalized quantization functions are also beneficial from a runtime perspective. Other standard autoregressive models, such as PixelCNN, require an arithmetic decoder (or other lossless decoder) to be executed every time a pixel is decoded using the context model. This is a serious performance issue in real-world applications of image and video compression. However, the generalized quantization function framework allows for the incorporation of autoregressive context models into AI-based compression without the runtime problems seen in PixelCNN and other frameworks. This is Q -1 This is because it operates automatically and recursively on the ξ (hat) that has been completely decoded from the bitstream. Therefore, there is no need to run the arithmetic decoder automatically and recursively, and the execution time problem is solved.

[0240] A generalized quantization function can be any invertible function. For example, • Invertible rational function: Q(·,Δ) = P(·,Δ) / Q(·,Δ), where P and Q are polynomials. • Logarithmic, exponential, and trigonometric functions, with their domains appropriately restricted so that they are invertible. • Invertible function of matrix Q(·,Δ) • A reversible function including the context parameter L predicted by the hyperdecoder: Q(·,Δ,L)

[0241] Furthermore, Q does not generally need to be closed or reversible. For example, Q enc (·,Δ) and Q dec (·,Δ) can be defined, and these functions do not necessarily have to be inverse functions of each other, allowing the entire pipeline to be trained end-to-end. In this case, Q enc and Q dec This can be modeled as a neural network, or as a specific process such as a Gaussian process or a probabilistic graphical model (a simple example being a Hidden Markov Model).

[0242] To train an Al-based compression pipeline using generalized quantization functions, many of the same tools described above are used. • Appropriately adjust the loss function (specifically the rate term). • When using partitioned quantization, use hard partitioned quantization. • Introduce Bayesian pliers and / or regularization terms (l1, l2, second moment penalty) into the loss function and, if necessary, control the distribution of the distribution parameter Δ and, if necessary, the context parameter L.

[0243] Depending on the choice of generalized quantization function, other tools may be required to train the Al-based compression pipeline. Techniques from reinforcement learning (Q-Learning, Monte Carlo Estimation, DQN, PPO, SAC, DDPG, TD3) • General proximity gradient method • Continuous relaxation. Here, discrete quantification residues are approximated by a continuous function. This function can have hyperparameters that control the smoothness of the function and are modified at various points in training to improve the training and final performance of the network.

[0244] There are several possibilities for context modeling that are compatible with the generalized quantization function framework. Q is an autoregressive neural network. The most common example that can be adapted to Q is a PixelCNN-style network, but other neural network building blocks such as Resnets, Transformers, Recurrent Neural Networks, and Fully-Connected networks can also be used for the autoregressive Q function. ·Δ can be predicted as a linear combination of previously decoded pixels.

number

[0245] For example, if important metadata is obtained from attention mechanisms / focus masks, this can be incorporated into delta prediction. In this case, the quantization bin size is more accurately adapted to the sensitive regions of the image and video, and knowledge of the sensitive regions is stored in this metadata. Thus, less information is lost from perceptually important regions, while information from unimportant regions is ignored, resulting in a more enhanced performance improvement compared to Al-based compression pipelines that do not have adaptable bin sizes.

[0246] Furthermore, we outline the relationship between the learned quantization bins and the variable-rate model. One form of the variable-rate model trains an Al-based compression pipeline with a free hyperparameter δ that controls the bin size. During inference, δ is transmitted as metadata to control the transmission rate (cost per bit).

[0247] In a variable rate framework, δ is a global parameter in the sense that it controls the size of all bins simultaneously. Our innovation locally calculates the tensor Δ of the predicted bin size for each pixel. In addition, a variable rate model that controls the transmission rate using δ is compatible with our framework because it can scale the local prediction element by element by the global prediction as needed to control the rate during interpolation.

number

[0248] Dataset bias This section details the training procedure applied to the generative adversarial network framework. This approach allows the generative compression model to be biased for any type of image data, and the quality of the resulting image can be controlled depending on the subject matter in the image.

[0249] Generalized Adversarial Networks (GANs) have shown excellent results when applied to various different generative tasks in the image, video, and audio domains. This approach is inspired by game theory, where two models, a generator and a critic, are pitted against each other, resulting in both becoming stronger. The first model in a GAN is a generator G, which takes a noise variable z as input and outputs a synthetic data sample x (hat), while the second model is a discriminator D, which is trained to distinguish between a sample from a real data distribution and data generated by the generator. An example of a GAN architecture outline is shown in Figure 10.

[0250] P x The data distribution on the actual sample x, P z The data distribution on the noise sample z, P g Let this be the generator distribution on data x.

[0251] The training of the GAN is shown as a minimax game in which the following function is optimized.

number

[0252] Applying a generative adversarial approach to the image compression task, Let's start by considering JPEG0007860563000057.jpg519, where C is the number of channels and H and W are the height and width in pixels.

[0253] A compression encoder pipeline based on an autoencoder is constructed from the following, and the encoder function f θ (x)=y represents the latent representation of image x. Encode it as JPEG0007860563000058.jpg721. Q is the quantization function required to send y as a bitstream, and the decoder function... JPEG0007860563000059.jpg616 is a reconstructed image of the quantized latent y (hat). This function decrypts the image into JPEG0007860563000060.jpg519.

number

[0254] In this case, encoder f θ , quantization function Q and decoder g θ This combination can be considered as a generative network. For simplicity of notation, this generative network will be denoted as G(x). This generative network is complemented by a discriminative network D, which is trained together with the generative network in a two-stage method.

number

[0255] An example of a standard generative adversarial compression pipeline is shown in Figure 11. Next, training is performed using a standard compression rate-strain loss function.

number

[0256] By complementing this trained compression network with a classifier model, the perceived quality of the output image can be improved. In this case, the compression encoder-decoder network can be considered a generative network, and the two models can be trained using a two-level approach in each iteration. For the classifier architecture, we decided to use a conditional classifier that generates higher quality reconstructed images. In this case, the classifier d(x,y(hat)) is conditioned on a quantized latent y(hat). First, the classifier is trained with the classifier loss.

number

[0257] The rate-distortion loss in (36) is augmented by adding an adversarial "unsaturated" loss used to train the GAN generator in order to train the generative network in (32).

number

[0258] By adding adversarial loss to rate distortion loss, we encourage the network to generate natural patterns and textures. Using an architecture that combines GANs and autoencoders, we were able to significantly improve the perceptual quality of reconstructed images and achieve excellent results in image compression. However, despite the impressive overall results of such architectures, there are numerous notable failure modes. These models have been observed to struggle with compressing areas of high visual importance, including but not limited to human faces and text. An example of such failure modes is shown in Figure 12. The image on the left is the original image, and the image on the right is the reconstructed image synthesized using a generative compression network. Note that the most distortion occurs in the human faces present in the image, i.e., the areas with high visual information. To address this problem, we propose a method that allows the model to be biased towards specific types of images, such as faces, thereby improving the perceptual quality of the reconstructed images.

[0259] Under this framework, we train the network on multiple datasets, each using a different classifier. First, we train N additional datasets X1, ... X that have biases in the model. N Select this. A good example of such a dataset useful for facial modeling would be a dataset consisting of portraits of people. Each dataset X i For this, discriminant model D i We will introduce each classifier model D. i Dataset X i While the encoder-decoder model is trained using only the data from one dataset, the encoder-decoder model is trained using images from the entire dataset.

number

[0260] Figure 13 shows a diagram of the compression pipeline using a cGAN multiple classifier with dataset bias. (Dataset X) i Image x i The data passes through an encoder, is digitized, and converted into a bitstream using a range coder. The decoder then decodes the bitstream, and x i An attempt is made to reconstruct the (hat). Then, the original image x and the reconstructed x (hat) are passed to the classifier corresponding to the dataset i.

[0261] For illustrative purposes, we will focus on the facial malfunction modes, as shown in Figure 12. Note that all the techniques described here are applicable to any number of regions of interest in an image (e.g., faces, eyes, lips, logos, cars, flowers, patterns, etc.).

[0262] As an example, consider biasing a dataset using only one additional dataset. In this case, X1- is a general training dataset, and X2- is a dataset containing only portrait images. Figure 14 shows a comparison of the reconstruction of the same generative model trained at the same bitrate with and without the multi-identifier dataset bias scheme. The image on the left is a reconstruction of the image synthesized by a standard generative compression network, and the image on the right is a reconstruction of the same image synthesized by the same generative network trained with the multi-identifier dataset bias. Using this scheme, the perceptual quality of human faces is improved without compromising the quality of the rest of the image.

[0263] [Table 1]

[0264] [Table 2]

[0265] The above approach can also be used in architectures that use a single classifier for all datasets. Furthermore, a classifier D for a specific dataset can be used. i It can be trained more frequently than a generator, increasing the effect of bias on its dataset.

[0266] Given the generative compression network described above, we define here an architectural modification that allows for higher bit allocation, conditional on the context of the image or frame. To enhance the effects of dataset bias and change the perceived quality of different regions of the image depending on the subject of the image, Lagrangian coefficients controlling the bitrate are applied to dataset X. i We propose a different training procedure for each. The generator modifies the loss function of (37) as follows:

number

[0267] This approach trains the model to allocate a higher percentage of the bitstream to the face region of the compressed image. The results of biasing the bitrate-adjusted dataset can be observed in Figure 15. Figure 15 is the same for the face dataset. While maintaining JPEG0007860563000070.jpg59, the background dataset is different. This image shows three images synthesized by a model trained on JPEG0007860563000071.jpg59. The left image was trained with a low bitrate on the background dataset, the middle image with a medium bitrate, and the right image with a high bitrate.

[0268] Extending the method proposed above, we suggest using a different distortion function d(x, x(hat)) for each dataset used for biasing. This method allows the model to focus on each specific type of data. For example, a linear combination of MSE, LPIPS, and MS-SSIM metrics can be used as the distortion function.

number

[0269] Coefficients of different components of the distortion function By modifying JPEG0007860563000073.jpg638, the perceived quality of the resulting image can be altered so that the generative compression model can reconstruct different regions of the image in different ways. Then, Equation 37 is given for each dataset X i This can be corrected by indexing the distortion function d(x, x (hat)) for that function.

number

[0270] Here, we discuss the use of a salience mask to bias the generational compression model towards particularly important regions within the image. This mask is generated using another pre-trained network, and its output can be used to further improve the performance of the compression model.

[0271] Take image x as input and a binary mask Consider network H that outputs JPEG0007860563000075.jpg517. Let's start from there.

number

[0272] X i Prominent pixels within m are indicated by 1s, while 0s indicate regions that the network does not need to focus on. This binary mask can be used to further bias the compression network towards those regions. Examples of such important regions include, but are not limited to, human facial features such as eyes and lips. Given m, the input image x can be modified to prioritize these regions. Modified image x H This is used as input to an adversarial compression network. An example of such a compression pipeline is shown in Figure 16.

[0273] Extending the approach proposed above, we propose an architecture that utilizes a pre-trained network to generate a salience mask that biases the compression pipeline. In this approach, the bitrate allocation to different parts of the reconstructed image can be changed by modifying the mask without retraining the compression network. In this modification, the mask m from Equation 41 is used as an additional input to train the network to allocate more bits to regions marked as salient (1) by m. In the inference phase after the network has been trained, the bit allocation can be adjusted by changing the mask m. An example of such a compression pipeline is shown in Figure 17.

[0274] We further propose a training scheme that ensures the model is exposed to a wide range of natural image examples. The training dataset is constructed from images from N different classes, each labeled accordingly. During training, images are sampled from the dataset according to their class. By sampling images equally from each class, the model can be exposed to underrepresented classes and learn the full distribution of natural images.

[0275] Area emphasis Modern trained image compression methods, such as VAE and GAN-based architectures, enable excellent compression at low bitrates and a significant improvement in the perceptual quality of reconstructed images. However, despite the impressive overall results of such architectures, there are numerous notable failure modes, as mentioned above. These models have been observed to struggle to compress areas of high visual importance, including, but not limited to, human faces or text. An example of such a failure mode is shown in Figure 12. The image on the left shows the original image, and the image on the right shows the reconstructed image synthesized using a generative compression network. Note that the most distortion occurs in the human faces present in the image, i.e., the visually important parts.

[0276] We propose an approach to improve the perceived quality of a region of interest (ROI) by allocating more bits to the bitstream through changing the quantization bins within the ROI.

[0277] To encode the latent y into a bitstream, we can first quantize it to ensure it is discrete. We propose controlling the number of bits (bpp) allocated to a region using a quantization parameter Δ, where Δ is the size of the quantization bins or quantization interval, representing the coarseness of quantization in the latent and hyperlatent spaces. The coarser the quantization, the fewer bits are allocated to the data.

[0278] The quantization of latent y is achieved as follows:

number

[0279] We propose using spatially varying delta to control the coarseness of quantization within an image. This allows us to control the number of bits allocated and thus the visual quality of different regions of the image. This proceeds as follows:

[0280] We begin by considering a function H that detects regions of interest, as is typically represented in neural networks. The function H(s) takes an image x as input and a binary mask. Outputs JPEG0007860563000078.jpg517. A value of 1 for m indicates that the corresponding pixel in image x is within the region of interest, while zero corresponds to a pixel outside of it.

[0281] In one example, the network H(x) is trained before the compression pipeline is trained, and in another example, it is trained in combination with the encoder-decoder. Map m is used to create a quantization map Δ in which each pixel is assigned a quantization parameter. If the value in m for a pixel is 1, the corresponding value in Δ is small. The function Q, defined in Equation 42, then quantizes y to y(hat) using the spatial map Δ before encoding it into a bitstream. As a result of such a quantization scheme, the bitrate of the region of interest is higher compared to the rest of the image.

[0282] The proposed pipeline is shown in Figure 18. Figure 18 illustrates the proposed compression pipeline, which has quantization using a quantization map Δ that depends on the binary mask of the ROI m.. Figure 19 shows the result of implementing such a pipeline using a face detector and network H(x) used for identifying the region of interest. The image on the left is a compressed image synthesized with faces assigned as ROIs in the quantization map Δ, and the image on the right is a compressed image using a standard generative compression model.

[0283] In another example, for the region identified by the ROI detection network H(x) from equation (41), different quantization functions Q m This can be used. An example of such an arrangement is shown in Figure 20. Figure 20 shows the quantization function Q for the entire image and the quantization function Q for the region of interest. m A diagram illustrating a proposed compression pipeline using [the specified method] is shown.

[0284] Discrete PMFS Encoding and decoding a stream of discrete symbols (such as latent pixels in an Al-based compression pipeline) into a binary bitstream may require access to a discrete probability mass function (PMF). However, it is widely believed that training such a discrete PMF in an Al-based compression pipeline is impossible because the training requires access to a continuous probability distribution function (PDF). Therefore, the de facto standard in training Al-based compression pipelines is to train a continuous PDF and only after training is complete to approximate the continuous PDF with a discrete PMF evaluated by the discrete number of quantization points.

[0285] The following describes the reverse procedure, where an Al-based compression pipeline can be directly trained on a discrete PMF by interpolating the discrete PMF into a continuous real-valued space. The discrete PMF can be trained or predicted and can also be parameterized.

[0286] The following description outlines the discrete probability mass function and the interpolation capabilities, scope, and future prospects for its use, not limited to AI-based image and video compression. It also discusses a high-level description of the discrete probability mass function, its inference and training of AI-based compression algorithms, and methods for interpolating functions (such as the discrete probability mass function).

[0287] In the literature on Al-based compression, the standard approach for constructing an entropy model is the continuous probability density function (PDF)p yThe first step is to start with (y) (such as a Laplace distribution or a Gaussian distribution). Shannon entropy is a discrete variable y (hat) (usually Since it is defined only in JPEG0007860563000079.jpg69), this PDF is a discrete probability mass function (PMF) Convert to JPEG0007860563000080.jpg610 for use in, for example, a reversible arithmetic encoder / decoder. This can be done by collecting all (continuous) masses in a (unit) bin centered on y (hat).

number

[0288] This approach was first proposed by Johannes Balle, Valero Laparra, and Eero P. Simoncelli. It is an end-to-end optimized image compression. OpenReview.net, 2017, in the Conference Track Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, is incorporated here by reference. This function is an integer Not only is it defined for JPEG0007860563000082.jpg69, but it also accepts any real number argument. This is very useful when training where a PDF is needed for the latent values ​​of a continuous real number output by the encoder. Hence the "new" function. JPEG0007860563000083.jpg523 is defined. By definition, this is a PMF defined on an integer. This function is an exact match to JPEG0007860563000084.jpg55. JPEG0007860563000085.jpg56 is a PMF (Product-Market Fitting) on ​​which an end-to-end AI-based compression algorithm is actually trained.

[0289] In summary, continuous real-valued PDF JPEG0007860563000086.jpg617 is a discrete PMF. Converted to JPEG0007860563000087.jpg517, then during training JPEG0007860563000088.jpg513 contains a continuous PDF It is evaluated as JPEG0007860563000089.jpg56. Figure 21 shows examples of three typical one-dimensional distributions used to train an Al-based compression pipeline. In reality, the original PDF p y It is never explicitly used during training or inference. What is actually used is: JPEG0007860563000090.jpg56 (used for inference) and There are only two functions: JPEG0007860563000091.jpg56 (used for training).

[0290] This thinking model can also be reversed. Instead of starting with a PDF, it's possible to start with discrete PMFs and reconstruct a continuous PDF (which is used only in training) by interpolating the PMFs.

[0291] PMF Suppose the image JPEG0007860563000092.jpg56 is given. This PMF consists of two vectors of length N, i.e., y i (Hat) and p i It can be expressed using a hat symbol, where i=1... and N represents discrete points. In the older way of thinking (when PMF is defined as a function), It is defined as JPEG0007860563000093.jpg517. However, generally JPEG0007860563000094.jpg58 can be any non-negative vector whose sum is 1. i The results are sorted in ascending order and do not necessarily have to be integer values.

[0292] Here is the query point Given the image JPEG0007860563000095.jpg619, the query points must be enclosed by discrete extrema. (Approximate) Training PDF To define JPEG0007860563000096.jpg58, an interpolation routine is used.

number

[0293] The following is a non-exhaustive list of possible interpolation routines: • Piecewise constant interpolation (nearest neighbor interpolation) Linear interpolation Polynomial interpolation • Spline interpolation (e.g., tertiary interpolation) • Gaussian process / Kriging

[0294] In general, a function defined by interpolation is not necessarily a PDF. Depending on the interpolation routine used, the interpolated value may be negative or lack unit mass. However, these problems can be mitigated by selecting an appropriate routine. For example, piecewise linear interpolation preserves mass, preserves positivity, and ensures that the interpolated function is indeed a PDF. Piecewise cubic Hermitian interpolation can be constrained to be positive if the interpolation points themselves are positive, as discussed by Randall L Dougherty, Alan S Edelman, and James M Hyman. Cubic and quintic Hermitian interpolation that preserves non-negativity, monotonicity, or convexity. Mathematics of Computation, 52(186):471-494, 1989, which is incorporated herein by reference.

[0295] However, piecewise linear interpolation has another problem. Its derivative is piecewise constant, and the interpolation error can be quite severe, as shown in the left figure of Figure 22, for example. For example, if the PMF is generated using Balle's method, the interpolation error is It is defined as JPEG0007860563000098.jpg523. Other interpolation schemes, such as piecewise cubic Hermitian interpolation, result in smaller interpolation errors, as shown in the right-hand figure of Figure 22, for example.

[0296] Discrete PMFs can be directly trained with AI-based image and compression algorithms. During training, the latent probability values ​​of the real values ​​output from the encoder are interpolated using the discrete values ​​of the PMF model. The PMF is learned by feeding the gradient from the rate (bitstream size) loss back into the parameters of the PMF model.

[0297] PMF models can be either learned or predicted. Learned means that the PMF model and its hyperparameters are independent of the input image. Predicted means that the PMF model conditionally depends on "side information" such as hyperlatencies. In this scenario, the parameters of the PMF may be predicted by a hyperdecoder. Furthermore, the PMF can also conditionally depend on adjacent latent pixels (in this case, the PMF is called a discrete PMF context model). Regardless of how the PMF is represented, the values ​​of the PMF can be interpolated during training to provide estimates of the probability values ​​of real-valued (non-quantized) points, which can then be input into the rate loss of the training objective function.

[0298] The PMF model can be parameterized in one of the following ways (however, this list is not exhaustive): • PMF is a categorical distribution, and the probability values ​​of a categorical distribution correspond to a finite number of quantization points on the solid line. The probability values ​​of a categorical distribution correspond to a finite number of quantization points on a solid line. The categorical distribution can be parameterized by a vector, which is projected onto a stochastic simplex. This projection may be a softmax style projection or any other projection onto the stochastic simplex. ·The PMF can be parameterized with several parameters. For example, if the PMF is defined for N points, n parameters (n < N) can be used to control the values of the PMF. For example, the PMF can be controlled with mean and scale parameters. This can be done, for example, by collecting the mass of a continuous-valued 1D distribution in quantization bins of discrete numbers. ·The PMF can be multivariate, in which case the PMF is defined over a multi-dimensional set of quantization points. ·For any of the previous items, the quantization points can be, for example, some integer values or can be at arbitrary intervals. The intervals of the quantization bins can also be predicted by an auxiliary network such as a hyper decoder or can be predicted from the context (adjacent latent pixels). ·For any of the previous items, the parameters controlling the PMF can be fixed or can be predicted by an auxiliary network such as a hyper decoder or can be predicted from the context (neighboring latent pixels).

[0299] This framework can be extended in several ways. For example, if the discrete PMF is multivariate (multi-dimensional), a multivariate (multi-dimensional) interpolation scheme can be used to interpolate the values of the PMF to real vector-valued points. For example, multi-linear interpolation (bilinear in 2D, trilinear in 3D, etc.) can be used. Also, multi-cubic interpolation (bicubic interpolation in 2D, tricubic interpolation in 3D, etc.) can be used.

[0300] This interpolation method is not restricted to modeling only discrete-valued PMFs. Any discrete-valued function can be interpolated anywhere in an AI-based compression pipeline, and the techniques described here are not strictly limited to modeling probability mass / density functions.

[0301] Context Model In Al-based compression, autoregressive context models offer powerful entropy modeling capabilities, but they suffer from extremely short execution times due to the need for serial execution. This paper describes how to overcome this difficulty by predicting autoregressive model components from a hyperdecoder (and conditioning these components on "side" information). This technique yields an autoregressive system that is both highly modelable and real-time executable. This real-time capability is achieved by decoupling the autoregressive system from the model required by the lossless decoder. Instead, the autoregressive system solves a linear equation during decoding, which can be solved extremely quickly using numerical linear algebra techniques. Encoding can also be performed quickly by solving a simple implicit equation.

[0302] This paper outlines the current and future applications of autoregressive probabilistic models using linear decoding systems, not limited to image and video data compression based on AI and deep learning, but also encompassing their functions and scope.

[0303] In AI-based image and video compression, an input image x is mapped to a latent variable y. This latent variable is encoded into a bitstream and set into a receiver that decodes the bitstream back into the latent variable. The receiver then converts the recovered latent variable into a representation (reconstruction) of the original image x (hat).

[0304] To perform the step of converting the latent to a bitstream, the latent can be quantized to an integer value representation y (hat). This quantized latent y (hat) is then converted to a bitstream via a lossless encoder / decoder scheme such as an arithmetic encoder / decoder or a range encoder / decoder.

[0305] Lossless coding / decoding schemes may require a model one-dimensional discrete probability mass function (PMF) for each element of the latent quantization variables. The optimal bitstream length (file size) is achieved when this model PMF matches the latent true one-dimensional data distribution.

[0306] Therefore, file size is closely tied to the power of the PMF model to match the true data distribution. A stronger PMF model results in smaller file sizes and better compression. In some cases, this leads to better reconstruction error (because more information can be sent to reconstruct the original image for a given file size). For this reason, much effort is being put into developing powerful PMF models (often called entropy models).

[0307] A typical approach to modeling one-dimensional PMF in Al-based compression is a parametric one-dimensional distribution. The method involves using JPEG0007860563000099.jpg519, where θ is a parameter of the one-dimensional PMF. For example, a quantized Laplacian or a quantized Gaussian can be used. In these two examples, θ constructs the position and scale o-parameters of the distribution. For example, if a quantized Gaussian (Laplacian) is used, the PMF can be expressed as follows:

number

[0308] By "conditioning" parameters θ, such as position μ or scale σ, with other information stored in the bitstream, more robust models can be created. In other words, instead of statically fixing the PMF parameters so that they remain constant across all inputs of the Al-based compression system, the parameters can respond dynamically to the inputs.

[0309] This is generally done in two ways. The first is to send extra side information z (hat) to the bitstream in addition to y (hat). This variable z (hat) is often called the hyperlatency. This variable can be used for encoding / decoding y (hat) because its entirety is decoded before y (hat) is decoded. Then μ and σ can be functions of z (hat), for example, and μ and σ are returned through the neural network. Then the one-dimensional PMF is said to be conditional on z (hat), Provided by JPEG0007860563000101.jpg532.

[0310] Another approach is to use an autoregressive probabilistic model. For example, there is PixelCNN, described by Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation using a pixelCNN decoder. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, ​​Spain, pages 4790-4798, 2016, which is herein incorporated by reference, David Minnen, Johannes Balle, and Joint. It is widely used in academic papers on AI-based compression, as done in Joint autoregressive and hierarchical priors for learned image compression. This is described in Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolo Cesa-Bianchi, and Roman Garnett, editors, *Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018*, NeurIPS 2018, December 3-8, 2018, Montreal, Canada, pages 10794-10803, 2018. In this framework, context pixels are used to condition the position μ and scale σ parameters of the PMF at the current pixel.These context pixels are previously decoded pixels that are adjacent to the current pixel. For example, suppose the previous k pixels have been decoded. Because images have inherent spatial correlations, these pixels often contain relevant information about the currently active pixel. Therefore, these context pixels can be used to improve the prediction of the current pixel's position and scale. The PMF of the current pixel is... The image is given as JPEG0007860563000102.jpg569, where μ and σ are functions of the previous k variables (usually a convolutional neural network).

[0311] These two approaches involve conditioning using hyperlatent or autoregressive context models, but both have their advantages and disadvantages.

[0312] One of the main advantages of hyperlatency conditioning is that quantization can be location-shifted. In other words, the quantization bins can be centered around the position parameter μ. Using integer bins, the quantization latent can be expressed as follows:

number

[0313] The main advantage of autoregressive context models is the ability to utilize contextual information (decoded neighboring pixels). Because images (and videos) are highly spatially correlated, these neighboring pixels can provide very accurate and precise predictions about what the current pixel should be. Most state-of-the-art academic AI-based compression pipelines use autoregressive context models due to their impressive performance results, measured in terms of bitstream length and reconstruction error. However, despite their impressive relative performance, they suffer from two problems.

[0314] In other words, the PMF of the current pixel depends on all previously decoded pixels. Furthermore, the position function μ(·) and scale function σ(·) are typically large neural networks. These two facts mean that autoregressive context models cannot be run in real time, taking orders of magnitude longer than the computational requirements needed for real-time performance on edge devices. Therefore, in its current state, autoregressive context models are not commercially viable despite the fact that they offer impressive compression performance.

[0315] Secondly, due to the effects of cascade errors, the autoregressive context model is rounded straight. You must use JPEG0007860563000106.jpg515. Position shift rounding JPEG0007860563000107.jpg527 is impossible because a small floating-point error introduced early in the decode pass is amplified during the serial decode pass, resulting in a large difference in predictions between the encoder and decoder. The lack of position shift rounding is problematic, and an autoregressive model with position shift rounding (if constructible) is considered superior to a straight-round autoregressive model, given that all other constructors are equal.

[0316] Therefore, we need to develop a PMF modeling framework that combines the advantages of conditioning on hyperlatencies (fast execution time, position shift rounding) with the excellent performance of autoregressive models (making strong predictions from pre-decoded context pixels).

[0317] The following describes a technique for modifying a hyperdecoder to additionally predict the parameters of an autoregressive model using the hyperdecoder. In other words, it conditions the parameters of the autoregressive model on a hyperlatent z (hat). This is in contrast to the standard setup in autoregressive modeling, where the autoregressive function is static and invariant, and does not change depending on the input of the compressed pipeline.

[0318] This section primarily deals with the following quasi-linear setup (where the decoding path is linear, but the encoding is not). The hyperdecoder can output a sparse matrix L called the context matrix, in addition to μ and σ predictions. This sparse matrix is ​​used as an autoregressive context modeling element in PMF, as follows. Given the order of latent pixels (e.g., raster scan order), the previous k latent pixels are encoded / decoded and available for autoregressive context modeling. The approach used is a modified position-shift quantization as follows. Quantization is performed as follows:

number

[0319] Probability models are, It is given as JPEG0007860563000109.jpg848. In matrix-vector notation, it is as follows:

number

[0320] Note that this is a form of autoregressive context modeling, because the one-dimensional PMF depends on previously decoded latent pixels. However, only the position parameter depends on previously decoded latent pixels; the scale parameter does not.

[0321] Note that the integer values ​​that can actually be encoded by an arithmetic encoder / decoder are quantization residues.

number

[0322] Therefore, in decoding, the arithmetic encoder returns ξ (hat) from the bitstream, not y (hat). Then, y (hat) can be reconstructed by solving the following system of linear equations.

number

[0323] Solving system (50) is decoupled from the arithmetic decoding process. That is, while the arithmetic decoding process must be performed serially as the bitstream is received, solving (50) is independent of this process and can be performed using any numerical linear algebra algorithm. The decoding path of the L context modeling step does not have to be a serial procedure and can be executed in parallel.

[0324] Looking at this result from another perspective, the arithmetic encoder / decoder is equivalent to operating on ξ (hat) where position is 0. In other words, the arithmetic encoder operates on residual ξ (hat), not on the latent y (hat). Considering this, the PMF is The image is JPEG0007860563000114.jpg524. Only after ξ (hat) has been recovered from the bitstream do we recover y (hat). However, since the recovery from the bitstream may not be autoregressive (the only dependency is σ, which is not context / autoregressively dependent), this procedure may be very fast. Then, we can recover y (hat) using a highly optimized linear algebra routine to solve (50).

[0325] In both coding and training L-context systems, (48) can be solved, but the unknown variable y(hat)-y(hat) is not explicitly given and must be determined. In fact, (48) is an implicit system. Here we outline several possible approaches to finding y(hat) that satisfies (48). The first approach is to solve (48) sequentially and manipulate the pixels according to the order of dependencies in the autoregressive model. In this setup, we simply iterate through all pixels in an autoregressive order and apply (47) at each iteration to extract the latent quantized in the current iteration. Since (48) is an implicit equation, the second approach is to employ an implicit equation solver (implicit coding solver). This is an iterative solver that finds fixed-point solutions to (48). Finally, in certain special cases, the components of the serial decoding path can be parallelized by utilizing an autoregressive structure defined by a sparse context matrix L. This approach first creates a Directed Acyclic Graph (DAG) that defines the dependencies between latent pixels. This dependency graph can be constructed based on a sparse structure of the L matrix. Next, note that pixels at the same level of the DAG are conditionally independent of each other. Thus, all pixels can be computed in parallel without affecting the computation of other pixels at that level. Therefore, in coding (and training), the graph is iterated through by starting from the root node and working through the levels of the DAG. At each level, all nodes are processed in parallel. This procedure provides a significant speedup compared to a naive serial implementation when a parallel computing environment (such as a graphics processing unit or neural processing unit) is available. This procedure is shown in Figure 23. The left image in Figure 23 shows the L context parameters related to the i-th pixel in the example. Neighboring context pixels are the pixels directly above the current pixel and the left neighbors. The right image shows the pixels listed in raster scan order. The image below shows an example of constructing a directed acyclic graph (DAG) given dependencies generated by an L-context matrix. Pixels at the same level are conditionally independent of each other and can be encoded / decoded in parallel.

[0326] Many of the techniques described in the previous section can also be applied during decoding. Specifically, linear equations

number

[0327] The following section provides a detailed example of an L context module within an Al-based compression pipeline.

[0328] Figure 24 shows the prediction context matrix L in an example of an Al-based compression pipeline. y This diagram illustrates the encoding process using [a specific method / tool]. A typical implicit solver is depicted in this figure. In encoding, the input image x (hat) is fed to an encoder function such as a neural network. The encoder outputs a latent y, which is then sent to a hyperencoder, which returns the hyperlatent z.

[0329] The hyperlatency is the learned position μ z and scale σ z The ID PMF, which depends on the quantized hyperlatency, is quantized to z (hat) via an arithmetic encoder and then a lossless encoder, and sent to the bitstream. Optionally (though not shown in Figure 24), a learned L context module can also be used in the entropy model for y (hat). The quantized hyperlatency is then sent to a hyperdecoder, which outputs the parameters of the entropy model for y. These include the position μ y , scale σ y and L context matrix L y This includes the residual, which is calculated by solving the coding equation using one of the methods described above. This quantized residual ξ (hat) is obtained by the zero mean and scale parameter σ. yIt is sent to the bitstream using an MF that has [a specific parameter].

[0330] Figure 25 shows the prediction context matrix L in an example of an Al-based compression pipeline. y This figure shows the decoding process using [a specific method / tool]. The linear equation solver is depicted in this figure. In decoding, the learned position parameter μ is used. z and σ z The hyperlatent z (hat) is first reconstructed from the bitstream using a one-dimensional PMF with a scale parameter and a lossless decoder. Optionally (though not shown in the diagram), the L context module used during encoding can also be employed. The hyperlatent is fed to the hyperdecoder, which then processes the position μ y , scale σ y and sparse context matrix L y The output is the lossless decoder, zero-mean PMF, and scale parameter σ. y The bitstream is reconstructed using [a specific method]. Then, as described above, the quantized latent is reconstructed by solving the linear decoding system. Finally, the reconstructed image is reconstructed by supplying the quantized latent y (hat) through another decoder function, such as a neural network.

[0331] In the previous sections, we assumed that L is a lower triangular matrix with respect to the pixel decoding order. As a generalization, we relax this assumption and assume a general matrix A that is not necessarily a lower triangular matrix. In this case, the coding equation is:

number

number

number

[0332] In general, context functions can be nonlinear. For example, in coding problems,

number

number

number

[0333] One interpretation of this latter extension is the implicit PixelCNN. For example, if f(·) has a triangular Jacobian (matrix of first derivatives), then (55) models an autoregressive system. However, (55) is more general than this interpretation and can indeed model not only autoregressive systems but also stochastic systems that have both forward and backward conditional dependencies on the order of pixels.

[0334] Learned AR sequence In AI-based image and video compression, autoregressive modeling is a powerful technique for modeling the entropy of latent spaces. Contextual models, conditional on previously decoded pixels, are used in modern AI-based compression pipelines. However, the autoregressive order in contextual models is often a predefined, elementary one, such as a raster scan order, which can impose undesirable biases on the learning process. Therefore, we propose fixed orders other than raster scans, conditional orders, learning orders, or direct optimization orders as autoregressive orders in contextual models.

[0335] In mathematical terms, the goal of lossy Al-based compression is to infer a prior probability distribution, an entropy model, that matches as closely as possible to the latent distribution generating the observed data. This can be achieved by training a neural network through optimization frameworks such as gradient descent. Entropy modeling underpins the entire Al-based compression pipeline, and better distribution matching corresponds to better compression performance, characterized by lower reconstruction losses and bit rates.

[0336] In image and video data exhibiting significant spatial and temporal redundancy, an autoregressive process called contextual modeling is highly useful for leveraging this redundancy in entropy modeling. At a high level, the general idea is to use existing available information to condition the explanation of subsequent information. The process of conditioning previous variables to realize subsequent variables implies an autoregressive information acquisition structure of a specific order. This concept has proven very powerful in AI-based image and video compression and is commonly used as part of state-of-the-art neural compression architectures.

[0337] However, the autoregressive ordering (AO) of autoregressive structures in AI-based image and video compression can sometimes be predetermined. In such contextual models, the so-called raster scan order is often adopted, which naturally follows the data sequence of the image data type (3D; such as RGB, height x width x channels). Figure 26 shows an example of raster scan order for a single-channel image. Gray squares represent conditionable variables, and white squares represent unconditionable variables. However, adopting raster scan order as the basic AO is arbitrary and can be detrimental because it cannot be conditioned on information from below and to the right of the current variable (or pixel). This can lead to inefficiencies or potentially undesirable biases in the training of neural networks.

[0338] Below, we describe many fixed or learnable AOs, along with many different frameworks that can formulate them. Contextual modeling AOs can be generalized through these frameworks, and latent variables can be addressed. (a) The theoretical aspects of AI-based image and video compression and the purpose of autoregressive modeling in contextual models are described in detail. (b) Describe and illustrate various conventional and unconventional AOs that can be assigned to context modeling. (c) This section describes many frameworks that can train an automated environment (AO) using gradient descent or reinforcement learning network optimization.

[0339] Al-based image and video compression pipelines typically follow an autoencoder structure, which is built by a convolutional neural network (CNN) that constructs encoding and decoding modules, the parameters of which can be optimized by training on natural image and video datasets. The (observed) data is generally denoted as p and is assumed to be distributed according to the data distribution p(x). The feature representation after the encoder module is called the latent and denoted as y. This is entropy-encoded into a bitstream during encoding and vice versa during decoding.

[0340] The true distribution of the latent space p(y|x) cannot be practically obtained. This is because the data distribution This is because it is difficult to marginalize the joint distribution with respect to y and x in order to compute JPEG0007860563000125.jpg637. Therefore, we can only find an approximate representation of this distribution, and that is what entropy modeling does.

[0341] The true latent distribution of JPEG0007860563000126.jpg612 can be expressed as a joint probability distribution with a conditional dependent variable without loss of generality.

number

number

[0342] Applying either of the two concepts imposes constraints that invalidate the equivalence of the joint probability and factorization into conditional components as described in Equation (59), but this is often done to trade off against the complexity of the modeling. The first concept is mostly actually done for high-dimensional data, such as modeling the context based on PixelCNN where only local receptive fields are considered. An example of this process is shown in Figure 27, which shows a 3×3 receptive field and the next pixel is conditioned on local variables (where the arrows occur) instead of all the preceding variables. However, in this paper, we do not consider imposing this constraint in order to generalize the innovations presented here.

[0343] The second concept includes cases that assume a factored entropy model (conditioning only on deterministic parameters without conditioning on random variables) and a hyperprior entropy model (latent variables are all conditionally independent by conditioning on a set of hyperlatent z), and both of these cases have a 1-step AO, that is, the inference of the joint distribution is performed in a single step.

[0344] In what follows, we will describe three different frameworks for specifying AO for the serial execution of any autoregressive process for applications in AI-based image and video compression. This includes, but is not limited to, entropy modeling by a context model. Each framework provides (1) a way to define AO and (2) a way to formulate optimization techniques for AO.

[0345] The data can be assumed to be in a two-dimensional format with dimensionality M = H × W (where H is the height dimension and W is the width dimension) (single-channel image or single frame, single-channel video). The concepts presented here can be similarly applied to data with multiple channels or multiple frames.

[0346] Graphical models, more specifically directed acyclic graphs (DAGs), are extremely useful for describing probability distributions and their conditional dependency structures. The graph is constructed with nodes corresponding to the variables of the distribution and directed links (arrows) indicating conditional dependencies (variables at the head of the arrow are conditioned on the variable at the tail). As a visual example, the joint distribution illustrating the example in Figure 28 looks like this:

number

[0347] The main constraint for a directed graph to correctly describe the joint probabilities is that it does not contain directed cycles. This means that there can be no paths that start at any node on a path and end at the same node, and therefore it is a directed, non-cycled graph. The raster scan ordering follows exactly the same structure as shown in Figure 28, which shows an example of a DAG describing the joint distribution of four variables {y1, y2, y3, y4}, and in equation (60) when the variables are sorted from 1 to N in the raster scan pattern. Given our assumptions, this is an M-step AO, meaning that M passes or runs of the context model are required to evaluate the complete joint distribution.

[0348] Another AO less than M steps is checkerboard ordering. Figure 29 shows a 2-step AO where all current variables at each step are conditionally independent and can be evaluated in parallel. In Figure 29, at step 1, the distribution of the current pixel is inferred in parallel without conditioning. At step 2, the distribution of the current pixel is inferred in parallel and conditioned on all pixels of the previous step. At each step, all current variables are conditionally independent. The corresponding directed graph is shown in Figure 30. As shown in Figure 30, step 1 evaluates the topmost nodes, and step 2 is represented by the bottommost nodes with arrows indicating conditioning.

[0349] The binary mask kernel for autoregressive modeling is a convenient framework for specifying an N-step AO with N << M. Given data y, the binamask kernel approach requires splitting it into N low-resolution sub-images {y1,…,y N}. Previously, each pixel was defined as variable y i , but here, JPEG0007860563000130.jpg513 is defined as a group of K pixels or variables that are conditionally independent (conditioned by combining for future steps).

[0350] Sub-image y i is k H ×k W stride and contains elements M i,pq is extracted by convolving the data with a binary mask kernel JPEG0007860563000131.jpg651. For this to define a valid AO, the binary mask kernel must follow the following constraints.

Number

Number

[0351] Here, 1 kH,kW is size k H ×k W It is a single matrix. In other words, each mask must be unique and have only one entry of 1 (the remaining elements are 0). These are sufficient conditions to ensure that the AO is exhaustive and follows a logical conditional order. To enforce constraints (61) and 62) while establishing the AO, k H k W We can learn logits, one at a time at each position (p,q) in the kernel, and then, for example, rank the logits from highest to lowest to order the autoregressive process. Alternatively, we can apply the Gumbel softmax trick to gradually lower the temperature and eventually force one-hot coding.

[0352] Figure 31 shows an example where a 2x2 binary mask kernel, following constraints (61) and (62), generates four subimages and is conditioned according to the graphical model (but with vector variables instead of scalars) described in Figure 28. This example is very similar to the checkerboard AO described in the previous section, but with several intermediate steps added. In fact, by not conditioning between y1 and y2, and between y3 and y4, it is possible to define the AO in the binary mask kernel so that it accurately reflects the previous checkerboard AO. Previous subimage y i The choice of whether or not to impose a condition can be determined by the adjacency matrix A, which is strictly a lower triangular matrix, and the threshold condition T. In this case, A has a more practical dimension k. H k W ×k H k W This is how it should be. Following the previous example, Figure 32 shows how this is possible. Figure 32 shows an example of an adjacency matrix A that determines the graph connectivity of an AO defined in a binary mask kernel framework. Links are ignored if the relevant adjacency terms are less than the threshold T. When links are ignored, a conditionally independent structure emerges, allowing the autoregressive process to be built in fewer steps.

[0353] It is also possible to represent conventional interlacing schemes using binary mask kernels such as Adam7, which are used in PNG. Figure 33 shows the index of the Adam7 interlacing scheme. Note that this notation is used to show how the kernels are positioned and grouped, and is not actually used directly in the model. For example, 1 corresponds to one mask in the top left position. 2 corresponds to another single mask in the same position. 3's corresponds to two masks grouped together, each at the same position as index 3's. The same principle applies to the remaining indices. The kernel size is 8x8, generating 64 mask kernels, and the grouping is indexed as shown in Figure 33, resulting in the generation of 7 subimages (hence 7-step AO). This demonstrates that the binary mask kernel framework is adapted to handle common interlacing algorithms that utilize low-resolution information to generate high-resolution data.

[0354] The raster scan order can also be defined within the framework of binary mask kernels with kernel size H × W. In this case, H × W = N steps of AO, and N binary mask kernels of size H × W are organized so that they are ordered in the raster scan.

[0355] In summary, binary mask kernels are well-suited to gradient descent-based learning methods and relate to further concepts of autoregressive ordering in frequency space, as described below.

[0356] The ranking table is a third framework that characterizes AO, and under fixed ranking, it is particularly effective in describing M-step AO without the representational complexity of binary mask kernels. The concept of the ranking table is simple, Given JPEG0007860563000134.jpg513 (flattened and corresponding to the total number of variables), each AO is a ranking system of elements q,qi It is determined based on. Indexing can be performed using the argsort operator, and the ranking can be either in descending or ascending order according to the interpretation of q. The largest index is assigned as y1, and the index with the second-largest q i is assigned as y2. Indexing is performed using the argsort operator, and the ranking becomes either in descending or ascending order according to the interpretation of q.

[0357] q can be an existing quantity that conveys specific information about the source data y, such as the entropy parameter of y (learned or predicted by hyper-preference), for example, the scale parameter σ. This is because the region with high uncertainty associated with the variable having a large scale parameter σ ij is an area with information that cannot be easily obtained by the context, so an example of unprocessed visualization is shown in FIG. 34. In the formula, AO is y1, y2,..., y 16 is defined as. FIG. 34 shows an example of the visualization of the scale parameter that defines the ranking table by the argsort operator after flattening q and sorts the variables in descending order according to the magnitude of each scale parameter.

[0358] q can also be derived from existing quantities, such as the first or second derivative of the position parameter μ. Both of these can be obtained by applying the finite difference method to obtain the gradient vector (in the case of the first derivative) or the Hessian matrix (in the case of the second derivative), and are obtained before calculating q and argsort(q). Then, the ranking is established by the norm of the gradient vector, the norm of the eigenvalues of the Hessian matrix, or any measure of the curvature of the latent image. Alternatively, in the case of the second derivative, the ranking can be based on the magnitude of the Laplacian equivalent to the trace of the Hessian matrix.

[0359] Finally, q can be an entirely different entity. A fixed q can be arbitrarily predefined before training and, like hyperparameters, remains static or dynamic during training. Alternatively, it can be learned and optimized using gradient descent, or parameterized by a hypernetwork.

[0360] How you access elements of y depends on whether you want to pass the gradient through the ranking operator: · When a gradient is not needed Sort :q in ascending / descending order to access its elements, and then access the elements of y based on that order. · If a gradient is required : The order is a discrete permutation matrix P or a continuous relaxation matrix Represented as JPEG0007860563000135.jpg55, and multiplied by the sorted y and matrix: y sort =Py

[0361] When a ranking matrix is ​​optimized using gradient descent, index operators such as argsort or argmax may not be differentiable. Therefore, permutation matrices are not differentiable. The continuous relaxation of JPEG0007860563000136.jpg55 needs to be used, which can be implemented with the soft sort operator.

number

[0362] The concept of the ranking table can be extended to work with binary mask kernels as well. Matrix q has the same dimension as the mask kernel itself, and AO is specified based on the rank of the elements of q. Figure 36 visualizes an example of the concept of the ranking table applied to the binary mask kernel framework.

[0363] Another possible autoregressive model is defined by hierarchical transformation of the latent space. From this perspective, the latent is transformed into a hierarchy of variables, with lower - level hierarchical levels being conditioned on higher - level hierarchical levels.

[0364] This concept can be best explained using wavelet decomposition. In wavelet decomposition, a signal is decomposed into high - frequency and low - frequency components. This is done via the wavelet operator W. Let the latent of size H×W pixels be y 0 with the superscript 0 indicating that the latent is at the lowest (or root) level of the hierarchy. Applying the wavelet transform once, the latent is converted into a set of four smaller images y 1 ll , y 1 lh , y 1 hl , and y 1 hh , each of size JPEG0007860563000141.jpg521. H and L represent the high - frequency and low - frequency components respectively. The first letter of the tuple corresponds to the first spatial dimension of the image (e.g., height), and the second letter corresponds to the second dimension (e.g., width). Thus, for example, y 1 hl is the wavelet component of the latent image y 0 corresponding to the high - frequency in the height dimension and the low - frequency in the width dimension.

[0365] In matrix notation, it looks like this:

number

[0366] Applying this procedure recursively to the low-frequency blocks constructs a hierarchical tree of decomposition. Figure 37 shows an example of a procedure with two hierarchies. Figure 37 shows a hierarchical autoregressive order based on wavelet transforms. The top figure shows that the forward wavelet transform creates a hierarchy of variables, in this case there are two levels. The middle image shows that the transform can be reversed to restore the low-frequency elements of the previous level. The bottom image shows an autoregressive model defined by an example of a DAG between elements at one level of the hierarchy. Thus, wavelet transforms can be used to create multi-level trees.

[0367] The important thing is that the transformation matrix W is an invertible matrix (in fact, in the case of wavelet transforms, W -1 =W T All procedures are reversible. Given the last level of a hierarchy, the low-frequency components of the previous level can be easily reconstructed by simply applying the inverse transform to the last level. Then, after reconstructing the low-frequency components of the next hierarchy, the inverse transform is applied to the second hierarchy, and this is repeated to reconstruct the original image.

[0368] So, how can we construct an autoregressive order using this hierarchical structure? At each hierarchical level, an autoregressive order is defined between the elements of that level. For example, referring to the bottom image in Figure 37, the low-frequency components are at the root of the DAG at that level. Note that the autoregressive model can also be applied to the constructing elements (pixels) of each variable within the hierarchy. The remaining variables of the level are conditioned on the preceding elements of the level. After describing all the conditional dependencies of the level, we use the inverse wavelet transform to reconstruct the lowest-frequency components of the preceding level.

[0369] An autoregressive process is recursively applied until another DAG is defined between the elements at the next lowest level and the original latent variables are restored.

[0370] Thus, the autoregressive order is defined on a variable given by the level of the wavelet transform of the image, using the DAG and inverse wavelet transform between the elements at the tree level.

[0371] This can be generalized in several ways. • It does not have to be a wavelet transform; any reversible transform can be used. This includes the following: -Wavelets- For example, permutation matrices defined by a binary mask. - Other orthonormal transformations such as the Fast Fourier Transform - Learned invertible matrices - Learned invertible matrix - For example, a reversible matrix predicted by hyperpreference. • Hierarchical decomposition can also be applied to video. In this case, each level of the tree has eight components corresponding to lll, hll, lhl, hhl, llh, lhh, hhh, and hlh, where the first letter in the expression represents the time component.

[0372] Extended Lagrangian Technical examples of constrained optimization and rate distortion annealing are described in international patent application PCT / GB2021 / 052770, which is incorporated herein by reference.

[0373] Al-based compression pipelines attempt to minimize rate (R) and strain (D). The objective function is as follows:

number

[0374] International patent application PCT / GB2021 / 052770 reformulates this problem as a constrained optimization problem. The method for solving this constrained optimization problem is the Augmented Lagrangian method, as described in international patent application PCT / GB2021 / 052770. The constrained optimization problem involves solving the following: Min D (66) R=c etc. (67) In the formula, c is the target compression ratio. Note that D and R are the mean of the entire data distribution. Inequality constraints can also be used. Furthermore, the roles of R and D can be reversed. Alternatively, the rate can be minimized according to a strain constraint (which may also be a system of constraint equations).

[0375] Typically, constrained optimization problems are solved using stochastic linear optimization. That is, the objective function is computed for a small number of training samples (rather than the entire dataset), and then the gradient is computed. Next, an update step is performed, modifying the parameters of the compression algorithm and, if necessary, other parameters related to constrained optimization, such as the Lagrange multiplier. This process is repeated thousands of times until a suitable convergence criterion is reached. For example, the following steps are performed: · Using SGD, mini-batch SGD, Adam (or other optimization algorithms used to train neural networks), the loss Perform one optimization step on JPEG0007860563000144.jpg747. • Update the Lagrange multiplier: JPEG0007860563000145.jpg530, in the formula, choose a small value for ε. Repeat the above two steps until the Lagrange multiplier converges based on the target rate r0.

[0376] However, there are several problems encountered when training constrained optimization problems in a stochastic small-batch linear optimization setting. First, it is not possible to compute constraints on the entire dataset in each iteration, but typically only on the small number of training samples used in each iteration. Using such a small number of training samples in each batch makes the updates of constrained optimization parameters (such as Lagrangian Multipliers in Augmented Lagrangian) extremely dependent on the current batch, leading to large variability in training updates, potentially preventing the acquisition of an optimal solution, or even making the optimization routine unstable.

[0377] The average constraint value (such as the average rate) can be calculated over the previous N iteration steps. This has the beneficial effect of extending the constraint information from the last many optimization iteration steps so that it can be applied to the current optimization step, particularly with respect to parameter updates related to constraint optimization algorithms (such as updating the Lagrange multiplier in the extended Lagrangian). A non-exhaustive list of ways to calculate the average of the past N iterations and apply it to the optimization algorithm is as follows: • It maintains a buffer of constraint values ​​for the most recent N training samples. This buffer is, for example, The image JPEG0007860563000146.jpg637 can be used to update the extended Lagrange multiplier λ over iteration t. In the formula, avg is the general mean operator. An example of the mean operator is as follows: - Arithmetic mean (often simply called "average") - Median -geometric mean -harmonic mean - Exponential moving average -Smoothed moving average -Linear weighted moving average However, any average operator may be used.

[0378] Instead of calculating the gradient at each training step and immediately applying it to the model weights, the gradient is accumulated over N iterations, and after N iterations, one training step is performed using an averaging function.

[0379] These averaged constraint values ​​are calculated over the previous N iterations and used to update parameters of the training optimization algorithm, such as the Lagrange multiplier.

[0380] A second problem with stochastic first-order optimization algorithms is that the dataset may contain images with extremely large or extremely small constraint values ​​(e.g., extremely small or extremely large rates R). When outliers are present in the dataset, the function being trained will have to account for these outliers, potentially leading to a poor fit for more general samples. For example, updating the parameters of the optimization algorithm (such as the Lagrange multiplier) can lead to large variance and suboptimal training if there are many outliers.

[0381] Some of these outliers can be removed from the calculation of the constraint mean described above. Several possible methods for filtering (removing) these outliers are as follows: When accumulating training samples, the mean can be replaced with a trimmed mean. The difference between a trimmed mean and a regular mean is that in the trimmed version, x% of the upper and lower values ​​are not considered in the mean calculation. For example, you can trim the top and bottom 5% of the samples (in sorted order). • Trimming is not required throughout the entire training procedure; a normal average (untrimmed) can be used in the later stages of the training procedure. For example, the trimmed average can be turned off after 1 million iterations. • An outlier detector can be fitted every N iterations so that the model can either completely remove or weight those samples from the mean.

[0382] Using constrained optimization algorithms such as Augmented Lagrangian, it is possible to target a specific mean constraint (such as rate) target c on the training dataset. However, convergence to that target constraint on the training set does not guarantee the same constraint value on the validation set. This can be caused, for example, by changes between the quantization function used in training and the quantization function used in inference (test / validation). For example, it is common to use uniform noise for quantization during training, but rounding for inference (known as "STE"). Ideally, it is desirable for the constraint to be satisfied in inference, but achieving this is difficult.

[0383] The following methods are possible: • In each training step, the constraint target value is calculated using the detach operator. Here, the inference value is used in the forward pass, but the training gradient is used in the backward pass (gradient calculation). For example, when discussing rate constraints, The value JPEG0007860563000147.jpg548 is used. Here, 'detach' indicates separating the value from the auto-differentiating graph. • Update the constraint algorithm parameters with a "holding set" of images used to evaluate the model, using the same settings as for inference.

[0384] The technique described above and in international patent application PCT / GB2021 / 052770 can also be applied to AI-based video compression. In this case, Lagrangian multipliers can be applied to the rate and distortion associated with each frame of the video used in each training step. One or more of these Lagrangian multipliers can be optimized using the technique described above. Alternatively, the multipliers may be averaged over multiple frames during the training process.

[0385] The target value of the Lagrange multiplier may be set to the same value for each frame of the input video used in the training step. Alternatively, different values ​​may be used. For example, different target values ​​can be used for the I-frames and P-frames of the video. A higher target rate can be used for the I-frames than for the P-frames. The same method can be applied to the B-frames.

[0386] Similar to image compression, the target rate may be initially set to zero for one or more frames of the video used for training. If a target value is set for distortion, the target value may be set so that the initial weighting for distortion is maximized (for example, the target rate may be set to 1).

[0387] Tensor Network Al-based compression relies on modeling discrete probability mass functions (PMFs). These PMFs may seem simple at first glance. Our usual mental model starts with one discrete variable X, which has D possible values ​​X1, ..., X D It is possible to take this. Next, the construction of PMF P(X) is P i =P(X i This can be easily done by creating a table defined as follows. Of course, Pi must be non-negative and the sum must be 1, but this can be achieved, for example, by using the softmax function. This can be achieved by using JPEG0007860563000148.jpg1631. For modeling purposes, each P in this table fits a specific data distribution. i Learning it doesn't seem to be that difficult.

[0388] What about PMF for two variables, X and Y? Entry P ij =P(X i ,Y j This also seems manageable, in that it requires a 2D table containing ). This is a bit more complex. Currently the table contains D 2There are entries, but even so, as long as D is not too large, JPEG0007860563000149.jpg2646 is manageable. Next, using three variables, a 3D table is needed, and entry P ijk It is indexed by a 3-tuple.

[0389] However, this naive table-creation "approach" quickly becomes unmanageable when trying to model more than a handful of discrete variables. For example, consider modeling PMF in a 1024x1024 RGB image space, where each variable is 256. 3 It can take the possible values ​​(each color channel has 256 possible values, and we have 3 color channels). Then the required lookup table is This will result in the entry JPEG0007860563000150.jpg615. Calculating in decimal gives approximately The result will be JPEG0007860563000151.jpg68. There are many approaches to dealing with this problem, but the textbook approach in discrete modeling is to use a probabilistic graphical model.

[0390] As an alternative approach, PMFs can be modeled as tensors. A tensor is simply another term for a very large table (which, though not discussed here, possesses algebraic properties). Discrete PMFs can always be described as tensors. For example, a 2-tensor (also called a matrix) is an array with two indices, i.e., a two-dimensional table. Therefore, the above PMF P for two discrete variables X and Y... ij =P(X i ,Y j ) is a 2-tensor. N-tensor T i1 ,…,T iN This is a PMF for N discrete variables if it is an array with N indices, and if the entries in T are positive and their sum is 1. Table 1 compares the standard PMF view and the tensor view for several probabilistic concepts.

[0391] The main appeal of this perspective is that huge tensors can be modeled using the framework of tensor networks. Tensor networks are used to approximate very high-dimensional tensors by contracting them with several lower-dimensional (i.e., easier-to-handle) tensors. In other words, tensor networks are used to approximate otherwise difficult-to-handle tensors at a low rank. [Table 3]

[0392] For example, if a matrix is ​​viewed as a 2-tensor, standard low-rank approximations (such as singular value decomposition (SVD) and principal component analysis (PCA)) are tensor network factorizations. Tensor networks are a generalization of low-rank approximations used in linear algebra to multilinear maps. An example of the use of tensor networks in probabilistic modeling in machine learning is shown in "Ivan Glasser, Ryan Sweke, Nicola Pancotti, Jens Eisert, and J Ignacio Cirac. Expressive power of tensor-network factorizations for probabilistic modeling, with applications from hidden markov models to quantum machine learning. arXiv preprint, arXiv:1907.03741, 2019," which is incorporated herein by reference.

[0393] Tensor networks can be considered an alternative to graphical models. There is a correspondence between tensor networks and graphical models; probabilistic graphical models can be reconstructed as tensor networks, but the reverse is not true. Tensor networks exist for computable connection density modeling, offering strong performance guarantees, although they cannot be recast as probabilistic graphical models. In many situations, tensor networks are more expressive than traditional probabilistic graphical models such as HMMs. Given a fixed number of parameters, tensor networks experimentally outperform HMMs. Furthermore, under certain low-rank approximations, tensor networks may theoretically surpass HMMs again.

[0394] Assuming all other modeling assumptions remain the same, tensor networks may be preferred over HMMs.

[0395] An intuitive explanation for this result is that probabilistic graphs factorize connections via conditional probabilities, but typically, exponential maps... By considering only JPEG0007860563000153.jpg642, we end up modeling the connection as a Boltzmann / Gibbs distribution. This is indeed a restrictive modeling assumption. A completely different approach offered by tensor networks is to model the connection as an inner product: for some Hermitian positive (semi)definite operator H, JPEG0007860563000154.jpg527 (This modeling approach is inspired by Born's law for quantum systems). The operator H can be written as a huge tensor (or tensor network). The key point is that the entries for H are complex. It is not at all clear how (or if at all) this can be translated into a graph model. However, it presents a completely different modeling perspective that cannot be obtained elsewhere.

[0396] Let's explain what tensor network decomposition is with a simple example. Item T ijThere is a large D×D matrix T (a 2-tensor) having [something], and it is desired to make a low-rank approximation (rank-r approximation with r < D) of T. One way to do this is to find an approximate T (hat).

Number

[0397] In other words, setting T(hat) = AB, A is a D×r matrix and B is an r×D matrix. We have introduced hidden dimensions shared between A and B, and these are summed. By setting r to be very small compared to dealing with a huge D×D matrix, instead of D 2 parameters, we can reduce the computational time and power significantly to 2Dr parameters. Furthermore, in many modeling scenarios, even if r is made very small, a "sufficient" approximation of T can be obtained.

[0398] Here, let's try to model a 3-tensor with the same approach. Let T ijk be a D×D×D tensor T given with entries.

Number

[0399] Here A and C are low-rank matrices, and B is a low-rank 3-tensor. There are two hidden dimensions between A - B and B - C. One is between A and B, and the other is between B and C. In tensor network terms, these hidden dimensions may be called binding dimensions. The sum of dimensions may be called reduction.

[0400] Continuing this example, a 4-tensor can be approximated as a product of lower-dimensional tensors, but the index notation becomes cumbersome to write immediately. Instead, use a tensor network diagram, which is a concise way to convey the same calculation diagrammatically.

[0401] In a tensor network diagram, tensors are represented as blocks, and each index dimension is represented as an arm, as shown in Figure 38. The dimensionality of a tensor can be determined simply by counting the number of empty (hanging) arms. The top row of Figure 38 shows, from left to right, vectors, matrices, and N-tensors. The tensor product (summation / contraction along a particular index dimension) is represented by connecting two tensor arms. Schematically, in Figure 38, the matrix-vector product in the lower left has one hanging arm, so the resulting product is a tensor, i.e., a vector, as expected. Similarly, the matrix-matrix product in the lower right has two hanging arms, so the result is a matrix, as expected.

[0402] The tensor decomposition of the three-tensor T (hat) given by equation (69) can be diagrammed as shown in the upper part of Figure 39. Here, a specific element T of T (hat) ijk Suppose we want to access (the hat). We can fix the free index to the desired value and perform the necessary contractions.

[0403] This notation allows us to explore the possibilities of tensor network factorization used in probabilistic modeling. The key idea is that the true connection distribution of high-dimensional PMFs is impractical. We must approximate it, and we do so using tensor network factorization. These tensor network factorizations can be learned to fit the training data. Not all tensor network factorizations are suitable. It may be necessary to constrain the tensor network entries to be non-negative and to sum to 1.

[0404] One example of such an approach is to use Matrix Product State (MPS) (also known as tensor training). PMF P(X1,…,X N ) to tensor Let's say we want to model it as JPEG0007860563000157.jpg611.

number

[0405] To ensure that the sum of the entries is 1, a normalization constant is calculated by summing it over all possible states. While calculating this normalization constant for a general N tensor might be impractical, in the case of MPS, its linear nature allows it to be calculated in O(N) time. "Linearity" here means that the rows of tensor training can be manipulated to perform tensor products sequentially, one by one. (Both tensors and their tensor network approximations are multilinear functions).

[0406] MPS is very similar to a Hidden Markov Model (HMM). In fact, there is a correspondence: an MPS with positive terms corresponds exactly to an HMM.

[0407] Further examples of tensor network models include Born Machines and Locally Purified States (LPS), both of which draw inspiration from quantum systems. Quantum systems assume Born's law, which states that the probability of an event X occurring is proportional to the square of the norm of the inner product <·,H·> of a positive (semi) constant volume Hermitian operator H with the event X. This is a powerful probabilistic modeling framework, but it does not have an obvious connection to graphical models.

[0408] The locally purified state (LPS) takes the form shown in Figure 40. In LPS, the construct element is A k There are no restrictions on the sign of a tensor; it can be positive or negative. In fact, A k It can have a complex number value. In this case, JPEG0007860563000159.jpg54 is a tensor obtained by taking the complex conjugate of entries A. α k Dimensions are sometimes called combined dimensions, β k Dimensions are sometimes called refined dimensions.

[0409] The elements of T (hat) are guaranteed to be positive by the fact that contraction along the purification dimension yields a positive value (for a complex number z, JPEG0007860563000160.jpg511). {i1,…,i N If we consider} as one huge multi-exponential I, then LPS is the diagonal of a huge matrix (after all hidden dimensions have been collapsed), and evaluating LPS is equivalent to an inner product operating on state space.

[0410] Similar to MPS, the calculation of the normalization constant for LPS is fast, taking O(N) time. The Born Machine is a special case of LPS, where the size of the purification dimension is 1.

[0411] Tensor trees are another example of tensor networks. In the leaves of the tree, dangling arms are contracted by data. However, hidden dimensions are located within the tree, and the tree nodes store tensors. The edges of the tree are the dimensions of the tensors being contracted. A simple tensor tree is shown in Figure 41. The tree nodes store tensors, and the edges represent contractions between tensors. The leaves of the tree have indices that are contracted by data. Tensor trees can be used to model multi-resolution and / or multi-scale probability distributions.

[0412] Each tensor node can have a refined dimension added so that it contracts with the complex conjugate of that node. This defines the inner product according to the Hermitian operators given by the tensor tree and its complex conjugate.

[0413] Another example of a tensor network is Projected Entangled Pair States (PEPS). In this tensor network, tensor nodes are arranged in a regular grid and contract with their immediate neighbors. Each tensor has an additional dangling arm (free index) that contracts with data (such as latent index values). In certain senses, PEPS is similar to Markov random fields and the Ising model. A simple example of PEPS in a 2x2 image patch is shown in Figure 42.

[0414] The calculations of tensor networks (such as calculating the connection probabilities, conditional probabilities, and marginal probabilities of PMFs, or the entropy of PMFs) can be greatly simplified and sped up by converting the tensors to canonical form, as will be explained in detail below. All of the tensor networks mentioned above can be placed in canonical form.

[0415] Since the basis on which the hidden dimensions are represented is not fixed (so-called gauge degrees of freedom), the basis on which these tensors are represented can be easily changed. For example, if a tensor network is placed in canonical form, almost all tensors can be transformed into orthonormal matrices (unitary matrices).

[0416] This is possible by sequentially performing a series of decompositions on the tensors of the tensor network. These decompositions include QR decomposition (and its variations, RQ, QL, and LQ), SVD decomposition, spectral decomposition (if available), Schur decomposition, QZ decomposition, and Takagi decomposition. The procedure for describing a tensor network in canonical form works by decomposing each tensor into orthonormal (unitary) components and other factors. The other factors are contracted with neighboring tensors, modifying them. Then, the same procedure is applied to neighboring tensors and their neighbors until all but one tensor is orthonormal (unitary).

[0417] The remaining tensors that are not orthonormal (unitary) are called core tensors. Core tensors are similar to the diagonal matrix of singular values ​​in SVD decomposition and contain spectral information of the tensor network. Core tensors can be used, for example, to calculate the normalization constant of the tensor network or the entropy of the tensor network.

[0418] Figure 43 shows an example of the procedure for converting an MPS to canonical form, from top to bottom. The core tensors are sequentially decomposed into QR forms. The R tensor is contracted with the next tensor in the chain. This procedure is repeated until everything except the core tensor C is in canonical form.

[0419] Here, we will discuss in more detail the use of tensor networks for probabilistic modeling in AI-based image and video compression. As mentioned above, in an AI-based compression pipeline, the input image (or video) x is mapped to a latent variable y via an encoding function (typically a neural network). The latent variable y is quantized to integer values ​​y (hat) using a quantization function Q. These quantized latent variables are converted into a bitstream using a reversible encoding method such as entropy coding, as described above. Arithmetic coding or decoding is an example of such encoding and will be used as an example in further discussion.

[0420] This lossless encoding process requires a probabilistic model: an arithmetic encoder / decoder requires a probabilistic mass function q(y(hat)) to convert integer values ​​into a bitstream. During decoding, a PMF is similarly used to convert the bitstream back into a quantized latent, which is then passed through a decoder function (typically a neural network) to return the reconstructed image x(hat).

[0421] The size (compression rate) of the bitstream is largely dependent on the quality of the probabilistic (entropy) model. A better, more powerful probabilistic model will result in a smaller bitstream for the same quality reconstructed image.

[0422] Arithmetic encoders typically operate in one-dimensional PMFs. To accommodate this modeling constraint, it is usually assumed that the combined PMF q(y(hat)) is independent, and the pixels Each element in JPEG0007860563000161.jpg55 is a one-dimensional probability distribution. It is modeled by JPEG0007860563000162.jpg615. The binding density is then modeled as follows:

number

[0423] In any case, this modeling approach is basically, Let's assume a one-dimensional distribution for each of the 55 pixels in JPEG0007860563000164.jpg. This can be restrictive. A better approach would be to fully model the combined distribution. Then, when encoding or decoding the bitstream, we can compute the one-dimensional distribution required for the arithmetic encoder / decoder as conditional probabilities.

[0424] Tensor networks can be used to model the connection distribution. This can be done as follows: Quantized latent Suppose we are given JPEG0007860563000165.jpg630. Each latent pixel is embedded (or lifted) into a higher-dimensional space. In this higher-dimensional space, integers are represented by vectors lying at the vertices of a probabilistic simplex. For example, y i D possible integer values Let's call it JPEG0007860563000166.jpg677. This embed is Map JPEG0007860563000167.jpg55 to a D-dimensional one-hot vector, where slots corresponding to integer values ​​are 1 and all other slots are 0.

[0425] For example, each JPEG0007860563000168.jpg55 takes values ​​of {-3,-2,-1,0,1,2,3}. Let's assume it's JPEG0007860563000169.jpg612. Next, the embedding is The filename will be JPEG0007860563000170.jpg536.

[0426] Therefore, the embedding is JPEG0007860563000171.jpg649 This maps to JPEG0007860563000172.jpg633. In effect, this maps y (hat) existing in M-dimensional space to D M Map it into a dimensional space.

[0427] Here, each of these terms in the embedding can be considered a dimension that indexes a higher-dimensional tensor. Therefore, the approach we take is to model the connection probability density via a tensor network T (hat).

number

[0428] During encoding / decoding, arithmetic encoders / decoders cannot use joint probabilities. Instead, they must use a one-dimensional distribution. Conditional probabilities can be used to calculate the one-dimensional distribution.

[0429] Conveniently, conditional probabilities can be easily calculated by marginalizing the hidden variables, fixing the precondition variables, and normalizing the results. All of this can be easily done using tensor networks.

[0430] For example, suppose we encode / decode in raster scan order. Then, for each pixel, we need the following conditional probability: JPEG0007860563000174.jpg565. Each of these conditional probabilities can be easily calculated by reducing the tensor network to hidden variables (unseen variables), fixing the index of the condition variable, and normalizing it with an appropriate normalization constant.

[0431] This is a particularly fast procedure if the tensor network is in canonical form, because in this case, the collapse along the hidden dimension is equivalent to multiplication by the unit.

[0432] Tensor networks can be applied to the joint stochastic modeling of PMF across all latent pixels or patches of latent pixels, or to the joint stochastic modeling across channels of latent representations, or any combination thereof.

[0433] Tensor network-based stochastic modeling can be easily incorporated into an Al-based compression pipeline as follows: The tensor network is learned during end-to-end training and fixed after training. Alternatively, the tensor network or its components can be predicted by a hypernetwork. The tensor network can be used additionally or alternatively for entropy coding and decoding of hyperlatencies in the hypernetwork. In this case, the parameters of the tensor network used for entropy coding and decoding of hyperlatencies may be learned during end-to-end training and fixed after training.

[0434] For example, a hypernetwork can predict the core tensor of a tensor network patch by patch. In this scenario, the core tensor changes between pixel patches, while the remaining tensors are learned and fixed between pixel patches. See, for example, Figure 44, which shows an Al-based compression encoder with a tensor network predicted by a hyperencoder / hyperdecoder, and Figure 45, which shows an Al-based compression decoder with a tensor network predicted by a hyperdecoder for the use of tensor networks in an Al-based compression pipeline. The features corresponding to the features shown in Figures 1 and 2 can be assumed to be the same as described above. In these examples, the residual ξ = y - μ is quantized, encoded, and decoded using a tensor network probabilistic model. In this case, the parameters of the tensor network are T y It is represented as follows. In the examples in Figures 44 and 45, the quantized hyperlatency z (hat) is further expressed as T z The data is encoded and decoded using a ssol network probabilistic model with parameters represented by .

[0435] Instead of using a hypernetwork to predict tensor network components (or possibly in combination with one), parts of the tensor network may be predicted using a context module that uses previously decoded latent pixels.

[0436] During training of an Al-based compressed pipeline using a tensor network probabilistic model, the tensor network can be trained for non-integer latents (where y is y, not y(hat) = Q(y), and Q is the quantization function). To do this, the embedding function e can be defined for non-integer values. For example, the embedding function can be constructed as a tent function that takes values ​​of I at appropriate integer values, is zero at all other integer values, and linearly interpolates between them. This performs multilinear interpolation. Other real-valued extensions to the embedding scheme are usable as long as they match the original embedding for integer-valued points.

[0437] The performance of a tensor network entropy model can potentially be improved during training through some form of regularization. For example, entropy regularization can be used. In this case, the entropy H(q) of the tensor network can be calculated, and its multiples can be added to or subtracted from the training loss function. Note that the entropy of a normal form tensor network can be easily calculated by calculating the entropy of the core tensor.

[0438] Hyper-Hyper Network The current functionality and scope of use of training techniques for auxiliary hyper-hyper-precision algorithms for use (but not limited to) in compressing image and video data based on AI and deep learning are described below.

[0439] A commonly used network configuration in AI-based image and video compression is the autoencoder. This consists of an encoder module that transforms input data into a "latency" (y), often modeled as a set of pixels, which is an alternative representation of the input data; and a decoder module that takes the set of latents and transforms them back into the input data (or something as close to it as possible). Because the "latent space," where each latent pixel represents one dimension, has high-dimensional properties, a so-called "entropy model" p(y(hat)) is used to "fit" a parametric distribution into the latent space. The entropy model is used to transform y(hat) into a bitstream using a lossless arithmetic encoder. The parameters of the entropy model ("entropy parameters") are learned within the network. The entropy model can be learned directly or predicted by a hyperplier structure. A diagram of this structure is shown in Figure 1.

[0440] The entropy parameter is most commonly constructed using a position parameter and a scale parameter (often expressed as positive real values) (although it is not limited to these). Naturally, there are many more distribution types, both parametric and nonparametric, and the types of parameters are also diverse.

[0441] The hyperplier structure (Figure 2) introduces an additional set of "latencies" (z) through a set of transformations that predict the entropy parameter. Assuming that y is modeled as a Gaussian distribution, the hyperplier model can be defined by the following equation:

number

[0442] Hyperpliers can be added to models that already have trained hyperpliers, sometimes called auxiliary hyperpliers. This technique is applied, but is not limited to, improving the model's ability to capture low-frequency features and enhancing performance on "low-rate" images. Low-frequency features are present in images when there are no abrupt color changes along their axis. Therefore, an image with many low-frequency features will have only one color throughout the entire image. The amount of low-frequency features in an image can be determined by extracting its power spectrum.

[0443] A hyperhyperplier can be trained in conjunction with a hyperplier and an entropy parameter. However, using a hyperhyperplier for all images can be computationally expensive. To give the network the ability to model low-frequency features while maintaining performance for non-low-rate images, an auxiliary hyperplier can be employed, used only when the image fits certain characteristics, such as being low-rate. An example of a low-rate image is one where the bits per pixel (bpp) is approximately less than 0.1. An example of this is shown in Algorithm 3.

[0444] A supplementary hyperhyperplier framework allows the model to be tuned only when necessary. Once trained, a flag indicating the need for a hyperhyperplier for a particular image can be encoded into the bitstream. This approach can be generalized to an infinite number of constructors of entropy models, such as hyperhyperpliers.

[0445] [Table 4]

[0446] The most direct way to train a hyper-hyperplier is to "freeze" an existing trained hyperplier network, including the encoder and decoder, and optimize only the weights of the hyper-hypermodules. In this book, "freezing" means that the weights of the frozen modules are not trained, and gradients for training the modules that are not frozen are not accumulated. By freezing the existing entropy model, the hyperplier can be modified to bias its parameters toward low-rate images, similar to μ and σ in the case of a normal distribution.

[0447] Using this training scheme offers several advantages. Because gradients need to be calculated and stored with fewer parameters, training time and memory consumption can scale better with respect to image size. • Hyperpliers can be trained indefinitely, and one hyperplier can be frozen, allowing another hyperplier to be trained on top of it.

[0448] One approach is to first train the hyperplier network N times. Once N iterations are reached, the entropy model can be frozen, and the system can switch to the hyperplier when the image is low-rate. This allows the hyperplier model to specialize in images it is already proficient in, and the hyperplier functions as intended. Algorithm 4 shows a training scheme. This training can also be performed by dividing the image into NxN size K blocks and then applying this scheme to those blocks, so that the system is trained only on low-frequency regions in the image.

[0449] Another possibility is not to wait N iterations before starting to train the hyperplier, as shown in Algorithm 5.

[0450] [Table 5]

[0451] [Table 6]

[0452] To classify an image as low-rate, various criteria can be chosen, such as using a rate calculated using a distribution selected as a prior distribution, using the mean or median of the image's power spectrum, or using the median or mean of the frequencies obtained from the Fast Fourier Transform.

[0453] Data augmentation can create sufficient data by increasing the number of samples with low-frequency features associated with low-rate images. There are various methods for correcting images. • Upsample all images to a fixed size NxN. • Use a constant upsampling factor. • Randomly sample upsampling coefficients from the distribution. Uniform distribution, Gaussian distribution, Gumbel distribution, Laplacian distribution, Gaussian mixture distribution, geometric distribution, Student's t-distribution, nonparametric distribution, CHI 2 You can choose any distribution from the following: the standard distribution, the beta distribution, the gamma distribution, the Pareto distribution, and the Cauchy distribution. • Apply a smoothing filter to the image. Any filter will work: average, weighted average, median, Gaussian, or bilateral. Unless it is smaller than a specific threshold N, use the actual image size, then upsample to a fixed size and use a constant upsampling coefficient, or sample the upsampling coefficient from the distribution as described above.

[0454] In addition to image upsampling or blurring, random cropping may also be performed.

Claims

1. A method for training one or more neural networks, wherein the one or more neural networks are for use in encoding, transmitting, and decoding lossy images or videos, and the method is The first computer system receives an input image, The steps include: encoding the input image using a first neural network to generate a latent representation; The steps include entropy encoding the aforementioned latent representation, The steps include sending the entropy-encoded latent representation to a second computer system, The steps include entropy decoding the entropy-encoded latent representation, A step of decoding the latent representation using a second neural network to generate an output image, wherein the output image is an approximation of the input image. A step of determining a quantity based on the difference between the output image and the input image, The steps include updating the parameters of the first neural network and the second neural network based on the determined quantity, The process includes the step of repeating the above steps using a first set of input images to generate a first trained neural network and a second trained neural network, The entropy decoding of the entropy-encoded latent representation is performed pixel by pixel. The order of decoding for each pixel is further updated based on the determined amount. method.

2. The method according to claim 1, wherein the order of decoding each pixel is based on the latent representation.

3. The method according to claim 1, wherein the entropy decoding of the entropy-encoded latent representation includes operations based on previously decoded pixels.

4. The method according to claim 1, wherein determining the order of the pixel-by-pixel decoding includes ordering a plurality of pixels of the latent representation in a directed aperiodic graph.

5. The method according to claim 1, wherein the determination of the order of the pixel-by-pixel decoding is performed on the latent representation using a plurality of adjacency matrices.

6. The method according to claim 1, wherein determining the order of the pixel-by-pixel decoding includes dividing the latent representation into a plurality of sub-images.

7. The method according to claim 6, wherein the plurality of sub-images are obtained by convolving the latent representation with a plurality of binary mask kernels.

8. The method according to claim 1, wherein the determination of the order of the pixel-by-pixel decoding includes ranking a plurality of pixels of the latent representation based on the magnitude of a quantity associated with each pixel.

9. The method according to claim 8, wherein the quantity associated with each pixel is a position or scale parameter associated with that pixel.

10. The method according to claim 8, wherein the amount associated with each pixel is additionally updated based on the evaluated difference.

11. The method according to claim 1, wherein the determination of the order of the pixel-by-pixel decoding includes wavelet decomposition of a plurality of pixels of the latent representation.

12. The method according to claim 11, wherein the order of decoding for each pixel is based on the frequency components of the wavelet decomposition related to the plurality of pixels.

13. The steps include: encoding the latent representation using a fourth trained neural network to generate a hyperlatent representation; The steps include transmitting the hyperlatent representation to the second computer system, A step of decoding the hyperlatent representation using a fifth trained neural network, further comprising the step of the sequence of pixel-by-pixel decoding being based on the output of the fifth trained neural network. The method according to claim 1.

14. A method for encoding, transmitting, and decoding lossy images and videos, wherein the method is The first computer system receives an input image, The steps include: encoding the input image using a first trained neural network to generate a latent representation; The steps include entropy encoding the aforementioned latent representation, The steps include sending the entropy-encoded latent representation to a second computer system, The step of entropy decoding the entropy-encoded latent representation: The process includes the step of decoding the latent representation using a second trained neural network to generate an output image, wherein the output image is an approximation of the input image. The first trained neural network and the second trained neural network are trained according to the method described in any one of claims 1 to 13. method.

15. A method for encoding and transmitting lossy images or videos, wherein the method is The first computer system receives an input image, The steps include: encoding the input image using a first trained neural network to generate a latent representation; The steps of entropy encoding the latent representation and The steps include: sending an entropy-encoded latent representation, The first trained neural network is trained according to the method described in any one of claims 1 to 13. method.

16. A method for receiving and decoding a lost image or video, wherein the method is The steps include receiving the entropy-encoded latent representation transmitted according to the method of claim 15 in a second computer system, The process includes the step of decoding the latent representation using a second trained neural network to generate an output image, wherein the output image is an approximation of the input image. The second trained neural network is trained according to the method described in any one of claims 1 to 13. method.

17. A data processing system configured to perform the method described in any one of claims 1 to 13.

18. A data processing device configured to perform the method described in claim 15.

19. A computer program that includes an instruction causing the computer to perform the method described in claim 15 when the program is executed by the computer.

20. A computer-readable storage medium that, when executed by a computer, includes an instruction causing the computer to perform the method described in claim 15.

Images