Method and apparatus for encoding or decoding data representing image data

Abstract

In some examples, a method for generating an input tensor representing image data to be encoded by an encoder includes: processing the image data using a selected global transform to generate a first tensor, the first tensor including the transformed image data; processing the first tensor through one or more layers of a selected neural network to generate the input tensor, wherein the input tensor includes a preprocessed tensor representation of the image data; providing a decoder with at least one of the following: a set of configuration parameters for the selected global transform, a first index representing the selected global transform, and a second index representing the selected neural network; and providing the input tensor to an encoder for generating a bitstream representing the encoded image data.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the technical field of encoding and decoding data based on neural network architectures. Specifically, some embodiments relate to methods and apparatus for encoding and decoding images and / or videos in a bitstream using multiple processing layers. Background Technology

[0002] Image and / or video decoding (e.g., encoding and decoding) is widely used in various scenarios, such as broadcasting digital TV, transmission over the Internet and mobile networks, real-time conversational applications such as video chat and video conferencing, and so on. In such applications, data representing images and / or video can be compressed. For example, images shared between participants in a video conference can be compressed to, for example, reduce latency during the image sharing process.

[0003] Lossy or irreversible image compression algorithms that use inaccurate approximations and / or partial data to represent images require a trade-off between compression and perceived quality. However, the concept of perceived quality differs for natural images and so-called screen content, which can include synthetic images containing text and graphic regions, such as text, graphics, and illustrations (e.g., presentation slides, spreadsheets, computer desktop images, etc.), where distortion leads to varying degrees of degradation.

[0004] With the rapid development and widespread adoption of smartphones and cloud computing, computer-generated screen content images (SCIs) have become an important medium for representing large amounts of information. Therefore, compression algorithms typically optimized for natural images often fail when processing screen content. For example, a slight defect might be invisible in a sunset image, but could render the captions in an SCI unreadable.

[0005] In learning-based image compression (AI codecs), the performance gap between natural content and screen content is particularly pronounced, even if the codec has been trained using data related to both natural and screen content. Summary of the Invention

[0006] The present invention provides a method and apparatus for improving image compression, such as lossy compression of a screen content image including a composite image, the composite image comprising at least one of a text region and a graphic region.

[0007] The foregoing and other objectives are achieved through the subject matter of the independent claims. Other implementations will be apparent from the dependent claims, the specification, and the drawings.

[0008] Specific embodiments are set forth in the appended independent claims, and other embodiments are set forth in the dependent claims.

[0009] According to a first aspect, the present invention relates to a method for generating an input tensor representing image data to be encoded by an encoder, the method comprising: processing the image data using a selected global transform to generate a first tensor, the first tensor including the transformed image data; processing the first tensor through one or more layers of a selected neural network to generate the input tensor, wherein the input tensor includes a preprocessed tensor representation of the image data; providing a decoder with at least one of the following: a set of configuration parameters for the selected global transform, a first index representing the selected global transform, and a second index representing the selected neural network; and providing the input tensor to an encoder for generating a bitstream representing the encoded image data.

[0010] Therefore, a transform space is introduced, in which typical codecs (e.g., autoencoders) are more efficient in compressing screen content images. Specifically, a preprocessing mechanism includes: first processing the image data using a global transform, which is a deterministic non-spatial transform (i.e., the output depends only on the input), the global transform potentially including a linear transform; then processing using a non-linear transform, which is a spatial transform, potentially an neural network-based transform. According to one example, this processing order is also preserved in the decoding operation (e.g., processing with a global transform first, followed by a non-linear transform). That is, during encoding, image data is preprocessed in a pipeline where the processing order includes processing with a global transform first, followed by a non-linear transform. For example, when decoding an encoded image, the data is first processed with a global transform, followed by a non-linear transform. In other words, the processing order in the decoding pipeline is preserved and follows the same order as the encoding pipeline; that is, post-processing involves first decoding the encoded image data using the inverse version of the global transform (if available), followed by decoding using a nonlinear transform symmetric to the nonlinear transform used in encoding. In one example, the neural networks used for encoding and decoding are symmetric and have the same size, the same number of channels, and the same number of layers. However, they can be trained separately and use separate sets of weights.

[0011] In one example, a global transformation may include a linear transformation that depends on the input value of a single pixel in the image, while a non-linear transformation may include a learnable transformation that may depend on, for example, the values ​​of pixels in a local neighborhood of the image.

[0012] In one possible implementation of the first aspect, the preprocessed tensor representation of the image data can be generated by combining the first tensor and the latent representation of the first tensor by adding corresponding components from the first tensor and the latent representation of the first tensor.

[0013] The first tensor may include a transformed version of the image to be compressed. For example, the first tensor may represent a forward transformation of the image, where the image data representing the image to be compressed has been transformed using a selected global transformation (e.g., a linear transformation).

[0014] In one implementation of the first aspect, the global transformation may include a color space transformation for modifying the color space of the image data. The global transformation may include a resolution transformation for modifying the resolution of the image data. The global transformation may include a desaturation transformation for reducing the color saturation values ​​of one or more color channels of the image data. The global transformation may include an alpha blending transformation for blending the image data with one or more background image datasets. The alpha blending coefficient may be selected in the range of 0 to 1. In an example, an alpha blending coefficient of 0.8 may be used. At least one of the one or more background image datasets may include a grayscale or desaturated version of the image data. The global transformation may include a linear global transformation. The global transformation may include a transformation that is identical for every pixel of the image represented by the image data.

[0015] According to one example, the first tensor can be processed using a convolutional kernel to generate a first intermediate latent tensor, the first intermediate latent tensor including an increased number of channels for the image data; the first intermediate latent tensor can be processed using a depthwise convolutional kernel to generate a second intermediate latent tensor; the second intermediate latent tensor can be processed using a convolutional kernel to generate the input tensor. In response to the determination of the content of the image data, the application of the global transformation and the processing of the first tensor through one or more layers of the selected neural network can be bypassed. In one example, the neural network for the nonlinear transformation may include a neural network based on mobile inverted bottleneck convolution (MBConv).

[0016] In other words, start / stop signals (e.g., 0 for off, 1 for on, etc.) can be provided to indicate whether the image data should be preprocessed using a linear / nonlinear combination of the transforms described herein. Similarly, the decoder can use a linear / nonlinear combination of the transforms to bypass the decoding of the encoded data when a "off" signal indicating that post-processing should be bypassed is present. Thus, in one example, data can be provided to the decoder representing the following indication: in response to a determination of the content of the image data, bypassing the application of the global transform and the processing of the first tensor through one or more layers of the selected neural network.

[0017] According to a second aspect, the present invention relates to a method for encoding an input tensor representing image data to generate encoded image data, the method comprising: providing an input tensor generated according to the method of the first aspect; and encoding the input tensor to generate a bitstream representing the encoded image data. For example, the input tensor may be provided to an image encoder to generate the encoded image data.

[0018] According to a third aspect, the present invention relates to a preprocessor structure for an encoder for encoding an input tensor representing image data, the preprocessor structure including processing circuitry for: processing the image data using a selected global transform to generate a first tensor, the first tensor including transformed image data; processing the first tensor through one or more layers of a selected neural network to generate the input tensor, wherein the input tensor includes a preprocessed tensor representation of the image data; providing a decoder with at least one of the following: a set of configuration parameters for the selected global transform, a first index representing the selected global transform, and a second index representing the selected neural network; and providing the input tensor to the encoder for generating encoded image data.

[0019] In one implementation of the third aspect, the preprocessor structure may be provided as part of an image capture and / or processing pipeline in a mobile device (e.g., a smartphone). The image data may be associated with screen content images.

[0020] According to a fourth aspect, the present invention relates to a method for decoding a bitstream representing encoded image data, the method comprising: receiving at a decoder at at least one of the following: a set of configuration parameters for selecting a global transform, a first index representing the selected global transform, and a second index representing a selected neural network; decoding the bitstream using the decoder to generate an initial representation of the decoded image data; processing the initial representation of the decoded image data using a global inverse transform selected based on the first index, wherein the global inverse transform is configured based on the set of configuration parameters to generate an output tensor, the output tensor comprising a preprocessed tensor representation of the image data; and processing the output tensor through one or more layers of a neural network to generate output image data.

[0021] In one implementation of the fourth aspect, the decoder may receive an index representing a bypass start / stop signal from an encoder as described herein. That is, the decoder obtains the index from the encoder, along with an index associated with a selected linear and / or nonlinear transformation, and transformation parameters for the linear and / or nonlinear transformation.

[0022] The decoder can decode the reconstruction of the original image encoded by the encoder. If the bypass start / stop signal / index is "on," indicating that post-processing should be performed (as opposed to "off," indicating that post-processing should not be performed because no pre-processing has occurred), this can include post-processing as described herein.

[0023] In one example, the neural network can be selected based on the second index. The neural network selected by the decoder based on the second index can be a symmetric or corresponding version of the neural network used by the encoder in the preprocessing stage.

[0024] The global inverse transform may include a color space transformation, which modifies the color space of the initial representation of the decoded image data. The global inverse transform may include a resolution transformation, which modifies the resolution of the initial representation of the decoded image data. The global inverse transform may include a saturation transformation, which increases the color saturation value of one or more color channels in the initial representation of the decoded image data. The global inverse transform may include a sub-alpha blending transformation, which generates a difference between the initial representation of the decoded image data and one or more background image datasets.

[0025] Sub-alpha mixing coefficients can include the multiplicative inverses of the alpha mixing coefficients used in the forward transform (i.e., in the encoder / preprocessing stage of the global transform). For example, when an alpha mixing coefficient of 0.8 is used in preprocessing, the sub-alpha mixing coefficients can be 1.25.

[0026] The output tensor can be processed using a convolutional kernel to generate a first intermediate latent tensor, the first intermediate latent tensor including an increased number of channels for the image data; the first intermediate latent tensor can be processed using a depthwise convolutional kernel to generate a second intermediate latent tensor; the second intermediate latent tensor can be processed using a convolutional kernel to generate the output image data.

[0027] In one example, the application of the global inverse transform and the processing of the output tensor through one or more layers of the selected neural network can be bypassed based on data received from the encoder (e.g., an index representing a bypass start / stop signal).

[0028] The global transformation may include a linear global transformation, and / or the inverse global transformation may include a linear inverse global transformation. The global transformation may include a transformation that is exactly the same for every pixel of the image represented by the image data, and / or the inverse global transformation may include an inverse transformation that is exactly the same for every pixel of the image represented by the image data.

[0029] According to a fifth aspect, the present invention relates to a method for decoding a bitstream representing encoded image data to generate output image data, the method comprising: providing a bitstream representing encoded image data; and, according to the fourth aspect, decoding the bitstream representing the encoded image data to generate the output image data.

[0030] The bitstream may include the bitstream generated according to the method described in the first aspect.

[0031] According to a sixth aspect, the present invention relates to a post-processor structure for a decoder, the decoder being used to decode a bitstream representing encoded image data, the post-processor structure including processing circuitry configured to: receive from an encoder at least one of: a set of configuration parameters for selecting a global transform, a first index representing the selected global transform, and a second index representing a selected neural network; receive an initial representation of the decoded image data, the initial representation of the decoded image data including a preprocessed tensor representation of the image data; process the initial representation of the decoded image data using a global inverse transform selected based on the first index, wherein the global inverse transform is configured based on the set of configuration parameters to generate the initial representation of the decoded image data; and process the initial representation of the decoded image data through one or more layers of a neural network to generate output image data. The post-processor structure may include processing circuitry configured to: select and / or configure the neural network based on the second index.

[0032] According to a seventh aspect, the present invention relates to a computer program stored on a non-transitory medium and comprising code instructions that, when executed on one or more processors, cause the one or more processors to perform the method according to the first aspect and / or the fourth aspect.

[0033] Optionally, the above and other embodiments may each individually or in combination include one or more of the following features.

[0034] Encoding and / or decoding and / or preprocessing and / or postprocessing can be implemented in hardware or in hardware that executes the corresponding software.

[0035] The method according to the first aspect of the present invention can be executed by the preprocessor structure according to the third aspect of the present invention. Other features and implementations of the method according to the first aspect of the present invention can correspond to corresponding features and implementations of the preprocessor structure according to the third aspect of the present invention. The advantages of the method according to the first aspect can be the same as the advantages of the corresponding implementations of the preprocessor structure according to the third aspect.

[0036] The method described in the fourth aspect of the present invention can be executed by the post-processor structure described in the sixth aspect of the present invention. Other features and implementations of the method described in the fourth aspect of the present invention can correspond to the corresponding features and implementations of the post-processor structure described in the sixth aspect of the present invention. The advantages of the method according to the fourth aspect can be the same as the advantages of the corresponding implementations of the post-processor structure according to the sixth aspect.

[0037] The following drawings and description illustrate one or more embodiments in detail. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0038] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings, in which: Figure 1 A schematic diagram of the processing channels of each layer of a neural network is shown; Figure 2 A schematic diagram of an autoencoder type of neural network is shown; Figure 3A A schematic diagram of an exemplary network architecture including an encoder and decoder with a super-prior model is shown. Figure 3B A schematic diagram of a general network architecture including a super-prior model encoding end is shown; Figure 3C A schematic diagram of a general network architecture including a super-prior model at the decoding end is shown; Figure 4 A schematic diagram of an exemplary network architecture including an encoder and decoder with a super-prior model is shown. Figure 5 A block diagram illustrating the structure of an encoder based on an example is shown; Figure 6 A block diagram of the global transformation based on the example is shown; Figure 7 A block diagram illustrating a neural network implementation based on an example is shown; Figure 8 A block diagram illustrating the structure of a decoder based on an example is shown; Figure 9 A block diagram of an example encoding system for implementing embodiments of the present invention is shown; Figure 10 A block diagram of another example of an encoding system for implementing embodiments of the present invention is shown; Figure 11 A block diagram illustrating an example of an encoding or decoding apparatus for implementing embodiments of the present invention is shown; Figure 12 The example shown can be used as Figure 9 A block diagram of one or both of the source and destination devices; Figure 13 A block diagram illustrating the coding used to implement an embodiment of the present invention is shown.

[0039] Similar reference numerals and names in different figures can represent similar elements. Detailed Implementation

[0040] In the following description, reference is made to the accompanying drawings, which form part of this invention, and which illustrate by way of description specific aspects of embodiments of the invention or aspects in which embodiments of the invention may be used. It should be understood that embodiments of the invention can be used in other aspects and include structural or logical changes not depicted in the drawings. Therefore, the following detailed description should not be construed in a limiting sense, and the scope of the invention is defined by the appended claims.

[0041] For example, it should be understood that the disclosure relating to the described method is equally applicable to the corresponding device or system used to perform the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units (e.g., functional units) to implement the described one or more method steps (e.g., one unit implements one or more steps, or multiple units respectively implement one or more of the one or more steps), even if such one or more units are not explicitly described or shown in the drawings. On the other hand, for example, if a particular apparatus is described based on one or more units (e.g., functional units), the corresponding method may include a step to perform the function of one or more units (e.g., one step implements the function of one or more units, or multiple steps respectively implement the function of one or more of a plurality of units), even if such one or more steps are not explicitly described or shown in the drawings. Furthermore, it should be understood that, unless otherwise stated, features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0042] The following provides an overview of some of the technical terms used and the framework for embodiments of the present invention that may be employed.

[0043] Artificial Neural Networks Artificial neural networks (ANNs), or connectionist systems, are computational systems vaguely inspired by the biological neural networks that make up the animal brain. These systems “learn” to perform tasks by considering examples, typically without being programmed using task-specific rules. For example, in image recognition, these systems might learn to recognize images containing cats by analyzing exemplary images manually labeled “cat” or “no cat” and using the results to identify cats in other images. These systems do this without prior knowledge about cats, such as that cats have fur, tails, whiskers, and faces. Instead, they automatically generate identifying features from the examples they process.

[0044] ANNs are based on a collection of connection units or nodes called artificial neurons, which loosely model neurons in the biological brain. Each connection, like a synapse in the biological brain, can transmit signals to other neurons. Artificial neurons receive signals, process them, and can then send signals back to the neurons connected to them.

[0045] In ANN implementations, the "signals" at connections are real numbers, and the output of each neuron is calculated using a non-linear function of the sum of its inputs. These connections are called edges. Neurons and edges typically have weights that adjust as they learn. These weights increase or decrease the strength of the signals at the connections. Neurons can have thresholds, such that a signal is only sent when the aggregated signal exceeds that threshold. Typically, neurons are grouped into layers. Different layers can perform different transformations on their inputs. The signal may travel from the first layer (input layer) to the last layer (output layer) after multiple traversals of the layers.

[0046] The initial goal of ANN methods was to solve problems in the same way the human brain does. Over time, attention shifted to performing specific tasks, leading to a departure from biology. ANNs have been used for a wide range of tasks, including computer vision, speech recognition, machine translation, social network filtering, board and video games, medical diagnosis, and even activities traditionally considered to be performed by humans, such as painting.

[0047] The term "convolutional neural network" (CNN) indicates that the network employs a mathematical operation called convolution. Convolution is a specialized linear operation. A convolutional network is a neural network that uses convolution instead of general matrix multiplication in at least one of its layers.

[0048] Figure 1 This diagram illustrates the general concept of processing in a neural network (e.g., a CNN). A convolutional neural network consists of an input layer, an output layer, and multiple hidden layers. The input layer provides input (e.g., ...). Figure 1 The input image shown (11) is processed by the following layers. The hidden layers of a CNN typically consist of a series of convolutional layers that convolve with multiplication or other dot products. The result of a layer is one or more feature maps (represented by empty solid-line rectangles), sometimes also called channels. Some or all layers may involve resampling (e.g., subsampling). Therefore, the feature maps may become smaller, such as... Figure 1As shown. It's worth noting that convolutions with strides can also reduce the size of the input feature map (resampling). The activation functions in CNNs are typically rectified linear unit (ReLU) layers or Leaky ReLU, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, called hidden layers, because their inputs and outputs are masked by the activation function and the final convolution. While these layers are colloquially called convolutions, this is just a convention. Mathematically, convolution is technically a sliding dot product or cross-correlation. This is important for indexing in a matrix because the sliding dot product or cross-correlation affects how weights are determined at a particular index point.

[0049] When programming a CNN to process images, such as Figure 1 As shown, the input is a tensor of shape (number of images) × (image width) × (image height) × (image depth). It should be understood that image depth can be composed of the image's channels. After passing through the convolutional layer, the image is abstracted into a feature map with a shape of shape (number of images) × (feature map width) × (feature map height) × (feature map channels). Convolutional layers within a neural network can have the following properties: a convolutional kernel defined by width and height (hyperparameters); the number of input and output channels (hyperparameters); and the depth of the convolutional filter (input channels), which can be equal to the number of channels (depth) of the input feature map.

[0050] In the past, image recognition was performed using traditional multilayer perceptron (MLP) models. However, due to the full connectivity between nodes, MLP models suffer from high dimensionality and do not scale well when processing high-resolution images. A 1000×1000 pixel image with RGB color channels has 3 million weights, which is too large to be efficiently processed on a large scale with full connectivity. Furthermore, such network architectures do not consider the spatial structure of the data, treating distant input pixels in the same way as nearby pixels. This ignores the reference locality in image data both computationally and semantically. Therefore, the full connectivity of neurons is wasted for purposes such as image recognition dominated by spatially local input patterns.

[0051] Convolutional Neural Networks (CNNs) are biologically inspired variants of multilayer perceptrons, specifically designed to mimic the behavior of the visual cortex. These models alleviate the challenges of MLP architectures by leveraging the strong spatial local correlations present in natural images. The convolutional layer is the core building block of a CNN. The parameters of this layer consist of a set of learnable filters (the kernel mentioned above) that have a small receptive field but extend across the entire depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, thus computing the dot product between the entries in that filter and the input, and producing a two-dimensional activation map of that filter. Therefore, the network learns filters that activate when a specific type of feature is detected at a certain spatial location in the input.

[0052] The activation maps of all filters stacked along the depth dimension form the complete output volume of the convolutional layer. Therefore, each entry in the output volume can also be interpreted as the output of a neuron that looks at a small region of the input and shares parameters with neurons in the same activation map. A feature map, or activation map, specifies the output activation of a filter. Feature maps and activations have the same meaning. In some papers, it's called an activation map because it's a mapping corresponding to the activations of different parts of an image; it's also called a feature map because it's a mapping that finds a certain feature in an image. High activation indicates that a certain feature has been found.

[0053] Another important concept in CNNs is pooling, which is a form of non-linear downsampling. Several non-linear functions exist to implement pooling, with max pooling being the most common. It divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such sub-region.

[0054] Intuitively, the exact location of a feature is less important than its approximate location relative to other features. This is the idea behind using pooling in convolutional neural networks. Pooling layers are used to progressively reduce the size of the representation space, reducing the number of parameters, memory footprint, and computational cost in the network, thereby also controlling overfitting. In CNN architectures, it is common practice to periodically insert pooling layers between consecutive convolutional layers. Pooling operations provide another form of translation invariance.

[0055] Pooling layers operate independently on each depth slice of the input and spatially resize it. The most common form is a pooling layer with a 2×2 filter that applies a stride of 2 to each depth slice of the input, along both width and height, thus discarding 75% of the activations. In this case, each max-pooling operation is performed on four numbers. The depth dimension remains unchanged. Besides max pooling, pooling units can also use other functions, such as average pooling or... 2-norm pooling. Average pooling was frequently used in the past, but it has been less common recently compared to max pooling, as max pooling generally performs better in practice. Due to the significant reduction in representation size, there is a recent trend towards using smaller filters or abandoning pooling layers altogether. Region of Interest (ROI) pooling is a variant of max pooling where the output size is fixed and the input rectangle is a parameter. Pooling is an important component of convolutional neural networks used for object detection based on the Fast R-CNN architecture.

[0056] The ReLU mentioned above stands for Corrected Linear Unit, which applies a non-saturating activation function. It effectively removes negative values ​​from the activation map by setting them to zero. It increases the non-linearity of the decision function and the overall network without affecting the receptive field of the convolutional layers. Other functions are also used to increase non-linearity, such as the saturating hyperbolic tangent and sigmoid functions. ReLU is generally superior to other functions because it trains neural networks several times faster without significantly reducing generalization accuracy.

[0057] Leaky Rectified Linear Unit (Leaky ReLU) is a type of ReLU-based activation function, but it has a small slope for negative values ​​instead of a flat slope. The slope coefficients are determined before training, meaning they are not learned during training. This type of activation function is common in tasks with sparse gradients (e.g., training generative adversarial networks). Leaky ReLU applies an element-wise function: LeakyReLU(x)=max(0,x)+negative_slope min(0,x), or

[0058] Among them, the parameters are: negative_slope: Controls the angle of the negative slope. Default: 1e–2 inplace: Optionally, the operation can be performed in place. Default: False.

[0059] After several convolutional and max-pooling layers, higher-order inference in the neural network is accomplished through fully connected layers. Neurons in a fully connected layer are connected to all activations in the previous layer, as shown in a conventional (non-convolutional) artificial neural network. Therefore, these activations can be computed as affine transformations, followed by matrix multiplication and then bias shifts (vector addition of learned or fixed bias terms).

[0060] A "loss layer" (including the calculation of the loss function) represents how training penalizes the deviation between the predicted (output) and the true label, and is usually the last layer of the neural network. Various loss functions suitable for different tasks can be used. Softmax loss is used to predict a single class among K mutually exclusive classes. Sigmoid cross-entropy loss is used to predict K independent probability values ​​in [0, 1]. Euclidean loss is used to regress to real-valued labels.

[0061] In short, Figure 1 The diagram illustrates the data flow in a typical convolutional neural network. First, the input image is passed through a convolutional layer and abstracted into a feature map, which includes several channels corresponding to multiple filters in a set of learnable filters for that layer. Then, the feature map is subsampled using a pooling layer or similar method, reducing the dimensionality of each channel. Next, the data reaches another convolutional layer, which may have a different number of output channels. As mentioned above, the number of input and output channels are hyperparameters of the layers. To establish connections in the network, these parameters need to be synchronized between two connected layers such that the number of input channels in the current layer should equal the number of output channels in the previous layer. For the first layer processing input data such as images, the number of input channels is typically equal to the number of channels in the data representation; for example, 3 channels for an RGB or YUV representation of an image or video, or 1 channel for a grayscale image or video representation. Channels obtained by one or more convolutional layers (and possibly one or more resampling layers) can be passed to the output layer. In some implementations, such an output layer can be a convolutional layer or a resampling layer. In exemplary and non-limiting implementations, the output layer is a fully connected layer.

[0062] Autoencoders and Unsupervised Learning An autoencoder is an artificial neural network used to learn efficient data encoding in an unsupervised manner. Its schematic diagram is shown below. Figure 2 As shown, the autoencoder includes an encoder 210 and a decoder 250. The encoder has an input x that is fed into the input layer of the encoder subnet 220, and the decoder has an output x' that is output from the decoder subnet 260. The goal of the autoencoder is to learn a representation (encoding) 230 of a set of data x by training networks 220 and 260 to ignore signal "noise," typically used for dimensionality reduction. Together with the reduction (encoding) subnet 220, a reconstruction (decoding) subnet 260 is learned, in which the autoencoder attempts to generate a representation x' from the reduction encoding 230 that is as close as possible to its original input x, hence its name. In the simplest case, given a hidden layer, the encoder level of the autoencoder takes the input... and map it to : .

[0063] The image This is commonly referred to as encoding 230, latent variable, or latent representation. Here, It is an element-wise activation function, such as a sigmoid function or a modified linear unit. It is a weight matrix. This is the bias vector. Weights and biases are typically initialized randomly and then iteratively updated during training via backpropagation. Afterward, the decoder level of the autoencoder maps h to a vector with shape and... Same reconstruction :

[0064] The decoder , and Possibly corresponding to the encoder , and Irrelevant.

[0065] Variational autoencoder models make strong assumptions about the distribution of latent variables. They use variational methods for latent representation learning, which introduces an additional loss component, and train the algorithm using a specific estimator called a stochastic gradient variational Bayes (SGVB) estimator. It is assumed that the data is generated by a directed graphical model. Generate, and the encoder is learning the posterior distribution. approximation ,in, and These represent the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vectors of a VAE is typically closer to the probability distribution of the training data than that of a standard autoencoder. The objective of a VAE has the following form:

[0066] here, This represents the Kullback-Leibler divergence. The priors of the latent variables are typically set to centrally isotropic multivariate Gaussian. Typically, the shape of the variational and likelihood distributions is chosen to resemble a factorized Gaussian distribution:

[0067]

[0068] in, It is the encoder output, and and ) is the decoder output.

[0069] Recent advances in the field of artificial neural networks, particularly convolutional neural networks, have sparked researchers' interest in applying neural network-based techniques to image and video compression tasks. For example, end-to-end optimized image compression has been proposed using networks based on variational autoencoders.

[0070] Therefore, data compression is considered a fundamental and well-studied problem in engineering, typically involving designing code for a given discrete data set (ensemble) with minimum entropy. This approach relies heavily on an understanding of the probabilistic structure of the data, thus the problem is closely related to probabilistic source modeling. However, since all practical code must have finite entropy, continuous-valued data (e.g., vectors of image pixel intensities) must be quantized into a finite set of discrete values, which introduces error.

[0071] In this context, known as the lossy compression problem, a trade-off must be struck between two competing costs: the entropy (rate) of the discretized representation and the error (distortion) introduced by quantization, as described above. This can be particularly challenging in the context of SCI. Different compression applications, such as data storage or transmission over channels with limited capacity, require different rate-distortion trade-offs.

[0072] Joint optimization of rate and distortion is difficult. The general problem of optimal quantization in high-dimensional spaces is intractable without other constraints. Therefore, most existing image compression methods operate by linearly transforming a data vector into a suitable continuous-valued representation, independently quantizing its elements, and then encoding the resulting discrete representation using lossless entropy coding. Due to the central role of the transform, this scheme is called transform coding.

[0073] For example, JPEG uses Discrete Cosine Transform (DCT) on pixel blocks, while JPEG 2000 uses multi-scale orthogonal wavelet decomposition. Typically, the three components of a transform coding method (transform, quantizer, and entropy coding) are optimized separately (usually by manually adjusting parameters). Modern video compression standards (such as HEVC, VVC, and EVC) also use the transformed representation to encode the predicted residual signal. Several transforms, such as the Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), and Low Frequency Non-Separable Manually Optimized Transform (LFNST), are used for this purpose.

[0074] Variational image compression The variational autoencoder (VAE) framework can be considered a nonlinear transform coding model. The transform process can be mainly divided into four parts. This is illustrated in the diagram of the VAE framework. Figure 3A An example was provided in the text.

[0075] The transformation process can be mainly divided into four parts: Figure 3A An example of a VAE framework is shown. In Figure 3A In this circuit, encoder 101 maps the input image x to a latent representation (denoted by y) using the function y=f(x). This latent representation can also be referred to as a portion or a point within the "latest space" below. The function f() is a transformation function that converts the input signal x into a more compressible representation y. Quantizer 102... Transform the latent representation y into a quantized latent representation with (discrete) values. Where Q represents the quantizer function. The entropy model, or superencoder / decoder (also known as superprior), estimates the quantized latent representation. The distribution is determined to obtain the minimum rate achievable through lossless entropy source encoding.

[0076] The latent space can be understood as a compressed representation of data, where similar data points are closer together. The latent space can be used to learn data features and find simpler data representations for analysis. Arithmetic coding (AE) is used to quantize the super-prior 3 latent representation T. and edge information Included in bitstream 2 (by binarization). Furthermore, a decoder 104 is provided that transforms the quantized latent representation into the reconstructed image. , .Signal This is an estimate of the input image x. We expect x to be as close as possible to... In other words, the desired reconstruction quality is as high as possible. However, The higher the similarity to x, the greater the amount of side information that needs to be transmitted. Side information includes, for example, Figure 3A The bitstreams 1 and 2 shown are generated by the encoder and sent to the decoder. Generally, the greater the amount of side information, the higher the reconstruction quality. However, a large amount of side information means a low compression ratio. Therefore, Figure 3A One objective of the system described in the text is to balance reconstruction quality with the amount of side information transmitted in the bitstream.

[0077] exist Figure 3A In this context, component AE 105 is the arithmetic coding module, used to encode the quantized latent representation. and edge information The sample is converted into binary representation bitstream 1. For example, and The samples can include integers or floating-point numbers. One purpose of the arithmetic encoding module is to convert the sample values ​​into binary number strings (through a binarization process) (which are then included in the bitstream, which may include other parts or other side information corresponding to the encoded image).

[0078] Arithmetic decoding (AD) 106 is the process of restoring the binarization process, in which binary numbers are converted back to sample values. Arithmetic decoding is provided by the arithmetic decoding module 106.

[0079] It should be noted that this invention is not limited to this specific framework. Furthermore, this invention is not limited to image or video compression, but can also be applied to object detection, image generation, and recognition systems.

[0080] exist Figure 3A In this context, two subnets are joined together. A subnet is a logical division between different parts of the entire network. For example, in... Figure 3A In this code, modules 101, 102, 104, 105, and 106 are called the "encoder / decoder" subnets. The "encoder / decoder" subnets are responsible for encoding (generating) and decoding (parsing) the first bitstream "bitstream 1". Figure 3A The second network, comprising modules 103, 108, 109, 110, and 107, is called the "super encoder / decoder" subnet. This second subnet is responsible for generating the second bitstream, "bitstream 2." The two subnets serve different purposes.

[0081] The first subnet is responsible for: Transform the input image x (101) into its latent representation y (which is easier to compress x). Quantize the latent representation y (102) into the quantized latent representation. , Arithmetic coding module 105 uses AE to process the quantized latent representation. Compress the stream to obtain stream "stream 1". The arithmetic decoding module 106 is used to parse the bitstream 1 via AD. The parsed data was used to reconstruct (104) the reconstructed image. ).

[0082] The purpose of the second subnet is to obtain the statistical properties of the samples in "stream 1" (e.g., the mean, variance, and correlation among the samples in stream 1) so that the compression of stream 1 by the first subnet is more efficient. The second subnet generates a second stream "stream 2", which includes the aforementioned information (e.g., the mean, variance, and correlation among the samples in stream 1).

[0083] The second network includes an encoding section that incorporates the quantized latent representation. Transform (103) into edge information z, and quantize the edge information z into quantized edge information. and the quantized edge information Encoding (e.g., binarization) (109) results in bitstream 2. In this example, binarization is performed using arithmetic encoding (AE). The decoding portion of the second network includes arithmetic decoding (AD) 110, which transforms the input bitstream 2 into decoded quantized side information. . Possibly with The same applies because the decoding operation at the arithmetic encoding end is a lossless compression method. Then, the decoded quantized side information is... Transform (107) into decoded side information . express Statistical properties (e.g., The mean of the samples, or the variance of the sample values, etc.). Then, the decoded latent representation Provided to the aforementioned arithmetic encoder 105 and arithmetic decoder 106 for control The probability model.

[0084] Figure 3A An example of a variational autoencoder (VAE) is described, the details of which may differ in different implementations. For example, in a particular implementation, additional components may exist to more efficiently acquire the statistical properties of samples from bitstream 1. In one such implementation, a context modeler may exist, the goal of which is to extract the cross-correlation information of bitstream 1. The statistical information provided by the second subnet can be used by the arithmetic encoder (AE) 105 and the arithmetic decoder (AD) 106 components.

[0085] Figure 3A The encoder and decoder are depicted in a single figure. As will be apparent to those skilled in the art, the encoder and decoder can and often are embedded in different devices.

[0086] Figure 3B and Figure 3C The encoder and decoder components of the VAE framework are described separately. According to some embodiments, the encoder receives an image as input. The input image may include one or more channels, such as color channels or other types of channels, such as depth channels or motion information channels. The encoder (e.g.) Figure 3B The outputs (shown) are bitstream 1 and bitstream 2. Bitstream 1 is the output of the first subnet of the encoder, and bitstream 2 is the output of the second subnet of the encoder.

[0087] Similarly, in Figure 3C In the process, two bitstreams (bitstream 1 and bitstream 2) are received as inputs and generated at the output. This refers to the reconstructed (decoded) image. As mentioned above, a VAE can be divided into different logical units that perform different operations. This is in Figure 3B and Figure 3C Examples are provided in the text so that... Figure 3B Describes the components involved in encoding the signal (e.g., an image or video), and the encoded information provided. For example, this encoded information is subsequently... Figure 3C The decoder component described herein receives data for decoding. It should be noted that the encoder and decoder components, represented by the numbers 12x and 14x, can functionally correspond to those described above. Figure 3A The component mentioned in the text is represented by the number 10x.

[0088] Specifically, such as Figure 3B As shown, the encoder includes encoder 121, which transforms the input x into a signal y, and then provides the signal y to quantizer 122. Quantizer 122 then processes the information... The information is provided to the arithmetic coding module 125 and the super encoder 123. The super encoder 123 provides the bitstream 2 discussed above to the super decoder 127, which in turn provides the information to the arithmetic coding module 105 (125).

[0089] The output of the arithmetic encoding module is bitstream 1. Bitstream 1 and bitstream 2 are the encoded outputs of the signal, and these two bitstreams are then provided (sent) to the decoding process. Although unit 101 (121) is called an "encoder", it can also be used to... Figure 3B The complete subnet described in the code is called an "encoder". The encoding process typically refers to the unit (module) that transforms the input into an encoded (e.g., compressed) output. From Figure 3BAs can be seen, unit 121 can actually be considered the core of the entire subnet because it performs the transformation of the input x to y, where y is a compressed version of x. Compression in encoder 121 can be achieved, for example, by applying a neural network or any processing network typically having one or more layers. In such processing networks, compression can be performed through cascaded processing including downsampling, which reduces the size of the input and / or the number of channels. Therefore, the encoder can be referred to, for example, as a neural network (NN) based encoder, etc.

[0090] The remaining parts in the diagram (quantization unit, super encoder, super decoder, arithmetic encoder / decoder) are all responsible for improving the efficiency of the encoding process or converting the compressed output y into a series of bits (bit stream). Quantization can be provided to further compress the output of the NN encoder 121 through lossy compression. The AE 125, combined with the super encoder 123 and super decoder 127 used to configure the AE 125, can perform binarization, which can further compress the quantized signal through lossless compression. Therefore, it is also possible to... Figure 3B The entire subnet in the code is called the "encoder".

[0091] Most deep learning (DL) based image / video compression systems perform dimensionality reduction on the signal before converting it into binary numbers (bits). For example, in a VAE framework, an encoder performing a non-linear transformation maps the input image x to y, where the width and height of y are smaller than x. Because y has a smaller width and height, it is smaller in size, reducing the signal's size and making it easier to compress. It's important to note that, in general, an encoder doesn't necessarily need to reduce size in two (or often all) dimensions. Instead, some exemplary implementations can provide an encoder that reduces the size of only one dimension (or often a subset of dimensions).

[0092] In J. Balle, L. Valero Laparra, and EP Simoncelli (2015), “Density Modeling of Images Using Generalized Divisive Transformation” (published on the arXiv preprint platform and presented at the 4th International Conference on Learning Representations in 2016) (hereinafter referred to as “Balle”), the authors proposed an end-to-end optimization framework for image compression models based on nonlinear transforms. The authors optimized for mean squared error (MSE) but used a more flexible transform constructed from a cascade of linear convolutions and nonlinearities. Specifically, they used a joint nonlinearity of generalized divisive normalization (GDN), inspired by neuron models in biological visual systems and proven effective for Gaussianizing image density. This cascaded transform is followed by uniform scalar quantization (i.e., rounding each element to the nearest integer), which efficiently implements the parametric form of vector quantization in the original image space. The compressed image is reconstructed from these quantized values ​​using an approximately parametric nonlinear inverse transform.

[0093] Examples of this type of VAE framework include: Figure 4 As shown, it utilizes six downsampling layers (401 to 406). The network architecture includes a hyperprior model. (Left side (g)) a g s The image autoencoder architecture is shown on the right (h). a h s This corresponds to the realization of a priori autoencoders. Factoring the prior model affects the analysis and synthesis transformation g. a and g s They use the exact same architecture. Q represents quantization, and AE and AD represent the arithmetic encoder and arithmetic decoder, respectively. The encoder takes the input image x and inputs it into g. a This produces a response y (latent representation) with spatially varying standard deviation. The encoding g... a It includes multiple convolutional layers, with subsampling and generalized divisive normalization (GDN) as the activation function.

[0094] Feed the response to h a In the first step, the standard deviation distribution of z is summarized. Then, it is quantized, compressed, and transmitted as side information z. The encoder then uses the quantized vector. To estimate The spatial distribution of standard deviation is used to obtain the probability (or frequency) values ​​of arithmetic coding (AE) and is used to compress and transmit the quantized image representation. (or potential representation). The decoder first recovers from the compressed signal. Then, the decoder uses h s To obtain This provides it with the correct probability estimate so that it can also be successfully recovered. Then, the decoder will Feed to g s In order to obtain the reconstructed image.

[0095] Layers that include downsampling are indicated by a down arrow in the layer description. The layer description "Conv N,k1,2↓" indicates that the layer is a convolutional layer with N channels and a kernel size of k1×k1. For example, k1 can be equal to 5, and k2 can be equal to 3. As mentioned above, 2↓ indicates that downsampling is performed by a factor of 2 in this layer. A factor of 2 downsampling results in one dimension of the input signal being reduced by half on the output side. Figure 4 In this context, 2↓ indicates that both the width and height of the input image are reduced by a factor of 2. Due to the presence of six downsampling layers, if the width and height of the input image 414 (also denoted by x) are given by w and h, then the output signal z 413 has a width and height equal to w / 64 and h / 64, respectively. The modules represented by AE and AD are the arithmetic encoder and arithmetic decoder, which will refer to... Figures 3A to 3C This will be explained. Arithmetic encoders and decoders are concrete implementations of entropy coding. AE and AD can be replaced by other entropy coding methods. In information theory, entropy coding is a lossless data compression scheme used to convert the value of a symbol into a binary representation, a process that can be reversed. Furthermore, "Q" in the diagram corresponds to the quantization operation, which is discussed above... Figure 4 This has already been mentioned and further explained in the "Quantization" section above. Furthermore, the quantization operation and corresponding quantization unit that are part of component 413 or 415 may not exist and / or may be replaced by another unit.

[0096] exist Figure 4 The diagram also shows a decoder including upsampling layers 407 to 412. Another layer 420, implemented as a convolutional layer, is provided between the upsampling layers 411 and 410 in the order of input processing, but does not upsample the received input. A corresponding convolutional layer 430 for the decoder is also shown. Such layers can be provided in a neural network to perform operations on the input that modify specific features without changing the input size. However, providing such layers is not necessary.

[0097] When viewed in the order in which bitstream 2 is processed by the decoder, the upsampling layers operate in reverse order, from upsampling layer 412 to upsampling layer 407. Each upsampling layer is shown here to provide an upsampling ratio of 2, indicated by ↑. Of course, not all upsampling layers necessarily have the same upsampling ratio; other upsampling ratios, such as 3, 4, 8, etc., can also be used. Layers 407 to 412 are implemented as convolutional layers (conv). Specifically, since these layers may be designed to perform the opposite operation on the input to that of the encoder, the upsampling layers can apply a deconvolution operation to the received input, increasing its size by a factor corresponding to the upsampling ratio. However, the invention is generally not limited to deconvolution, and upsampling can be performed in any other way, such as by bilinear interpolation between two adjacent samples, or by copying nearest neighbor samples, etc.

[0098] In the first subnet, some convolutional layers (401 to 403) undergo generalized divisive normalization (GDN) after the encoder and inverse GDN (IGDN) after the decoder. In the second subnet, the activation function applied is ReLU. It should be noted that this invention is not limited to this implementation, and other activation functions can generally be used instead of GDN or ReLU.

[0099] In this specification, "parameters" are values ​​used in the operations of each layer of a neural network; for example, they may include weights used when applying input values ​​to a given expression. Here, parameters can be represented in matrix form. Parameters are values ​​set during training and can be updated as needed using separate training data.

[0100] As mentioned above, compression algorithms are typically optimized for natural images, and therefore often fail when processing SCI. That is, such compression techniques can introduce distortion into the compressed image, leading to varying degrees of degradation; for example, it may render certain parts of the SCI unreadable or incomprehensible.

[0101] In traditional screen content coding used for compressing SCI, intra-block duplication is a technique employed by HEVC. For related information, see X. Xu et al., “Intra-block duplication in HEVC screen content coding extensions,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, Vol. 6, No. 4, pp. 409-419, December 2016, doi: 10.1109 / JETCAS.2016.2597645.

[0102] During encoding, the algorithm searches for similar blocks in reconstructed copies of the original SCI frames. The search area is limited, meaning that reconstructed blocks need to be decoded before the current block in the decoder. If such a block is found, the encoder encodes a shift vector pointing to the found block, rather than encoding the current block. The decoder can then use this vector to copy the decoded block to reconstruct the current block.

[0103] Another traditional screen content encoding method is palette mode encoding. For related information, see W. Pu et al., “Palette Mode Encoding in HEVC Screen Content Encoding Extensions”, published in IEEE Journal on Emerging and Selected Topics in Circuits and Systems, Vol. 6, No. 4, pp. 420-432, December 2016, doi: 10.1109 / JETCAS.2016.2605661.

[0104] This technique aims to utilize repeating colors in SCI by encoding these repeating colors in a table. First, a table is created that corresponds to the possible colors in each block. Then, using this table, pixel colors are mapped to indices. Finally, instead of encoding the image blocks as is, the encoder sends the indices along with the table to save some bitrate.

[0105] However, traditional image compression algorithms typically rely on hand-crafted feature extraction. This means they must depend on assumptions about the data, which may not always hold true. The performance of these methods is also limited because it is impractical to determine every pattern in the data and create filters to achieve better compression gain.

[0106] On the other hand, thanks to the vast amounts of data accumulated over the years and end-to-end training techniques, learning-based image compression algorithms can create more efficient representations of images. However, once these models are deployed, it becomes difficult to update them for new types of data without breaking compatibility with existing files. Furthermore, the underrepresentation of screen content data in most learning-based compression datasets leads to poor performance on SCI. Finally, the model architecture in learning-based compression algorithms is not well-suited to fully utilize the redundancy and tile similarity of SCI. Even when SC images are introduced into the training set of the learning-based codec, it is difficult to generalize them together because the image distributions of SC and natural images are very different. This results in very poor performance for both types of data.

[0107] Finally, there are hybrid approaches, such as training ComCNN and RecCNN around traditional codecs, or combining neural networks to accelerate coding unit prediction. However, these methods are inherently highly complex. Because most modules in these systems contain non-differentiable operations (e.g., rounding, selection, or sampling), they are not well-suited for end-to-end training, which can easily achieve higher coding performance by optimizing each module together. Furthermore, downscaling the original input is not suitable for screen content images, which have sharp and / or thin edges that are highly susceptible to degradation during interpolation to lower resolutions.

[0108] According to one example, two neural network (NN) modules (CR and RS) can be employed to help compress screen content data. Implementing these two NN modules improves compression quality compared to existing techniques. The former module enhances the compressibility of the image containing the SCI, while the latter module restores these enhancements and some compression noise to recover an approximation of the original input. Forward linear transforms and inverse linear transforms can be used before these modules to leverage the internal structure of the SCI to produce a smaller bitstream. In one example, the parameters of the forward and linear transforms can be computed from the image and / or pre-fixed, increasing flexibility during compression. The pipeline provides the option to turn additional filters on or off, so there is no performance degradation compared to the baseline image encoder and decoder.

[0109] As an example, CR and RS modules, which will be described in more detail below, can be provided at the same logical location in the encoder / decoder architecture. At the encoding end, a global linear transform can be applied before further processing using the CR module to generate linearly transformed image data. At the decoding end, a global inverse linear transform can be applied before further processing using the RS module.

[0110] The example method and system utilize a differentiable image codec as a baseline, meaning the implementation is end-to-end trainable. Therefore, there is no need to rely on approximations to obtain optimal parameters. Instead, a large number of available datasets can be used to improve the compression performance of the baseline codec on SCI. Furthermore, the provided method and system are also applicable to other image types due to the ease of module replacement. Additionally, compatibility with existing codec infrastructure is ensured by using a fixed image codec.

[0111] Figure 5 A schematic diagram of an encoder based on an example is shown. (As shown) Figure 5The encoder 500 shown can be used to implement a method for encoding an input tensor representing image data 501, thereby generating encoded image data 503. The input data 501 can be provided to a start / stop decision module 505, which determines whether to implement the enhanced encoding process of the encoder 500, and generates data 507 representing a "start / stop" flag, which can be used to indicate whether the enhanced encoding process according to the example should be used.

[0112] The start / stop decision module 505 can make a decision based on the input image data 501. If the decision is negative, the enhanced coding (509) can be bypassed, and the image data can be compressed using the underlying baseline codec implemented by the image encoder module 527. That is, if the decision is negative, the image data 501 can bypass other processing and be directly input to the image encoder module 527, which outputs encoded image data 503 in the form of an output bitstream.

[0113] If the start / stop decision module 505 makes a positive decision, then enhanced encoding can be activated. That is, the input image data 501 can be passed (509) to other modules of the encoder 500 for processing, as will be described in more detail below.

[0114] The start / stop decision module 505 can make decisions by running the system once for each decision and comparing their encoding performance. Alternatively, the start / stop decision module 505 can include an NN-based classifier or a traditional classifier to check whether a given image provided by the input image data 501 is an SCI, or whether it benefits from activating additional modules based on the baseline codec. In one example, the start / stop decision module 505 can be implemented using a heuristic algorithm. The output signal 507 of the start / stop decision module 505 can be transmitted to or otherwise provided to the decoder.

[0115] If the start / stop decision module 505 generates an enable signal, it provides the input image data 501 to the transform selection module 513. This module 513 generates transform parameters 517 and transform index 519 for the forward transform module 515 of the encoder 500. The forward transform module 515 also takes the input image data 501 as input to apply a selected linear transform. The transform selection module 513 takes the input image data 501 as input and determines the transform that the forward transform module 515 should apply. In one example, this decision may or may not be based on image or patch statistics determined from the input image data 501. For example, the transform selection module 513 may process the input image data 501 to determine whether it corresponds to an SCI.

[0116] The transform selection module 513 can compare options in the candidate transform list based on predicted or actual performance. Optimal transforms can be provided for different image types. For example, specific linear transforms can be associated with SCI and can be selected as a combination of such linear transforms and subsequent processing, as described below, to obtain the optimally coded image.

[0117] Once the optimal transform is selected, index 519 is output and can be passed to the decoder. If the selected transform requires additional parameters 517, such as the color transform matrix, scaling factor, or α-mixing coefficient, these parameters can be calculated and additionally passed to the decoder. Alternatively, the transform can use a fixed set of parameters that do not need to be passed to the decoder (i.e., the decoder can be pre-configured with this information).

[0118] According to one example, the forward transformation module 515 can be used to process the input image data 501 using a selected global linear transformation to generate a first tensor that includes the linearly transformed image data. In one example, the selected global linear transformation may include a color space transformation, where, for example, a 3×3 transformation matrix can be learned, generated through Principal Component Analysis (PCA), or obtained from a known transformation such as "RGB to HSV" or "RGB to YUV". Another type of transformation from which the global linear transformation can be selected may change the resolution of some or all channels of the image represented by the input image data 501. Alternatively, the selected global linear transformation may include an alpha blend of the original image represented by the input image data 501 and the transformed image. The selected transformation may be a combination of one or more such transformations in any order, which can be specified by transformation parameters. In one example, the selected global linear transformation may include a transformation in which the original image represented by the input image data 501 is converted from color (e.g., RGB) to grayscale and blended with the original image using an alpha blend. In one example, the alpha blending coefficient may be approximately 0.8. This specific combination of transformations is called desaturation transformation.

[0119] Figure 6 The example is shown. Figure 5 A schematic diagram of the global linear transformation of the forward transformation module. Figure 6In the example, input 601 includes input image data 501. Input 601 is fed to grayscale conversion module 603. Grayscale conversion module 603 converts the input image data 501 from its color space to grayscale, such as from RGB, YCbCr, HSV, or LAB. For example, gamma compression functions can be removed by gamma expansion (linearization) to transform the input image data 501 into a linear RGB color space, allowing appropriate weighted sums to be applied to the linear color components to calculate linear luminance. Y linear Each channel of the input image data 501 can be weighted using a predefined α value such as 0.8 (605). The gray-level components of the input image data 501, determined using the gray-level transformation module 603, are weighted by a factor of 1 minus the α value (607), and the two weighting results (for the original input image data and the gray-level transformation) are combined (609) to generate a first tensor 520, which includes the linearly transformed image data, referenced to... Figure 5 The linearly transformed image data is then passed to the CR module 521. As described above, the output represents the index 519 of the selected linear transformation and is transmitted to the decoder. If the selected transformation requires additional parameters 517, such as the color transformation matrix, scaling factor, or α-mixing coefficient (e.g., as referenced...), then... Figure 6 The parameters described (0.8) can then be calculated and additionally transmitted to the decoder.

[0120] According to one example, the CR module 521 includes a neural network (NN) for processing a first tensor 520, which includes linearly transformed image data, to generate a second tensor 522, which includes a preprocessed tensor representation of the input image data 501. The second tensor may include an input tensor of an image encoder 527. In one example, this input tensor may be provided as input to the image encoder 527 for generating a bitstream representing the encoded image data. That is, the input tensor may be provided as input to the image encoder 527 for encoding the image data.

[0121] In one example, CR module 521 may include a neural network (NN) comprising at least one convolutional layer. In one implementation, the NN or a portion thereof may include an inverse residual structure. The specific implementation of the NN in CR module 521 may be predefined or determined based on the input image data 501. That is, in one example, CR selection module 523 may take the input image data 501 as input. CR selection module 523 may use the input image data 501 to determine the NN that should be implemented or applied in CR module 521. In one example, this decision may or may not be based on image or patch statistics and / or related to a selected linear transformation determined by transformation selection module 513. CR selection module 523 may compare options in a list of alternative NN implementations based on predicted or actual performance. Once the optimal NN implementation is selected, an index 525 may be output and transmitted to the decoder. The index 525 may be provided to CR module 521 to enable the implementation of the selected NN. In one example, the NN implementation may use a set of predefined parameters, which need not be transmitted to the decoder as long as the decoder knows which NN to implement from the index 525. According to one example, CR module 521 includes a neural network (NN) for computing a more compressible version of the transformed image represented by the first tensor 520.

[0122] Figure 7 A schematic diagram of a NN architecture based on an example is shown. Figure 7 The example depicts the neural network architecture implemented by the CR module 521. The input image 701 (i.e., the first tensor 520 comprising the linearly transformed input image data 501) is provided to the CR module 521. Figure 7 In the example, the first tensor 520 includes a three-channel image with dimensions of 256×256 pixels. Figure 7The NN architecture downsamples the first tensor 520 representing image 701 and increases the number of channels with convolutional layers 703. The result of layer 703 is processed using a set of mobile inverse residual bottleneck (MBConv) convolutional layers 705. This set of layers 705 forms an inverse residual block that maps the relatively narrow input layer 703 to a wider inner layer, which in turn maps to a narrower output layer. The output of layer 705 is passed through another convolution 707 to reduce the number of channels to 12, and its output can then be reshaped (709) to produce an output with the same shape as the input 701. The original input 701 and this intermediate result 710 (which includes an image-like displacement tensor) are added (711) to form the final output 713 of CR module 521. In other words, the image data 701 (i.e., the first tensor 520) and the image-like displacement tensor (which is derived from the first tensor 710, which includes the image data from the output of layer 709) can be added (711), that is, the corresponding components are added together, thereby generating a preprocessed tensor representation 713 of the input image data 501. Therefore, the final output 713 of the CR module 521 includes a preprocessed tensor representation of the input image data 501 in the form of a second tensor 522. The number of layers and the number of intermediate channels may vary depending on design requirements.

[0123] Therefore, the CR module 521 essentially generates a compact representation of the input image data 501, which defines a second tensor 522 that includes the preprocessed tensor representation of the input image data 501. The output of the CR module 521 (i.e., the second tensor 522) can be passed to the image encoder module 527, which can be used to encode the preprocessed tensor representation (522) of the image data (501) to generate encoded image data 503. That is, the image encoder module 527 converts the second tensor 522 into a bitstream that includes the encoded image data 503.

[0124] According to one example, the combination of the forward transform module 515 and the CR module 521 effectively forms a preprocessor structure for an image encoder 527, which encodes an input tensor representing image data. Such a preprocessor structure can be implemented using processing circuitry that processes the input image data 501 using module 515 through a selected global transform to generate a first tensor 520, which includes the transformed image data. The first tensor 520 can then be processed through one or more layers of a selected neural network in the CR module 521 to generate an input tensor (i.e., a second tensor 522), which can be provided as input to the image encoder module 527 for generating the encoded image data. In one example, at least one of the following can be provided to the decoder: a set of configuration parameters for selecting the global transform, a first index representing the selected global transform, and a second index representing the selected neural network.

[0125] For example, image encoder module 527 can be the encoder portion of a learning image codec with frozen weights. It should be noted that, as described above, if start / stop switch signal 507 is in the off state, encoder 500 skips transform selection module 513 and CR selection module 523, as well as forward transform module 515 and CR module 521. In this case, the original image 501 is fed to image encoder module 527 instead of a compact representation of the second tensor 522. The bitstream 503 can then be transmitted to the decoder.

[0126] Figure 8 A schematic diagram of a decoder based on an example is shown. Figure 8In the example, after acquiring bitstream 503, start / stop switch signal 507, transform index 519, transform parameter 517, and CR / RS transform index 525, decoder 800 can decode the reconstruction of the original image 501. That is, decoder 800 can decode bitstream 503 through image decoder 801 to reconstruct a compact representation of the original image. Then, according to start / stop switch signal 507, the remaining modules of decoder 800 can be skipped, and the output of image decoder 810 can be directly output (803). When start / stop switch signal 507 is on, the other modules of decoder 800 continue the process. That is, inverse transform selection module 805 uses transform index 519, transform parameter 517, and CR / RS transform index 525. This module (805) selects the appropriate inverse transform based on what is used at the encoding end. In the case of transforms (e.g., color space conversion or alpha mixing), the inverse transform can be exact. However, for some loss of linear transform information (e.g., downsampling or weighted mixing of linear transforms), there is no exact inverse transform. In such cases, the Moore-Ponros inverse of the forward linear transformation can be achieved. However, other types of inverse transformations are also possible.

[0127] The selected transform is passed to the inverse transform module 807 so that it can be applied to the reconstructed compact representation. In one example, a desaturation transform with an α coefficient of 1.25 can be used. The desaturation transform is its own inverse, and its α coefficient is the multiplicative inverse of the α coefficient in the forward transform module 515.

[0128] After applying the inverse transform, the original input image data is reconstructed from the inversely transformed representation using RS module 809. As the corresponding module in encoder 500, this module (809) uses a neural network (NN) to produce the final output 803, which is selected based on transform index 525 using the inverse CR / RS selection module 811. The architecture of the NN in RS module 809, as shown in the example, can be a symmetric inverse version of the NN used in CS module 521. That is, the NNs in CS module 521 and RS module 809 can have the same size, the same number of channels, and the same number of layers. However, they can be trained separately and use separate weight sets. The output of RS module 809 is returned as the decoded reconstruction of the original image.

[0129] According to one example, the combination of inverse transform module 807 and RS module 809 effectively forms the post-processor structure of image decoder 801, which decodes an input tensor representing image data. Such a post-processor structure can be implemented using processing circuitry configured to: receive from an encoder at least one of the following: a set of configuration parameters for selecting a global transform, a first index representing the selected global transform, and a second index representing the selected neural network; and receive an initial representation of the decoded image data, which includes a preprocessed tensor representation of the image data. The initial representation of the decoded image data (e.g., the first tensor 520) can be processed using a global inverse transform selected based on the first index, which can be configured based on the set of configuration parameters to generate the initial representation of the decoded image data. The initial representation of the decoded image data can be processed through one or more layers of the NN of RS module 809 to generate output image data 803.

[0130] Based on one example, Figure 9 A corresponding system is shown that can be deployed with the above encoder-decoder processing chain. Figure 9 A schematic block diagram of an exemplary encoding system (e.g., a video, image, audio, and / or other encoding system (or simply a short encoding system)) that can utilize the techniques of this application is shown. The video or image encoder 20 (or simply encoder 20) and video or image decoder 30 (or simply decoder 30) of the video or image encoding system 10 represent examples of devices that can be used to perform various techniques according to the various examples described in this application. For example, video and / or image encoding and decoding can employ a neural network, which can be distributed, and the aforementioned bitstream parsing and / or bitstream generation can be applied to transmit feature maps between (two or more) distributed computing nodes.

[0131] like Figure 9 As shown, the encoding system 10 includes a source device 12, which is used to provide encoded picture or image data 21 to, for example, a destination device 14 via a communication channel 13 for decoding the encoded picture or image data.

[0132] Source device 12 includes encoder 20 (e.g., as referenced above). Figure 5 The encoder described herein may additionally (i.e., optionally) include an image source 16, a preprocessor (or preprocessing unit) 18 (e.g., an image preprocessor 18), and a communication interface or communication unit 22.

[0133] Image source 16 may include or may be any type of image capture device such as a camera for capturing real-world images, and / or any type of image generation device such as a computer graphics processor for generating computer-animated images, or any other device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images) and / or any combination thereof (e.g., augmented reality (AR) images). Image source may be any type of memory or storage for storing any of the aforementioned images.

[0134] To distinguish between the processing performed by the preprocessor 18 and the preprocessing unit 18, the image or image data 17 may also be referred to as the raw image or raw image data 17.

[0135] The preprocessor 18 receives (raw) image data 17 and performs preprocessing on the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. The preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It should be understood that the preprocessing unit 18 may be an optional component. It should also be noted that preprocessing may employ neural networks (e.g., as described above). Figures 1 to 8 (as described in any of them).

[0136] Encoder 20 is used to receive preprocessed image data 19 and provide encoded image data 21.

[0137] The communication interface 22 of the source device 12 can be used to receive encoded image data 21 and transmit the encoded image data 21 (or any other processed version thereof) to another device (e.g., destination device 14) or any other device via the communication channel 13 for storage or direct reconstruction.

[0138] Destination device 14 includes decoder 30 (e.g., video or image decoder 30, as referenced above). Figure 8 The decoder described may additionally (i.e., optionally) include a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0139] The communication interface 28 of the destination device 14 is used to receive encoded image data 21 (or any other processed version thereof), for example, directly from the source device 12 or any other source (e.g., a storage device such as an encoded image data storage device), and to provide the encoded image data 21 to the decoder 30.

[0140] Communication interfaces 22 and 28 can be used to send or receive encoded image data 21 or encoded data 13 via a direct communication link (e.g., a direct wired or wireless connection) between source device 12 and destination device 14, or via any type of network (e.g., a wired network or wireless network or any combination thereof, any type of private network and public network or any combination thereof).

[0141] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing so as to transmit it over a communication link or communication network.

[0142] For example, the communication interface 28 corresponding to the communication interface 22 can be used to receive the transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21.

[0143] Both communication interface 22 and communication interface 28 can be used as Figure 9 The arrow pointing from source device 12 to destination device 14 indicates a one-way communication interface, or it can be used as a two-way communication interface. It can be used to send and receive messages, establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission (e.g., encoded image data transmission). Decoder 30 is used to receive encoded image data 21 and provide decoded image data 31 or decoded image 31.

[0144] The post-processor 32 of the destination device 14 is used to post-process the decoded image data 31 (also referred to as reconstructed image data) (e.g., decoded image 31) to obtain post-processed image data 33 (e.g., post-processed image 33). The post-processing performed by the post-processing unit 32 may include color format conversion (e.g., from YCbCr to RGB), color correction, trimming or resampling, or any other processing, such as preparing the decoded image data 31 for display by a display device 34, etc. For example, such post-processing (and the pre-processing described above with reference to source device 12) may be a reference... Figures 5 to 8 Any supplementary preprocessing or postprocessing described herein.

[0145] The display device 34 of the destination device 14 is used to receive the post-processed image data 33 to display the image to a user or viewer. The display device 34 can be or may include any type of display (e.g., an integrated or external display or screen) to represent the reconstructed image. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display.

[0146] Although Figure 9 Source device 12 and destination device 14 are described as separate devices; however, embodiments of the devices may also include two devices or two functions, namely source device 12 or its corresponding function and destination device 14 or its corresponding function. In these embodiments, source device 12 or its corresponding function and destination device 14 or its corresponding function may be implemented using the same hardware and / or software, or by separate hardware and / or software or any combination thereof.

[0147] According to the description, Figure 9 The different units or functions that are (accurately) divided in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application, which is obvious to those skilled in the art.

[0148] Encoder 20 (e.g., video or image encoder 20) or decoder 30 (e.g., video or image decoder 30), or both encoder 20 and decoder 30, can be transmitted via, for example, Figure 10The processing circuitry shown can be implemented as, for example, one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video encoding-specific processors, or any combination thereof. Encoder 20 can be implemented via processing circuitry 46 to embody various modules including neural networks or portions thereof. Decoder 30 can be implemented via processing circuitry 46 to embody any encoding system or subsystem described herein. The processing circuitry can be used to perform various operations discussed below. If these techniques are partially implemented in software, the device can store the instructions for that software in a suitable non-transitory computer-readable storage medium, and these instructions can be executed in hardware by one or more processors to perform the techniques of this invention. Either video encoder 20 or video decoder 30 can be integrated into a single device as part of a combined encoder / decoder (codec), for example, as... Figure 10 As shown.

[0149] Source device 12 and destination device 14 can include any of a variety of devices, including any type of handheld or fixed device, such as a laptop or notebook computer, mobile phone, smartphone, tablet or tablet computer, camera, desktop computer, set-top box, television, display device, digital media player, video game console, video streaming device (e.g., content service server or content distribution server), broadcast receiving device, broadcast transmitting device, etc., and may or may not use any type of operating system. In some cases, source device 12 and destination device 14 can be configured for wireless communication. Therefore, source device 12 and destination device 14 can be wireless communication devices.

[0150] In some cases, Figure 9 The video encoding system 10 shown is merely an example, and the technology of this application can be applied to video encoding setups (e.g., video encoding or video decoding) that do not necessarily involve any data communication between the encoding and decoding devices. In other examples, data is retrieved from local memory, streamed over a network, etc. The video encoding device may encode data and store it in memory, and / or the video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other but merely encode data into memory and / or retrieve data from memory and decode it.

[0151] Figure 11A schematic diagram of a decoding device 8000 according to an embodiment of the present invention is shown. The decoding device 8000 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the decoding device 8000 may be a decoder (e.g., Figure 9 or Figure 10 The decoder 30 in the code can also be an encoder (e.g., Figure 9 or Figure 10 Encoder 20 in the middle.

[0152] The video decoding device 8000 includes: an input port 8010 (or input port 8010) and a receiving unit (Rx) 8020 for receiving data; a processor, logic unit, or central processing unit (CPU) 8030 for processing data; a transmitting unit (Tx) 8040 and an output port 8050 (or output port 8050) for transmitting data; and a memory 8060 for storing data. The decoding device 8000 may also include optical-to-electrical (OE) components and electro-optical (EO) components coupled to the input port 8010, the receiving unit 8020, the transmitting unit 8040, and the output port 8050, serving as outputs or inputs for optical or electrical signals.

[0153] Processor 8030 is implemented through hardware and software. Processor 8030 can be implemented as one or more CPU chips, one or more cores (e.g., as a multi-core processor), one or more FPGAs, one or more ASICs, and one or more DSPs. Processor 8030 communicates with ingress port 8010, receiver unit 8020, transmitter unit 8040, egress port 8050, and memory 8060. Processor 8030 includes a neural network-based codec 8070. The neural network-based codec 8070 implements the disclosed embodiments described above. For example, the neural network-based codec 8070 is used to implement, process, prepare, or provide various encoding operations. Therefore, including the neural network-based codec 8070 significantly improves the functionality of the video decoding device 8000, enabling transitions between different states of the video decoding device 8000. Optionally, the neural network-based codec 8070 is implemented with instructions stored in memory 8060 and executed by processor 8030.

[0154] The memory 8060 may include one or more disks, tape drives, and solid-state drives, and may be used as an overflow data storage device to store programs as programs are selected for execution, as well as instructions and data read during program execution. For example, the memory 8060 may be volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0155] Figure 12 A simplified block diagram of an apparatus according to an exemplary embodiment is shown, which can be used as... Figure 9 The source device 12 and the destination device 14 are either or both of them.

[0156] The processor 9002 in device 9000 can be a central processing unit. Alternatively, the processor 9002 can be any other type of device or multiple devices, existing or to be developed in the future, capable of manipulating or processing information. Although the disclosed implementation can be carried out using a single processor such as the processor 9002 shown in the figure, using more than one processor can improve speed and efficiency.

[0157] In one implementation, the memory 9004 in the device 9000 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 9004. Memory 9004 may include code and data 9006 accessed by the processor 9002 via bus 9012. Memory 9004 may also include an operating system 9008 and an application program 9010, which includes at least one program that causes the processor 9002 to perform the methods described herein. For example, application program 9010 may include applications 1 to N, including a video encoding application that performs the methods described herein.

[0158] The device 9000 may also include one or more output devices, such as a display 9018. In one example, the display 9018 may be a touch-sensitive display that combines a display with a touch-sensitive element capable of sensing touch input. The display 9018 may be coupled to the processor 9002 via a bus 9012.

[0159] Although bus 9012 of device 9000 is described herein as a single bus, bus 9012 may include multiple buses. Furthermore, auxiliary memory may be directly coupled to other components in device 9000 or accessible via a network, and may include a single integrated unit (e.g., a memory card) or multiple units (e.g., multiple memory cards). Therefore, device 9000 can be implemented in a variety of configurations.

[0160] Figure 13 A block diagram of an encoding system 10000 according to an embodiment of the present invention is shown.

[0161] Platform 10002 in system 10000 can be a cloud server or a local server. Alternatively, platform 10002 can be any other type of device, or multiple devices, capable of computing, storing, transcoding, encrypting, rendering, decoding, or encoding. While the disclosed implementation can be implemented using a single platform such as platform 10002 as shown in the figure, using more than one platform can improve speed and efficiency.

[0162] The content delivery network (CDN) 10004 in system 10000 can be a group of geographically distributed servers. Alternatively, CDN 10004 can be any other type of device, or multiple devices, capable of buffering, scheduling, propagating, or accelerating the distribution of web page content by bringing the content closer to the user's location. While the disclosed implementation can be achieved using a single CDN such as CDN 10004 as shown in the figure, using more than one CDN can improve speed and efficiency.

[0163] The terminal 10006 in device 10000 can be a mobile phone, computer, television, laptop, or camera. Alternatively, the terminal 10006 can be any other type of device capable of displaying video or images, or multiple devices.

[0164] It should be noted that RGB and YUV are common color spaces. Conversion between RGB and YUV can be performed according to the equations specified in standards such as CCIR601 and BT.709.

[0165] References to images and / or videos in this document should include references to one or more images (e.g., SCI images). That is, for example, a video may include multiple images (e.g., multiple SCI images).

[0166] Although the accompanying drawings depict operations in a specific order, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or requiring the execution of all the operations shown to achieve the desired result. In some cases, multitasking and parallel processing can be advantageously performed. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0167] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions described in the claims can be performed in a different order and still achieve the desired result. For example, the processes described in the drawings do not necessarily require execution in the specific order shown to achieve the desired result. In some implementations, multitasking and parallel processing can be advantageously performed.

Claims

1. A method for generating an input tensor representing image data to be encoded by an encoder, characterized in that, The method includes: The image data is processed using a selected global transform to generate a first tensor, which includes the transformed image data. The first tensor is processed by selecting one or more layers of a neural network to generate the input tensor, wherein the input tensor includes a preprocessed tensor representation of the image data. The decoder is provided with at least one of the following: a set of configuration parameters for the selected global transformation, a first index representing the selected global transformation, and a second index representing the selected neural network; The input tensor is provided to the encoder, which generates a bitstream representing the encoded image data.

2. The method according to claim 1, characterized in that, Also includes: The preprocessed tensor representation of the image data is generated by adding the corresponding components from the first tensor and the latent representation of the first tensor.

3. The method according to claim 1 or 2, characterized in that, Also includes: The global transformation is selected based on at least one of the following: the content of the image data and a portion of the content of the image data.

4. The method according to any one of the preceding claims, characterized in that, The global transformation includes a color space transformation, which is used to modify the color space of the image data.

5. The method according to any one of the preceding claims, characterized in that, The global transformation includes a resolution transformation, which is used to modify the resolution of the image data.

6. The method according to any one of the preceding claims, characterized in that, The global transformation includes a desaturation transformation, which is used to reduce the color saturation value of one or more color channels of the image data.

7. The method according to any one of the preceding claims, characterized in that, The global transformation includes an alpha blending transformation, which is used to blend the image data with one or more background image datasets.

8. The method according to claim 7, characterized in that, At least one of the one or more background image datasets includes a grayscale or desaturated version of the image data.

9. The method according to any one of the preceding claims, characterized in that, The global transformation includes linear global transformation.

10. The method according to any one of the preceding claims, characterized in that, The global transformation includes a transformation that is exactly the same for every pixel of the image represented by the image data.

11. The method according to any one of the preceding claims, characterized in that, Also includes: The first tensor is processed using a convolution kernel to generate a first intermediate latent tensor, the first intermediate latent tensor including an increased number of channels for the image data; The first intermediate latent tensor is processed using a depthwise convolution kernel to generate a second intermediate latent tensor; The second intermediate latent tensor is processed using a convolution kernel to generate the input tensor.

12. The method according to any one of the preceding claims, characterized in that, Also includes: In response to the determination of the content of the image data, the application of the global transformation and the processing of the first tensor through the selected neural network layer or layers are bypassed.

13. The method according to claim 12, characterized in that, Also includes: Data is provided to the decoder, the data representing the following indications: in response to the determination of the content of the image data, bypassing the application of the global transformation and the processing of the first tensor through the one or more layers of the selected neural network.

14. A method for encoding an input tensor representing image data to generate encoded image data, characterized in that, The method includes: Provide an input tensor generated by the method according to any one of the preceding claims; The input tensor is encoded to generate a bitstream representing the encoded image data.

15. A preprocessor structure for an encoder, characterized in that, The encoder is used to encode the input tensor representing image data, and the preprocessor structure includes a processing circuit, which is used to: The image data is processed using a selected global transform to generate a first tensor, which includes the transformed image data. The first tensor is processed by selecting one or more layers of a neural network to generate the input tensor, wherein the input tensor includes a preprocessed tensor representation of the image data. The decoder is provided with at least one of the following: a set of configuration parameters for the selected global transformation, a first index representing the selected global transformation, and a second index representing the selected neural network; The input tensor is provided to the encoder, which generates encoded image data.

16. A method for decoding a bitstream representing encoded image data, characterized in that, The method includes: The decoder receives at least one of the following: a set of configuration parameters for selecting a global transformation, a first index representing the selected global transformation, and a second index representing the selected neural network; The decoder is used to decode the bitstream to generate an initial representation of the decoded image data; The initial representation of the decoded image data is processed using a global inverse transform selected based on the first index, wherein the global inverse transform is configured based on the set of configuration parameters to generate an output tensor, the output tensor comprising a preprocessed tensor representation of the image data; The output tensor is processed through one or more layers of a neural network to generate output image data.

17. The method according to claim 16, characterized in that, The neural network is selected based on the second index.

18. The method according to claim 16 or 17, characterized in that, The global inverse transform includes a color space transformation, which is used to modify the color space of the initial representation of the decoded image data.

19. The method according to any one of claims 16 to 18, characterized in that, The global inverse transform includes a resolution transform, which is used to modify the resolution of the initial representation of the decoded image data.

20. The method according to any one of claims 16 to 19, characterized in that, The global inverse transform includes a saturation transform, which is used to increase the color saturation values ​​of one or more color channels in the initial representation of the decoded image data.

21. The method according to any one of claims 16 to 20, characterized in that, The global inverse transform includes an α-mix transform, which is used to generate an initial representation of the decoded image data and the difference between one or more background image datasets.

22. The method according to any one of claims 16 to 21, characterized in that, Also includes: The output tensor is processed using a convolution kernel to generate a first intermediate latent tensor, the first intermediate latent tensor including an increased number of channels for the image data; The first intermediate latent tensor is processed using a depthwise convolution kernel to generate a second intermediate latent tensor; The second intermediate latent tensor is processed using a convolution kernel to generate the output image data.

23. The method according to any one of claims 16 to 22, characterized in that, Also includes: Based on the data received from the encoder, the application of the global inverse transform and the processing of the output tensor through one or more layers of the selected neural network are bypassed.

24. The method according to any one of claims 16 to 23, characterized in that, The global transformation includes a linear global transformation, and / or the global inverse transformation includes a linear global inverse transformation.

25. The method according to any one of claims 16 to 24, characterized in that, The global transformation includes a transformation that is exactly the same for every pixel of the image represented by the image data, and / or the global inverse transformation includes an inverse transformation that is exactly the same for every pixel of the image represented by the image data.

26. A method for decoding a bitstream representing encoded image data to generate output image data, characterized in that, The method includes: Provides a bitstream representing the encoded image data; The method according to any one of claims 16 to 25 decodes the bitstream representing encoded image data to generate the output image data.

27. The method according to claim 26, characterized in that, The bitstream includes the bitstream generated by the method according to any one of claims 1 to 14.

28. A post-processor structure for a decoder, characterized in that, The decoder is used to decode the bitstream representing encoded image data, and the post-processor structure includes a processing circuit, which is used to: Receive at least one of the following from the encoder: a set of configuration parameters for selecting a global transformation, a first index representing the selected global transformation, and a second index representing the selected neural network; Receive an initial representation of the decoded image data, wherein the initial representation of the decoded image data includes a tensor representation of the image data after preprocessing; The initial representation of the decoded image data is processed using a global inverse transform selected based on the first index, wherein the global inverse transform is configured based on the set of configuration parameters to generate the initial representation of the decoded image data. The initial representation of the decoded image data is processed by one or more layers of a neural network to generate output image data.

29. The post-processor structure according to claim 28, characterized in that, The post-processor structure includes a processing circuit, the processing circuit being used for: The neural network is selected and / or configured based on the second index.

30. A computer program, characterized in that, The method is stored on a non-transient medium and includes code instructions that, when executed on one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 14 or 16 to 27.

Images